An Astro blog Google and ChatGPT can find — robots.txt, JSON-LD, Markdown twins, IndexNow
How this Astro blog got into Google and Bing: a robots.txt naming each AI crawler, JSON-LD in the head, Markdown copies of articles, and IndexNow.

On this page
Introduction
I searched Google for the exact title of one of my own articles and got nothing back. The blog had been live for months in three languages, and none of it was in Google’s index.
My first guess was the oharu-tech-blog.vercel.app host. The causes were duller: the site had never been submitted to a search engine, it had no inbound links, and /robots.txt returned 404.
Fixing that raised a second question: what decides which pages ChatGPT and Claude cite? Search indexes, Bing’s and Brave’s, and not llms.txt, which almost nothing fetches.
The blog now names each AI crawler in robots.txt, serves every article as Markdown beside the HTML page, and tells Bing about new pages on each deploy.
Google rate-limits a vercel.app host but still crawls it
*.vercel.app is a subdomain shared by a very large number of projects, some of them spam, so search engines rate-limit and deprioritise it as a class. It also cannot be registered in Search Console as a Domain property, only as a URL prefix.
That caps how well a site ranks. A site on a discounted host still gets crawled, so the host could not explain why my blog was missing from the index altogether. A search engine finds a new site in two main ways:
- Following a link to it from another site
- Reading a sitemap someone submitted for it
My blog had neither, so Google had no way to find it.
I listed the problems hurting the site’s search visibility, ordered by how much each one mattered:
| Problem | Impact |
|---|---|
| Never submitted a sitemap to Search Console or Bing, and no inbound links from other sites | Google had no way to find the site |
*.vercel.app shared subdomain | Search engines limit the crawl rate and cap the ranking |
/robots.txt returned 404 | Blocks nothing, but crawlers are not told where the sitemap is, and AI crawlers get no policy |
hreflang and canonical named different URLs | Google may drop each article’s language alternates |
Every page was static HTML with no X-Robots-Tag. sitemap-index.xml answered 200 with all three locales, and every article already carried a BlogPosting node. So the blog had been crawlable all along, and I had never told a crawler it existed.
A missing robots.txt blocks nothing, because absence means “crawl everything”. It costs the Sitemap: line, which tells an arriving crawler where the sitemap is, and any statement of policy for AI crawlers.
Search engines: hreflang, the domain, and Search Console
Why hreflang and canonical named different URLs
The canonical tag, <link rel="canonical">, names the one URL a page declares as its official address. Search engines index only that URL. When the same page has two or more URLs, the others are treated as copies and are not indexed.
localeUrl() in src/i18n/config.ts builds the paths for navigation, hreflang and the RSS feed, and it returned them without a trailing slash. The canonical tag and og:url come from Astro.url.pathname, which in a static build does end in a slash. The build writes each page as an index.html inside a folder, such as blog/x/index.html, and a folder’s URL is /blog/x/.
Every article in all three languages shipped both the slashed and the slashless URL:
<link rel="canonical" href="https://oharu121.com/blog/rag-vs-finetuning-kb-search/"><link rel="alternate" hreflang="en-US" href="https://oharu121.com/blog/rag-vs-finetuning-kb-search">The slashless URL was a second copy of the page. Vercel serves the same HTML at both paths with a 200, and the canonical tag names only the slashed one.
Google treats hreflang as a hint, and may ignore one that points at a non-canonical URL. Every article’s language alternates were at risk of being dropped, so a reader searching in Japanese could be shown the English article.
I changed localeUrl to always append the slash, so the URLs built by localeUrl() and by Astro.url.pathname match. The one exception is a final path segment containing a dot, so that /rss.xml and /blog/<slug>.md keep their extensions:
export function localeUrl(locale: Locale, path = "/"): string { const suffix = path.startsWith("/") ? path : `/${path}`; const isFile = suffix.slice(suffix.lastIndexOf("/") + 1).includes("."); const normalized = isFile || suffix.endsWith("/") ? suffix : `${suffix}/`; return `${localePrefix(locale)}${normalized}`;}After a rebuild, the canonical and hreflang tags name the same URL, and every navigation link points at the canonical one:
<link rel="canonical" href="https://oharu121.com/blog/rag-vs-finetuning-kb-search/"><link rel="alternate" hreflang="en-US" href="https://oharu121.com/blog/rag-vs-finetuning-kb-search/">A sitemap tags each URL with a lastmod, the date the page last changed, which tells a crawler which pages to fetch again. scripts/lib/article-lastmod.ts keys those dates by URL and used to append its own slash. With localeUrl emitting one, a second append would key every entry on a // that no <loc>, the URL of a sitemap entry, carries. Every article would then lose its lastmod without an error, so I changed the script to take localeUrl’s output as it is.
Moving to oharu121.com with Cloudflare DNS only
My first pick was a .click domain at three dollars a year, and the agent argued against it. Cheap novelty TLDs carry no direct ranking penalty, but they are heavily spam-registered, which makes them slower to index and less likely to be linked. So I bought oharu121.com.
I chose the bare apex over a subdomain such as blog.oharu121.com or www.oharu121.com. I have no other site that would use the apex, so leaving it empty and putting the blog on a subdomain made no sense. www.oharu121.com redirects to the apex with a 308.
I kept DNS at Cloudflare and turned its proxy, which routes traffic through Cloudflare’s servers first, off permanently. Vercel already serves the site from its own edge network, so the proxy would add a second CDN that can serve stale HTML after a deploy.
Cloudflare’s bot and AI crawler rules also act only on proxied traffic. New domains are onboarded with presets that can block agent crawlers or send a no-training signal, which would override the robots.txt allow-list described below. DNS-only records keep robots.txt as the only crawler policy.
Pointing the apex at Vercel with a CNAME
Vercel asked for a CNAME record on the apex, pointing at a per-project hostname rather than the generic cname.vercel-dns.com that many guides still quote.
A CNAME record makes one name an alias of another. A lookup for oharu121.com is told “this is an alias of xxxx.vercel-dns-017.com”, a hostname Vercel generated for this project, and looks that up instead to get the IP address. An A record pins a name to one fixed IP address, so it has to be edited whenever the IP changes. With a CNAME, Vercel keeps the target on its current IP, and nothing on my side changes.
The DNS rules forbid a CNAME at the apex, though. The apex must carry NS records, naming the servers that answer for the domain, and an SOA record, holding its administrative details. A CNAME means “everything about this name is defined by another name”, which conflicts with both.
The setup works because Cloudflare never sends the CNAME out. CNAME flattening is a Cloudflare feature, not a DNS standard: Cloudflare looks up the CNAME target’s IP itself and answers with a plain A record. For a DNS provider without such a feature, Vercel’s knowledge base recommends an A record at the apex, 76.76.21.21. The Vercel dashboard also notes that this A record still works.



The old host answers every path with a 308, a permanent redirect, to the same path on the new domain, so /blog/x/ on the old host redirects to /blog/x/ on the new one. A crawler following the redirect never reads the old page, which merges the two hosts more firmly than a canonical tag can.
In the code, the move took me one line. Every absolute URL on the site, including the Sitemap: line in robots.txt, the sitemap, RSS, llms.txt, the canonical tags and the IndexNow script, is built from one constant, SITE:
export const SITE = "https://oharu121.com";Submitting a Domain property to Search Console
A Search Console Domain property is verified by a DNS TXT record, and it covers the apex, www, HTTP and HTTPS, and any future subdomain at once. Google and Cloudflare share a one-time authorization flow that writes the record for you.

I first submitted the sitemap as the bare path sitemap-index.xml, and it failed with “Invalid sitemap address. Please enter a valid path to a sitemap in your site.” It needs the full URL: https://oharu121.com/sitemap-index.xml.

I imported the verification and the sitemap into Bing Webmaster Tools from Search Console, so Bing needed no verification or sitemap submission of its own.

Answer engines: robots.txt, llms.txt, JSON-LD, Markdown twins, IndexNow, and Brave
Naming eleven AI crawlers in robots.txt
robots.txt is now generated by a route, src/pages/robots.txt.ts, so its Sitemap: line is built from the SITE constant and follows a domain move. I set the policy: allow every AI crawler, including the ones that collect training data. This blog exists for reach, and a training corpus is one more place its reasoning gets read.
I still listed all eleven by name, so each has its own switch, even though User-agent: * with Allow: / already lets every crawler fetch the whole site:
# OpenAI — training data for future modelsUser-agent: GPTBotAllow: /
# OpenAI — the index behind ChatGPT searchUser-agent: OAI-SearchBotAllow: /Training and retrieval are separate agents with separate switches: blocking one leaves the others running. Anthropic’s documentation lists three:
ClaudeBot: collects training dataClaude-SearchBot: builds the search indexClaude-User: fetches a page a person asked for
With each agent named, changing one line to Disallow is the edit. A site that wants to be quoted in answers but kept out of training corpora blocks only the training agents.
ChatGPT-User and Claude-User both fetch a page a person asked for, but the two vendors treat robots.txt differently. OpenAI’s documentation says “Because these actions are initiated by a user, robots.txt rules may not apply”, so a Disallow on ChatGPT-User is a request. Anthropic’s documentation says all of its bots honor robots.txt, Claude-User included.
The eight agents from three vendors, each its own robots.txt switch:
| Vendor | Training | Search index | User-initiated |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User (robots.txt may not apply) |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | Not published | PerplexityBot | Perplexity-User |
Does anything read llms.txt?
llms.txt is a site index written in Markdown, served at /llms.txt in the site root, and almost no crawler reads it. Limy recorded about 408 requests for llms.txt against more than 500 million AI bot visits over ninety days. No major AI vendor has said it reads the file in production.
Publishing is far more common than fetching. Rankability found that about 8.7% of the Tranco top 1,000, a widely used ranking of popular sites, serve an llms.txt, and over half of the sites in some developer-heavy samples do.
IDE agents and MCP servers do fetch llms.txt, though, and developers read this blog. So I shipped llms.txt as a small index, one per locale, built the same way as the RSS feeds.
JSON-LD in <head>, linked by @id
Every article already had a BlogPosting node, but it rendered into <body> and carried no publisher, url or keywords. I moved the JSON-LD into <head>, built by src/lib/schema.ts, which adds BreadcrumbList on articles and WebSite, Blog and ItemList on the index pages.
Here is an excerpt of the JSON-LD in one article’s <head>:
{ "@type": "BlogPosting", "headline": "RAG vs fine-tuning for KB search — …", "url": "https://oharu121.com/blog/rag-vs-finetuning-kb-search/", "datePublished": "2026-08-11T15:07:58+09:00", "inLanguage": "en-US", "author": { "@type": "Person", "@id": "https://oharu121.com/#person", "name": "oharu" }, "isPartOf": { "@id": "https://oharu121.com/#website" }, "keywords": "RAG, LLM, Generative AI, Machine Learning"}Every page points at one author node and one site node by a stable @id: the site URL plus a fragment such as #person or #website:
const personId = (site: URL) => new URL("#person", site).href;const websiteId = (site: URL) => new URL("#website", site).href;A crawler reading any number of pages then resolves one person, instead of several people who share a name.
The serializer escapes every < as \u003c. JSON.stringify leaves </script> intact, so a string containing it would close the tag early and spill the rest of the JSON-LD into the page as markup.
The WebSite node carries no SearchAction. That property used to give a site a sitelinks search box in Google’s results, but Google removed the feature in November 2024, and this blog’s search page ignores a ?q= parameter anyway, so adding it would only advertise a search URL that does not work.
Search Console’s URL inspection reads the breadcrumb node back:

BreadcrumbList node that schema.ts emitted.Serving every article as Markdown
Adding .md to any article’s URL, as in /blog/<slug>.md, returns that article as a plain-text Markdown version, and the article’s <head> points to it with <link rel="alternate" type="text/markdown">. An answer engine then gets the prose directly, without parsing it out of the HTML page.
Every article here imports components, and an Astro component has no Markdown form. src/lib/mdx-to-markdown.ts turns each one into text:
- Imports are stripped.
<Callout>becomes a labelled blockquote.<Figure>becomes its caption plus the diagram’s name.- A raster keeps its alt text and drops its URL. The built asset path is a content hash the transform cannot know, and a model quoting a 404 link passes it on.
import Callout from '@/components/article/Callout.astro';import Figure from '@/components/article/Figure.astro';import RagVsFinetuning from './_figures/RagVsFinetuning.astro';
<Callout type="know">Fine-tuning changes behaviour, not facts.</Callout>
<Figure> <RagVsFinetuning /> <Fragment slot="caption">RAG keeps facts outside the model.</Fragment></Figure>
> **HEADS UP**>> Fine-tuning changes behaviour, not facts.
*Figure — RagVsFinetuning: RAG keeps facts outside the model.*
*Image: A bar chart of answer accuracy*Prose, headings, tables and code fences pass through unchanged, and they hold nearly all the quotable content.
IndexNow pushes each deploy into Bing’s index
Bing’s index is one source ChatGPT search cites from, and how much it accounts for is disputed. Seer Interactive found 87% of SearchGPT citations matched Bing’s top results in a 100-query sample in February 2025. A 400-query replication in April 2026 found 27%. Either way, a page Bing has not crawled cannot come from Bing.
IndexNow is a protocol for a site to tell search engines that URLs have changed. Sending URLs to the one endpoint api.indexnow.org notifies Bing, Yandex, Seznam, Naver and others at once. Google does not take part in IndexNow, so Google still has to be handled separately through Search Console.
IndexNow checks that whoever submits URLs controls the site through a key file at the site root, whose filename is the key and whose contents are the same key. Verification is “fetch it and compare”. I committed it under public/, and scripts/indexnow.ts reads it from that same file, so the key submitted always matches the key served.
buildLastmodMap already lists every published article URL with its lastmod for the sitemap. The script filters that map by a date window rather than diffing git.
70 URL(s) for oharu121.com: …IndexNow accepted the batch (200).I set CI to run pnpm indexnow after every production deploy. The first run returned 202, meaning received with key validation pending. Later runs return 200.
Bing’s URL inspection then showed a red cross:

“Discovered but not crawled” means the page is queued for Bing to crawl. The Discovered on 18 Aug 2026 on screen is the day IndexNow submitted the URL. A domain a few days old with no inbound links has almost no crawl budget, the number of pages a search engine is willing to fetch from a site. IndexNow tells Bing a URL exists, and Bing still decides when to fetch it.
Claude’s web search draws on Brave, which takes no submissions
Claude’s web search almost certainly runs on Brave Search. Anthropic has not named its provider, but as Simon Willison spotted, it listed Brave as a web search subprocessor, an outside company that processes data on Anthropic’s behalf, on 19 March 2025, the day before Claude’s web search launched. Claude’s citations also match Brave’s top results.
Brave has no webmaster console, accepts no sitemap, and is not part of IndexNow. According to MERJ’s analysis, Brave’s crawler sends no user agent of its own and follows whatever robots.txt tells Googlebot. This blog allows every crawler with User-agent: *, so Brave’s crawler was already admitted.
Brave finds new pages mostly through their Google and Bing rankings. Another source is Brave browser users who opted into its Web Discovery Project, who report the pages they visit. search.brave.com/submit-url only asks Brave to re-fetch a page it already knows, and cannot submit a new URL.
Summary
- The site was missing from Google because it had never been submitted and had no inbound links.
- A shared
*.vercel.apphost limits how well a site ranks, and Google still crawls it. llms.txtgot about 408 fetches against 500 million AI bot visits in one sample. It ships here as a small index for IDE agents.- IndexNow pushes each deploy into Bing’s index, one of the sources ChatGPT search cites from.
- Claude’s web search draws on Brave, which accepts no submissions and finds pages mostly through their Google and Bing rankings.
- A Markdown copy of each article gives an answer engine the prose without the page around it.
References
- Google Search Console sitemaps report, including the difference between Domain and URL-prefix properties
- Google Search Central’s “To slash or not to slash”, on trailing-slash URLs being separate pages
- OpenAI’s crawler documentation, listing GPTBot, OAI-SearchBot and ChatGPT-User and the robots.txt caveat on user-initiated fetches
- Anthropic’s crawler documentation, covering ClaudeBot, Claude-SearchBot and Claude-User as independently blockable agents
- Limy’s llms.txt guide, with the 408 requests against 500 million AI bot visits
- Rankability’s llms.txt adoption data for the Tranco top 1,000
- Seer Interactive’s comparison of SearchGPT citations with Bing and Google results
- A 400-query replication of the Bing and ChatGPT citation overlap
- Simon Willison on Brave Search appearing in Anthropic’s subprocessor list the day before Claude’s web search launched
- MERJ’s analysis of how Brave Search discovers pages, quoting Brave’s crawler documentation on Googlebot rules
- Cloudflare’s announcement of the AI crawler presets for new domains
- Cloudflare’s CNAME flattening documentation, on answering an apex CNAME with A records
- Vercel’s guide to pointing a domain at Vercel with an A record
- The IndexNow protocol specification, including the key file requirements and response codes
- Vercel’s guidance on avoiding duplicate content between a vercel.app URL and a custom domain



