An Astro blog Google and ChatGPT can find — robots.txt, JSON-LD, Markdown twins, IndexNow

How this Astro blog got into Google and Bing: a robots.txt naming each AI crawler, JSON-LD in the head, Markdown copies of articles, and IndexNow.

Updated

The Google Search Console logo, a yellow magnifying glass overlapping red, green and blue rounded bars of rising height, beside the Google Search Console wordmark in grey on a white card
On this page

Introduction

I searched Google for the exact title of one of my own articles and got nothing back. The blog had been live for months in three languages, and none of it was in Google’s index.

My first guess was the oharu-tech-blog.vercel.app host. The causes were duller: the site had never been submitted to a search engine, it had no inbound links, and /robots.txt returned 404.

Fixing that raised a second question: what decides which pages ChatGPT and Claude cite? Search indexes, Bing’s and Brave’s, and not llms.txt, which almost nothing fetches.

The blog now names each AI crawler in robots.txt, serves every article as Markdown beside the HTML page, and tells Bing about new pages on each deploy.

Two submission paths, one siteThe site feeds two separate paths. Google is reached by submitting a sitemap in Search Console. ChatGPT is reached the long way round: IndexNow pushes changed URLs into the Bing index, and ChatGPT search cites what Bing has indexed. Google does not participate in IndexNow, so neither path substitutes for the other.Search engineAnswer engineoharu121.com67 pages
Search Console
sitemap submitted
Google
organic search
IndexNowchanged URLs pushedBingthe index behind itChatGPTcites what Bing indexedGoogle does not participate in IndexNow. Neither path substitutes for the other.
One site, two kinds of reader: search engines, which list links, and answer engines such as ChatGPT and Claude, which answer directly and cite their sources. Each needs its own route in.

Google rate-limits a vercel.app host but still crawls it

*.vercel.app is a subdomain shared by a very large number of projects, some of them spam, so search engines rate-limit and deprioritise it as a class. It also cannot be registered in Search Console as a Domain property, only as a URL prefix.

That caps how well a site ranks. A site on a discounted host still gets crawled, so the host could not explain why my blog was missing from the index altogether. A search engine finds a new site in two main ways:

  1. Following a link to it from another site
  2. Reading a sitemap someone submitted for it

My blog had neither, so Google had no way to find it.

I listed the problems hurting the site’s search visibility, ordered by how much each one mattered:

ProblemImpact
Never submitted a sitemap to Search Console or Bing, and no inbound links from other sitesGoogle had no way to find the site
*.vercel.app shared subdomainSearch engines limit the crawl rate and cap the ranking
/robots.txt returned 404Blocks nothing, but crawlers are not told where the sitemap is, and AI crawlers get no policy
hreflang and canonical named different URLsGoogle may drop each article’s language alternates

Every page was static HTML with no X-Robots-Tag. sitemap-index.xml answered 200 with all three locales, and every article already carried a BlogPosting node. So the blog had been crawlable all along, and I had never told a crawler it existed.

A missing robots.txt blocks nothing, because absence means “crawl everything”. It costs the Sitemap: line, which tells an arriving crawler where the sitemap is, and any statement of policy for AI crawlers.

Search engines: hreflang, the domain, and Search Console

Why hreflang and canonical named different URLs

The canonical tag, <link rel="canonical">, names the one URL a page declares as its official address. Search engines index only that URL. When the same page has two or more URLs, the others are treated as copies and are not indexed.

localeUrl() in src/i18n/config.ts builds the paths for navigation, hreflang and the RSS feed, and it returned them without a trailing slash. The canonical tag and og:url come from Astro.url.pathname, which in a static build does end in a slash. The build writes each page as an index.html inside a folder, such as blog/x/index.html, and a folder’s URL is /blog/x/.

Two code paths, two URLs for one pageTwo lanes feed one page head. In the top lane, localeUrl() builds the paths for navigation links, hreflang and RSS, ending /blog/rag-vs-finetuning-kb-search with no trailing slash. In the bottom lane, Astro.url.pathname comes from the built <dir>/index.html and feeds the canonical tag and og:url, ending /blog/rag-vs-finetuning-kb-search/ with a trailing slash. The two URLs differ only in that final slash.localeUrl()builds paths itselfnav linkshreflang, RSSAstro.url.pathnamefrom <dir>/index.htmlcanonicalog:urlOne page's <head>/blog/rag-vs-finetuning-kb-searchno trailing slash/blog/rag-vs-finetuning-kb-search/trailing slashOne page, two URLs. They differ only in the final slash.
Two code paths build the same article’s URL. Only one of the two URLs ends in a slash.

Every article in all three languages shipped both the slashed and the slashless URL:

blog/rag-vs-finetuning-kb-search/index.html, before the fix
<link rel="canonical" href="https://oharu121.com/blog/rag-vs-finetuning-kb-search/">
<link rel="alternate" hreflang="en-US" href="https://oharu121.com/blog/rag-vs-finetuning-kb-search">

The slashless URL was a second copy of the page. Vercel serves the same HTML at both paths with a 200, and the canonical tag names only the slashed one.

Google treats hreflang as a hint, and may ignore one that points at a non-canonical URL. Every article’s language alternates were at risk of being dropped, so a reader searching in Japanese could be shown the English article.

I changed localeUrl to always append the slash, so the URLs built by localeUrl() and by Astro.url.pathname match. The one exception is a final path segment containing a dot, so that /rss.xml and /blog/<slug>.md keep their extensions:

src/i18n/config.ts
export function localeUrl(locale: Locale, path = "/"): string {
const suffix = path.startsWith("/") ? path : `/${path}`;
const isFile = suffix.slice(suffix.lastIndexOf("/") + 1).includes(".");
const normalized = isFile || suffix.endsWith("/") ? suffix : `${suffix}/`;
return `${localePrefix(locale)}${normalized}`;
}

After a rebuild, the canonical and hreflang tags name the same URL, and every navigation link points at the canonical one:

blog/rag-vs-finetuning-kb-search/index.html, after the fix
<link rel="canonical" href="https://oharu121.com/blog/rag-vs-finetuning-kb-search/">
<link rel="alternate" hreflang="en-US" href="https://oharu121.com/blog/rag-vs-finetuning-kb-search/">

A sitemap tags each URL with a lastmod, the date the page last changed, which tells a crawler which pages to fetch again. scripts/lib/article-lastmod.ts keys those dates by URL and used to append its own slash. With localeUrl emitting one, a second append would key every entry on a // that no <loc>, the URL of a sitemap entry, carries. Every article would then lose its lastmod without an error, so I changed the script to take localeUrl’s output as it is.

Moving to oharu121.com with Cloudflare DNS only

My first pick was a .click domain at three dollars a year, and the agent argued against it. Cheap novelty TLDs carry no direct ranking penalty, but they are heavily spam-registered, which makes them slower to index and less likely to be linked. So I bought oharu121.com.

I chose the bare apex over a subdomain such as blog.oharu121.com or www.oharu121.com. I have no other site that would use the apex, so leaving it empty and putting the blog on a subdomain made no sense. www.oharu121.com redirects to the apex with a 308.

I kept DNS at Cloudflare and turned its proxy, which routes traffic through Cloudflare’s servers first, off permanently. Vercel already serves the site from its own edge network, so the proxy would add a second CDN that can serve stale HTML after a deploy.

Cloudflare’s bot and AI crawler rules also act only on proxied traffic. New domains are onboarded with presets that can block agent crawlers or send a no-training signal, which would override the robots.txt allow-list described below. DNS-only records keep robots.txt as the only crawler policy.

Cloudflare proxied versus DNS only, in front of VercelTwo rows. With the proxy on, a request goes from the browser or crawler through the Cloudflare edge, with its own cache and bot rules, then through the Vercel edge to oharu121.com. Three problems sit on the Cloudflare edge: it can serve stale HTML after a deploy, its bot rules can override robots.txt, and Vercel cannot issue the certificate. With DNS only, Cloudflare DNS answers the address lookup and nothing more, and the request goes from the browser straight to the Vercel edge.Proxy on (orange cloud)Browseror crawler
Cloudflare edge
cache + bot rules
Vercel edge
cache + TLSoharu121.com× Can serve stale HTML after a deploy× Bot rules override robots.txt× Vercel cannot issue the certificateDNS only (grey cloud)Browseror crawler
Vercel edge
cache + TLSoharu121.comDNS lookup
Cloudflare DNS
answers the address onlyDNS only: one edge network, and robots.txt is the only crawler policy.
Proxied, Cloudflare’s edge sits in front of Vercel’s. DNS only, it answers the lookup and steps out of the path.

Pointing the apex at Vercel with a CNAME

Vercel asked for a CNAME record on the apex, pointing at a per-project hostname rather than the generic cname.vercel-dns.com that many guides still quote.

A CNAME record makes one name an alias of another. A lookup for oharu121.com is told “this is an alias of xxxx.vercel-dns-017.com”, a hostname Vercel generated for this project, and looks that up instead to get the IP address. An A record pins a name to one fixed IP address, so it has to be edited whenever the IP changes. With a CNAME, Vercel keeps the target on its current IP, and nothing on my side changes.

The DNS rules forbid a CNAME at the apex, though. The apex must carry NS records, naming the servers that answer for the domain, and an SOA record, holding its administrative details. A CNAME means “everything about this name is defined by another name”, which conflicts with both.

The setup works because Cloudflare never sends the CNAME out. CNAME flattening is a Cloudflare feature, not a DNS standard: Cloudflare looks up the CNAME target’s IP itself and answers with a plain A record. For a DNS provider without such a feature, Vercel’s knowledge base recommends an A record at the apex, 76.76.21.21. The Vercel dashboard also notes that this A record still works.

A CNAME on a subdomain versus a flattened CNAME at the apexTwo rows. On a subdomain, the DNS resolver asks Cloudflare DNS for www.oharu121.com and gets back a CNAME pointing at xxxx.vercel-dns-017.com, then asks Vercel DNS for that name and gets the IP address: two lookups. At the apex, the resolver asks Cloudflare DNS for oharu121.com. Cloudflare holds the NS and SOA records and the CNAME, looks the CNAME target up at Vercel DNS itself, and answers with a plain A record: one lookup, and the CNAME never leaves Cloudflare.CNAME on a subdomain: two lookupsDNS resolverfor a browser or crawler① www.oharu121.com ?CNAME → xxxx.vercel-dns-017.com
Cloudflare DNS
answers for oharu121.com② xxxx.vercel-dns-017.com ?A → IP address
Vercel DNS
answers for vercel-dns-017.comCNAME at the apex, flattened by Cloudflare: one lookupDNS resolverfor a browser or crawler① oharu121.com ?A → IP address
Cloudflare DNS
NS · SOACNAME → xxxx.vercel-dns-017.comthis CNAME stays insidelooks up
Vercel DNS
returns the IPFlattened, the apex answers with a plain A record, so NS and SOA never sit beside a CNAME.
On a subdomain the resolver follows the CNAME itself. At the apex, Cloudflare follows it and answers with an A record.

The Vercel Domains page for the project, with oharu121.com flagged Invalid Configuration and redirecting with a 308 to www.oharu121.com. Its DNS Records tab asks for a CNAME on @ pointing at a redacted per-project value with the proxy disabled, and notes that the legacy cname.vercel-dns.com and 76.76.21.21 records still work. www.oharu121.com is also flagged Invalid Configuration, and oharu-tech-blog.vercel.app is marked Valid Configuration

Before the record exists: Vercel names the CNAME it wants at Cloudflare, with the proxy disabled.

The Cloudflare Add record dialog for oharu121.com, filled in as a CNAME on @ whose target is a redacted per-project vercel-dns-017.com hostname, with the proxy status toggle off showing DNS only and TTL set to Auto

The same record entered at Cloudflare: @, meaning the apex, pointing at the per-project hostname, proxy off.

The same Vercel Domains page after the records propagated, with oharu121.com marked Valid Configuration and Production, and both www.oharu121.com and oharu-tech-blog.vercel.app marked Valid Configuration and issuing a 308 redirect to oharu121.com

After. The apex is the production domain, and both www and the old vercel.app host redirect into it.

The old host answers every path with a 308, a permanent redirect, to the same path on the new domain, so /blog/x/ on the old host redirects to /blog/x/ on the new one. A crawler following the redirect never reads the old page, which merges the two hosts more firmly than a canonical tag can.

In the code, the move took me one line. Every absolute URL on the site, including the Sitemap: line in robots.txt, the sitemap, RSS, llms.txt, the canonical tags and the IndexNow script, is built from one constant, SITE:

src/lib/site.ts
export const SITE = "https://oharu121.com";

Submitting a Domain property to Search Console

A Search Console Domain property is verified by a DNS TXT record, and it covers the apex, www, HTTP and HTTPS, and any future subdomain at once. Google and Cloudflare share a one-time authorization flow that writes the record for you.

The Cloudflare one-time authorization screen listing the DNS record Google will add for oharu121.com: a TXT record on the apex containing a redacted google-site-verification value, with a one hour TTL and proxy status DNS only

Google writes the verification record itself, once, and leaves the proxy off.

I first submitted the sitemap as the bare path sitemap-index.xml, and it failed with “Invalid sitemap address. Please enter a valid path to a sitemap in your site.” It needs the full URL: https://oharu121.com/sitemap-index.xml.

The Google Search Console sitemaps page for oharu121.com showing a green Sitemap submitted successfully confirmation for https://oharu121.com/sitemap-index.xml

The full URL, accepted. The same field rejected the bare path.

I imported the verification and the sitemap into Bing Webmaster Tools from Search Console, so Bing needed no verification or sitemap submission of its own.

The Bing Webmaster Tools welcome page offering two ways to add a site. On the left, Import your sites from GSC lists No site verification required, Import sitemaps instantly and Saves a lot of time, above an Import button. On the right, Add your site manually has a site URL field and an Add button

Bing’s welcome page: importing from Search Console skips verification and brings the sitemap along.

Answer engines: robots.txt, llms.txt, JSON-LD, Markdown twins, IndexNow, and Brave

Naming eleven AI crawlers in robots.txt

robots.txt is now generated by a route, src/pages/robots.txt.ts, so its Sitemap: line is built from the SITE constant and follows a domain move. I set the policy: allow every AI crawler, including the ones that collect training data. This blog exists for reach, and a training corpus is one more place its reasoning gets read.

I still listed all eleven by name, so each has its own switch, even though User-agent: * with Allow: / already lets every crawler fetch the whole site:

robots.txt (excerpt)
# OpenAI — training data for future models
User-agent: GPTBot
Allow: /
# OpenAI — the index behind ChatGPT search
User-agent: OAI-SearchBot
Allow: /

Training and retrieval are separate agents with separate switches: blocking one leaves the others running. Anthropic’s documentation lists three:

  1. ClaudeBot: collects training data
  2. Claude-SearchBot: builds the search index
  3. Claude-User: fetches a page a person asked for

With each agent named, changing one line to Disallow is the edit. A site that wants to be quoted in answers but kept out of training corpora blocks only the training agents.

ChatGPT-User and Claude-User both fetch a page a person asked for, but the two vendors treat robots.txt differently. OpenAI’s documentation says “Because these actions are initiated by a user, robots.txt rules may not apply”, so a Disallow on ChatGPT-User is a request. Anthropic’s documentation says all of its bots honor robots.txt, Claude-User included.

The eight agents from three vendors, each its own robots.txt switch:

VendorTrainingSearch indexUser-initiated
OpenAIGPTBotOAI-SearchBotChatGPT-User (robots.txt may not apply)
AnthropicClaudeBotClaude-SearchBotClaude-User
PerplexityNot publishedPerplexityBotPerplexity-User

Does anything read llms.txt?

llms.txt is a site index written in Markdown, served at /llms.txt in the site root, and almost no crawler reads it. Limy recorded about 408 requests for llms.txt against more than 500 million AI bot visits over ninety days. No major AI vendor has said it reads the file in production.

Publishing is far more common than fetching. Rankability found that about 8.7% of the Tranco top 1,000, a widely used ranking of popular sites, serve an llms.txt, and over half of the sites in some developer-heavy samples do.

IDE agents and MCP servers do fetch llms.txt, though, and developers read this blog. So I shipped llms.txt as a small index, one per locale, built the same way as the RSS feeds.

JSON-LD in <head>, linked by @id

Every article already had a BlogPosting node, but it rendered into <body> and carried no publisher, url or keywords. I moved the JSON-LD into <head>, built by src/lib/schema.ts, which adds BreadcrumbList on articles and WebSite, Blog and ItemList on the index pages.

Here is an excerpt of the JSON-LD in one article’s <head>:

blog/rag-vs-finetuning-kb-search/index.html, JSON-LD in <head> (excerpt)
{
"@type": "BlogPosting",
"headline": "RAG vs fine-tuning for KB search — …",
"url": "https://oharu121.com/blog/rag-vs-finetuning-kb-search/",
"datePublished": "2026-08-11T15:07:58+09:00",
"inLanguage": "en-US",
"author": { "@type": "Person", "@id": "https://oharu121.com/#person", "name": "oharu" },
"isPartOf": { "@id": "https://oharu121.com/#website" },
"keywords": "RAG, LLM, Generative AI, Machine Learning"
}

Every page points at one author node and one site node by a stable @id: the site URL plus a fragment such as #person or #website:

src/lib/schema.ts
const personId = (site: URL) => new URL("#person", site).href;
const websiteId = (site: URL) => new URL("#website", site).href;

A crawler reading any number of pages then resolves one person, instead of several people who share a name.

Every page, one author node, one site nodeA sample of pages sits in the middle row: two article pages, each with a BlogPosting node, and an index page with a Blog node. Every page points up at the same author node, oharu121.com/#person, and down at the same site node, oharu121.com/#website. Because every page uses the same @id, a crawler resolves one author and one site, however many pages it reads.authorisPartOfAuthoroharu121.com/#personSiteoharu121.com/#websiteArticleBlogPostingArticleBlogPostingIndex pageBlogSame @id on every page: one author and one site, however many pages a crawler reads.
Every page resolves to the same author node and the same site node.

The serializer escapes every < as \u003c. JSON.stringify leaves </script> intact, so a string containing it would close the tag early and spill the rest of the JSON-LD into the page as markup.

The WebSite node carries no SearchAction. That property used to give a site a sitelinks search box in Google’s results, but Google removed the feature in November 2024, and this blog’s search page ignores a ?q= parameter anyway, so adding it would only advertise a search URL that does not work.

Search Console’s URL inspection reads the breadcrumb node back:

Google Search Console URL inspection for a published article, showing URL is available to Google, Page availability Page can be indexed, and under Enhancements and Experience a Breadcrumbs row reading 1 valid item detected

Google reporting the BreadcrumbList node that schema.ts emitted.

Serving every article as Markdown

Adding .md to any article’s URL, as in /blog/<slug>.md, returns that article as a plain-text Markdown version, and the article’s <head> points to it with <link rel="alternate" type="text/markdown">. An answer engine then gets the prose directly, without parsing it out of the HTML page.

The same article, with and without the page around itOn the left, the HTML page: a header, a table of contents, the article prose, a share row and a footer, so the prose is one band among five. On the right, the Markdown twin at the same slug with a dot md suffix: the prose and a short metadata header, and nothing else. Both carry identical text; only the second is mostly article.HTML page/blog/slug/header and navigationtable of contentsthe articleshare rowfooterMarkdown twin/blog/slug.mdtitle, dates, tags, source URLthe articlethe prose is one band among fivethe prose, and a header naming where it came from
The same text either way. The Markdown copy is almost all article.

Every article here imports components, and an Astro component has no Markdown form. src/lib/mdx-to-markdown.ts turns each one into text:

  1. Imports are stripped.
  2. <Callout> becomes a labelled blockquote.
  3. <Figure> becomes its caption plus the diagram’s name.
  4. A raster keeps its alt text and drops its URL. The built asset path is a content hash the transform cannot know, and a model quoting a 404 link passes it on.
MDX in the article
import Callout from '@/components/article/Callout.astro';
import Figure from '@/components/article/Figure.astro';
import RagVsFinetuning from './_figures/RagVsFinetuning.astro';
<Callout type="know">
Fine-tuning changes behaviour, not facts.
</Callout>
<Figure>
<RagVsFinetuning />
<Fragment slot="caption">RAG keeps facts outside the model.</Fragment>
</Figure>
![A bar chart of answer accuracy](./_images/accuracy.webp)
The Markdown twin (output)
> **HEADS UP**
>
> Fine-tuning changes behaviour, not facts.
*Figure — RagVsFinetuning: RAG keeps facts outside the model.*
*Image: A bar chart of answer accuracy*

Prose, headings, tables and code fences pass through unchanged, and they hold nearly all the quotable content.

IndexNow pushes each deploy into Bing’s index

Bing’s index is one source ChatGPT search cites from, and how much it accounts for is disputed. Seer Interactive found 87% of SearchGPT citations matched Bing’s top results in a 100-query sample in February 2025. A 400-query replication in April 2026 found 27%. Either way, a page Bing has not crawled cannot come from Bing.

IndexNow is a protocol for a site to tell search engines that URLs have changed. Sending URLs to the one endpoint api.indexnow.org notifies Bing, Yandex, Seznam, Naver and others at once. Google does not take part in IndexNow, so Google still has to be handled separately through Search Console.

IndexNow checks that whoever submits URLs controls the site through a key file at the site root, whose filename is the key and whose contents are the same key. Verification is “fetch it and compare”. I committed it under public/, and scripts/indexnow.ts reads it from that same file, so the key submitted always matches the key served.

How the IndexNow key is submitted and verifiedThe key file public/<key>.txt sits at the top; its content is the key itself. In CI, scripts/indexnow.ts reads that file and POSTs host, key, keyLocation and urlList to api.indexnow.org. The deploy serves the same file at oharu121.com/<key>.txt, and IndexNow fetches it and compares it with the submitted key. The first submission returns 202, received with verification pending, and later ones return 200. One file feeds both sides, so the key submitted always matches the key served.public/<key>.txtits content is the keyread in CIdeployed asscripts/indexnow.tsruns in CI after deployoharu121.com/<key>.txtserved publicly① POST{ host, key, keyLocation, urlList }② fetch and compareapi.indexnow.orgBing, Yandex, Seznam, Naver…202 first run200 afterwardsOne file feeds both sides, so the key submitted always matches the key served.
The key file is read twice: by the script when it submits, and by IndexNow when it verifies.

buildLastmodMap already lists every published article URL with its lastmod for the sitemap. The script filters that map by a date window rather than diffing git.

pnpm indexnow, output
70 URL(s) for oharu121.com:
…
IndexNow accepted the batch (200).

I set CI to run pnpm indexnow after every production deploy. The first run returned 202, meaning received with key validation pending. Later runs return 200.

Bing’s URL inspection then showed a red cross:

Bing Webmaster Tools URL inspection showing a red Discovered but not crawled status with the note URL cannot appear on Bing, a green Discover step dated 18 August 2026, and a failed Crawl step with a generic message recommending the Bing Webmaster Guidelines

Bing discovered the page at that URL when IndexNow submitted it. The red crawl failure is not a real error: it only means Bing has noticed the URL but has not fetched the page yet.

“Discovered but not crawled” means the page is queued for Bing to crawl. The Discovered on 18 Aug 2026 on screen is the day IndexNow submitted the URL. A domain a few days old with no inbound links has almost no crawl budget, the number of pages a search engine is willing to fetch from a site. IndexNow tells Bing a URL exists, and Bing still decides when to fetch it.

Claude’s web search draws on Brave, which takes no submissions

Claude’s web search almost certainly runs on Brave Search. Anthropic has not named its provider, but as Simon Willison spotted, it listed Brave as a web search subprocessor, an outside company that processes data on Anthropic’s behalf, on 19 March 2025, the day before Claude’s web search launched. Claude’s citations also match Brave’s top results.

Brave has no webmaster console, accepts no sitemap, and is not part of IndexNow. According to MERJ’s analysis, Brave’s crawler sends no user agent of its own and follows whatever robots.txt tells Googlebot. This blog allows every crawler with User-agent: *, so Brave’s crawler was already admitted.

Brave finds new pages mostly through their Google and Bing rankings. Another source is Brave browser users who opted into its Web Discovery Project, who report the pages they visit. search.brave.com/submit-url only asks Brave to re-fetch a page it already knows, and cannot submit a new URL.

Summary

  • The site was missing from Google because it had never been submitted and had no inbound links.
  • A shared *.vercel.app host limits how well a site ranks, and Google still crawls it.
  • llms.txt got about 408 fetches against 500 million AI bot visits in one sample. It ships here as a small index for IDE agents.
  • IndexNow pushes each deploy into Bing’s index, one of the sources ChatGPT search cites from.
  • Claude’s web search draws on Brave, which accepts no submissions and finds pages mostly through their Google and Bing rankings.
  • A Markdown copy of each article gives an answer engine the prose without the page around it.

References

Share this article