Making an Astro blog findable by Google and ChatGPT — robots.txt, JSON-LD, and IndexNow

Introduction

I searched Google for the exact title of one of my own articles and got nothing back. Twenty-one articles had been published across three languages, the site had been live for months, and none of it existed as far as Google was concerned.

My first guess was the domain. The blog was served from oharu-tech-blog.vercel.app, and a free subdomain shared across Vercel projects felt like the sort of thing a search engine would quietly discount. That guess was right about the mechanism and wrong about the weight. The domain was real but second-order. The reasons were duller: the site had never been submitted to any search engine, it had no inbound links, and /robots.txt returned 404.

Fixing that turned into a second question I had not planned on. If the site was going to be made legible to Google, what would make it legible to ChatGPT and Claude? That half produced a more useful surprise: the file everyone reaches for, llms.txt, is fetched by almost nothing, and the thing that actually moves an article into an AI answer is Bing.

This article walks through both halves: what was really stopping the site being found, and what got built for search engines and answer engines once the diagnosis was right.

The domain was the obvious suspect

*.vercel.app is a shared subdomain used by a very large number of projects, some of them spam, so search engines rate-limit and deprioritise it as a class. It also cannot be registered in Google Search Console as a Domain property, only as a URL prefix, which is a smaller and more awkward thing to own.

So the domain was a genuine ceiling. But a ceiling is not a floor, and nothing about it explained an absence from the index. A site on a discounted host still gets crawled.

The tell was that I had never done the one thing that starts the process. A search engine does not find a new site by accident. It follows a link to it, or it reads a sitemap you handed it. This blog had no inbound links and had submitted no sitemap, which left Google with no route in at all.

What the audit found instead

Checking the live site rather than reasoning about it produced a short list, ordered by how much each one mattered:

The technical side was mostly fine and that was the useful part. Every page was static HTML with no X-Robots-Tag, sitemap-index.xml answered 200 with all three locales and their alternates, and every article already carried a BlogPosting node. The site was crawlable and nobody had been invited to crawl it.

The robots.txt 404 is worth being precise about, because it is easy to over-read. A missing robots.txt blocks nothing; the convention is that absence means “crawl everything”. What it costs is the Sitemap: line, which is the one place a crawler that arrived without a sitemap can find one, and any statement at all about AI crawlers.

Every article advertised two URLs for itself

The audit turned up one defect nobody had been looking for.

localeUrl() in this repo builds site-relative paths for navigation, hreflang and the RSS feed. It returned paths without a trailing slash. The canonical tag and og:url are built from Astro.url.pathname, which in a static build does carry one, because the build emits <dir>/index.html.

The result shipped on every article in three languages:

<link rel="canonical" href="https://oharu121.com/blog/rag-vs-finetuning-kb-search/">
<link rel="alternate" hreflang="en-US" href="https://oharu121.com/blog/rag-vs-finetuning-kb-search">

Two URLs, one page, handed to a crawler that was being asked to reconcile them. Nothing failed. No check caught it, because no check compared those two strings to each other.

The fix normalises inside localeUrl and exempts anything with a dot in its final segment, so /rss.xml still resolves:

export function localeUrl(locale: Locale, path = '/'): string {
	const suffix = path.startsWith('/') ? path : `/${path}`;
	const isFile = suffix.slice(suffix.lastIndexOf('/') + 1).includes('.');
	const normalized = isFile || suffix.endsWith('/') ? suffix : `${suffix}/`;
	return `${localePrefix(locale)}${normalized}`;
}

One consequence had to be fixed in the same change. scripts/lib/article-lastmod.ts keys the sitemap’s lastmod values by URL and had been appending its own slash to compensate. Left alone it would have produced // on every entry, and the failure mode there is silent: the keys simply stop matching and every article quietly loses its date.

Naming eleven AI crawlers to allow them

robots.txt is generated by a route rather than a static file, so the Sitemap: line derives from the site origin and survives a domain move. I decided the policy: allow every AI crawler, including the ones that collect training data.

The reasoning is that this is a personal blog whose entire purpose is reach. Being in a training corpus is the upside rather than the cost. It is how an answer about Pagefind and Japanese IME composition carries this site’s reasoning whether or not a crawler happens to be online at the time.

The agent’s design choice was to list all eleven by name anyway, even though User-agent: * with Allow: / already covers them:

User-agent: GPTBot
Allow: /

# OpenAI — the index behind ChatGPT search
User-agent: OAI-SearchBot
Allow: /

That looks redundant and is not. Training and retrieval are separate agents with separate switches, and blocking one does not block the others. Anthropic documents this explicitly for its three bots: ClaudeBot collects training data, Claude-SearchBot builds the search index, and Claude-User fetches a page because a person asked for it. The interesting question is never “AI crawlers, yes or no”. It is whether you want to be quoted without being trained on, and that is only expressible per token.

Two of them come with a caveat worth knowing. ChatGPT-User and Claude-User act on behalf of a human who asked for a specific page, and OpenAI’s own documentation says robots.txt “may not apply” to those. A Disallow there is a request rather than a guarantee.

llms.txt gets the attention and almost no traffic

The obvious move for AI visibility is llms.txt, a Markdown index of your site at a well-known path. I asked the agent to check whether it works before building it.

The evidence was worse than expected. Instrumentation across a ninety-day window recorded roughly 408 requests for llms.txt against more than 500 million AI bot visits, and as of early 2026 no major vendor had publicly committed to reading it in production. Adoption figures vary wildly by sample, from about 8.7% of the Tranco top 1,000 to over half of some fixed panels, but adoption is not consumption.

Where it does get fetched is IDE agents and MCP servers. That is a real audience for a blog read by developers, so llms.txt shipped, one per locale, mirroring the existing RSS routes. It shipped sized to what it earns, which is a cheap index rather than a centrepiece.

The lever that actually matters turned out to be two other things.

Serving every article as Markdown

An answer engine handed an HTML page has to find the article inside a navigation shell, a table of contents, a theme toggle and a share row. Handed /blog/<slug>.md it gets the prose.

So every article is now served twice: once as a page, once as Markdown at the same path with .md on the end, linked from the article’s own <head>. That is 67 files, one per published locale page.

The complication is that every published article on this blog imports components. There are 26 Figure imports, 109 <Figure> usages, and roughly 40 one-off inline-SVG diagram components living in each article’s own _figures/. There is no Markdown for an Astro component to degrade into.

The agent’s transform strips imports, converts <Callout> to a labelled blockquote, and turns a figure into its caption plus the diagram’s name:

*Figure — RagVsFinetuning: RAG keeps facts outside the model; fine-tuning folds behaviour into it.*

Rasters keep their alt text and lose their URL, because the built asset path is a content hash the transform cannot know, and a relative path does not survive the move to /blog/slug.md. A link that 404s is worse than no link, since a model quoting it passes the broken URL on.

This is lossy on purpose. Prose, headings, tables and code fences pass through untouched, and that is where essentially all the quotable content is.

The checker shared the bug’s blind spot

The transform had a bug, the agent wrote it, and the way it was found is the most useful thing in this article.

Code fences have to be masked before any tag rewriting, or an article quoting <Figure> as an example would have its own snippet rewritten. That masking existed. The inline-code half of it used this pattern:

`[^`\n]+`

Which requires at least one non-backtick character between the delimiters. On a doubled span it fails, and this blog’s house style uses doubled spans routinely, because that is how you write a backtick inside inline code:

`` `{ foo: 1 }` ``

The pattern matched backtick-space-backtick instead, and every backtick after it in the paragraph paired one position out.

Two files shipped the damage. In the Japanese and Traditional Chinese translations of the content-collections article, an inline `<Callout>text</Callout>` fell outside the mask and got rewritten into a real blockquote inside the code span:

`> **Note**
>
> text`はさらに2言語への翻訳に耐えますが、``は翻訳者に…

Here is the part worth keeping. The verification I had been shown reported clean, because it stripped code with the same single-backtick pattern. The checker carried the exact blind spot it existed to catch. A code review agent reading the diff found it by going back to the built artifact rather than trusting the check, which is the only reason it was caught before release.

Fixing it introduced a second bug immediately. The widened pattern could swallow a fence placeholder, and three files came out with raw NUL bytes in them. Both are now handled by matching on run length, the rule Markdown itself uses: a span opens on a run of N backticks and closes on the next run of exactly N.

A green check is evidence about the checker as much as about the code. When a check and the thing it checks share an assumption, it will report clean at exactly the moment it matters.

IndexNow, because Bing is what ChatGPT reads

The single highest-leverage item for AI visibility has nothing to do with AI file formats. Bing’s index is reported to be behind the large majority of ChatGPT’s citations, which means a page Bing has not crawled cannot be cited in a ChatGPT answer regardless of how clean its schema is.

IndexNow is the push channel into that index. One endpoint serves Bing, Yandex, Seznam and Naver. Google does not participate, so this is the other half of Search Console rather than a substitute for it.

The key is a file at the site root whose contents match its own filename, and verification is “fetch it and check”. That makes the key public by construction, so it is committed to the repo, and the submission script reads it back out of public/ rather than from an environment variable. Key and proof cannot drift apart if the file that serves it is the file that is read.

Which URLs to submit came free. buildLastmodMap already existed to give the sitemap its per-article lastmod values, and it already skips anything not status: published, so the script filters that map by a date window instead of diffing git.

70 URL(s) for oharu121.com:
  …
IndexNow accepted the batch (200).

CI runs it after every production deploy. The first deploy submitted 45 URLs inside the default seven-day window and returned 202, meaning received with key validation pending. A later full run returned 200, meaning validated and accepted.

Moving to oharu121.com

I bought the domain once the diagnosis was clear. My first pick was a .click at three dollars a year, and the agent argued against it: cheap novelty TLDs carry no direct ranking penalty, but they are heavily spam-registered, which makes them harder to get indexed and makes other people less willing to link to them. Those are precisely the two things that were already the bottleneck. I took the argument and bought oharu121.com.

I also decided to use the bare apex rather than a blog. subdomain. Subdomains do not automatically share authority with the parent, and with zero authority anywhere, splitting it into two piles was the wrong shape.

DNS stayed at Cloudflare, with the proxy off. Grey cloud is not a temporary state here, it is the correct permanent one. Vercel already fronts the site with its own edge network, so the proxy adds a second CDN that can serve stale HTML after a deploy. More pointedly, Cloudflare began default-blocking AI training and agent crawlers on newly-onboarded domains, which would silently undo the allow-list described above. Bot rules only apply to proxied traffic, so DNS-only keeps it inert.

Vercel wanted a CNAME on the apex pointing at a per-project hostname, not the generic cname.vercel-dns.com that most guides still quote. Cloudflare flattens CNAMEs at the zone apex, so this works despite DNS forbidding a literal CNAME there.

The old host now issues a per-path 308 to the new one, so /blog/x/ there lands on /blog/x/ here. That consolidates the two hosts outright, which is stronger than the canonical-tag arrangement usually recommended for this. A redirected request never reaches a page whose head a crawler could read.

In the repo the move was one line, because every absolute URL on the site resolves from a single constant:

export const SITE = 'https://oharu121.com';

Submitting to Google and Bing

Search Console takes a Domain property verified by DNS TXT, which covers the apex, www, both protocols, and any future subdomain in one. Google and Cloudflare have a one-time authorization flow that writes the record for you.

Then the part that made the structured-data work feel worth it. Google’s URL inspection reports the page as available, and reports the BreadcrumbList node it found:

Bing was faster to set up, because it imports verification and sitemaps straight from Search Console.

Two messages that looked like failures

Two things during submission read as errors and were not, and both cost time.

The first was mine to walk into. Search Console rejected the sitemap with “Invalid sitemap address. Please enter a valid path to a sitemap in your site.” The advice I had been given was to enter the bare path, sitemap-index.xml. That is correct for a URL-prefix property, where the console shows a fixed host beside the field. A Domain property spans several hosts, so a bare path is ambiguous and gets rejected. It wants the full URL.

The second was Bing reporting “Discovered but not crawled” with a red cross and the line “URL cannot appear on Bing”.

Reading it as a fault is the natural mistake. Discovered on 18 Aug 2026 is the IndexNow submission landing, which means the pipeline worked. The message body recommends the general guidelines and names no specific issue, which is what that panel says when the answer is “not fetched yet”. Checking directly confirmed there was nothing to fix: robots.txt has zero Disallow lines, and a request carrying Bingbot’s user agent gets 200 on the home page, an article and the sitemap alike.

The honest reading is that a domain a few days old with no inbound links has a crawl budget near zero. IndexNow can tell Bing a URL exists. It cannot make Bing want it.

Summary

The work split cleanly in two, and both halves had the same shape: the loud answer was not the load-bearing one.

• Being absent from Google was not a technical fault. The site was crawlable from the day it launched. It had never been submitted and had no inbound links, and no amount of markup substitutes for either.
• The host mattered less than it looked. A shared *.vercel.app subdomain is a real ceiling, and it was nowhere near the binding constraint.
• llms.txt is not the AI lever. Roughly 408 fetches against 500 million AI bot visits. It is worth shipping for IDE agents and worth sizing accordingly.
• Bing is the AI lever, because its index feeds ChatGPT’s search, and IndexNow is the push channel into it.
• Markdown twins beat file formats. Serving the prose without the page chrome is what an answer engine can actually use.
• A check that shares an assumption with the code will pass when it matters. The Markdown transform’s verification stripped code with the same broken pattern the transform used, and reported clean over two corrupted files.

What is still outstanding is the slowest item and the one that decides whether any of the rest converts: inbound links. Everything above makes the site legible. None of it makes anyone link to it.

References

• Google Search Console sitemaps report, including the difference between Domain and URL-prefix properties
• OpenAI’s crawler documentation, listing GPTBot, OAI-SearchBot and ChatGPT-User and the robots.txt caveat on user-initiated fetches
• Anthropic’s crawler documentation, covering ClaudeBot, Claude-SearchBot and Claude-User as independently blockable agents
• The IndexNow protocol specification, including the key file requirements
• Vercel’s guidance on avoiding duplicate content between a vercel.app URL and a custom domain
• The CommonMark specification’s code span rules, which are the run-length rule the fixed transform implements