Pagefind's stemming warning was a red herring — Japanese search broke on segmentation

Introduction

Every build of this site printed the same two notes, and I had been reading past them for weeks:

Note: Pagefind doesn't support stemming for the language ja-jp.
Search will still work, but will not match across root words.

One for ja-jp, one for zh-tw, and nothing at all for en-us. Eventually I asked the question the notes invite: is this a real limitation of Pagefind in Japanese and Chinese, and can it be fixed? I wanted an honest answer rather than a reassuring one, because search is the only way anyone finds an old article here.

The honest answer turned out to be two answers. The warning is permanent, harmless, and about something that was never broken. Underneath it sat a genuine defect that printed nothing at all: searching Japanese compound words returned the wrong article, confidently, and had been doing so since Pagefind 1.5. This article covers why English and CJK go through different machinery, why the warning names the wrong half of it, how the real failure was found by measuring rather than reading, and the one-line patch that fixed it.

Stemming is for English. Segmentation is for Japanese

The two words sound interchangeable and describe opposite problems.

Stemming strips suffixes so that inflected forms collapse onto one root. It is what makes searching repository also match repositories, and it works because English marks grammar by adding letters to the end of a word. Pagefind implements it with Snowball, which ships algorithms for around twenty-five languages.

Segmentation is the reverse problem: finding where the words are at all. Japanese and Chinese are written without spaces, so before anything can be indexed something has to decide that 画像変換 is two words and not one. English never needs this step, because the spaces already did the job.

Snowball has no Chinese, Japanese, or Korean algorithm, and it never will, because suffix-stripping is not a meaningful operation on a language that does not inflect by suffix. So the note is accurate and permanent. It is also irrelevant: nothing in Japanese search depends on the step the note is about.

The segmentation half is handled, and handled well. The pagefind npm package always installs the extended binary, which the build output confirms on its first line:

Running Pagefind v1.5.2 (Extended)

That binary is 55 MB because it carries dictionaries. Pulling the strings out of it shows what is inside:

charabia-0.9.9/src/segmenter/chinese.rs   -> jieba-rs 0.8.1
charabia-0.9.9/src/segmenter/japanese.rs  -> lindera 0.43.3 + lindera-unidic

Jieba for Chinese, Lindera with the UniDic dictionary for Japanese. The capability the warning does not mention was there the whole time.

The measurement the warning would never have prompted

Reading the note leads nowhere, so the agent proposed measuring the search instead: build the site, query each locale’s index directly through the JavaScript API, and compare the result counts against grep counts on the built HTML. If a word appears on eight pages and the search returns eight results, the language works.

Most of Japanese did work. Counting pages under dist/ja/blog against what the index returned:

リポジトリ is not an exotic word. It appears in eight articles on this site, and the single result it returned was an article about the history of language models that does not contain it anywhere.

A search that returns nothing tells you it failed. This one handed back an answer.

The index had the word. The query never asked for it

The excerpt Pagefind returned for リポジトリ explained itself. It had highlighted リレー, an unrelated word that merely looks similar. The engine had failed to match anything and fallen through to a fuzzy guess.

The next test separated the two halves. Searching ガベージ, a prefix of the compound ガベージコレクション, returned the correct article with the full compound highlighted in the snippet. Searching the complete word ガベージコレクション returned a different article entirely, matched on コレクション.

A shorter query found what the longer, more precise query could not. That inverts the usual relationship, and it puts the fault on the query side, because the index demonstrably held the whole compound.

Asking the browser what it does to those queries gave the mechanism in one line:

[...new Intl.Segmenter('ja-JP', { granularity: 'word' }).segment('ガベージコレクション')]
  .filter((s) => s.isWordLike)
  .map((s) => s.segment);
// ['ガ', 'ベ', 'ージ', 'コレクション']

// リポジトリ  -> ['リ', 'ポジ', 'トリ']
// ドキュメント -> ['ドキュメント']
// 画像        -> ['画像']

Pagefind 1.5.0 began segmenting queries in the browser with Intl.Segmenter, on the reasonable-sounding grounds that a query should be cut the same way the content was. But the two sides do not use the same dictionary. The index is cut by Lindera and UniDic inside the Rust binary. The query is cut by whatever ICU dictionary the reader’s browser happens to ship. Where those two disagree, the query loses.

That is the defect. The fragments ガ, ベ and ージ are not indexed words, Pagefind requires every query term to match, so the article containing the compound is eliminated. What survives is whatever the fuzzy fallback finds for the one fragment that is a real word.

The correlation held across every word tested. Where Intl.Segmenter kept the word intact, search was exact. Where it shredded the word, search was wrong.

Typing the word is what breaks it

Because prefixes work and complete words do not, the failure has an unusually cruel shape. Searching each prefix of ガベージコレクション in turn, and checking whether the one genuinely matching article is in the results:

The reader is punished for being precise. This is also why the bug survived so long unnoticed: anyone half-typing a query got a working search, and nothing in any log ever mentioned it.

Chinese, meanwhile, was fine. ICU cuts 儲存庫 into 儲存 and 庫, which looks like the same problem, but Jieba splits Traditional Chinese into short units too, so the two sides mostly agree. Ten zh-TW queries returned identical counts before and after everything that follows.

Downgrading to 1.4 fixed search and broke the search box

Since 1.5.0 introduced the query segmentation, 1.4.0 should not have the bug. Re-indexing the same dist/ with pnpm dlx pagefind@1.4.0 and running the same queries confirmed it:

So the regression was real and pinning was tempting. It was also unusable. This site’s search UI is the pagefind-searchbox web component, and 1.4.0 ships a version of it that calls a function its own pagefind.js does not define:

TypeError: s.createInstance is not a function

That is not a stale bundle being served. The agent re-fetched every file with cache: 'reload' and got the same error. Downgrading would have meant rewriting both search entry points against the older PagefindUI class API, which is a large change to buy back one dictionary.

Two other approaches failed before the working one. Quoting the query as "リポジトリ" does not bypass segmentation; it returned zero results, because the quoted phrase is still assembled from the same shredded fragments. And patching the segmentation list inside pagefind.js produced output byte-identical to stock, which read as the patch silently failing. It had not failed. Search runs in a Web Worker since 1.5.0, and pagefind-worker.js carries its own copy of the language list. Patching one file and not the other changes nothing a user can see.

Removing one language from one list

The guard itself is three lines of minified JavaScript:

needsWordSegmentation = (lang) => {
  if (!lang) return false;
  const primaryLang = lang.split('-')[0].toLowerCase();
  return ['zh', 'ja', 'th'].includes(primaryLang);
};

There is no configuration option for it. --force-language is not an alternative either, since it collapses the per-<html lang> index split that keeps /ja/search from returning English articles.

So the fix is to rewrite that list in the built bundle, in both files, and only for Japanese. I chose to ship this rather than wait for upstream, and scripts/patch-pagefind-ja.ts now runs after every pagefind invocation as part of pnpm build.

Editing build output is a workaround, and workarounds rot quietly. The script is written to fail loudly instead:

• If the list is missing, the build fails with a message naming the cause. Shipping a silently unpatched bundle would restore the exact bug this exists to fix.
• If Pagefind is not the version this was measured against, it warns.
• If the list already reads ['zh', 'th'] on an unverified version, that is fatal, because it might be upstream’s own list rather than an earlier run of the script, in which case the step would exit 0 having done nothing.

One further change came out of code review, and it is the kind of thing that only shows up when someone looks at the whole pipeline rather than the diff. pagefind was on a caret range, Dependabot auto-merges here, and this repository has no required status checks. A Pagefind release that reshaped its minified output would therefore have merged itself and then blocked every production deploy, including article publishes that have nothing to do with search. The agent proposed pinning to an exact version, which turns that into a bump someone has to look at. It also proposed making any version mismatch fatal, and I left that one out for the same reason: it would cause the outage rather than prevent it.

What the numbers say now

English still stems, Chinese still segments, and Japanese now matches the way its own index was built. The last row is the one worth keeping an eye on: it confirms the patch did not reach past the language it was aimed at.

The build still prints both stemming notes on every run, and prints each of them twice, which is a cosmetic quirk of Pagefind’s own output. It always will, and that is correct.

Summary

• A warning that prints on every build is not evidence of the bug you are hunting. This one was accurate, permanent, and irrelevant. The defect it sat next to logged nothing.
• Stemming and segmentation solve opposite problems. English arrives pre-split and needs endings trimmed; Japanese arrives unbroken and needs cutting. A tool can be missing the first and perfect at the second.
• A search engine that indexes with one dictionary and queries with another will fail on exactly the words its dictionaries disagree about, and the failure mode is a plausible wrong answer rather than an empty result.
• Test a full word, not a prefix. Prefix queries hid this bug completely, and a prefix is what anyone spot-checking a search box actually types.
• Verify a language you cannot read by counting. Comparing grep counts on built HTML against result counts needs no reading ability and would have caught this on day one.
• If you run Pagefind 1.5.x on a Japanese site, check a compound noun before assuming search works. Upstream issue #1237 tracks the same root cause from the opposite direction, where Lindera splits a kanji compound that ICU keeps whole.

References

• Pagefind: Multilingual search, including the list of languages with and without stemming support
• Pagefind 1.5.0 release notes, where query-side Intl.Segmenter segmentation was introduced
• Pagefind issue #1237: index-time Lindera and query-time Intl.Segmenter disagree on Japanese compounds
• MDN: Intl.Segmenter, the browser API Pagefind uses to cut queries
• Snowball, the stemming library behind Pagefind’s supported languages