Pagefind's stemming warning was a red herring — Japanese search broke on segmentation
Pagefind warned about missing stemming for Japanese and Chinese on every build. The real defect was a query and index tokenizer mismatch returning wrong pages.

On this page
Introduction
Every build of this site printed the same two notes, and I had been reading past them for weeks:
Note: Pagefind doesn't support stemming for the language ja-jp.Search will still work, but will not match across root words.One for ja-jp, one for zh-tw, and nothing at all for en-us. Eventually I asked the question the notes invite: is this a real limitation of Pagefind in Japanese and Chinese, and can it be fixed? I wanted an honest answer rather than a reassuring one, because search is the only way anyone finds an old article here.
The honest answer turned out to be two answers. The warning is permanent, harmless, and about something that was never broken.
Underneath it sat a genuine defect that printed nothing at all. In Pagefind 1.5, searching a Japanese compound word returned the wrong article, confidently. I fixed it with a one-line patch to Pagefind’s built bundle. Pagefind 1.6.0 has since fixed the missing results upstream, and the patch now only changes the order they come back in.
Stemming is for English. Segmentation is for Japanese
The two words sound interchangeable and describe opposite problems.
Stemming strips suffixes so that inflected forms collapse onto one root. It is what makes searching repository also match repositories, and it works because English marks grammar by adding letters to the end of a word. Pagefind implements it with Snowball, a stemming library, and lists 33 languages with stemming support.
Segmentation is the reverse problem: finding where the words are at all. Japanese and Chinese are written without spaces, so before anything can be indexed something has to decide that 画像変換 is two words and not one. English never needs this step, because the spaces already did the job.
Snowball has no Chinese, Japanese, or Korean algorithm, and it never will, because suffix-stripping is not a meaningful operation on a language that does not inflect by suffix. So the note is accurate and permanent. It is also irrelevant: nothing in Japanese search depends on the step the note is about.
The segmentation half is handled. The pagefind npm package installs the extended binary, the build that carries the CJK segmenters, and the build output confirms it on its first line:
Running Pagefind v1.5.2 (Extended)On 1.5.2 that binary is 55 MB because it carries dictionaries. Pulling the strings out of it shows what is inside:
charabia-0.9.9/src/segmenter/chinese.rs -> jieba-rs 0.8.1charabia-0.9.9/src/segmenter/japanese.rs -> lindera 0.43.3 + lindera-unidicJieba for Chinese, Lindera with the UniDic dictionary for Japanese. The capability the warning does not mention was there the whole time.
The measurement the warning would never have prompted
Reading the note leads nowhere, so I measured the search instead. I built the site, queried each locale’s index directly through Pagefind’s JavaScript API, and compared the result count for each word against the number of Japanese articles that contain it. If a word appears on eight pages and the search returns eight results, the language works.
On 1.5.2, most of Japanese did work:
| Query | Pages containing it | Results | Correct |
|---|---|---|---|
ドキュメント | 47 | 47 | yes |
ブラウザ | 38 | 38 | yes |
画像 | 21 | 21 | yes |
コンポーネント | 19 | 19 | yes |
リポジトリ | 49 | 1 | no |
リポジトリ is not an exotic word. It appears in 49 articles on this site, and the single result it returned was an article about the history of language models that does not contain it anywhere.
A search that returns nothing tells you it failed. This one handed back an answer.
The index had the word. The query never asked for it
The excerpt Pagefind returned for リポジトリ explained itself. It had highlighted リレー, an unrelated word that merely looks similar. The engine had failed to match anything and fallen through to a fuzzy guess.
The next test separated the two halves. Searching ガベージ, a prefix of the compound ガベージコレクション, returned both articles that contain the compound. Searching the complete word returned neither.
A shorter query found what the longer, more precise query could not. That inverts the usual relationship. It puts the fault on the query side, because the index demonstrably held the whole compound.
Asking the browser what it does to those queries gave the mechanism in one line:
[...new Intl.Segmenter('ja-JP', { granularity: 'word' }).segment('ガベージコレクション')] .filter((s) => s.isWordLike) .map((s) => s.segment);// ['ガ', 'ベ', 'ージ', 'コレクション']
// リポジトリ -> ['リ', 'ポジ', 'トリ']// ドキュメント -> ['ドキュメント']// 画像 -> ['画像']Pagefind 1.5.0 began segmenting queries in the browser with Intl.Segmenter, the browser’s built-in word splitter. The 1.5.0 release notes gave the reason: a query should be cut the same way the content was. But on 1.5 the two sides do not use the same dictionary.
The index is cut by Lindera and UniDic inside the Rust binary. The query is cut by whatever ICU dictionary the reader’s browser ships. Where those two disagree, the query loses.
That is the defect. The fragments ガ, ベ and ージ are not indexed words, and Pagefind requires every query term to match, so the articles containing the compound are eliminated. What survives is whatever the fuzzy fallback finds, if anything.
The correlation held across every word tested. Where Intl.Segmenter kept the word intact, search was exact. Where it shredded the word, search was wrong.
Typing the word is what breaks it
Because prefixes work and complete words do not, the failure has an unusually cruel shape. I searched each prefix of ガベージコレクション in turn and checked whether the two articles that contain it were in the results:
The reader is punished for being precise. This is also why the bug survived so long unnoticed: anyone half-typing a query got a working search, and nothing in any log ever mentioned it.
Chinese failed too, in a way I did not catch at first. ICU cuts 儲存庫 into 儲存 and 庫, while Jieba splits Traditional Chinese into short units of its own. On a small corpus the two mostly agreed. On today’s corpus, 搜尋 appears in 35 zh-TW articles and 1.5.2 returned one result, and 環境變數 appears in 10 and returned none.
Downgrading to 1.4 fixed search and broke the search box
Since 1.5.0 introduced the query segmentation, 1.4.0 should not have the bug. I re-indexed the same build with pnpm dlx pagefind@1.4.0 and ran the same queries:
| Query | 1.4.0 | 1.5.2 |
|---|---|---|
リポジトリ | 49, all correct | 1, wrong page |
ガベージコレクション | 2, both correct | 0 |
ドキュメント | 47 | 47 |
So the regression was real and pinning was tempting. It was also unusable. This site’s search UI is the pagefind-searchbox web component, and 1.4.0 ships a version of it that calls a function its own pagefind.js does not define:
TypeError: s.createInstance is not a functionThat is not a stale bundle being served. I re-fetched every file with cache: 'reload' and got the same error.
Two other approaches failed before the working one. Quoting the query as "リポジトリ" does not bypass segmentation. It returned zero results, because the quoted phrase is still assembled from the same shredded fragments.
Patching the segmentation list inside pagefind.js produced output byte-identical to stock, which read as the patch silently failing. It had not failed. Search runs in a Web Worker since 1.5.0, and pagefind-worker.js carries its own copy of the language list. Patching one file and not the other changes nothing a user can see.
Removing one language from one list
The guard itself is three lines of minified JavaScript:
needsWordSegmentation = (lang) => { if (!lang) return false; const primaryLang = lang.split('-')[0].toLowerCase(); return ['zh', 'ja', 'th'].includes(primaryLang);};There is no configuration option for it. --force-language is not an alternative either, since it indexes the whole site as one language and collapses the per-<html lang> index split that keeps /ja/search from returning English articles.
So the fix is to rewrite that list in the built bundle, in both files, and only for Japanese. I chose to ship this rather than wait for upstream, and scripts/patch-pagefind-ja.ts now runs after every pagefind invocation as part of pnpm build.
Editing build output is a workaround, and workarounds rot quietly. The script is written to fail loudly instead:
- If the list is missing, the build fails with a message naming the cause. Shipping a silently unpatched bundle would restore the exact bug this exists to fix.
- If Pagefind is not the version this was measured against, it warns.
- If the list already reads
['zh', 'th']on an unverified version, that is fatal. The list might be upstream’s own rather than an earlier run of the script, in which case the step would exit 0 having done nothing.
I also pinned pagefind to an exact version. It had been on a caret range, Dependabot auto-merges here, and this repository has no required status checks. A Pagefind release that reshaped its minified output would have merged itself and then blocked every production deploy, including article publishes that have nothing to do with search.
I did not make a version mismatch fatal, for the same reason: a fatal mismatch would cause that outage rather than prevent it.
What the patch changed on 1.5.2
| Query | Stock 1.5.2 | Patched 1.5.2 |
|---|---|---|
リポジトリ | 1, wrong page | 49, all correct |
ガベージコレクション | 0 | 2, both correct |
ドキュメント / アーキテクチャ / コンポーネント | 47 / 9 / 19 | unchanged |
ワークツリー | 3 of 3 found | 1 of 3 found |
搜尋 / 環境變數 (zh-TW) | 1 / 0 | 1 / 0 |
repository / repositories | 38 / 38 | 38 / 38 |
Chinese is untouched, so its failures stayed, and English still stems. The ワークツリー row is the cost: Lindera indexed that compound in pieces, so passing it through whole missed two of its three pages.
Pagefind 1.6.0 indexes with ICU4X, and the patch now only reorders results
Pagefind 1.6.0 replaced Lindera and Jieba with ICU4X, a Rust rewrite of the ICU libraries that Chromium’s Intl.Segmenter is built on. The extended binary dropped from 55 MB to 12 MB, and strings in it now name icu_segmenter-2.3.0.
The 1.6.0 release notes say the index side now “segments a lot closer to the Intl.Segmenter used in the browser”, and the maintainer closed the upstream issue tracking the mismatch (Pagefind/pagefind#1237).
On this corpus, 1.6.0 finds every page in both languages, with or without the patch. Chinese recovered with it:
| zh-TW query | Pages containing it | 1.5.2 | 1.6.0 |
|---|---|---|---|
搜尋 | 35 | 1 | 35 |
環境變數 | 10 | 0 | 10 found, in 16 results |
快取 | 24 | 24 found, in 31 results | 24, exact |
For Japanese, the patch now changes ranking only. In the table, “on top” counts how many of the correct pages land in the first N results, where N is the number of pages containing the word:
| Query | Pages containing it | Stock 1.6.0, on top | Patched 1.6.0, on top |
|---|---|---|---|
ガベージコレクション | 2 | 0 | 2 |
ワークツリー | 3 | 1 | 3 |
型チェック | 9 | 4 | 8 |
環境変数 | 10 | 8 | 3 |
依存関係 | 12 | 8 | 6 |
ブランチ保護 | 2 | 2 | 1 |
The browser still shreds a katakana compound, and stock 1.6.0 ranks pages matching the fragments above the ones holding the whole word. Kanji compounds go the other way: ICU4X now cuts them in the index the way the browser cuts the query, so passing the query through whole is what costs ranking.
I kept the patch, because katakana loanwords are the commoner query on a technical blog. Since 1.6.0 the build also prints the stemming notes only when run with --verbose.
Summary
- A warning that prints on every build is not evidence of the bug you are hunting. This one was accurate, permanent, and irrelevant. The defect it sat next to logged nothing.
- Stemming and segmentation solve opposite problems. English arrives pre-split and needs endings trimmed; Japanese arrives unbroken and needs cutting. A tool can be missing the first and perfect at the second.
- A search engine that indexes with one dictionary and queries with another will fail on exactly the words its dictionaries disagree about. The failure mode is a plausible wrong answer rather than an empty result.
- Test a full word, not a prefix. Prefix queries hid this bug completely, and a prefix is what anyone spot-checking a search box actually types.
- Verify a language you cannot read by counting. Comparing page counts against result counts needs no reading ability, and it caught the Chinese failure that a small corpus had hidden.
- Pagefind 1.6.0 fixed the missing results by indexing with ICU4X. Removing
jafrom the segmentation list now trades better katakana ranking for worse kanji ranking, and I re-measure that trade on every Pagefind bump.
References
- Pagefind: Multilingual search, including the list of languages with and without stemming support
- Pagefind 1.5.0 release notes, where query-side
Intl.Segmentersegmentation was introduced - Pagefind 1.6.0 release notes, where index-side segmentation moved to ICU4X
- Pagefind issue: index-time Lindera and query-time Intl.Segmenter disagree on Japanese compounds (Pagefind/pagefind#1237)
- MDN: Intl.Segmenter, the browser API Pagefind uses to cut queries
- Snowball, the stemming library behind Pagefind’s supported languages


