Pagefind's stemming warning was a red herring — Japanese search broke on segmentation

Pagefind warned about missing stemming for Japanese and Chinese on every build. The real defect was a query and index tokenizer mismatch returning wrong pages.

Updated

The pagefind compass mark, an ellipse enclosing a needle, beside the lowercase pagefind wordmark in white on a near-black card
On this page

Introduction

Every build of this site printed the same two notes, and I had been reading past them for weeks:

Note: Pagefind doesn't support stemming for the language ja-jp.
Search will still work, but will not match across root words.

One for ja-jp, one for zh-tw, and nothing at all for en-us. Eventually I asked the question the notes invite: is this a real limitation of Pagefind in Japanese and Chinese, and can it be fixed? I wanted an honest answer rather than a reassuring one, because search is the only way anyone finds an old article here.

The honest answer turned out to be two answers. The warning is permanent, harmless, and about something that was never broken.

Underneath it sat a genuine defect that printed nothing at all. In Pagefind 1.5, searching a Japanese compound word returned the wrong article, confidently. I fixed it with a one-line patch to Pagefind’s built bundle. Pagefind 1.6.0 has since fixed the missing results upstream, and the patch now only changes the order they come back in.

Stemming is for English. Segmentation is for Japanese

The two words sound interchangeable and describe opposite problems.

Stemming strips suffixes so that inflected forms collapse onto one root. It is what makes searching repository also match repositories, and it works because English marks grammar by adding letters to the end of a word. Pagefind implements it with Snowball, a stemming library, and lists 33 languages with stemming support.

Segmentation is the reverse problem: finding where the words are at all. Japanese and Chinese are written without spaces, so before anything can be indexed something has to decide that 画像変換 is two words and not one. English never needs this step, because the spaces already did the job.

Stemming and segmentation across two languagesTwo columns. English splits on spaces and then stems "repositories" down to "repository". Japanese is cut by a dictionary into four tokens and then reaches a stemming step that is greyed out and dashed, because no algorithm exists for it. The build warning is about that empty cell.Englishgit repositoriesSegmentationby spaces, freeStemmingSnowball, EnglishgitrepositoryJapanese画像を変換するSegmentationLindera + UniDicStemmingnone, and never will be画像を変換するThe build note reports the greyed cell. Japanese search never depended on it.
English arrives pre-split and needs its endings trimmed. Japanese arrives as an unbroken run and needs cutting first. The warning is about the trimming step, which Japanese never uses.

Snowball has no Chinese, Japanese, or Korean algorithm, and it never will, because suffix-stripping is not a meaningful operation on a language that does not inflect by suffix. So the note is accurate and permanent. It is also irrelevant: nothing in Japanese search depends on the step the note is about.

The segmentation half is handled. The pagefind npm package installs the extended binary, the build that carries the CJK segmenters, and the build output confirms it on its first line:

Running Pagefind v1.5.2 (Extended)

On 1.5.2 that binary is 55 MB because it carries dictionaries. Pulling the strings out of it shows what is inside:

charabia-0.9.9/src/segmenter/chinese.rs -> jieba-rs 0.8.1
charabia-0.9.9/src/segmenter/japanese.rs -> lindera 0.43.3 + lindera-unidic

Jieba for Chinese, Lindera with the UniDic dictionary for Japanese. The capability the warning does not mention was there the whole time.

The measurement the warning would never have prompted

Reading the note leads nowhere, so I measured the search instead. I built the site, queried each locale’s index directly through Pagefind’s JavaScript API, and compared the result count for each word against the number of Japanese articles that contain it. If a word appears on eight pages and the search returns eight results, the language works.

On 1.5.2, most of Japanese did work:

QueryPages containing itResultsCorrect
ドキュメント4747yes
ブラウザ3838yes
画像2121yes
コンポーネント1919yes
リポジトリ491no

リポジトリ is not an exotic word. It appears in 49 articles on this site, and the single result it returned was an article about the history of language models that does not contain it anywhere.

A search that returns nothing tells you it failed. This one handed back an answer.

The index had the word. The query never asked for it

The excerpt Pagefind returned for リポジトリ explained itself. It had highlighted リレー, an unrelated word that merely looks similar. The engine had failed to match anything and fallen through to a fuzzy guess.

The next test separated the two halves. Searching ガベージ, a prefix of the compound ガベージコレクション, returned both articles that contain the compound. Searching the complete word returned neither.

A shorter query found what the longer, more precise query could not. That inverts the usual relationship. It puts the fault on the query side, because the index demonstrably held the whole compound.

Asking the browser what it does to those queries gave the mechanism in one line:

[...new Intl.Segmenter('ja-JP', { granularity: 'word' }).segment('ガベージコレクション')]
.filter((s) => s.isWordLike)
.map((s) => s.segment);
// ['ガ', 'ベ', 'ージ', 'コレクション']
// リポジトリ -> ['リ', 'ポジ', 'トリ']
// ドキュメント -> ['ドキュメント']
// 画像 -> ['画像']

Pagefind 1.5.0 began segmenting queries in the browser with Intl.Segmenter, the browser’s built-in word splitter. The 1.5.0 release notes gave the reason: a query should be cut the same way the content was. But on 1.5 the two sides do not use the same dictionary.

The index is cut by Lindera and UniDic inside the Rust binary. The query is cut by whatever ICU dictionary the reader’s browser ships. Where those two disagree, the query loses.

Two dictionaries disagreeing about one compoundOne Japanese compound feeds two branches. On the index side Lindera stores it as a single word. On the query side the browser’s Intl.Segmenter cuts it into four fragments, of which only the last is an indexed word anywhere on the site. Since every term must match, the page holding the compound is eliminated, and the one real fragment pulls back an unrelated page instead.One query, one indexed pageガベージコレクションIndex, at build timeLindera + UniDic, in the Rust binaryガベージコレクションstored as one wordQuery, in the browserIntl.Segmenter, using the browser’s ICUガベージコレクションcut into four, only the last one indexed✗ Not every term matchesEvery term has to match, so the page holding the word is eliminatedand a fuzzy guess is returned in its place.
One compound, two dictionaries. Lindera stored it whole; ICU cut it into four pieces, only the last of which is an indexed word. Every term has to match, so the page holding the compound drops out and that one real fragment pulls back something else.

That is the defect. The fragments ガ, ベ and ージ are not indexed words, and Pagefind requires every query term to match, so the articles containing the compound are eliminated. What survives is whatever the fuzzy fallback finds, if anything.

The correlation held across every word tested. Where Intl.Segmenter kept the word intact, search was exact. Where it shredded the word, search was wrong.

Typing the word is what breaks it

Because prefixes work and complete words do not, the failure has an unusually cruel shape. I searched each prefix of ガベージコレクション in turn and checked whether the two articles that contain it were in the results:

The correct result disappearing as the word is completedTen columns, one per keystroke of a Japanese compound. The row underneath shows how many results came back, and the row below that whether both articles genuinely containing the word were among them. They are present at the first keystroke, absent at the second, present for the six consecutive keystrokes from the third to the eighth, and absent for the final two, where no results come back at all.Characters typed, one keystroke per columnResults returnedBoth correct articles among themガ59✓ベ51✗ー2✓ジ2✓コ2✓レ2✓ク2✓シ2✓ョ0✗ン0✗Stopping early finds the articles. Finishing the word loses them.
Both correct articles are present for six consecutive keystrokes and disappear on the last two. A reader who stops early finds them; a reader who finishes the word does not.

The reader is punished for being precise. This is also why the bug survived so long unnoticed: anyone half-typing a query got a working search, and nothing in any log ever mentioned it.

Chinese failed too, in a way I did not catch at first. ICU cuts 儲存庫 into 儲存 and 庫, while Jieba splits Traditional Chinese into short units of its own. On a small corpus the two mostly agreed. On today’s corpus, 搜尋 appears in 35 zh-TW articles and 1.5.2 returned one result, and 環境變數 appears in 10 and returned none.

Since 1.5.0 introduced the query segmentation, 1.4.0 should not have the bug. I re-indexed the same build with pnpm dlx pagefind@1.4.0 and ran the same queries:

Query1.4.01.5.2
リポジトリ49, all correct1, wrong page
ガベージコレクション2, both correct0
ドキュメント4747

So the regression was real and pinning was tempting. It was also unusable. This site’s search UI is the pagefind-searchbox web component, and 1.4.0 ships a version of it that calls a function its own pagefind.js does not define:

TypeError: s.createInstance is not a function

That is not a stale bundle being served. I re-fetched every file with cache: 'reload' and got the same error.

Two other approaches failed before the working one. Quoting the query as "リポジトリ" does not bypass segmentation. It returned zero results, because the quoted phrase is still assembled from the same shredded fragments.

Patching the segmentation list inside pagefind.js produced output byte-identical to stock, which read as the patch silently failing. It had not failed. Search runs in a Web Worker since 1.5.0, and pagefind-worker.js carries its own copy of the language list. Patching one file and not the other changes nothing a user can see.

Removing one language from one list

The guard itself is three lines of minified JavaScript:

needsWordSegmentation = (lang) => {
if (!lang) return false;
const primaryLang = lang.split('-')[0].toLowerCase();
return ['zh', 'ja', 'th'].includes(primaryLang);
};

There is no configuration option for it. --force-language is not an alternative either, since it indexes the whole site as one language and collapses the per-<html lang> index split that keeps /ja/search from returning English articles.

So the fix is to rewrite that list in the built bundle, in both files, and only for Japanese. I chose to ship this rather than wait for upstream, and scripts/patch-pagefind-ja.ts now runs after every pagefind invocation as part of pnpm build.

Editing build output is a workaround, and workarounds rot quietly. The script is written to fail loudly instead:

  • If the list is missing, the build fails with a message naming the cause. Shipping a silently unpatched bundle would restore the exact bug this exists to fix.
  • If Pagefind is not the version this was measured against, it warns.
  • If the list already reads ['zh', 'th'] on an unverified version, that is fatal. The list might be upstream’s own rather than an earlier run of the script, in which case the step would exit 0 having done nothing.

I also pinned pagefind to an exact version. It had been on a caret range, Dependabot auto-merges here, and this repository has no required status checks. A Pagefind release that reshaped its minified output would have merged itself and then blocked every production deploy, including article publishes that have nothing to do with search.

I did not make a version mismatch fatal, for the same reason: a fatal mismatch would cause that outage rather than prevent it.

What the patch changed on 1.5.2

QueryStock 1.5.2Patched 1.5.2
リポジトリ1, wrong page49, all correct
ガベージコレクション02, both correct
ドキュメント / アーキテクチャ / コンポーネント47 / 9 / 19unchanged
ワークツリー3 of 3 found1 of 3 found
搜尋 / 環境變數 (zh-TW)1 / 01 / 0
repository / repositories38 / 3838 / 38

Chinese is untouched, so its failures stayed, and English still stems. The ワークツリー row is the cost: Lindera indexed that compound in pieces, so passing it through whole missed two of its three pages.

Pagefind 1.6.0 indexes with ICU4X, and the patch now only reorders results

Pagefind 1.6.0 replaced Lindera and Jieba with ICU4X, a Rust rewrite of the ICU libraries that Chromium’s Intl.Segmenter is built on. The extended binary dropped from 55 MB to 12 MB, and strings in it now name icu_segmenter-2.3.0.

The 1.6.0 release notes say the index side now “segments a lot closer to the Intl.Segmenter used in the browser”, and the maintainer closed the upstream issue tracking the mismatch (Pagefind/pagefind#1237).

On this corpus, 1.6.0 finds every page in both languages, with or without the patch. Chinese recovered with it:

zh-TW queryPages containing it1.5.21.6.0
搜尋35135
環境變數10010 found, in 16 results
快取2424 found, in 31 results24, exact

For Japanese, the patch now changes ranking only. In the table, “on top” counts how many of the correct pages land in the first N results, where N is the number of pages containing the word:

QueryPages containing itStock 1.6.0, on topPatched 1.6.0, on top
ガベージコレクション202
ワークツリー313
型チェック948
環境変数1083
依存関係1286
ブランチ保護221

The browser still shreds a katakana compound, and stock 1.6.0 ranks pages matching the fragments above the ones holding the whole word. Kanji compounds go the other way: ICU4X now cuts them in the index the way the browser cuts the query, so passing the query through whole is what costs ranking.

I kept the patch, because katakana loanwords are the commoner query on a technical blog. Since 1.6.0 the build also prints the stemming notes only when run with --verbose.

Summary

  • A warning that prints on every build is not evidence of the bug you are hunting. This one was accurate, permanent, and irrelevant. The defect it sat next to logged nothing.
  • Stemming and segmentation solve opposite problems. English arrives pre-split and needs endings trimmed; Japanese arrives unbroken and needs cutting. A tool can be missing the first and perfect at the second.
  • A search engine that indexes with one dictionary and queries with another will fail on exactly the words its dictionaries disagree about. The failure mode is a plausible wrong answer rather than an empty result.
  • Test a full word, not a prefix. Prefix queries hid this bug completely, and a prefix is what anyone spot-checking a search box actually types.
  • Verify a language you cannot read by counting. Comparing page counts against result counts needs no reading ability, and it caught the Chinese failure that a small corpus had hidden.
  • Pagefind 1.6.0 fixed the missing results by indexing with ICU4X. Removing ja from the segmentation list now trades better katakana ranking for worse kanji ranking, and I re-measure that trade on every Pagefind bump.

References

Share this article