Measuring TypeSafe's Jev before wiring it in — it can tag an article, but not judge a translation

TypeSafe's Jev scored AUC 0.97 and stayed calibrated picking this blog's tags, but only 0.57 to 0.63 judging translated prose, so now it suggests draft tags.

The TypeSafe AI logo, a white geometric outline of interlocking hexagonal blocks on a black rounded tile, beside the black TypeSafe AI wordmark on a white card
On this page

Introduction

I had heard that TypeSafe’s Jev could sit between Claude Code and its tools and pick which ones to use, and I wanted to know whether it had a place in this blog’s pipeline. Jev returns no text, only a probability per question, and the costliest judgment here is an Opus agent reading every translation for sentences that sound translated.

So I measured Jev against labels on two candidate jobs before wiring it into anything. On picking an article’s tags it scored AUC 0.97 and its probabilities held up; on judging translated prose it scored 0.57 to 0.63, barely better than chance.

/blog new now asks Jev to pick a finished draft’s tags, using a threshold read off that measurement rather than chosen by feel, and translation review stays with the reviewer.

What Jev is

Jev takes one piece of state and a set of named, typed questions, and returns one typed answer per question: never a sentence. TypeSafe calls it a System One model. There are three question types:

  1. choice, a pick among fixed options.
  2. score, one level from a low-to-high scale you define, up to 10 levels.
  3. noul, a yes/no question answered with the probability of yes.

The context window is 32,000 tokens, and OpenRouter notes that the probabilities vary somewhat from call to call.

OpenRouter’s own worked example asks all three types about one support ticket in a single request:

Jev's three question typesOn the left, a state card holding a support ticket about a double charge, and a questions card with three questions: team, a choice among three teams; urgency, a score on three levels; and refund, a yes/no noul. Both feed one box labelled Jev. On the right, an answers card: team is billing with probability 1 and confidence 1; urgency is level 2 with probability 1 and confidence 1; refund is a probability of yes of 0.82. No text is returned.StateticketHi, I was charged twice for my Prosubscription this month. Please fix thisbefore my next payroll run on Friday.QuestionschoiceteamWhich team handles it? 3 optionsscoreurgencyHow urgent is it? 3 levels, 0 to 2noulrefundIs it asking for money back?JevAnswerschoiceteam = billingbilling 1 · technical 0 · account 0confidence 1scoreurgency = 2P(0) 0 · P(1) 0 · P(2) 1confidence 1noulrefundP(yes) = 0.82No text: each named question gets one answer of its own type
One state and any number of named questions go in, and each question comes back in its own type: an option for choice, a level for score, a probability of yes for noul. The values are from OpenRouter’s example.

The first request: textbook translationese at 0.56

This is the first request I sent, through OpenRouter’s TypeSafe-compatible endpoint, with one noul asking whether a sentence reads as translated and one meaningless choice as a control:

POST https://openrouter.ai/api/v1/systemone
{
  "model": "~typesafe/jev-latest",
  "state": {
    "language": "Japanese",
    "sentence": "このアプローチを使用することによって、私たちはパフォーマンスを改善することができます。"
  },
  "questions": {
    "a": {
      "type": "noul",
      "instructions": "Does this sentence read as translated from English rather than written natively?",
      "criteria": { "true": "Translationese", "false": "Natural native prose" }
    },
    "b": { "type": "choice", "instructions": "pick", "criteria": { "x": "one", "y": "two" } }
  }
}

The response carried only numbers for each question (probabilities, plus how concentrated they are for the choice) and no text:

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "a": { "type": "noul", "noul": 0.56 },
    "b": { "type": "choice", "choice": "x", "probabilities": { "x": 0.72, "y": 0.28 }, "confidence": 0.44 }
  },
  "usage": { "input_tokens": 404, "output_tokens": 47, "cost": 0.000016968 }
}

A noul returns only the probability of yes. A choice returns a probability for each option (probabilities, summing to 1), the most probable option (choice), and confidence, which says how concentrated those probabilities are: 1 means all of it on one option, 0 means spread evenly across them.

Input costs $0.042 per million tokens and output is free, so the request above cost less than two thousandths of a cent. Several questions about the same state share one request, so later, when picking tags, one call per article covered every tag in the registry.

The sentence in that request is textbook translationese, a stack of 〜することによって and 〜することができます that no native writer would produce, but Jev put the chance that it reads as translated at only 0.56. The meaningless control choice drew a more decided 0.72 against 0.28. On this example, Jev could not tell whether a sentence reads as translated.

Calibration and AUC, checked against labels

A model is calibrated when its numbers mean what they say: across many answers given 0.8, about 80% turn out true. TypeSafe trains Jev for this. Calibration is a claim about a population of answers, so it can only be checked against a population of labeled examples.

Two numbers summarise such a check, and both appear in the results below:

  1. ECE, expected calibration error. Group the answers into ten probability buckets, compare each bucket’s average stated probability with how often its items were actually positive, and average the gaps. 0 is perfect.
How ECE is computedA chart of stated probability against the real rate of yes, with a dashed diagonal for perfect calibration. The horizontal axis is divided into ten bands, and four of them hold a bucket of answers, labelled A to D, each off the diagonal: stated 0.25 with real 0.15, stated 0.35 with real 0.40, stated 0.65 with real 0.45, and stated 0.85 with real 0.90. A red bar joins each point to the diagonal, marking gaps of 0.10, 0.05, 0.20 and 0.05. Their average, 0.10, is the ECE.Real rate of yes000.50.511ABCDStated probability (bucket average)Four buckets with answersbucketstatedrealgapA0.250.150.10B0.350.400.05C0.650.450.20D0.850.900.05Dashed line: perfect calibrationRed bar: the gapECE = average gap = (0.10 + 0.05 + 0.20 + 0.05) ÷ 4 = 0.10
ECE over the ten probability bands, four of which hold answers. Each gap is the distance from a bucket’s point to the diagonal, and with buckets of equal size ECE is their plain average; empty bands drop out. The values are illustrative.
  1. AUC, area under the curve, measures ranking. Sort the items by Jev’s score: AUC is the share of yes–no pairs where the actually-yes item ranks first, so 0.97 means 97%. At 1.0 a cutoff splits yes from no perfectly, at 0.5 no cutoff can, and at 0 the order is exactly reversed.
What AUC measuresTwo number lines from 0 to 1. On each, four items that are actually yes sit above the line as green circles and four that are actually no sit below it as red crosses. On the first line every yes is above 0.7 and every no is below 0.4, and a dashed cutoff at 0.55 separates them: AUC 1.0. On the second line yes and no alternate along the whole range, so no cutoff separates them: AUC 0.5. AUC 0 would be the reverse of the first line: every yes below every no.Ranked well: one cutoff splits them (AUC 1.0)actually yesactually no00.51probability of yes Jev gave each itemcutoffMixed: every cutoff lets some through (AUC 0.5)actually yesactually no00.51probability of yes Jev gave each itemAUC 1.0: every yes above every no · 0.5: mixed · 0: every yes below every no
AUC on two example sets of eight items. When every yes scores above every no, one cutoff splits them; when they are interleaved, none does. The values are illustrative.

Neither number transfers from TypeSafe’s data to anyone else’s. Alex Molas makes the argument that calibration is a property of the data distribution as much as of the model, so the same probabilities can be calibrated on TypeSafe’s training data and wrong on yours. Every number in this article comes from this blog’s own labels.

First candidate: translation fluency review

Every Japanese and Traditional Chinese translation on this blog is read by a translation-native-reviewer agent, which runs on Opus and returns each passage that reads as translated with a replacement for it. Claude Code proposed putting Jev in front of it as a screen: score every line, and send the reviewer only the doubtful ones.

Git history held rule sweeps, not judgments

The obvious labels were the lines I had already rewritten. A translation line replaced in a commit that left the English source alone looked like a fluency judgment, so the first extraction took the pre-edit text of every such line as “rejected” and the untouched lines in the same files as “kept”.

It produced 592 rejected zh-TW lines, and reading them showed what they were. Most were glossary sweeps applied across the corpus, such as 工作流程 becoming workflow. Dropping changes where a term only switched script left 225; dropping changes where the new side was a glossary rendering, such as 智能體 becoming AI 代理, left 34.

Even the remaining 34 were mostly the em-dash budget, the zh-TW spacing rule and terminology. This repository squash-merges, so the draft a reviewer corrected inside a pull request never reaches history, and what history keeps is the rule sweeps that ran later. Japanese had 6 rejected lines in total.

A Jev run against those labels came back at AUC 0.54 for zh-TW. That says nothing about Jev, because the labels were not about fluency.

Labels from a reviewer that reads every line

Claude Code proposed taking labels from the reviewer, and I chose that route with a 10-file pilot. The reviewer reads the target file end to end before it opens the English source, so every line it does not flag has been read and passed. Its kept lines are real negatives, which is the half that git history could never supply.

I ran it on 10 published translations, five Japanese and five Traditional Chinese, from 10 different articles. A small script, fluency-record.ts, parses each report’s NOW: lines and matches them to the file’s prose lines:

Terminal window
IN
pnpm fluency:record --reviewer src/content/blog/<slug>/ja.mdx --report report.txt
OUT
reviewer <slug> ja: 41 prose lines, 11 rejected

The ten reports came to 613 prose lines, 144 rejected and 469 kept. The same step now runs inside /blog publish before any finding is applied, and inside /blog polish after I finish reading a translation line by line, so the label set keeps growing and will eventually hold my judgments beside the reviewer’s.

Fluency labels from git history and from the reviewerTwo columns. Git history, zh-TW: the reviewer fixes the draft inside the pull request, the squash merge keeps only the final text, and history keeps the rule sweeps, 592 lines filtered to 34. Result: no line judged as fine, and none of the reviewer’s fixes. Opus reviewer, 10 translations: it reads each translation start to end and then opens the English, flags 144 lines that read as translated, and every other line, 469, was read and passed. Result: both yes and no are labeled, first-hand.Git history (zh-TW)Reviewer fixes the draft inside the pull requestSquash mergekeeps only the final textHistory keeps the rule sweeps592 lines, 34 after filtering✗No line judged as fineand none of the reviewer’s fixesOpus reviewer (10 translations)Reads the translation, then compares the EnglishFlags lines that read as translated144 rejectedEvery other line was read and passed469 kept✓Both yes and no are labeledand judged first-handUsable labels need both sides, judged by something that read the line
Why git history could not supply fluency labels and the reviewer could: the squash merge left only rule sweeps, while the reviewer judged every line it read.

Jev could not tell translated prose from native

Each line went to Jev on its own, as one noul:

scripts/jev-fluency-calibration.ts
const QUESTION = {
  type: "noul",
  instructions:
    "Would a native technical blogger writing in this language rewrite this passage because it reads as translated from English rather than written natively?",
  criteria: {
    true: "It reads as translated: English word order or sentence structure carried over, calques of English idioms, stiff or unnatural phrasing, or wording a native writer would not choose.",
    false:
      "It reads as written natively in this language. Technical terms, product names and code identifiers left in English do not count against it.",
  },
};

Against the reviewer’s labels:

LinesRejectedAUCECE
Japanese29568 (23%)0.6270.267
Traditional Chinese31876 (24%)0.5730.275

Jev placed all 613 lines between 0.2 and 0.8, nearly all between 0.3 and 0.7. In Japanese the rejection rate does climb, from 16% of the lines in the 30–40% bucket to 41% in the 60–70% bucket. In Traditional Chinese the climb is nearly flat. Stated probabilities averaged around 50% against a real rejection rate under 25%.

Jev cannot take work off the reviewer. Say we want to miss at most 5% of the lines the reviewer would reject. Sort the lines by Jev’s score and move the cutoff down until it covers 95% of those rejections: it ends up at 0.34 in Japanese and 0.40 in Traditional Chinese.

By then most lines sit above it, so only 6% to 7% are left below it to skip review.

Moving the cutoff down until it covers 95% of the yes linesThree copies of the same row of 60 lines, sorted by Jev’s score from high on the left to low on the right, with yes lines as circles and no lines as crosses mixed along the whole row. In the first copy the cutoff at 0.60 covers 8 of the 20 yes lines, 40%; in the second, at 0.50, it covers 14, 70%; in the third, at 0.40, it covers 19, 95%. In the third copy only the last 4 lines, 7%, are below the cutoff.60 lines, sorted by Jev’s scorehighlowcutoff 0.60: 8 of 20 yes covered (40%)cutoff 0.50: 14 of 20 yes covered (70%)cutoff 0.40: 19 of 20 yes covered (95%)below the cutoff: 4 lines (7%) could skip reviewactually yes (the reviewer rejects it)actually noCovering 95% of the yes lines pushes the cutoff so far down that only 7% are left below it
A schematic: 60 lines sorted by Jev’s score, yes and no mixed along the row. The cutoff has to move down until it covers 19 of the 20 yes lines (95%), and by then only 4 lines (7%) are left below it. The 7% matches the Traditional Chinese measurement.

Jev saw each line alone, and that is the likeliest reason. Many of the reviewer’s findings exist only across sentences: a missing connective between two lines, a pronoun whose referent is in the previous sentence, three short sentences that should have been one.

Second candidate: tags, where the labels already existed

Claude Code proposed tagging next, because the labels were free. Every published article already carries tags I chose from a closed registry of 56, so a tag the article has is a positive and a tag it lacks is a negative.

One request per article asks one noul per registry tag, with the title and the first 12,000 characters of the body as the state:

scripts/lib/jev-tags.ts
instructions: `Is "${label}" one of the main subjects of this technical blog article, such that a reader browsing the "${label}" tag would expect to find it?`,

The first article I tried carried git and developer-tooling. Jev returned git at 0.98, developer-tooling at 0.95 and github at 0.46, then nothing above 0.23, in 1.2 seconds for $0.0002. Across all 65 published articles the run cost $0.014.

Across 3,640 article × tag pairs it scored AUC 0.970 with ECE 0.049.

At both ends, where nearly all its answers fell, the numbers meant what they said: 89% of the pairs above 0.9 were real tags, and under 0.5% of the 2,884 pairs below 0.1 were.

Calibration of Jev on tags and on translation fluencyTwo charts of stated probability against the real rate, with a dashed diagonal for perfect calibration. On tags, the points start near zero and end near 0.95, close to the diagonal at both ends. On fluency, every point for both Japanese and Traditional Chinese sits between stated 0.27 and 0.73, with real rates between 0 and 0.41, a nearly flat line well below the diagonal.Tags (65 articles × 56 tags)000.50.511Real rateJev’s stated probabilityTranslation fluency (613 lines)000.50.511Real rateJev’s stated probabilityJapaneseTraditional Chinese
Stated probability against the real rate, per ten-point bucket. Tag answers sit near the diagonal at both ends, where most of them fall; fluency answers stay between 27% and 73% and well below it.

My own tagging could have been wrong too, so I checked the 30 pairs where Jev and I disagreed most and fixed 11 tags across 10 articles. Re-scored against the corrected tags, AUC rose to 0.977 and the share of real tags above 0.9 to 95%.

Tag suggestions in /blog new

/blog new now leaves the tags to pnpm tags:suggest <slug> instead of the interview. Once the draft is written, the command scores it against every registry tag and offers each tag at 0.5 or above for me to accept or refuse. Jev only scores existing tags, so the skill then asks whether the article needs a new one.

Tag suggestions in /blog newFive steps left to right: the interview settles the title and slug but no tags; the draft is written and revised; tags:suggest scores every registry tag against it and offers those at 0.5 or above; the owner decides on each and adds a tag the registry lacks if needed; en.mdx gets only the approved tags. Without a key the command prints skipped and the tags are picked by hand.Interviewtitle and slugno tags yetDraftbody writtenand revisedtags:suggestscores every tagoffers p ≥ 0.5Ownerdecides on eachadds a missing tagen.mdxgets the tagsthe owner approvedNo key: tags:suggest prints “skipped”, and the tags are picked by hand
The interview leaves the tags empty; Jev scores the finished draft, and only the tags I approve reach the file.

The threshold comes off the tag measurement: of all the pairs at 0.5 or above, two thirds were real tags, rising to 95% above 0.9. The command also flags any existing tag below 0.16, where only 3% of pairs were real tags, useful when re-checking a published article. On the newest published article, already tagged, it printed:

Terminal window
IN
pnpm tags:suggest terraform-rebuild-console-aws-account-prod-spec-parity-oidc
OUT
terraform-rebuild-console-aws-account-prod-spec-parity-oidc: 3 tag(s) now: terraform, aws, github-actions
  0.98  terraform
  0.97  aws
  0.28  github-actions


Jev would add (p ≥ 0.5):
  0.79  opensearch
  0.76  bedrock
  0.75  automation
  0.61  developer-tooling
  0.57  agentcore
  0.54  generative-ai


Jev doubts (p < 0.16):
  nothing

A suggestion at 0.54 sits in the 0.5–0.6 bucket, where only 30% of pairs were real tags, which is why each one is a question rather than an edit. Without an OPENROUTER_API_KEY the command prints skipped and exits 0, and the tags are picked from the registry by hand instead.

Summary

Jev answers typed questions about one input with probabilities and no text, which makes it cheap enough to ask every tag in a registry about every article in one call each. Whether those probabilities mean anything depends on the job, and the only way to find out was to label examples and compare.

On a narrow question whose answer is in the input, picking an article’s tags, it ranked at AUC 0.977 and stayed calibrated, so it now suggests tags in /blog new with thresholds read off that measurement.

On an open-ended judgment, whether a sentence reads as translated, it ranked at 0.57 to 0.63 and gave nearly every line the same middling number, so translation review stays with the reviewer. The hardest part of both measurements was the labels, and git history could not provide them.

References

Share this article

More

More