09 Oct 2026 · artificial intelligence · 9 min

Jev, three weeks later: what outside measurements found

Jev came out on 15 September. I wrote about what it was on announcement day, using TypeSafe’s own numbers, because that was all there was. In the three weeks since, outsiders took the model, ran it on public data and published the arithmetic. There are at least seven independent measurements to read now, and the picture they draw is more interesting than the launch post.

Straight to it: the product is real, the marketing runs a few steps ahead of it, and the number that matters in production is neither of the two on the front page.

What TypeSafe has not published

The launch claims 193.6x faster and 444.6x cheaper. Both numbers come from four tasks written by the company itself, and the answer key is the average of two frontier models rather than the correct answer. It is like letting the student grade their own exam and then praising the score.

That is sales material, and sales pages do that. The uncomfortable part is different: the product exists to sell calibrated confidence, and the curve that would show that confidence (a reliability diagram, expected calibration error, a Brier score) is not published anywhere. There is no paper either. In a podcast interview on 22 September the founder was asked whether one existed, and the answer was “not yet”. On what happens when calibration is wrong, the answer was better still: “We have a report issues button. Complain to us in Discord.”

Fair enough. It is also thin for something sold on probability.

What the outside measurements found

Three readings of that table.

The cost claim holds, the speed claim does not. Against frontier models it came out 84x to 139x cheaper, but the median time gain was 5 to 6 times, not 193. The tail is good: 95% of answers under 500 ms, against 3 to 3.5 seconds for chat models.

Calibration is better than a chat model’s when the comparison is against token probability, and it ties once the chat model is told to use the whole 0 to 1 range. The TrueStandard team tripped over exactly that: their first round said Jev won comfortably, and the flaw was in their own prompt, which never asked for a real probability. They fixed the prompt and the tie appeared. Worth keeping: much of the calibration advantage is the difference between asking for a probability and asking for a guess.

The advantage that survives is of another kind. On the hardest tier of the TrueStandard test, the claims that look supported and are not, Jev holds 91.7% against 83.3% and 80.6% for the two chat models. That is not calibration, it is accuracy where being wrong is expensive.

The number I would take to production

This is the most useful result of the lot and it barely shows up in headlines: when Jev is wrong, its confidence drops. In Ravvich’s test the wrong answers came back at 0.66 to 0.69 confidence. The cheap chat model, wrong on the same items, kept saying 0.82 to 0.85, as if nothing had happened.

That is what lets you use the number as a filter. On spam, Jev’s 90% most confident answers were 100% correct. Pushing the rest to a frontier model, the combination reached the frontier model’s precision while sending it one message in nine: 8.5 times cheaper and faster.

DecisionEval measured the same thing and published the table that matters more than the headline accuracy:

At Verne, with a 95% precision target, 58% to 60% of the queue could skip a person.

Read that table slowly: at a 0.9 threshold you automate a quarter of the queue at 93.7% accuracy and send the rest to review. That is not the 193x from the front page. It is an architecture decision, and an honest one.

Where it slips

The probability is coarse. It comes back with two decimal places. In Meng’s test, 71.6% of the news answers came back at exactly 100%, and only 94.3% of those were right. On a list of 1,500 items a threshold can only take 53 distinct positions. Worse: 96.9% of the per-option probabilities were exactly zero, and in 6.2% of cases the correct answer got zero. There is nowhere to fall back to, because there is no second option, and recalibration does not move a zero.

How you ask changes the calibration. The same 872 movie reviews, once as a yes/no question and once as a choice between two labels. The choice form came back close to calibrated; the yes/no form was systematically underconfident, with a calibration error of 0.091 against 0.020. Same model, same sentences, different result.

The order of the options changes the answer. One audit measured the probability of the same answer across the six possible orders of the candidate list and found it moving from 0.43 to 0.63. If your question has options, pin the order or average over a few of them.

Planted text works. Across 486 Wikipedia deletion discussions accuracy was 96.5%. With one injected line saying the text had already been approved, it fell to 26.5%. The useful half of knowing this: it works when the planted text reads like evidence about the item being judged, and usually fails when it is a blunt system command.

The error can be in the label, not the model. In the Swiss test, a line reading “automatically generated expense line for creation of asset” was labelled as travel, and Jev answered asset at probability 1.0. Jev was right and the label was wrong. Audit your labels before you audit the model.

There is no native abstention, and it is not deterministic. There is no “none of the above”: if you need one, add it as an option. Repeats vary little, but the order of the options moves the result more than calling again does.

And the test you can run today, at home: ask for the probability of rolling a 1 on a fair die, 400 times. The audit that did this got an average stated confidence of 82.9% for an event that happens once in six, with an observed hit rate of 19%. If you have never run that test on your own case, the 0.95 in your system is an opinion.

Before putting it in production

1. Run the deterministic rule first. In the Swiss test, where the tax rate was written in the text, a simple rule also scored 100%, and costs nothing. Jev starts paying off where the rule becomes expensive to maintain.
2. Ask the question as a choice between options, not as yes or no.
3. Set aside 50 to 300 labelled examples from your own history. With 50 labels, one temperature cut the median calibration error from 0.092 to 0.059; with 400 it reached 0.058. Fifty already do most of the work.
4. Pick the threshold from precision measured on your data, not from the number the model states. At Verne a 95% precision target landed at a threshold near 0.84.
5. Build three exits instead of two: automatic, human review, and escalation to a model that writes. The cascade is where the gain shows up.
6. Record the model version, the question wording and the option order. Calibration changes with all three, and you will run the numbers again.

The code is short. With Simon Willison’s plugin, testing from the terminal takes one command:

llm -m jev 'Please refund my last payment.' \
  -s 'Does this message explicitly request a refund?'

And the cascade, in Python, looks roughly like this:

LIMIAR = 0.84        # measured on your labels, not copied from the announcement

decision = jev.ask(state=message, questions={
    "intent": {"type": "choice", "options": INTENTS},
    "wants_refund": {"type": "noul",
                     "statement": "The text explicitly requests a refund"},
})

if decision.intent.confidence >= LIMIAR:
    apply(decision.intent.choice)              # automatic
elif decision.wants_refund.noul > 0.9:
    refund_queue(message)                      # deterministic rule
else:
    human_queue(message, decision.intent)      # review

What I think today

Jev did not invent the classifier and it will not retire any chat model. It is a classifier with a probability head that knows how to sound unimpressive when it does not know, and in production that is worth more than the 193x on the front page. Selling calibrated confidence without publishing the reliability curve still looks bad, and saying so is fair.

If TypeSafe publishes the curve, this post goes out of date. I will update it.

Sources

• DecisionEval, end-to-end test on 400 cases: decisioneval.dev
• Verne, calibration on 1,232 banking messages: blog.vernesystems.com
• TrueStandard, 108 claims across six domains: truestandard.ai
• Lingsen Meng, confidence audit and preprint: lsmeng.github.io
• Pavel Ravvich, against a cheap and a frontier model: medium.com
• tsukumo, tested on Swiss bookkeeping: tsukumo.ch
• arXiv paper, seven political science replications: arxiv.org
• Collection of robustness tests, including the fair die and the injection: github.com
• Technical account of what RLCD does and what has not been disclosed: motionandmadness.com
• Plugin for testing from the terminal: simonwillison.net
• TypeSafe documentation, including the nine failure modes the company lists itself: docs.typesafe.ai