How we use Jev for two jobs in AI code review
Jev, a decision model, answers with numbers, not text. We use it for two jobs, and our measurements give one rule: it compares, it does not judge.
Summary. A decision model is an AI model that writes no text and answers typed questions with a number. diffox ultra uses TypeSafe’s Jev model in the fourth stage of a review, when it decides which findings to post. In September 2026 we measured where it helps and where it does not; this post covers what we found and the lessons we drew from it.
- Two jobs. Before selection, it answers ten questions about each candidate finding. After selection, it spots the findings that share a root cause and keeps only one of them.
- Where it helps: comparing. When we asked whether two findings describe the same bug, 46 of the 47 pairs it flagged really were the same bug.
- Where it does not help: judging. On “is this finding real?”, it separated real findings from the rest no better than the confidence score of the model that wrote them.
- The rule. Ask a decision model to choose, not to judge: “are these two the same”, “which one”, “what level”.
How a decision model answers
The first three posts covered diffox’s results, its price and the four stages of a review. This one is about a model that runs in one of those stages and does not write a single sentence.
Ask a language model a question and you get text back. A decision model instead takes a state and typed questions, and answers each one with a single number: a probability of yes, a distribution over options or a level on a scale. It answers all the questions in a request together. TypeSafe’s Jev is such a model; TypeSafe calls it a “System One” model.
Because the answer is not text, two things get easier: deciding with a threshold, and combining the answer with other signals. One thing gets harder: the model answers only the question you asked and never tells you the question was wrong. So we had to measure to find out which questions are worth asking.
What Jev does in diffox ultra
A review goes through four stages: context, specialist review, evidence checks, and selection and publishing. Jev runs only in diffox ultra and only in the fourth stage. The evidence checks are still plain code; diffox, the main product, uses no decision model.
The first job comes before selection. In diffox ultra, two separate models read the change. For every candidate finding from those two reviews, the decision model gets two rounds of questions, and its ten answers go into the selection score alongside the finding’s own signals. A finding whose score clears the threshold is posted. On its own, the decision model neither posts a finding nor removes one; its answers are one of the score’s inputs.
The second job comes after selection. The findings about to be posted go to the decision model in pairs: “do these two describe the same root cause?” If the answer is 0.5 or higher, the later finding is dropped and the higher-scoring one stays. A candidate that was waiting for the freed place takes it if it passes the same test. In the evidence checks, Second Witness merges the same objection made at the same place; the decision model finds findings that sit on different lines but lead to the same bug.
If the decision model does not answer, the review carries on: selection runs without its answers, the duplicate check is skipped, and the review’s record notes it.
What we ask the decision model
The questions in the first job do not ask whether a finding is correct. They ask what kind of claim it makes and what that claim rests on. For every candidate finding, the decision model gets four things: the pull request’s title, the agent that wrote the finding along with that agent’s brief, the change in the finding’s file (at most 15,000 characters), and the finding’s text.
| Question | What it looks at | Answer |
|---|---|---|
| On the main path? | Does the bug happen for ordinary, valid input, or only when something else fails, on hostile input or under an unusual setting? | probability of yes |
| A precaution? | Does the finding report a wrong result, or suggest a precaution: a validation, a defensive check, a different error type? | probability of yes |
| A visible effect? | Left unfixed, would a user or a caller see any difference? Dead code, an unused import or outdated docs make none. | probability of yes |
| Rests on the unseen? | Does the claim lean on code, a caller or a setting that is not shown here? | probability of yes |
| Caused by this change? | Does the problem come from lines this diff adds or changes, or was it there before? | probability of yes |
| Would a reviewer comment? | Would an experienced reviewer comment on this point? The scale runs from not mentioning it to blocking the merge. | a level from 1 to 5 |
| Contrasted with a sibling? | Does the finding rest on sibling code in the same change that does the same thing differently and shows the intended form? | probability of yes |
| A small fix? | Would changing one name, value, operator or condition make the code correct? | probability of yes |
| Hedged? | Does the claim say “may, might”, or does it state definitely what goes wrong? | probability of yes |
| Deliberate? | Judging by the pull request’s title and the shape of the diff, is the behaviour the finding objects to the purpose of this change: something removed on purpose, a new limit? | probability of yes |
Nine of the ten are yes-or-no questions, answered with a probability of yes. The remaining one, the reviewer question, asks for a level from 1 to 5. The ten answers go into the selection score together with the finding’s own signals: its severity, its kind, the result of the evidence checks, and whether other findings point at the same place. The score then decides which findings are posted. Each agent is also asked a question or two about its own subject, but only these ten answers count toward the score.
What these questions have in common is that the answer is already in front of the model. Whether a bug is on the main path is spelled out in the finding’s scenario; whether a claim is hedged shows in the sentence itself. We do not ask the decision model to judge the finding again, only to read the text in front of it and sort it into a box.
Where the decision model helps
In September 2026 we measured where the decision model helps. The data was findings that four different review setups produced on pull requests in open-source projects, each matched against bugs that people had marked beforehand. The clearest result came on one question: do two findings describe the same bug? It is the question that keeps a bug from getting a second comment.
We asked “same root cause?” about 374 pairs of findings, of which 58 really described the same bug. At a threshold of 0.5, the decision model flagged 47 pairs and got 46 right. Its separation score was 0.99; a rule that only asks “same file?” scores 0.73. The separation score is the chance that a randomly chosen same-bug pair ranks above a randomly chosen different-bug pair: 0.5 is a coin toss, 1 is perfect.
The same kind of question helped elsewhere too: placing a finding’s severity on one of four levels. Exact agreement with the severity people assigned was 49.5% for the engine’s own call and 56.8% with the decision model’s correction (222 findings). We measured this, but today it is switched on in neither product.
Where it does not help
The same model was weak on “is this finding real?”. Of 656 findings, 315 matched a known bug. The decision model’s separation score was 0.64, against 0.65 for the confidence score of the model that wrote the finding. In other words, it was no better than a signal we already had.
Narrowing the question did not change that: twelve narrow questions and a learned model combining them lifted the separation score only to 0.67.
The difference lies in the question itself. When the model compares two texts, all the evidence is in front of it. Whether code really behaves a certain way is written nowhere in that text; to know it, you have to read the code and build a scenario. In diffox that is the specialist agents’ job, and the claim is proved by a quote from the file.
Which tool for which job?
Four kinds of tool run in a review, and each is good at a different job.
| Tool | Produces | Same input, same answer | Its job in diffox |
|---|---|---|---|
| Plain code | a definite yes or no | always | evidence checks, rule-based detectors |
| Decision model | a probability, a choice or a level | nearly: asked again, it moved by at most 0.07 | in diffox ultra, ten questions and the duplicate check |
| Language model (LLM) | text | no | specialist agents; titles and fix suggestions |
| Learned selector | a score | always | which findings are posted |
The division of labour follows from the table. A question that a fixed rule can answer goes to code. Comparing two things side by side, or picking one of a few options, goes to the decision model. Reading code, building a scenario and explaining it to a reader goes to the language model. Turning all these signals into a single “post it or not” decision goes to the learned selector.
Seven lessons for using a decision model
What our own experiments taught us while bringing a decision model into the product:
- Ask it to choose, not to judge. The answer to “are these two the same”, “what level”, “which one” is in front of it; the answer to “is this right” is not.
- Send questions that share a state in one request. An extra question barely slows the answer. One of our stages did not do this: it sent one request per finding, with the same diff every time. Switching to a single request cut the state it sent by 55%.
- Pin the version. Thresholds are measured on one version’s probabilities. diffox ultra runs one fixed version, not a “latest” alias; when the version changes, the threshold is measured again.
- Before adding a signal, compare it with what you have. A new signal is worth what it adds to the ones you already have. Our “is it real” question added nothing to the finding’s own confidence score.
- Make it an input, not a gate. In diffox ultra the decision model’s answers neither remove nor post a finding on their own; they feed the selection score. The one thing it removes is a second comment on the same bug.
- Choose the threshold with the reader in mind. The duplicate check uses a threshold of 0.5: 46 of the 47 pairs it flags are right. At 0.9 it would make no mistakes at all, but it would find only 36% of the duplicates.
- If no answer comes, let the review go on without it. And have it say so in the record: a missing stage is better than one skipped in silence.
Frequently asked questions
What is a decision model?
An AI model that writes no text. You give it a state and typed questions, and it answers each one with a number: a probability of yes, a distribution over options or a level on a scale. It answers all the questions in a request together. TypeSafe’s Jev is such a model; diffox ultra uses it in the selection stage of a review.
How is a decision model different from LLM-as-a-judge?
With LLM-as-a-judge, a language model reads an output and writes its verdict as text, often with its reasoning. A decision model writes no text: it returns a probability of yes, a choice or a level for each question, and answers all the questions in a request together. That makes its answer easy to compare with a threshold and to combine with other signals, which is how diffox ultra uses it.
Does diffox use a decision model too?
No. The decision model runs only in diffox ultra. In diffox, the findings’ own signals decide which ones are posted, and plain code merges the same objection made at the same place.
Does the decision model see my code?
For every candidate finding it sees the finding’s text, the change in the finding’s file (at most 15,000 characters), the pull request’s title and the brief of the agent that wrote the finding. This goes to TypeSafe, which serves the model, directly or through OpenRouter; our privacy page lists the services used.
Can the decision model remove a finding?
Not on its own. Its answers feed the selection score, and the score decides which findings are posted. The one thing it removes is a second finding with the same root cause; the higher-scoring one is posted.
Why not ask the decision model whether a finding is real?
We measured it: on that question it separated real findings from the rest no better than the confidence score of the model that wrote them. Knowing whether code really behaves a certain way means reading the code and building a scenario; in diffox the specialist agents do that work, and the claim is proved by a quote from the file.
Does it give the same answer to the same question every time?
Nearly. Asked the same questions again, its answers moved by at most 0.07. That is why we measure thresholds on one version’s probabilities and pin the version.