What is LLM-as-a-judge, and when can you trust the judge?

What is LLM-as-a-judge?

A language model’s output often can’t be compared with a single right answer: two comments that describe the same bug can use completely different words. LLM-as-a-judge fills that gap: you hand the evaluation to another model and give it an instruction (a rubric). In this post we call that model the judge model, or simply the judge.

The method was popularised by a 2023 paper by Zheng and colleagues. They showed that strong judge models agree with human preferences over 80% of the time, the same level at which humans agree with each other. Today most benchmarks rely on a judge model. So does ReviewBench, which GitHub published for code review: it has a language model grade findings with a single rubric, and writes that it had senior engineers relabel its judgments and measured 96.6% agreement.

A judge model isn’t the first choice. Anthropic’s evaluation documentation gives the order plainly: grade with code first, because that is the fastest and most reliable way; then use a judge model for judgments that can’t be written as a rule; and test the judge’s reliability before you scale it up.

Source: Anthropic’s evaluation documentation, 18 October 2026.

Judge models have known biases

The same paper also lists the limits of judge models: position bias, verbosity bias, self-enhancement bias and limited reasoning. When you set up a judge, take a precaution against each one.

Self-enhancement bias is the easiest to prevent: the model that evaluates should be different from the model that produced the output. Anthropic’s example code notes this as good practice. Position bias shows up in questions that compare two answers; swap the order of the answers and ask twice, and its effect becomes visible. The most effective guard against verbosity bias is to ask the judge not for a score but for an answer that rests on a single fact.

Ask the judge whether two things match, not whether one is good

The question decides most of how reliable a judge is. “Is this review comment good? Score it from 1 to 10” has no right answer; two judges, or even two runs of the same judge, give different scores. Anthropic’s advice points the same way: have the judge say only “correct” or “incorrect”, or pick a level from 1 to 5, because purely qualitative evaluations are hard to assess quickly and at scale.

Our judge’s question is a comparison. It is given a review comment and a known bug in the same pull request, and asked one thing. The core of the rubric looks like this:

You are given a review comment and a known bug in the same pull request.

Do they describe the SAME defect: the same code, failing for the same reason?
- A comment about the same file but a different problem is NOT a match.
- A comment that names the symptom while the known bug names its cause IS a match.
- If the comment is too vague to tell, answer no.

Answer with one word: yes or no.

The three lines in the rubric settle in advance the cases where a judge most often hesitates: the same file but a different problem, a symptom against a cause, a vague comment. When those lines aren’t written, two judges split on exactly these cases.

Why overall agreement between two judges is misleading

The first step in testing a judge is to ask a second judge about the same pairs. We used two different models as judges with the same rubric, and they agreed on 98.8% of 945 shared pairs. On its own, that number is reassuring, and it is misleading.

Most pairs are an obvious “no”: the comment is about one file, the known bug about a completely different one. Two judges agree on those anyway. What decides the result is the pairs where at least one judge said “yes”, and there they agreed 93.5% of the time. When you report how reliable a judge is, name the denominator: all pairs, or the positive pairs.

How we audited a cheap judge with a strong one, group by group

We run our day-to-day measurements with a cheap judge. To answer “what if this judge gets it wrong?”, we didn’t need to replace the judge outright, which would have meant a full matrix of 4,185 pairs. Instead, we asked a stronger model, at a high effort setting, about just 891 pairs, and we chose the pairs not at random but from the groups that would answer the question.

• 91 pairs where both judges said “yes”: the strong judge overturned none of them. A match both judges agreed on stayed a match with a more expensive judge too.
• 400 pairs where both judges said “no”: only 1 was overturned (upper bound 1.4%). The judge does not miss matches.
• 400 of the “yes” pairs counted in the score: 24 were rejected, which is 6.0%. The cheap judge is generous in counting matches.

The result is two separate statements, and they shouldn’t be mixed up. Absolute scores are about 0.02–0.03 optimistic. Changing the judge didn’t move a configuration’s score by more than 0.033, which is below the measurement’s noise floor (0.09). But we measured this on a single configuration, so we don’t claim that the ranking of two configurations will hold in every case. Whether a judge’s flaw matters depends on what you do with the judge.

Rerunning moves the score more than changing the judge

When we rescored the same configuration with three different judges, the largest shift in F1 was 0.033. When we reran the same configuration, on the same pull requests and with the same judge, the run-to-run standard deviation came out at 0.094. Once, with nothing changed, we saw a difference of 0.058 between two runs.

So the real uncertainty in the measurement comes not from the judge but from the system’s own randomness. The rule that follows is simple: if two configurations differ by less than about 0.09, a single run can’t show the difference, and buying a better judge won’t change that.

A finding the judge can’t match isn’t necessarily wrong

In most benchmarks, a judge model compares a finding with a reference: the bugs marked in advance. A finding that isn’t in the reference counts as a “false alarm”. But that only tells you one thing: whoever prepared the reference didn’t write that point down. The finding may still be right.

That is why ReviewBench publishes two families of metrics: one uses only the existing labels, the other also evaluates findings that have no match in the golden set. Make the same distinction in your own measurements. “Doesn’t match the reference” and “wrong” are two different labels, and to say the second, you have to look at the finding itself.

Store every verdict, and test rule changes without a judge

A judge call costs money, and there is no point asking the same question about the same pair twice. Store every verdict together with the pair and the judge’s identity: the model, the effort setting, the version of the rubric. When one of them changes, don’t show the old verdicts and the new ones in the same table; two numbers measured with two different judges can’t be compared.

Not every change needs a judge, either. When you change a threshold or a filter, you can replay the old and the new rule over results you have already stored, and look only where they disagree. That is how we measured a gate change: across 1,405 stored findings, the old and new rules disagreed in only 6 places, and reading those six by eye was enough. A new run would have been neither free nor this exact.

These lessons came from diffox’s own measurements: we don’t decide whether a change works on the strength of a single run and a judge nobody has audited. In the product itself, the quote a finding rests on is checked by code, not by a judge model (see the architecture post).