AI coding benchmarks, and what SWE-bench really measures

What is SWE-bench, and how is it scored?

A coding benchmark is a fixed list of tasks plus a scoring rule. The model or coding agent works through the tasks, the rule marks each solution as passed or failed, and the score is the percentage that passed. This post starts with the benchmark people talk about most, SWE-bench, then reads the others with the same four questions. We covered the structure around the model (the harness) in the post on harness engineering, and evaluations scored by a judge model in the post on LLM-as-a-judge.

SWE-bench is a benchmark built from real GitHub issues. According to its page, the first version came out in October 2023 and has 2,294 tasks collected from issues and pull requests in 12 popular Python repositories. Each task is taken from a pull request that closed an issue and also changed test files.

Scoring works like this: the model gets the text of the issue and the repository as it was before the pull request, and it changes the code. When the model has finished, two groups of tests run:

• FAIL_TO_PASS: tests that failed before the pull request and pass after it. They show that the issue is really fixed.
• PASS_TO_PASS: tests that must pass both before and after. They show that the change did not break something that already worked.

If both groups pass, the task counts as solved. The score is the percentage of tasks solved; leaderboards write it as “% Resolved”.

SWE-bench today is not one set but a family. We read the numbers in the table on each set’s own page on 20 October 2026:

Source: SWE-bench’s page, Multilingual’s announcement and Scale AI’s SWE-bench Pro page, 20 October 2026. Results on Pro’s private set are published on a separate leaderboard.

One more detail matters when you read the leaderboard. Verified’s page says the leaderboard puts every kind of coding system on the same list, from simple agent loops to systems that make several attempts. To compare models with each other there is a separate “Bash Only” view: there every model runs with the same small agent (mini-SWE-agent) and nothing but a bash shell.

Why is SWE-bench Verified no longer enough?

Verified came out with OpenAI in August 2024. According to Epoch AI, its 500 tasks were picked from the 2,294 in the original set through an independent review by three experts, to weed out tasks that had problems.

On 3 September Epoch reviewed Verified again and rated it “Flawed”. It gives three reasons:

• The tests reject correct solutions. In an audit it published on 23 February 2026, OpenAI looked at the tasks models often failed to solve. It reviewed 27.6% of the tasks and found that in 59.4% of those, the test rejected a solution that worked correctly. According to Epoch, that means at least 16.4% of all tasks are broken. Epoch’s bar is that fewer than 20% of the questions have a defect like this.
• The tasks have leaked into training data. Verified is a fixed set; all of its tasks and all of its solutions are public. According to OpenAI’s audit, every frontier model it tested had seen at least some of these problems and their solutions in training. OpenAI’s conclusion: progress on Verified no longer shows real software development skill, but how much of this benchmark the model saw in training.
• The score can be gamed. The RDI team at Berkeley found a reward hack that reaches a perfect score.

Model makers acted on this. According to Cursor’s post, OpenAI stopped reporting Verified results entirely once it saw that frontier models could reproduce the reference solutions from memory. Anthropic’s Opus 5 system card of 24 July still reported Verified: 96.0%, the average of five trials. The Opus 5.5 card of 22 September has no Verified; it has only Pro, Multilingual and Multimodal. In the announcement of the same model Anthropic also writes that at this level of capability, differences on benchmarks have become a less reliable guide to differences in real use.

A benchmark on which everyone is close to the ceiling can no longer tell models apart. The people who build benchmarks see it too. Aider replaced its old editing benchmark with a harder set once scores passed 80%. Terminal-Bench 4.0 counted the tasks that all the newest models solved in 5 out of 5 attempts as “saturated” and removed them from the set. DeepSWE’s page also says the leading models are squeezed into a narrow band of scores, and that the confidence intervals of neighbouring configurations often overlap.

In the same week Epoch gave two more coding benchmarks the same rating. It rates a benchmark “Flawed” when defects could affect 20% or more of the score:

Source: Epoch AI’s reviews. For Verified, Epoch relies on OpenAI’s audit. Terminal-Bench’s tasks come from the community; Epoch counted the problems publicly reported in its repository. For BFCL it took a random sample from 5,088 items.

Here is what a defect looks like, with an example from BFCL. In the task “What is the current time in Sydney, Australia?” the model is given a function called get_local_time(location). The answer key marks the task “irrelevant”. Only a model that calls no function at all gets full marks; a model that calls the function correctly scores zero.

What do the other coding benchmarks measure?

SWE-bench looks at one kind of work: a patch that closes an issue in a repository. What you expect from a coding agent is broader, so each benchmark measures a different slice of it. We read the details in the table on each benchmark’s own page on 20 October 2026:

Source: each benchmark’s own page or announcement, 20 October 2026. “Not stated on its page”: on the page we read that day, we found no sentence that says so. Cognition’s and Cursor’s assessments come from the posts in which they introduce their own benchmarks.

The leaderboards are not equally fresh either. Aider’s leaderboard was last updated on 20 November 2025, BFCL’s leaderboard on 12 April 2026. The newest release in LiveCodeBench’s repository (release_v6) covers 1,055 problems published between May 2023 and April 2025. If a model is missing from a leaderboard like this, that may mean the leaderboard is old, not that the model is bad.

The oldest name not on the list is HumanEval: a set OpenAI released in 2021 that has models write code from a docstring. Its repository is now marked “Deprecated and unmaintained”. The LiveCodeBench team also found that some models that do well on HumanEval may be overfitting to it.

Code review tools have benchmarks too. GitHub released ReviewBench on 5 October; there a judge model decides whether a comment is a correct finding, and GitHub published the judge’s prompt as well. We covered how to read a benchmark that a judge model scores in the post on LLM-as-a-judge.

Does code that passes the tests get merged?

SWE-bench’s score comes from an automatic grader: if the tests pass, the task counts as solved. In a study it published on 10 March 2026, METR measured how well that number matches a real maintainer’s review. 4 active maintainers of 3 repositories in Verified (scikit-learn, Sphinx, pytest) read a total of 296 AI-written PRs the way they would in a real review, and decided whether they would merge each one. They did not know who had written the PR.

The result: the maintainers would not have merged about half of the PRs that passed the tests. Because a maintainer’s decision can vary from person to person, METR also had the same maintainers read human PRs that had really been merged in the repository; they approved about 68% of those. With the scores scaled to that baseline, the automatic grader sits on average 24.2 points above the maintainers’ decision. Without the scaling, the gap is 34.9 points.

The maintainers rejected PRs for three kinds of reason: the change does not really fix the problem, it touches unrelated code and breaks it, or the code does not meet the repository’s standards. METR also states its limits plainly: the agents got no chance to fix their work in response to feedback, and the study looked at part of a single benchmark and at a single agent. So the result does not say that models cannot do this. What it says is this: read on its own, a score makes an agent look more useful than it is.

FrontierCode is a benchmark that puts these questions into the score. Cognition prepared 150 tasks with the maintainers of 36 open-source repositories; in each repository the maintainers set the criteria for a change to count as mergeable. Alongside the unit tests there is a scope check: does the change touch only the files it is allowed to, and does the diff go over a set size? The tests the agent writes are also run on the old, broken code; a test that does not fail there is not testing anything.

Anthropic’s own report shows what the scope penalty does. According to the Opus 5 system card, at effort levels above high the model made more changes than the task asked for, and its FrontierCode score dropped. When a short instruction telling it to stay within the scope of the task was added to the prompt, the score came back on most of those tasks. This is a result the vendor reports for Opus 5; the card does not say how many points came back.

This gap is exactly where code review works: a test shows that the change produces the behaviour you asked for; a review asks whether the change belongs there. In diffox, specialist agents ask that question. One compares the changed lines, one rule at a time, with the rules in the repository’s rules file (AGENTS.md, CLAUDE.md); a rule that is not written in the file does not count. Another looks for a test that does not check what its name says, or an assertion that can never fail; a missing test, on the other hand, is not a finding for it. We covered how to tell whether a test really checks something in the post on reviewing AI-generated code.

How should you read a benchmark score?

When you come across a score, ask these six questions. If two scores differ on any one of them, they are no longer measuring the same thing:

1. Which set or subset? “SWE-bench” on its own says nothing: Verified, Pro’s public set and a subset a company picked are different numbers. Anthropic’s documentation says that scores on the 482-problem subset it chose from Pro cannot be compared with the public leaderboard. They cannot be compared with the Pro results in the same company’s system card either, because those were measured on a different problem set and at max effort.
2. Inside which agent? A score belongs not to the model but to the pair of the model and the agent that runs it. According to Epoch, Verified results depend heavily on the agent used; on Terminal-Bench’s leaderboard every row is a model and an agent. Settings can differ even on the same benchmark: Epoch noted that on Terminal-Bench, Codex entries retry safety refusals while the default setting does not. Artificial Analysis also writes that its results belong to specific configurations, not to product names, and that the model, the settings and the way it is run can change the result materially. Epoch also counts an agent managing its context poorly as a setup problem that lowers the score; we covered what coding agents do when the context fills up in the post on auto-compact.
3. At which effort level? Leaderboards often show a model at the level where it does best. The results in Anthropic’s Opus 5.5 announcement were measured at max effort unless noted otherwise; the Terminal-Bench row is at the level that gave each model its highest score. FrontierCode also runs every model 5 times at every effort level and reports the best level. If you use the tool at its default level, you may not see the result in the table.
4. How many runs, and how much uncertainty? The same configuration can give different scores in two runs. Terminal-Bench’s leaderboard shows the average of 5 runs with a 95% confidence interval; Epoch recommends at least 5 runs per model for error bars. A difference that is not bigger than the noise is not a difference; we covered the noise between two runs in our own measurements in the post on LLM-as-a-judge.
5. Who measured it? A model maker’s table is measured under conditions it chose. Independent measurements try to run every model under the same rules: DeepSWE runs all models with the same small agent. Look at the details in both: Artificial Analysis lets another model finish when a model refuses a task, and counts that attempt too; Anthropic also writes that when its safeguards stepped in, other models completed some of Opus 5.5’s tasks.
6. Is the set saturated, leaked, up to date? A set on which everyone is close to the ceiling does not tell models apart; a set whose tasks and solutions are public may also be measuring memory. Look at the tasks’ release dates (LiveCodeBench keeps one for every problem), ask whether the set has a closed part, and check when the leaderboard was last updated.

An example that shows what effort does on the same model and the same tasks is in Anthropic’s document on cost and intelligence. The only thing that changes is the effort level:

On the same model, with only the effort changing, the score moves by 9.3 points, and the cost per task goes up from $0.11 to $0.74. In a table that compares two models, first check the level each one was measured at: if one was measured at max and the other at its default level, the difference between them may come from the setting, not from the model.

Which benchmark should you look at for which job?

Choosing a benchmark means choosing what you want from the agent. The table below matches common jobs with the benchmark closest to each:

Source: the tables in this post and each benchmark’s page, 20 October 2026.

None of them replaces the work in your own repository. Anthropic’s documentation also recommends trying the effort setting on your own workload. The most reliable measurement is to run a few real tasks from your own repository with the same agent, at the same effort level, several times.

None of them measures whether a change should be merged, either. FrontierCode comes closest, but there too the decision is made by criteria the maintainers wrote down in advance. In your repository, that decision is made by review.

Frequently asked questions

What is SWE-bench?

SWE-bench is a benchmark that measures whether AI models and coding agents can solve real GitHub issues. The model is given the text of the issue and the repository as it was at that point, and it changes the code. If the change passes the FAIL_TO_PASS tests, which show that the issue is fixed, and the PASS_TO_PASS tests, which show that nothing broke, the task counts as solved. The first version has 2,294 tasks from 12 Python repositories; there are subsets and sibling sets such as Verified, Pro and Multilingual.

What is the difference between SWE-bench Verified and SWE-bench Pro?

Verified is a 500-task subset of the original set that people reviewed; its tasks and solutions are public. Pro is Scale AI’s harder set of 1,865 tasks from 41 repositories: solutions average 107.4 lines and 4.1 files. Pro’s tasks come from copyleft-licensed repositories and from companies’ closed code, and the results on part of it are never published. Epoch AI rated Verified flawed in September 2026; Anthropic’s latest model card also reports Pro instead of Verified.

What percentage counts as a good SWE-bench score?

There is no single threshold. A score depends on which set, which agent and which effort level it belongs to: on Anthropic’s own subset the same model moves 9.3 points with the effort level alone, and Anthropic writes that these scores cannot be compared with the public leaderboard. Compare two scores only if they were measured on the same set, with the same agent and similar settings; do not look at a number on its own, but at its distance from other models measured under the same conditions.

What does Terminal-Bench measure?

Terminal-Bench measures whether an agent can complete end-to-end tasks in the terminal: software, science, ML, operations, security, hardware and media tasks. Each task has its own verifier. With version 4.0 it became a continuously updated benchmark; it removes tasks that are saturated, tasks that get refused and tasks whose solutions have become public. The leaderboard shows the average of 5 runs for each model and agent pair, with a 95% confidence interval. On 4 September Epoch AI recorded a publicly noted scoring defect in 30 of 66 tasks.

What is the difference between LiveCodeBench and SWE-bench?

LiveCodeBench uses problems collected from LeetCode, AtCoder and Codeforces contests to look at a model’s ability to write code, repair its own code, run code and predict test output; a task is a single problem. SWE-bench, by contrast, asks for a change that closes a real issue in a repository. Because LiveCodeBench tags every problem with its release date, you can measure a model on problems released after its training cutoff; that way the model cannot have seen the solution in training.

How far can you trust a benchmark score?

A score answers the narrow question the benchmark asks. Before you trust it, check six things: which set, which agent, which effort level, how many runs and what confidence interval, who measured it, and whether the set is saturated or leaked. That is why independent audits matter: in September 2026 Epoch AI rated SWE-bench Verified, Terminal-Bench 4.0 and BFCL v4 flawed; METR measured that about half of test-passing SWE-bench PRs would not get through a maintainer’s review.

Which benchmark is diffox measured on?

diffox is an AI code review tool, so it is measured on code review benchmarks, not on coding benchmarks. We ran Martian’s open-source code review benchmark ourselves and published the result with its evidence: every review is a public pull request, the scoring was done by the benchmark’s own code, and you can rebuild the table on your own machine with the commands in the evidence repository. That post also says how many runs each row comes from: we did the public run at what was then the High sensitivity setting, and the Medium and Low rows are the average of several internal runs.