diffox ultra ranks first among 23 AI code review tools
We ran Martian’s open-source code review benchmark ourselves. diffox ultra caught 119 of 158 real bugs and ranked first. Every review is public.
Summary. On 7 October 2026 we ran Martian’s open-source Code Review Benchmark ourselves. It covers 50 pull requests from Sentry, Grafana, Cal.com, Discourse and Keycloak, with 158 known bugs; the reviews are judged by Claude Opus 4.5. At High sensitivity, diffox ultra scored 66.5 on F2 and ranked first among 23 AI code review tools. It caught 119 of the 158 bugs (75%), and no tool caught more. All 100 reviews are public pull requests, scored by Martian’s own code: github.com/diffox-bench/code-review-benchmark.
- Closest competitor. Qodo Extended, with 65.1.
- Critical bugs. diffox caught 11 of the 12 bugs Martian rates critical.
- You choose which findings are posted. New installations start at “Critical and warnings”: critical findings and warnings are posted, suggestions are not. One step in the console takes you to “Everything” to get suggestions as well, or to “Critical only” for fewer comments.
diffox ultra scored highest and caught the most bugs
We scored diffox’s two products exactly the way Martian scores the 21 tools on its published board. diffox ultra ranks first with 66.5 points, and diffox ranks fifth among the tools with 60.9. diffox ultra also caught the most bugs (119), followed by diffox (106) and GitHub Copilot (105).
| Rank | Tool | F2 | Bugs caught | False alarms |
|---|---|---|---|---|
| 1 | diffox ultra · High sensitivity | 66.5 | 119 | 144 |
| 2 | Qodo Extended | 65.1 | 102 | 50 |
| 3 | Cubic v2 | 64.9 | 104 | 65 |
| 4 | Augment | 64.0 | 103 | 70 |
| 5 | diffox ultra · Medium sensitivity1 | 62.6 | 98 | 51 |
| 6 | diffox · High sensitivity | 60.9 | 106 | 132 |
| 7 | Qodo v2 | 60.6 | 98 | 79 |
| 8 | GitHub Copilot | 58.0 | 105 | 168 |
| 9 | diffox · Medium sensitivity1 | 57.4 | 89 | 55 |
| 10 | Gitar | 53.6 | 97 | 176 |
| 11 | Gemini | 52.8 | 87 | 105 |
| 12 | GitLab Duo Code Review | 52.7 | 83 | 72 |
| 13 | Sourcery | 51.5 | 87 | 125 |
| 14 | Greptile v4 | 51.2 | 81 | 78 |
| 15 | CodeRabbit | 51.1 | 94 | 194 |
| 16 | Macroscope | 50.8 | 77 | 49 |
| 17 | diffox ultra · Low sensitivity1 | 48.8 | 70 | 19 |
| 18 | Cursor Bugbot | 48.6 | 74 | 56 |
| 19 | Claude Code (CLI) | 47.7 | 76 | 88 |
| 20 | Devin | 44.8 | 65 | 29 |
| 21 | Kodus | 41.5 | 61 | 42 |
| 22 | diffox · Low sensitivity1 | 41.5 | 59 | 19 |
| 23 | CodeAnt | 40.9 | 65 | 97 |
| 24 | Baz | 40.5 | 59 | 37 |
| 25 | Claude Code | 38.9 | 60 | 79 |
| 26 | KG | 18.4 | 25 | 21 |
| 27 | Graphite | 9.3 | 12 | 0 |
1 Settings are ranked by F2 as well. “23 tools” counts only tools, which is why diffox is fifth among the tools but sixth in this list. When two scores shown are equal, the public result is listed first. The Medium and Low sensitivity rows come from our internal measurement on the same 50 pull requests and 158 bugs: each is the rounded average of 5 separate runs for diffox ultra and 2 for diffox. The High sensitivity rows come from the public run. High, Medium and Low were diffox’s sensitivity settings at the time; now you choose which findings are posted by their class.
diffox ultra caught more bugs than GitHub Copilot and CodeRabbit while raising fewer false alarms. At Low sensitivity1 it caught 70 bugs with only 19 false alarms; no tool with fewer than 30 false alarms caught more than 65. Results for the other tools are Martian’s own, as published at codereview.withmartian.com on 7 October 2026.
How the Martian Code Review Benchmark works
The Martian Code Review Benchmark (github.com/withmartian/code-review-benchmark) is an open-source benchmark that tests AI code review tools under identical conditions. It is made up of 50 pull requests from five well-established open-source projects: Sentry, Grafana, Cal.com, Discourse and Keycloak. People went through those pull requests beforehand and marked 173 known issues. The board’s default scoring counts 158 of them, and those are the bugs this post counts.
Each tool reviews the pull requests in its own way. A judge model, Claude Opus 4.5, then checks every comment against the known bugs. A comment that matches a bug counts as a catch; one that matches none counts as a false alarm. The board ranks tools by F2, which gives catches more weight than false alarms, because a bug that reaches production costs far more than a comment someone reads and dismisses.
Verify it yourself
Claiming first place in a benchmark is easy; proving it is not. That is why every part of the High-sensitivity run is public.
- One hundred reviews, one hundred public pull requests. We copied each of the benchmark’s 50 pull requests into its own public repository, once per product: 100 in all. The live diffox app reviewed each copy at High sensitivity, just as it reviewed a customer’s pull request. Only the layout of the comment is different (see the questions below). The list of all 100 reviews
- Scored by Martian’s own code. We ran the scoring steps without changing them. On the same files, Martian’s dashboard script reproduces the published numbers of 20 of the other 21 tools exactly.
- Reproduce the table on your machine:
git clone https://github.com/diffox-bench/code-review-benchmark && cd code-review-benchmark/offline
uv sync --frozen
uv run python analysis/benchmark_dashboard.py # Martian’s own dashboard script, now with our two rows
git diff --diff-algorithm=histogram e616e849 -- results # additions only: no other tool’s line changesWho caught the most severe bugs?
Not all bugs weigh the same. The table below compares diffox’s two products with the four competitors that caught the most bugs. diffox caught the most critical bugs: 11 of 12. On high-severity bugs, diffox ultra leads by a clear margin: 46 of 54.
| Severity | diffox ultra | diffox | GitHub Copilot | Cubic v2 | Augment | Qodo Extended |
|---|---|---|---|---|---|---|
| Critical (12 bugs) | 10 | 11 | 9 | 10 | 8 | 9 |
| High (54 bugs) | 46 | 37 | 38 | 38 | 41 | 40 |
| Medium (59 bugs) | 44 | 40 | 42 | 37 | 39 | 37 |
| Low (33 bugs) | 19 | 18 | 16 | 19 | 15 | 16 |
| Total (158 bugs) | 119 | 106 | 105 | 104 | 103 | 102 |
Which tool leads on each project?
In three of the five projects, a diffox product caught the most bugs: diffox ultra on Cal.com and Keycloak, and diffox on Discourse. Cubic v2 leads on Grafana and Sentry; on Sentry, diffox ultra trails by a single bug. Each project contributes 10 pull requests.
| Project | diffox ultra | diffox | GitHub Copilot | Cubic v2 | Augment | Qodo Extended |
|---|---|---|---|---|---|---|
| Cal.com (37 bugs) | 32 | 25 | 29 | 19 | 29 | 25 |
| Discourse (38 bugs) | 27 | 31 | 27 | 22 | 22 | 22 |
| Grafana (25 bugs) | 17 | 17 | 15 | 24 | 17 | 17 |
| Keycloak (25 bugs) | 19 | 12 | 16 | 14 | 13 | 17 |
| Sentry (33 bugs) | 24 | 21 | 18 | 25 | 22 | 21 |
Four bugs most tools missed
The tables put a number on the gap; the four bugs below show what it looks like. Martian’s repository has bug-by-bug results for 20 other tools on the board, and each of these four bugs was caught by at most five of them. diffox ultra caught all four. Each card links to the comment diffox posted on that pull request.
Discourse · security · medium severity
The new code sends an X-Frame-Options: ALLOWALL header. Browsers don’t recognize that value, so clickjacking protection is effectively off and any site can frame the page. diffox and diffox ultra both caught it. Only 2 of the 20 other tools caught it.
Sentry · high severity
The check meant to kill hung processes never fires. The processes are created with the spawn context, so they are not instances of multiprocessing.Process; the isinstance check always returns false, and hung processes keep running. Only 4 of the 20 other tools caught it.
Grafana · high severity
The function that fills the cache writes its result even when the read fails, overwriting a good cached value with nil. Only 3 of the 20 other tools caught it.
Keycloak · high severity
The permission check looks the resource up under the wrong owner: resources are created under the client’s ID but looked up under the server’s. The lookup never finds the resource, so the check always answers “no”. Only 5 of the 20 other tools caught it.
Keycloak: View the comment →More bugs caught, or fewer false alarms?
Every code review tool faces the same trade-off: the more bugs it catches, the more false alarms it raises. Every other tool on the board appears once, at the setting it was benchmarked at. diffox’s two products appear three times each, because we measured every sensitivity setting on the same 50 pull requests.
Real bugs caught vs. false alarms
Each diffox line shows High, Medium and Low; the other tools are single points.
The settings of that time decided how sure diffox had to be before it posted a finding. High caught the most bugs, and it was the setting that took first place. Medium posted fewer comments and still caught most real bugs: comments per pull request dropped from 5.4 to 2.9. Low posted only the findings it was surest of: 1.7 comments per pull request.
| Average comments per PR | High | Medium | Low |
|---|---|---|---|
| diffox ultra | 5.4 | 2.9 | 1.7 |
| diffox | 4.7 | 2.9 | 1.6 |
Now you choose which findings are posted, by their class. diffox gives every finding a class: Critical, Warning or Suggestion. The setting has three levels: “Critical only”, “Critical and warnings” and “Everything”. New installations start at “Critical and warnings”. The account owner or an organization admin can change it on the Settings tab of the diffox console, and the change applies from the next review on. Nothing the setting holds back is lost: it waits on your Findings page for 30 days, and you can publish any of it from there.
If you’d rather not deal with the setting, set “How it is chosen” to “Automatic” in the console: the level then moves between “Everything” and “Critical and warnings” on its own, by the 👍 and 👎 you give findings. And a finding your team turned down with a 👎 is not posted again in the same repository as long as its line does not change.
Frequently asked questions
Can I check these results myself?
Yes. All 100 reviews are public pull requests, and every result file and command is at github.com/diffox-bench/code-review-benchmark. If you run Martian’s own dashboard script there, it prints the table with our two rows, and git diff shows that no other tool’s results changed.
How do I choose which findings are posted?
diffox gives every finding a class: Critical, Warning or Suggestion. You choose which of them are posted. “Critical only” posts only critical findings. “Critical and warnings” posts warnings too, but not suggestions; every new installation starts there. “Everything” posts suggestions as well. You can change the setting at any time on the console’s Settings tab, and whatever the setting holds back waits on your Findings page for 30 days.
What is diffox ultra, and how do I use it?
diffox ultra reads each change with two models instead of one and finds more. Every installation starts on diffox. To review every pull request with diffox ultra, choose it under “Review with” on the console’s Settings tab. To try it on one pull request, comment @diffoxai ultra review on it, and diffox ultra reviews it again. A review with diffox ultra costs more; every review’s price is on its own row in your console.
What is F2, and why F2?
F2 is a score that counts a missed bug more heavily than a false alarm. Martian’s board ranks tools by it, so we report our results on the same measure.
Why do the comments on the copies look different from diffox’s usual comments?
Martian’s harness treats everything a bot writes on a pull request as a candidate finding, including summary tables and the list of findings held back. So on the copies, diffox posted only its published findings, in a single plain comment. The findings are the same; only the presentation differs.
How do I install diffox?
diffox installs from GitHub in one click, with no key and no config file, and one install covers every repository. From then on, every pull request you open is reviewed without you asking: the verdict first, then each finding under the exact line it is about.
Which platforms does diffox support?
diffox is a GitHub App and works on public and private repositories on GitHub. Every review arrives in the language your team reads: English, Turkish, German, French, Spanish or Italian.
The prices, compared
The second post compares prices on the same 50 pull requests: what each tool costs per review, and what you get for it. Our current prices are on the pricing page.