Nelson Benchmark Leaderboard

16 competitors — 95 case hits across 351 audited cases

16
Competitors
351
Cases audited
95
Case hits
$560.73
Competitor spend
$240.14
Judge spend

Leaderboard

#CompetitorSizeDetectdet+½Hits/EligPartialFalse posPrecisionFP/caseOther realCost/caseLatencyTokens/caseCases
1gpt-5.6-sol/codex ★large68%
3 passes, de-duped
70%15/2211573%0.6818$0.00352s022
2gpt-5.6-sol/apilarge41%
3 passes, de-duped
45%9/2222469%1.0942$8.02584s1.5M22
3opus-4.8/api ★large36%
3 passes, de-duped
36%8/22——100%0.0013$6.43272s1.2M22
4gpt-5.5large36%
3 passes, de-duped
41%8/2222367%1.0534$3.77504s2.4M22
5mimo-v2.5-prolarge35%
3 passes, de-duped
38%7/201391%0.1422$0.441520s2.2M22
6opus-4.8/claude-code†large32%
3 passes, de-duped
†		34%6/191384%0.149$2.47395s1.6M21
7glm-5.2large30%
3 passes, de-duped
32%6/201292%0.0916$0.96941s1.5M22
8deepseek-v4-prolarge27%
3 passes, de-duped
34%6/2231562%0.6816$0.99738s2.4M22
9gemma4-31bsmall24%
3 passes, de-duped
29%5/2121267%0.5518$0.003454s934k22
10qwen3.6-27bsmall23%
3 passes, de-duped
30%5/2231359%0.5913$0.002804s1.8M22
11minimax-m3large23%
3 passes, de-duped
25%5/2211159%0.5010$0.71839s2.2M22
12qwen3.7-maxlarge20%
3 passes, de-duped
25%4/202969%0.4114$1.221027s1.8M22
13gemma4-12b-thinksmall18%
3 passes, de-duped
25%4/2232628%1.186$0.002203s1.3M22
14deepseek-v4-flashmedium16%
3 passes, de-duped
18%3/191756%0.323$0.36453s2.5M22
15laguna-s-2.1small9%
3 passes, de-duped
9%2/22—736%0.322$0.231050s2.2M22
16gemma4-12b-nothink ★small9%
3 passes, de-duped
9%2/22—3116%1.413$0.00101s261k22

Every count below is de-duped across the --repeat passes — the same bug (or the same false positive) reported in several passes is counted once, never once per pass. This is the point of multipass scanning: more bites at the apple to find a bug, without inflating the tally. Detect = distinct cases HIT in any pass / eligible cases (hits + partials + genuine misses); undetermined, refused and auth/infra-excluded cases are out of the denominator. Partial = distinct cases localized to the right spot but judged a different bug — right place, wrong bug; an eligible non-hit that never moves Detect or the ranking, but still points a human at the vulnerable line. False pos = absolute distinct confirmed false positives — the human-time-wasted number, to weigh against the bugs found. Precision = true / (true + false positives). Other real = confirmed real bugs that are not the planted target CVE (extra capability, not counted as detection). det+½ = (hits + 0.5·partials) / eligible, informational. Cost/latency are the competitor's own spend per audited case. ★ = on a Pareto frontier below.

This run used repeated trials (--repeat). Detect and every finding count are the de-duped union across passes (best-of-N pooled), not a per-trial mean — hover a Hits/Elig cell for the per-trial spread. In the per-case matrix, a HIT marked n/N was found in only n of N trials (flaky). See bench noise for the full per-trial breakdown.

† Partial coverage: this competitor completed fewer than the full 22 cases (see the Cases column). Its detection rate is therefore based on fewer audited cases and is not directly rank-comparable with full-corpus competitors — read it alongside the Cases count, not the rank.

Pareto frontier

Quality vs cost / case

0.00.51.0quality (det x prec)$0.00$8.02cost / case (lower is better →)gpt-5.6-sol/codexgpt-5.6-sol/apiopus-4.8/apigpt-5.5mimo-v2.5-proopus-4.8/claude-codeglm-5.2deepseek-v4-progemma4-31bqwen3.6-27bminimax-m3qwen3.7-maxgemma4-12b-thinkdeepseek-v4-flashlaguna-s-2.1gemma4-12b-nothink

Quality vs latency / case

0.00.51.0quality (det x prec)101s3454slatency / case (lower is better →)gpt-5.6-sol/codexgpt-5.6-sol/apiopus-4.8/apigpt-5.5mimo-v2.5-proopus-4.8/claude-codeglm-5.2deepseek-v4-progemma4-31bqwen3.6-27bminimax-m3qwen3.7-maxgemma4-12b-thinkdeepseek-v4-flashlaguna-s-2.1gemma4-12b-nothink

Quality = detection rate x precision (precision treated as 1.0 when a competitor reported no scorable findings). Green points are non-dominated — no other competitor is at least as good on quality while also cheaper/faster. Size is shown in the table; it is categorical, so it is not used as a numeric Pareto axis.

Tokens & time per case

deepseek-v4-flash2.5M · 453sgpt-5.52.4M · 504sdeepseek-v4-pro2.4M · 738sminimax-m32.2M · 839slaguna-s-2.12.2M · 1050smimo-v2.5-pro2.2M · 1520sqwen3.7-max1.8M · 1027sqwen3.6-27b1.8M · 2804sopus-4.8/claude-code1.6M · 395sgpt-5.6-sol/api1.5M · 584sglm-5.21.5M · 941sgemma4-12b-think1.3M · 2203sopus-4.8/api1.2M · 272sgemma4-31b934k · 3454sgemma4-12b-nothink261k · 101sgpt-5.6-sol/codex0 · 352s

Mean total tokens (prompt + completion, with the ReAct loop's resent context counted each turn) per audited case; the trailing number is mean latency/case. Bars are linear, so brute-force models dwarf frugal ones. Data-quality caveat: these are the tokens the provider's API reported — some OpenAI-compatible endpoints under-report usage (a near-zero bar with input ≈ output is the tell), so a suspiciously short bar may mean broken metering rather than a frugal model, and that competitor's cost/case is then an underestimate.

Per-case results

Competitor CVE-2026-27654 CVE-2026-32316 CVE-2026-33721 CVE-2026-5199 CVE-2026-5446 CVE-2026-5447 CVE-2026-5448 CVE-2026-5466 CVE-2026-5477 CVE-2026-5479 CVE-2026-5500 CVE-2026-5501 CVE-2026-7474 GHSA-9f49-8x56-jmjc GHSA-cc7p-2j3x-x7xf GHSA-chgx-jx3p-rf73 GHSA-crr4-7rm4-8gpw GHSA-f26g-jm89-4g65 GHSA-j273-m5qq-6825 GHSA-mpxh-8fq3-x8mh GHSA-w52v-v783-gw97 GHSA-x9h5-r9v2-vcww
gpt-5.6-sol/codex miss HIT 2/3 HIT miss miss miss miss miss HIT 2/3 HIT 2/3 HIT 1/3 HIT HIT HIT 2/3 part HIT 2/3 HIT HIT 2/3 HIT 2/3 HIT HIT HIT 2/3
gpt-5.6-sol/api miss HIT HIT 1/3 miss miss miss miss HIT 1/3 miss miss part HIT 1/3 HIT 2/3 miss part miss HIT HIT HIT 1/3 miss HIT miss
opus-4.8/api miss HIT 1/3 HIT miss miss HIT 1/3 HIT 1/3 miss miss miss miss miss HIT 1/3 miss miss miss miss miss HIT 1/3 HIT 2/3 HIT miss
gpt-5.5 miss HIT HIT 1/3 HIT 2/3 miss miss miss miss miss miss miss HIT 2/3 HIT 1/3 miss miss part part HIT HIT 2/3 miss HIT miss
mimo-v2.5-pro miss jerr jerr miss miss miss HIT 1/3 miss miss miss miss miss HIT 2/3 HIT 1/3 miss miss part HIT HIT 1/3 HIT 1/2 HIT 2/3 miss
opus-4.8/claude-code miss jerr jerr miss miss excl miss miss miss miss miss miss HIT 2/3 miss miss miss part HIT 1/3 HIT HIT 1/3 HIT HIT 1/3
glm-5.2 miss jerr jerr miss miss miss HIT 1/3 miss miss miss miss miss HIT HIT 1/3 miss miss part HIT HIT 1/3 miss HIT miss
deepseek-v4-pro miss HIT 1/3 HIT 2/3 miss miss miss part miss miss miss miss miss HIT 2/3 miss miss part part HIT 1/2 HIT 1/3 miss HIT miss
gemma4-31b miss jerr miss miss miss miss miss miss miss miss miss miss HIT 1/3 miss part part HIT 1/3 HIT 1/3 HIT 2/3 miss HIT 2/3 miss
qwen3.6-27b miss HIT HIT 2/3 miss miss miss part miss miss miss miss miss miss miss miss part part HIT HIT miss HIT miss
minimax-m3 miss HIT 1/3 miss miss miss miss miss miss miss miss miss miss HIT 1/3 miss miss miss part HIT 2/3 HIT miss HIT miss
qwen3.7-max miss jerr jerr miss miss miss part miss miss miss miss miss HIT 2/3 miss miss miss part HIT HIT 1/3 miss HIT miss
gemma4-12b-think miss HIT 1/3 miss HIT 1/3 miss miss miss miss miss miss miss miss miss miss part part part miss HIT 1/3 miss HIT 2/3 miss
deepseek-v4-flash miss jerr miss miss miss miss jerr miss miss miss miss miss jerr miss miss part miss HIT 2/3 HIT miss HIT miss
laguna-s-2.1 miss miss miss miss miss miss miss miss miss miss miss miss miss miss miss miss miss HIT 1/3 miss miss HIT 2/3 miss
gemma4-12b-nothink miss miss miss miss miss miss miss miss miss miss miss miss miss miss miss miss miss miss HIT 2/3 miss HIT miss

HIT = detected; part = right spot, wrong bug (half credit, eligible non-hit); miss = looked, found nothing; jerr = judge undetermined (out of denominator); refu = model refused the task (out of denominator, never a miss); excl = auth/infra failure (never a miss); · = not run. A HIT marked n/N was found in only n of N trials (flaky); a bare HIT was found in every trial.