Nelson Benchmark Leaderboard

18 competitors — 42 case hits across 155 audited cases

Leaderboard

Detect = case hits / eligible (hits + genuine misses); undetermined, refused, and auth/infra-excluded cases are not in the denominator. Precision = true findings / (true + false positives). Other real = confirmed real bugs the model found that are not the planted target CVE (extra capability, but not counted as detection). Cost/latency are the competitor's own spend per audited case. ★ = on a Pareto frontier below.

† Partial coverage: this competitor completed fewer than the full 9 cases (see the Cases column). Its detection rate is therefore based on fewer audited cases and is not directly rank-comparable with full-corpus competitors — read it alongside the Cases count, not the rank.

Pareto frontier

Quality = detection rate x precision (precision treated as 1.0 when a competitor reported no scorable findings). Green points are non-dominated — no other competitor is at least as good on quality while also cheaper/faster. Size is shown in the table; it is categorical, so it is not used as a numeric Pareto axis.

These charts omit gpt-5.5-pro (4/9): a competitor that audited fewer than 75% of the 9 cases measures its quality over a smaller, self-selected subset, so the point is not comparable with the full-corpus competitors — and a cost-capped probe sits so far out on the cost axis that every other competitor collapses into one indistinguishable cluster, making the trade-off unreadable. Its position would also imply a quality ranking the partial run does not establish. It remains in the leaderboard table above (marked †).

Tokens & time per case

Mean total tokens (prompt + completion, with the ReAct loop's resent context counted each turn) per audited case; the trailing number is mean latency/case. Bars are linear, so brute-force models dwarf frugal ones. Data-quality caveat: these are the tokens the provider's API reported — some OpenAI-compatible endpoints under-report usage (a near-zero bar with input ≈ output is the tell), so a suspiciously short bar may mean broken metering rather than a frugal model, and that competitor's cost/case is then an underestimate.

Per-case results

HIT = detected; miss = looked, found nothing; jerr = judge undetermined (out of denominator); refu = model refused the task (out of denominator, never a miss); excl = auth/infra failure (never a miss); · = not run.