Gemma 4 (QAT-4bit) prompt-lab — full 9-case corpus

The two new Quantization-Aware-Training 4-bit Gemma 4 models (gemma4-31b-qat dense, gemma4-26b-a4b-qat MoE), self-hosted and free, over all 9 corpus cases. Three non-leaking prompting arms (open / plan-first / CWE-checklist) × 2 trials/arm at temperature 0.5. Multi-file cases (nomad/craft/ghost) audit every baseline file; a case is detected if ANY file localizes within 10 lines of the planted hunk (the benchmark's gate). Motivation: at 8-bit the 26B-A4B MoE tied the field-leading 4/9 detection, so we feel out the QAT-4bit edges as we did Qwen 3.6.

Detection matrix

Cell = case-detected trials / completed trials for that arm (a case counts if any of its files hit). green = every trial, amber = some, red = none, — = no data. Hard-miss cases (baseline ≤1) bold.

gemma4-26b-a4b-qat

gemma4-31b-qat

Arm summary (union over 2 trials)

Cases each arm detected in at least one trial, split by whether the baseline solved the case (regression guard) or missed it (the real question). Denominators count only cases with a completed run.

Off-target findings: real secondary bugs vs false positives

Detection above is only half the story. Every finding a model reports that isn't the planted CVE still lands in front of a human (or a downstream model) who burns time and tokens triaging it — so a model that floods the queue with hallucinations is expensive even when it finds the real bug. But these are real in-the-wild OSS files, not synthetic single-bug fixtures, so an off-target finding is not automatically a false positive: it may be a genuine, never-fixed secondary bug. We judged every off-target finding with the code-grounded Opus FP-judge (it reads the pre-patch source via git show, never the advisory) and classified each as a real bug, a false positive, or undetermined. Distinct sites are deduped (the verdict is a property of the code, not which model/arm/trial surfaced it), so this counts how many real secondary bugs actually live in each file.

green = judge-confirmed real bug, red = false positive, grey = undetermined. Hard-miss cases bold. The on-target CVE hits (31 findings) are not shown here — those are the planted bug, already known real.

Does one model hallucinate more? Off-target rows by model

Per-model off-target finding volume split by verdict (row-level, so repeated reports of the same site count each time). A lower % real means more of that model's noise is hallucination — the cost a human pays per report.

Method & caveats

• Models: gemma4-31b-qat (dense, 10.20.30.1) and gemma4-26b-a4b-qat (MoE, 10.20.30.2), self-hosted llama-server, free, QAT-4bit weights.
• 15 target files over 9 cases (nomad 2, craft 5, ghost 2, rest 1); a case is a hit if any file localizes. 90 runs/model = 15 files × 3 arms × 2 trials.
• Temperature 0.5 so repeats explore different reasoning paths; llama-server seed=-1 randomizes per request. A timeout/infra_error reads as no-data, never a miss.
• Detection is localization-only (free). Hard-case hits still need the truth judge to confirm SAME-bug; this report is the cheap first pass.
• Off-target findings are FP-judged once per distinct site by the code-grounded Opus judge (fpjudge_gemma_promptlab.py), which reads the pre-patch source and never the advisory; verdicts persist on judge_fp_verdict so the report is reproducible from the DB. On-target CVE hits are not judged.

Generated from nelson-gemma-promptlab.db · 2 models · 9 cases × 3 arms × 2 trials.