Why I Had to Switch to GPT for Nelson

I like Claude. I use it every day. Opus 4.8 is a great model, excellent code, aware of security without being prompted most of the time, doesn’t need babysitting to complete huge categories of task. The $100 or $200 plans are a no-brainer for a software developer regularly performing any of the many coding tasks where LLMs excel. But, I recently resubscribed to ChatGPT and switched to Codex and GPT 5.5 for development on Nelson. Not because Opus can’t do the job, but because it began refusing to do the job. I’d previously found that Gemini in Antigravity refuses to do any security work, even basic auditing, so it was right out.

I’ve been doing a bunch of benchmarks of models and their capabilities for finding bugs. It’s been fun, if occasionally expensive, and has provided some surprising results (Gemma 4 31b is an itty bitty little bug finding savant, crazily good at the task for its size). The first set of benchmarks was a manageable size and expense: 9 cases selected from the bugs that have been disclosed that Mythos found. But, because that small set of cases produced sort of lackluster differentiation (the top models all found 4/9 consistently, sometimes 5, everything else limped along with 0-3 findings). And, my earliest benchmarks only tested the model once on each case; this is cheap and fast, but in a non-deterministic system, noisy. Later benchmarks proved, for example, that MiMo 2.5 Pro is spikier than DeepSeek V4 Pro, sometimes best in class, sometimes mediocre. That’s also useful information, so I started doing two or three tests against eash case. Getting more expensive and still noisy for other reasons.

The other reason for the noise, and increasing expense, is the reason I can’t use Opus for working on Nelson, anymore: We need a better way to judge.

Until now, we were focused on “how many of the known bugs can the model find?” Useful information. But, not the whole picture. They also find other bugs in the corpus, or think they do. Sometimes they get lucky and smell something in the right place, but report it as the wrong kind of bug, which is less useful, as it wastes a human reviewers time. The way I solved that problem, or attempted to solve that problem, was to have a very good judge model (Opus 4.8) read the report and decide whether it’s correct or not. Is it a real bug? Is it identified in a way that would allow a human or other model to implement a fix? I’m sure you see the problem here. Opus 4.8 is also imperfect, and expensive once I run out of my included subscription usage (as happened several times in recent runs). Opus could fail to understand the bug. Software vulnerabilities can be quite complicated to reason about and cross several file boundaries. The best models are really good at it, and getting better. But they are not infallible (and, did I mention “expensive”?). Opus also probably grades itself on a curve, not because it’s cheating, but because its understanding of the bug is already baked into its report. It would be surprising if it often found its own bug reports to be bogus.

So, judging has to change. First I asked Fable, though I knew it would switch me to Opus, because it’s got overzealous guardrails (which is fine, new model, very good at finding security bugs, government is scared because of all the scare-mongering from Anthropic and OpenAI, etc.). I wanted to see the guardrails in action, and did, immediately. Fable won’t even look at security work. But, here’s the thing, Opus 4.8 also shut down on this task. Not immediately, but once it came time to implement the bug verification harness.

Going into it, I knew the general shape of what we needed, but in the interest of maybe learning something, I asked Opus to recommend a solution. And, what it came up with was basically that for every known bug in the corpus (we’ll cover the “other bug” problem later), we reproduce the exploit. For the known bugs, we have a before and after picture of the source code, so it’s usually pretty easy to cook up some tests that exercise the vulnerable code. Which makes it possible for a model to prove, beyond a reasonable doubt, that it understands the bug by fixing the bug.

This is, of course another confounding variable. Whether a model can find bugs is a different skill from fixing those same bugs, which is also a different skill from being able to exercise the bug by writing a proof of concept. But, it is proof. And it’s proof without a fallible (and expensive) judge, or at least putting the fallible judge a step away, at a point where it is feasible to manually review the code or have multiple models assess whether the tests accurately and exhaustively prove the contestant model has provided a fix.

So, our new judging workflow for every reported bug that matches the corpus of known bugs would be:

1. Model reports a bug.
2. Model is asked to fix the bug based on their own report in a new prompt. Basically, “This bug was reported, can you fix it?”
3. Tests are run to verify the fix: it compiles, it mitigates the vulnerability, and doesn’t break other tests.

No judge model is required for known bugs with this type of verification; we now have a bounded model spend for the verification of known bugs, which is the time spent building up the harness to test whether the contestants were able to fix the bug based on their own bug report about it. And, it mostly eliminates the fuzziness of judging.

Other bugs are bugs that were identified by the contestant somewhere else in the code. A naive judging process might just mark those as false positives. But, the corpus is made up of Open Source projects. There may be other bugs. We can’t easily exhaustively detect and build tests for every bug in the corpus; we don’t know how many there are (if our contestants are to be believed, there are hundreds spread across all of the cases). We’ve been letting Opus 4.8 just decide whether those other bugs are real or false positives, and it’s pretty good at it, but not infallible and not cheap or fast. We also can’t just ignore them. A dozen false positives for every one accurate finding is probably worse than nothing. So, those bugs need something else.

So, the logical next step for solving “other bugs” is to have the models themselves write a proof of concept, and write a fix that closes the hole, and submit both of those. One or more judging models can then audit both the proof of concept and the solution to make sure the prover actually exercises a real bug without tricks, and that applying the suggested fix actually closes the hole. This probably results in a report upstream to the project, after human review, assuming it hasn’t already been fixed (this will be another step), as it would be a novel bug.

Opus described this framework, called out the various caveats, etc. It started on a good plan for implementation. And, then refused to implement it. Opus will not build a framework for discovering security bugs that also builds a proof of concept for exercising that bug and it won’t build tools that ask other models to do so. And, Opus won’t participate on either side of the benchmark process. It won’t create a proof of concept, so it can’t be competitive as a contestant, despite being one of the strongest models at reporting bugs. And, it won’t participate as a creator of verification tests.

I get it. Anthropic is trying to be the AI company that thinks about safety. But, I feel like we’ve been through this in the past in the Open Source community and came to the conclusion that putting as much information as possible and the most capable tools in the hands of everyone, especially developers, was the only way to move forward. The government and industry folks kinda came to a similar conclusion with the CVE system, etc. The bad guys are well-equipped and highly motivated, with or without Anthropic models.

So, I’m not really mad about it, Anthropic is making what they consider the safe decisions. But, if Opus won’t help me exhaustively find security bugs in my Open Source projects, I need something that will. Nelson is where I ran into the problem, and I think Nelson is also a useful tool and the benchmarks provide a useful bit of data in an area that isn’t well-covered by existing benchmarks, but the software I work on that a million or two people depend on is where it really matters. So, I guess I use GPT for security issues now. 🤷🏻‍♂️

I’m also worried about the general trend of the best models being exclusively for a select group of major corporations and governments, especially the best models for security auditing and vulnerability discovery. I’m supposed to trust the S&P 500 and the Trump administration with this power, but normal folks can’t use it to defend their own privacy? Slippery slope, yada yada yada.

I am aware of their special dispensation for security work, but I don’t want to belong to any club that would have me as a member. Seriously, though, I guess I will apply for access to Mythos, as I’d like to give it the chance of rebuttal in the benchmarks where it’s currently just a shadow on the wall of the cave in the current benchmarks. And, I work on a project running on a million or two servers that installs with administrative privileges, so I’d like to be able to use the best tools to find security bugs (that’s why I started down this path with Nelson in the first place).