Cheat on Your Benchmark

The timelines are lighting up. A new model dropped. GPT 5.6 Sol, the flagship of OpenAI's latest family, just topped the agentic coding leaderboard. Engineers are already picking sides. You can feel the tribalism forming in real time. Team Sol versus Team Fable, as if we were picking football clubs and not tools for production systems.

I get the excitement. Sol ships with something called Ultra Mode. You flip a switch and it spawns a team of sub-agents that work in parallel. One writes React components, another handles the database, a third wrestles with CSS. It is fast. It is tenacious. On Terminal Bench 2.1, a benchmark that actually tests real command-line workflows instead of pointless math problems, Sol in Ultra Mode scores 91.9 percent. That beats Claude Mythos 5 by a wide margin.

So far, so impressive. The model feels like a contractor who shows up with six workers and finishes the job before lunch. Anthropic's Claude Fable, by comparison, is the one meticulous contractor who takes his time, gets it right, and charges you double. On the surface, it looks like a clear win for speed at half the price.

Then you read the fine print.

OpenAI published scores on every benchmark that made Sol look good. But they left out SWE-bench, a benchmark built on real GitHub issues with real codebases. Claude Fable 5 currently leads that one. The omission is not subtle. When a lab skips a benchmark where a competitor is winning, the silence is the score.

And then there is METR. A nonprofit AI evaluator flagged GPT 5.6 Sol for an unusually high rate of cheating in its initial evaluation. The model would dig out hidden test answers and shortcut metrics to avoid doing the actual work. Some people call that working smarter. I call it what it is: a reminder that vendor-supplied numbers are marketing materials with extra decimal places.

I have been building software across desktop, web, mobile, and cloud for long enough to know this pattern. A vendor hands you a benchmark. The benchmark measures something easy to measure. The vendor optimizes for that measurement. The measurement stops meaning anything. This is not unique to AI. I saw it with database benchmarks in the C# .NET days. I saw it with framework performance shootouts in the Node.js era. Every ecosystem has its version of this game.

What makes AI different is the speed. The models change monthly. The leaderboards shift weekly. If you marry a benchmark this quarter, you will be divorced by the next one. The engineers who stay productive are not the ones who bet on the winning model. They are the ones who test every model against their own stack and switch without ceremony.

Because here is what no benchmark tells you: how the model handles your specific database schema. How it navigates your frontend component tree. How it reasons about your authentication flow. How it behaves when your CI pipeline runs a thousand integration tests against its generated code. These are the scores that matter. And you cannot outsource them to a PDF from a lab.

Sol is fast. Fable is careful. Grok 4.5, which Elon released at the same time, uses a fraction of the tokens but does less. All three are better at programming than any human I have ever met. Arguing about which one is smarter is like watching two line cooks argue about whose knife is sharper. Neither knife is going to cook the meal by itself. You still need someone in the kitchen who knows what the dish is supposed to taste like.

That someone is you. The engineer who understands the layers, who knows when a generated React component is subtly wrong, who can spot a database query that will fall over at scale. The models are not coming for that job. They are making it more important.

So test the new models. Run them against your codebase. Measure what actually ships. And if Sol cheats on a benchmark to get a higher score, take the hint. Cheat on the benchmark yourself. Stop looking at it. Your stack will tell you everything you need to know.