Naked in the PR

I spent some time testing a new AI coding model called Grok 4.5, released jointly by SpaceXAI and Cursor. Before this version, Grok had no reputation in software engineering. It rarely appeared in discussions about the best models for the job. Its predecessors were known for general conversation, not for writing production code. Then version 4.5 arrived, and suddenly it was producing code that rivaled Claude Opus and GPT. Faster, cheaper, and competitive. I was skeptical, so I put it through real TypeScript and Python projects instead of relying on published scores.

Here is what I learned about the model and about evaluating AI code in general.

Start with the benchmarks. The Cursor team published impressive numbers with the launch. But if you read the footnotes, one benchmark was quietly excluded: CursorBench. The reason was an earlier snapshot of the Cursor codebase had been accidentally included in the training data. The team could not know how much of the performance was real capability versus memorization. They disclosed this themselves, and they removed the contaminated data for future models. That level of transparency is rare and respectable. But it also proves a point that every engineer should remember. Benchmarks are not reality. It is like a student who memorizes past exam papers instead of learning the subject. The scores look great until a new question appears.

The only evaluation that matters is real code in real projects, reviewed by someone who reads every line.

So I tested Grok 4.5 on real TypeScript and Python projects. The results were mixed in a way that tells you more about AI coding in general than about this specific model.

When it was good, it was genuinely good. In one TypeScript project, a tool that pulls data from an external API and processes it, Grok 4.5 generated clean code with solid error handling. It sorted API results using a stable tie-break on record IDs, which mattered because the API did not guarantee the order of its responses. Without that one line, running the same operation twice could produce different results. The model understood the requirement and wrote the selection logic as clean functional expressions instead of the tangled conditionals that AI models tend to produce. All tests passed on the first run, and the linter was completely clean.

But when it was bad, the mistakes were the kind that benchmarks never catch. They are the kind that only appear when you read the code itself.

In the same project, the model wrote a function to pick the latest compatible version from a list of dependencies. It compared version numbers as strings. This means version 2.9.11 would be considered smaller than 2.9.9, because string comparison reads character by character. The project already had a version comparison utility imported and ready to use. The model ignored it and wrote its own string-based implementation instead. A human reviewer spotted it immediately. A benchmark would have reported a passing score because the tests were written to pass against the code the model generated, not against edge cases the model never considered.

In a Python project, a data processing pipeline, Grok 4.5 re-used an existing configuration schema instead of creating a parallel one, which showed good judgment. But it also left behind an imported module that was never called anywhere in the file. The model started with one approach, switched to another mid-task, and never cleaned up the dead import from the first attempt. This pattern appears across every model I have tested, not just this one.

Prompt adherence was another differentiator. I asked both Grok 4.5 and Fable 5 to build a data visualization dashboard using only HTML and vanilla JavaScript, no frameworks or build tools. Grok 4.5 followed the instruction exactly. The resulting dashboard was simple, visually basic, but it was pure vanilla JavaScript from start to finish. Fable 5, despite producing a better looking dashboard with a more sensible architecture, ignored the constraint and introduced React and a bundler. It optimized for developer experience over obedience to the specification. Prompt fidelity is a capability worth measuring in its own right, but no benchmark tracks it.

This gets at a deeper difference in how these models work. Some models like Grok 4.5, Opus, and GPT are designed for fast back-and-forth interaction. You give them a task, and within a minute they return a result. You review, refine, and iterate. The cycle is tight, and you stay in the loop the entire time. Other models like Fable 5 are built for long-running agentic loops. You give them an end goal and let them deploy sub-agents that work for minutes or hours, researching, coding, testing, and debugging on their own. Neither approach is better. They suit different workflows. The mistake is treating one as universally superior and dismissing the other.

What connects all of these observations is a single truth. No AI model today produces code that can be shipped without human review. Models that pass every benchmark produce string comparison bugs, dead code, and unnecessary reimplementations. Models that follow instructions perfectly produce simpler output that may need refinement. Models that optimize for quality sometimes ignore what you asked for. There is no universal winner. There is only the right tool for the specific job, evaluated by a human who reads the output.

I still use these tools every day. They make me faster. Grok 4.5 genuinely impressed me in several projects this time, and I will keep using it. But I read every line before it ships. The benchmark scores are marketing material. The diff is the truth.