Careful, Not Thorough: Claude Opus 5.5 vs GPT-6 Sol on Real Code

Anthropic announced Claude Opus 5.5 at 9:31 AM PT on September 22. OpenAI announced GPT-6 Sol and Luna at 11:12 AM PT. Each launch came with a benchmark table, and neither table includes the other company’s new model. Anthropic compares against GPT-6 Astra and last generation’s GPT-5.6 Sol. OpenAI compares against Opus 5 and Fable 5.1.

So the obvious question has no vendor answer: for coding work, is Opus 5.5 worth twice the price of Sol?

The Sticker and the Bill

Opus 5.5 is Anthropic’s cheaper model, and on Anthropic’s own table it beats the $10 Fable 5.1 on most rows. The launch page adds its own hedge: “the gap between Opus 5.5 and Claude Fable 5.1 is narrower than these scores suggest.”

The per-token ratio against Sol is 2x. The per-task ratio is not. Artificial Analysis puts Opus 5.5 at max effort first on its index (58, against Sol’s 48), and measures about 119k output tokens per task against Sol’s 31k. That makes it $5.98 per task against $1.06: 2x the price and nearly 4x the tokens.

It’s hard to overstate how competitive this pricing is.

— Simon Willison, on the Sep 22 pricing

Four Tasks With Hidden Tests

I wrote four tasks, each with a hidden test suite the models never saw, from a small bug fix to a from-spec library graded against 4,176 assertions from the real node-semver. Both models ran on their default medium effort, from the CLIs I use daily (Claude Code, Codex), loading the same global rules file. Two runs per model per task. Cost is at API list price.

Every run passed everything. The only thing that separated them was the bill, and the gap grew with scope. On the largest task Opus wrote 36k output tokens to Sol’s 11k, the same ratio Artificial Analysis found.

Self-contained tasks with a complete spec don’t find a ceiling. So I went looking for one.

Six Real Changes in a Large Monorepo

The second round used a production TypeScript monorepo: about 3,700 files across several workspaces. I picked six changes that had already been merged, checked out the commit before each, and gave both models a ticket-style prompt describing the problem and the interfaces the tests would import. Each run started from a clean snapshot with no git history, so the real fix could not leak. I graded the result with the tests the original fix shipped with, and compared the files each model changed against the files the original fix changed.

Three or four runs per model per change, 20 each in total. “Clean” means the run passed every test, old and new.

Across all 20 runs each:

That is not the result my first two changes suggested. With two runs each, Opus looked like the model that follows a change into code no test covers. With twenty, that mostly washes out.

• Opus breaks less. It never broke a passing test. Sol did in five runs. Four were the same habit on Change A: rewriting an existing user-facing error message and dropping the instruction it carried. The old test caught it every time. A review pass without that test might not.
• Coverage of the change is a tie. Opus changed 85% of the files the original fix changed, Sol 84%. On Change B, one screen displayed a classification the change redefined, and no test covered it: Opus fixed it in three of four runs, Sol in none. On Change F it went the other way, and Sol covered more of the original fix.
• Neither finished the biggest change. Change C spread one setting across 26 files. Both models fixed under half its new tests, and both skipped the same part of the codebase, one the ticket named. Opus cost six times as much to get there.
• The gap grows with the mess. Opus cost 1.6x more on the tidiest change and 6.1x on the widest.

What This Doesn’t Show

• Three or four runs per change. Enough to see a habit repeat, not enough for tight rates. Four of Sol’s five regressions are one habit on one change.
• Medium effort only. Both models have higher levels. Willison’s max-effort pelican on Opus 5.5 hit the 128K output cap twice at $2.56 a try, and he suspects max is “effectively useless”. I didn’t test high or xhigh.
• One codebase, my prompts. Six changes from one monorepo, prompted by someone who knows the fixes. A different codebase or a vaguer ticket could move every number.
• List prices, not my bill. Both ran on subscriptions. On a Claude Max plan, Opus 5.5’s appetite comes out of a weekly limit. The launch raised the five-hour window and said nothing about the week.
• Vendor harnesses differ. Vals found 30 of 198 Opus 5.5 Terminal-Bench attempts were served by an older fallback model. Counting those as failures drops it from 61.6% to 53.5%. None of my 28 Opus runs fell back, but a security-adjacent codebase might.

What I’m Doing With It

• Sol where the tests and review are strong. Every regression Sol made failed a test that already existed. The one miss no test caught is what a review step is for: a second model, a review agent or a person. With both in place, Sol cost 3.5x less for nearly the same result.
• Opus 5.5 where the tests are thin and nothing reviews. Unattended runs, long agent loops, code with sparse coverage. Zero regressions in 20 runs is the property you pay for there.
• Neither for a wide change from a thin ticket. Both stalled on the 26-file change in the same place.
• Medium effort as the default for both. The published max-effort numbers measure a setting I wouldn’t use for either.

The benchmark tables will keep not comparing these two. On my own work, the difference between them was not intelligence and not coverage. It was how often each one broke something that already worked, and Sol’s price is a fair discount for doing that five times in twenty.