GPT-5.6 Sol Wins One Benchmark by 13 Points, Loses Another by 15. OpenAI Retracted the One It Lost.

What actually shipped on July 9

OpenAI previewed GPT-5.6 in late June and took the full family generally available on July 9, 2026: Sol (the flagship, built for complex reasoning and long-running agentic work), Terra (the balanced everyday tier), and Luna (the fast, cheap tier). All three share a 1,050,000-token context window, a 128,000-token max output, and a February 16, 2026 knowledge cutoff.

Source: Simon Willison, “The new GPT-5.6 family: Luna, Terra, Sol”, July 9, 2026. GitHub Copilot rolled the same three tiers out the same day, though Sol is restricted to Copilot Pro+, Max, Business, and Enterprise; Terra and Luna are available on the base Pro plan.

The benchmark OpenAI put in the announcement

The headline result OpenAI led with is Agents' Last Exam, a benchmark that evaluates long-running professional workflows across 55 fields, scoring models across five functional layers it calls Brain, Eyes, Body, Hands, and Feet (reasoning, visual perception, orchestration, tool invocation, and runtime substrate). It is a hard, deliberately broad test of whether a model can carry a real, multi-step professional task from start to finish.

A 13.1-point margin on a hard, broad benchmark is a real result, and it is the number that led OpenAI's own announcement post. It is also, on its own, an incomplete picture of how Sol compares to Anthropic's current top model.

The benchmark OpenAI un-recommended two days before launch

On real-world coding tasks measured by SWE-Bench Pro, the pattern reverses hard.

Two days before GPT-5.6's GA launch, on July 8, 2026, OpenAI published its own audit of SWE-Bench Pro, the Scale AI-built benchmark it had previously recommended as a SWE-bench Verified replacement. The audit's finding: using automated screening plus human review, OpenAI found 27.4% of SWE-Bench Pro tasks (200 of 731) flawed. Independent developers it commissioned to re-review the same tasks found an even higher rate: 34.1% (249 tasks).

OpenAI is withdrawing its endorsement of SWE-Bench Pro entirely rather than proposing a fix, the same move it made against the original SWE-bench Verified earlier in 2026 after finding that benchmark contaminated and saturated. Artificial Analysis reached its own, independent decision to pull SWE-Bench Pro from its rankings around the same time, citing a different problem: it found some models were copying the correct solution directly out of a project's commit history instead of solving the task.

Source: the-decoder.com, “OpenAI finds roughly 30 percent of popular AI coding test is broken”, July 2026.

Read the two results together, not separately

None of this means SWE-Bench Pro's 80% score for Claude Fable 5 is wrong, or that Sol's 64.6% is secretly higher. A broken benchmark does not reliably favor one model over another; OpenAI's own audit does not claim the flaws explain the 15.4-point gap. What it does mean is that the one benchmark in this comparison where Sol clearly loses is also the one benchmark that got formally discredited, by both the company that lost on it and an independent evaluator, within 48 hours of the model that lost being announced. Reasonable people can read that as a coincidence of timing. It is at minimum a reason not to take either headline number, the 13.1-point win or the 15.4-point loss, as the full story on its own.

Journalist Simon Willison, who has used both models directly, put it plainly: “it hasn't struck me as better than Fable at the kind of complex coding tasks” he tests against. That is one person's hands-on impression, not a benchmark, but it lines up with the SWE-Bench Pro gap more than with Agents' Last Exam.

The benchmark that is not in the announcement at all

There is a third comparison worth noting precisely because OpenAI did not publish it. GDPval is OpenAI's own benchmark, built earlier in 2026 at real cost to measure model performance against expert-graded, real-world professional deliverables, the same category of test AutomationBench-AA and Snorkel's GDPVal+ now cover industry-wide. Economist and AI researcher Ethan Mollick flagged the omission directly: “I appreciate the clarification about bad benchmarks, but they spent a lot of money developing a very good benchmark of autonomous model ability at hard tasks, GDPval, and haven't reported it for GPT-5.6.” A company that will publish a detailed audit explaining why it is retracting a benchmark it lost on, in the same week it declines to publish a benchmark it built itself, is telling you something about which numbers it expects to help the story and which it expects to hurt it.

The calmer picture: two more independent scoreboards

Zoom out past the head-to-head and Artificial Analysis's own composite indices tell a more measured story, closer to “strong new model at a lower price” than either extreme above.

Sol lands one point below Claude Fable 5's 60 on the Intelligence Index, at roughly a third of Fable 5's cost. Terra scores within 4 points of Sol on the Coding Agent Index while costing roughly 60% less per completed task. Luna gives up 5 points versus Sol and cuts cost per task by roughly 80%. None of that is a scandal in either direction, it is a normal price-performance curve, and it is a more honest input for a routing decision than either single headline benchmark above.

A third, independent data point points the same direction. Terminal-Bench 2.1 grades models on real terminal sessions, multi-step shell tasks executed in a sandboxed environment and checked against the actual end state, closer in spirit to AutomationBench-AA than to a single-turn coding puzzle. Aggregated published scores put Sol Ultra at 91.9%, base Sol at 88.8%, Terra at 84.3%, and Luna at 82.5%, against GPT-5.5 at 83.4%, Claude Mythos 5 at 88.0%, and Claude Opus 4.8 at 78.9%. On this eval, Sol edges Mythos 5, and Terra alone beats GPT-5.5 while costing roughly half as much per token.

It is also worth remembering why any of this benchmark churn is happening at all. OpenAI deprecated the original SWE-bench Verified earlier in 2026 after finding it contaminated and saturated, the same reasoning it is now applying to SWE-Bench Pro. Two retirements inside a year is not a one-off embarrassment, it is the shape of every popular coding eval's life cycle right now: ship, get optimized against, saturate or get gamed, get retired. Treat any single benchmark number in this piece, including the ones favorable to Sol, as a snapshot with a shelf life, not a verdict.

The routing decision this data actually supports

Three independent scoreboards (Agents' Last Exam, the Intelligence and Coding Agent Indices, and Terminal-Bench 2.1) point at a version of GPT-5.6 that is genuinely competitive at the top end and meaningfully cheaper in the middle and bottom tiers. The one benchmark that would have told a cleanly negative story, SWE-Bench Pro, is the one benchmark two independent parties spent the days around launch discrediting. Whatever the truth of Sol's real-world coding ability turns out to be once a trustworthy successor benchmark exists, the Terra and Luna pricing is not in dispute, and it is the more actionable number for anyone running agents at volume.

# Illustrative shape of a tier-routing rule for GPT-5.6

def choose_tier(task):
    if task.is_long_horizon_agentic or task.requires_deep_reasoning:
        return "sol"      # ✓ OK: pay for the reasoning ceiling when the task needs it
    if task.is_interactive_coding or task.is_everyday_agentic:
        return "terra"    # ✓ OK: ~60% cheaper per task, within 4 points of Sol on Coding Agent Index
    return "luna"          # ⚠ FIX: don't default to Sol for docstrings, lint fixes, boilerplate

# The mistake this guards against isn't picking the wrong tier once.
# It's defaulting every request to Sol because it's the one in the announcement post.

That is exactly the decision an auto-router is built to make on every request instead of once at design time. MegaBrain's auto-balanced routing already treats Sol, Terra, and Luna as separate options to weigh per task against the Coding Agent Index and cost-per-task numbers above, rather than pinning every call to whichever tier a launch post led with. If you are building an agent that runs continuously rather than answering one prompt at a time, the tier decision compounds every single call, and getting it right matters more than any single benchmark headline, contested or not.