AI “Solved” Coding at 95.5%. This Week's Real-Work Benchmarks Say 48.6% and 21%.

The number everyone is quoting

If you have followed AI coding news at all in the past few weeks, you have seen this claim in some form: AI has solved software engineering. The evidence is SWE-bench Verified, the benchmark that grades whether a model can fix a real, historical GitHub issue by producing a correct patch.

As of July 6, 2026, Claude Mythos 5 tops that leaderboard at 95.5%. Claude Fable 5 follows at 95%, Claude Opus 4.8 at 88.6%. When the benchmark launched in 2024, the top score was 4.4%. Two years later, the field is bunched within a few points of the ceiling.

That is a genuine engineering achievement, and it is worth being precise about what it measures. Every SWE-bench task is a single, well-defined bug in a single repository. The target patch already exists in the historical record. Grading is automatic: does the generated diff pass the same test suite the real fix passed? There is no ambiguity about what “done” means, no multi-app state to track, and no business rule to avoid breaking along the way.

That narrowness is exactly why two independent groups spent the past few months building something harder.

The benchmark built to grade the rest of the job

On July 6, 2026, Artificial Analysis published its run of AutomationBench-AA, built on top of Zapier's AutomationBench. The premise: stop asking whether a model can write correct code, and start asking whether an agent can run a business process end to end, the way a human operator would.

The task set: 657 workflow-automation tasks pulled from real patterns on Zapier's platform, spanning six departments (Finance, HR, Marketing, Operations, Sales, Support), executed across 40 simulated SaaS environments including Gmail, Google Sheets, Slack, Salesforce, Zendesk, Jira, and HubSpot. Nearly 12,000 individual assertions check whether the correct data landed in the correct system, capped at 50 conversation turns per task so a model cannot retry its way to a pass. Grading is deterministic: automated checks against the actual final state of each simulated app, not a second model judging the first model's output.

Source: Artificial Analysis, “Announcing AutomationBench-AA”, July 6, 2026.

The best model in the world, tasked with running an actual multi-app business workflow end to end, completes fewer than half of its objectives. Compare that to the 95.5% isolated-bug-fix number from the same week and the gap is not subtle.

The highest-stakes work is where agents fail most

AutomationBench-AA breaks results down by department, and the pattern is not random. It tracks directly with how much a mistake costs.

Finance tasks, the ones where an error means money moved to the wrong place or a ledger entry that does not reconcile, complete at roughly half the rate of support and operations tasks. The category where accuracy matters most is the category current agents handle worst.

Agents do not only miss the goal. They break the rules getting there.

AutomationBench-AA does not only score whether an agent reaches the objective. Each task also carries explicit guardrails, business rules the agent is told not to violate regardless of whether it completes the task. Things like: do not email a customer twice, do not modify a record you were not authorized to touch, do not close a ticket before the linked invoice is settled.

A guardrail-based grading approach is a useful mental model for anyone building an agentic pipeline, whether or not you ever run this exact benchmark. It separates two different failure modes that a single pass/fail score hides: the agent that does nothing, and the agent that does the wrong thing. Something like this:

# Illustrative shape of an AutomationBench-style assertion set
task = {
    "objective": "Refund customer #4021 for order #88213 and notify them",
    "guardrails": [
        "do_not_refund_twice",
        "do_not_email_customer_more_than_once",
        "do_not_modify_orders_outside_scope",
    ],
}

def grade(final_state, actions_taken):
    objective_met = final_state.refund_issued and final_state.customer_notified
    violations = [g for g in task["guardrails"] if violates(g, actions_taken)]
    return {
        "objective_met": objective_met,   # ✓ OK or ⚠ FIX
        "guardrail_violations": violations,  # graded independently of the objective
    }

# A model can score "objective_met: True" and still rack up guardrail_violations.
# Pass/fail alone would hide that. AutomationBench-AA grades both.

A second scoreboard, two days later, says the same thing

On July 8, 2026, Snorkel AI published results from GDPVal+, its expert-created dataset of professional workplace tasks. The methodology is different from AutomationBench-AA but the target is the same: measure real work, not a proxy for it. Roughly 2,000 tasks span documents, spreadsheets, and presentations authored by domain experts across sectors including legal, education, healthcare, and QA analysis. Every model ran inside a full agent harness, tool use, document generation, and live web search all enabled, and a pass required a perfect score against an expert rubric. Partial credit did not count.

Source: Snorkel AI, GDPVal+ Edition 1 results, July 8, 2026.

Grok 4.5, released the day before as SpaceXAI's new frontier model, tops the field. It is a genuinely strong result relative to the other frontier models, and it still means the best agent in the world, doing real professional deliverables against an expert rubric, is right fewer than one time in three.

The spread by profession tells its own story. Even the category where every model does best, education, tops out at 58% for the leader and the rest of the field is still below half.

Why the same models score 95% and 21% in the same week

This is not a contradiction between two benchmarks. It is a definition problem, and once you see it, it explains most of the confusion in how “AI agents” get talked about right now.

SWE-bench measures a narrow, well-specified, automatically verifiable skill, and frontier models are, by now, extremely good at it. AutomationBench-AA and GDPVal+ measure whether an agent can operate with the ambiguity, the multi-step state tracking, and the implicit rules that an actual job requires. Those are different skills. The industry has spent most of 2026 quoting the score for the easier one.

The cost angle nobody is benchmarking yet

There is a fourth data point worth pulling into this picture, because it changes how you should read the GDPVal+ table above. Grok 4.5, the model that topped GDPVal+ at 29%, is also the cheapest frontier model in the comparison. Artificial Analysis puts it at $2 per million input tokens and $6 per million output tokens, and measures its cost per completed Intelligence Index task at $0.31, roughly a fifth of what Claude Sonnet 5 (max) costs per task on the same index, while scoring higher.

That matters because “best score” and “best model to run in production” are not the same question, and the gap between them is exactly the gap AutomationBench-AA and GDPVal+ expose. A 657-task, 40-app benchmark with a 50-turn cap per task is not something you run once. It is something you run continuously, against live workflows, at whatever volume your business generates invoices, tickets, and leads. At that volume, a model that is a few points behind on raw pass rate but a fifth of the price per task is frequently the better production choice, especially once you route the narrow, well-specified sub-steps described above to smaller models entirely and reserve the frontier tier for the genuinely ambiguous calls.

Why the timing is not a coincidence

Artificial Analysis and Snorkel did not build these benchmarks at random. They built them in the same week SWE-bench Verified effectively saturated. When every frontier model clears 90%+ on a benchmark everyone has already optimized against, that benchmark stops differentiating anything. The only honest response is a harder, less gameable test, one built from real deliverables instead of historical GitHub commits.

What this means if you are building agents

The practical takeaway is not “agents don't work.” Claude Fable 5 completing 48.6% of end-to-end, multi-app business objectives with zero human intervention is still a real capability that did not exist two years ago. The takeaway is about task design.

The tasks agents handle at SWE-bench-level reliability share a shape: narrow scope, unambiguous success criteria, verifiable output. The tasks agents still fail at scale share the opposite shape: multi-step, cross-system, judgment-dependent, with implicit rules nobody wrote down as a checklist. If you are shipping an agent into production today, the highest-leverage move is decomposing the second kind of task into more of the first kind, narrow, checkable sub-steps with explicit guardrails, rather than handing an agent one big ambiguous mandate and hoping the guardrails hold.

# Two ways to hand an agent the same job

# Ambiguous mandate (what AutomationBench-AA tasks look like by default)
agent.run("Reconcile this month's invoices and notify anyone with a discrepancy")

# Decomposed into narrow, checkable steps with explicit guardrails
invoices = agent.run("List invoices with status != 'reconciled'", verify=is_list)
for inv in invoices:
    delta = agent.run(f"Compute expected vs actual for invoice {inv.id}", verify=is_number)
    if delta != 0:
        agent.run(
            f"Draft (do not send) a discrepancy notice for invoice {inv.id}",
            guardrails=["do_not_send_without_review", "do_not_modify_invoice_amount"],
        )
# Every step has a narrow, verifiable output and an explicit rule the agent cannot cross.
# This is closer to a SWE-bench task than an AutomationBench task, by design.

That is also, not coincidentally, the argument for routing different steps of an agentic pipeline to different models rather than one frontier model for everything. A narrow, well-specified sub-task does not need the most expensive model available; it needs a model that is reliable on narrow, well-specified tasks, which is a much cheaper bar to clear than reliable on ambiguous, real-world judgment calls.

Expect both benchmarks to move fast over the next few quarters, the same way SWE-bench did. AutomationBench-AA and GDPVal+ are three and one days old at time of writing; there is no reason to think 48.6% and 29% are ceilings rather than starting lines. The number worth tracking is not whether some future model eventually clears 90% on these too. It is whether the gap between “narrow task” scores and “real workflow” scores closes, or whether every harder benchmark gets replaced by a still-harder one the moment it saturates. Either way, the score to trust right now is the one measured against your actual workflow, not the one on the leaderboard someone else optimized for.