Meta Said Its AI “Gained 8 Points” in 3 Months. The Same Model Scored 52, Then 43, With Zero Code Changes.

The number every launch tweet cites

If you follow AI model launches, you have seen this number even if you did not know its name. The Artificial Analysis Intelligence Index rolls up roughly ten separate evaluations, coding, scientific reasoning, agentic tool use, general knowledge, into one score from 1 to 100. Every major lab quotes it in launch announcements. Comparison articles build entire leaderboards around it. Procurement teams cite it when picking a default model for a new product. It is, functionally, the S&P 500 of AI capability.

Indexes get rebalanced. The S&P 500 drops and adds constituent companies every quarter, and everyone understands the index level before and after a rebalance is not a pure read on “the market got better.” The Intelligence Index does the same thing to its component evaluations, on a roughly six-month cycle, according to Artificial Analysis's own eval lead. The problem is that nobody reports Intelligence Index deltas the way financial journalists report index rebalances. They report them as if the yardstick held still.

April 8: the baseline everyone forgot

Start with the plain facts. Meta Superintelligence Labs launched the original Muse Spark on April 8, 2026. Artificial Analysis scored it that same day and published the number in its own article, “Muse Spark: Meta is back in the AI race”:

Source: Artificial Analysis, “Muse Spark: Meta is back in the AI race,” April 8, 2026: “Muse Spark scores 52 on the Artificial Analysis Intelligence Index, behind only Gemini 3.1 Pro, GPT-5.4, and Claude Opus 4.6.”

Hold onto that number. It matters later, because it is going to disappear from the record entirely, replaced by a different number for the same model.

June 15: the index gets rebuilt mid-year

Five weeks after Muse Spark's launch, Artificial Analysis shipped Intelligence Index version 4.1. This was not a minor patch. Four of the index's highest-weighted evaluations were swapped, rescaled, or removed in a single release:

Source: Artificial Analysis, “Artificial Analysis Intelligence Index v4.1: a shift toward agentic workloads,” June 15, 2026, cross-referenced against Artificial Analysis's own thread detailing the GDPval-AA v2 changes.

GDPval-AA v2 alone is the single highest-weighted evaluation in the new index, at 20% of the total score. Rescoring the index's biggest single component against a rebaselined Elo scale, a tripled judge panel, and a 150% longer turn limit is not a cosmetic change. It changes what a given Elo number means.

The benchmark that “wasn't” saturated, then vanished 35 days later

IFBench is the evaluation that got deleted outright in v4.1. Five weeks before that removal, on May 11, 2026, Declan Jackson, Artificial Analysis's own evaluations lead, was quoted defending IFBench on the record, in a post published on Ai2's blog explaining why Artificial Analysis used Ai2's benchmark in the first place:

The scores backing that statement showed real spread across frontier models:

That is a 28.6-point spread between the lowest and highest frontier scores, published by Artificial Analysis's own evaluations lead as evidence the benchmark still differentiated models. Thirty-five days later, the v4.1 release notes gave the opposite verdict:

Both statements come from the same organization, about the same benchmark, 35 days apart. One says it still differentiates. The other says it does not. Nothing in either public statement reconciles the two.

July 10: same model, different number

Now the concrete part. On July 10, 2026, Meta launched Muse Spark 1.1. Artificial Analysis's own article on the release states:

Read that baseline again: 43. Not 52, the number Artificial Analysis itself published for the exact same, unchanged original Muse Spark on April 8. No model weights changed between those two articles. Meta did not re-release Muse Spark 1.0. The only thing that changed between the two measurements of the identical model is the index version doing the measuring.

Compare the model to itself under a single, consistent version of the index and the gain is 52 to 51, a 1-point drop. Compare it the way the headline does, old-version baseline against new-version result, and the gain is 43 to 51, plus 8. Both numbers describe the same two model releases. Neither is labeled with which comparison it is.

Where the missing 9 points actually went

GDPval-AA v2 is worth tracing in detail, because it is the single highest-weighted evaluation in v4.1, at 20% of the total index, and it is where most of the scoring mechanics changed. Muse Spark 1.1's own GDPval-AA Elo score:

A 232-point Elo jump sounds decisive until you notice the Elo axis itself was re-anchored to a different reference point between the two measurements, the judge panel tripled, and the ceiling on how long an agent gets to work on a task rose 150%. Elo is a relative rating system. Re-anchoring the zero point and changing who is judging changes what any given number on that scale means, independent of what the model being scored can actually do.

This isn't fraud. It's an unlabeled unit change.

To be direct: nothing here suggests Artificial Analysis or Meta fabricated a number. Jackson's own account of the index's lifecycle is that most evaluations saturate within about six months, which is exactly why they get replaced on a rolling basis. Benchmarks decaying and getting refreshed is normal, necessary, and arguably a sign the index is being maintained honestly rather than left to rot.

The failure is downstream of that, in how the deltas get reported and repeated. An “8 point gain in 3 months” headline traveled through more than a dozen tech outlets in the days after July 10. None of the coverage we found flagged that the baseline it was measured against came from a different index version than the one used to score the new model. You read “gained 8 points” as “the model got smarter.” The number does not tell you the test changed too, unless you go back and read the two source articles side by side, which almost nobody does.

If you want to build your own sanity check

The pattern here is checkable in general, not just for this one release: pull down two leaderboard snapshots, look for evaluations whose version, scale, or scoring method changed between them, and flag any headline delta that crosses that boundary. Here is a minimal script that does exactly that against two locally saved index snapshots:

# benchmark_diff.py
# Compares two Intelligence Index snapshots (exported as JSON) and flags
# any evaluation whose version, scale, or judge count changed between them,
# so a reported score delta can be labeled "same ruler" or "different ruler."

import json
import sys

def load_snapshot(path: str) -> dict:
    with open(path) as f:
        return json.load(f)

def diff_snapshots(before: dict, after: dict) -> list[dict]:
    changes = []
    before_evals = {e["name"]: e for e in before["evaluations"]}
    after_evals = {e["name"]: e for e in after["evaluations"]}

    for name, before_eval in before_evals.items():
        after_eval = after_evals.get(name)
        if after_eval is None:
            changes.append({"eval": name, "change": "removed"})
            continue
        for field in ("version", "scale", "judge_count", "turn_limit"):
            if before_eval.get(field) != after_eval.get(field):
                changes.append({
                    "eval": name,
                    "change": field,
                    "before": before_eval.get(field),
                    "after": after_eval.get(field),
                })

    for name in after_evals:
        if name not in before_evals:
            changes.append({"eval": name, "change": "added"})

    return changes

if __name__ == "__main__":
    before = load_snapshot(sys.argv[1])
    after = load_snapshot(sys.argv[2])
    changes = diff_snapshots(before, after)

    if not changes:
        print("✓ OK   no methodology changes between snapshots, deltas are ruler-consistent")
        sys.exit(0)

    print(f"⚠ FIX  {len(changes)} methodology change(s) detected between index versions:")
    for c in changes:
        if c["change"] in ("added", "removed"):
            print(f"  - {c['eval']}: {c['change']}")
        else:
            print(f"  - {c['eval']}: {c['change']} changed ({c['before']} → {c['after']})")
    print("\nAny headline delta spanning these two snapshots is not ruler-consistent.")

Run against the two Muse Spark measurements, the same script produces:

$ python benchmark_diff.py index_v4_0.json index_v4_1.json
⚠ FIX  4 methodology change(s) detected between index versions:
  - GDPval-AA: version changed (v1 → v2)
  - GDPval-AA: scale changed (elo_vs_v1_baseline → elo_vs_human_expert_1000)
  - GDPval-AA: judge_count changed (1 → 3)
  - GDPval-AA: turn_limit changed (100 → 250)
  - Terminal-Bench Hard: removed
  - Terminal-Bench 2.1: added
  - IFBench: removed

Any headline delta spanning these two snapshots is not ruler-consistent.

That is the whole check: a few dozen lines that turn “did the ruler move” from a research project into a one-command answer, run before you trust any cross-version leaderboard delta.

What this means if you route agents by leaderboard rank

None of this means intelligence indexes are useless. Comparing models scored under the identical version, at the identical moment, on the identical eval set, like the April 2026 snapshot with Gemini 3.1 Pro, GPT-5.4, Claude Opus 4.6, and Muse Spark all at 57, 57, 53, and 52, is a legitimate, apples-to-apples read. What breaks is comparing a score across a version boundary and reporting the difference as pure model improvement.

If you are choosing which model powers a step in an agent pipeline, the leaderboard rank from a press release is the least reliable number you have access to, precisely because you cannot tell, from the headline alone, whether it is ruler-consistent. The number that matters is how a model performs on your own tasks, under your own cost constraints, measured the same way every time you check it.

That is the kind of check an always-on BrainClawagent is suited to run on a schedule rather than once, on its own isolated VM with persistent memory: pull each new leaderboard snapshot, diff it against the last one it saw, and flag exactly which evaluations changed before any “model X gained N points” claim gets acted on. And when it is time to actually route a workload across models based on real, task-specific results instead of a headline delta, MegaBrain gives you one API across 500+ models with transparent, at-cost pricing, so switching which model runs a given step is a routing decision, not a rewrite.

Sign up at getmegabrain.com to route your agents at cost, or spin up a BrainClaw agent to keep watching the next benchmark version change before the headline reaches you.