160 Ideas, One Pipeline, Two AI Judges, Two Different Scores

An AI research agent proposed 160 new hypotheses from 1,475 NASA satellite datasets. Grade those same 160 ideas with GPT-5.2 and you get one scorecard. Grade them again with Claude Sonnet 4.6, and the numbers change, even though nothing else did. The rankings didn't move. The scores did. If you've ever cited an agent benchmark's "quality score" as if it were an objective fact, this is the paper that says you probably shouldn't have.

How EO-Agents works: three agents, 1,475 datasets, 160 hypotheses

EO-Agents(arXiv 2607.01584, accepted to the ICML 2026 AI for Science Workshop) chains a filter agent, a generator agent, and an evaluator agent, all grounded in NASA's Earth Observation Knowledge Graph plus a GNN trained on dataset co-usage patterns. Pointed at roughly 1,475 NASA Earth-observation datasets, it produced 160 hypotheses spanning glaciology, vegetation phenology, and aerosol-cloud interactions. It is a real attempt at the unglamorous part of science that agents rarely get evaluated on: proposing what to study next, not just answering a question someone already framed.

The judge problem, in plain numbers

The authors scored all 160 hypotheses twice: once with GPT-5.2 as judge, once with Claude Sonnet 4.6. The relative ranking of hypotheses stayed consistent between judges, so the pipeline reliably surfaces its best ideas over its worst ones. What didn't stay consistent was the absolute score each judge assigned. That is a small, honest admission with a big implication: a single "quality score" in an agentic-science paper only means something once you know which judge produced it, and most benchmark write-ups don't disclose that the number would move with a different one.

Aurora 1.5: one open weather model, 26 variables

Microsoft Research expanded Aurora from 4 to 26 weather and Earth-system variables at hourly resolution and added probabilistic ensemble forecasting, training on ECMWF HRES analysis data from 2018–2023. Checkpoints are open on Hugging Face — a real, weights-available model a local kernel can actually run, not just a benchmark table.

Flint: a chart language built for agents, not humans

Microsoft also open-sourced Flint, an intermediate language and compiler that lets an agent emit a constrained specification instead of freehand plotting code, then compiles that spec deterministically to Vega-Lite, ECharts, or Chart.js. It ships with an MCP server (flint-chart-mcp) for direct use from an agent or chat session. A chart an agent can regenerate identically from the same spec is a small thing, but it's the same instinct behind a reproducibility record: don't trust an agent's output just because it looks right once.

What this means for local-first, reproducible science

Three unrelated releases, one shared theme: none of them trust agent output by default anymore. EO-Agents discloses that its scores are judge-dependent instead of hiding it. Flint makes chart generation deterministic instead of vibes-based. Aurora ships weights, not just a leaderboard row, so its 88.9% number is something you can rerun yourself. That's the exact gap a workbench with an independent reviewer and an auditable export trail is built to close — verification isn't a feature you bolt on after the agent is done, it has to be structural.