Claude, GPT, and Gemini Miss Dangerous Actions 30x More Often After 800K Tokens

The pitch everyone is selling

Every lab shipping an agent product right now is selling some version of the same promise: longer sessions, bigger context windows, less human babysitting. An agent that used to need a person reviewing every step now runs for hours, reading its own history, calling tools, and only checking in when something looks wrong. The entire value proposition of an always-on agent depends on one assumption holding: that whatever is watching the agent, whether that is the model itself, a separate monitor, or a set of guardrails, keeps working as the session gets longer.

Three studies published in the past few weeks tested that assumption directly, from three different angles, using three different methodologies and three different sets of models. They agree with each other in a way that is hard to dismiss as noise.

The paper that measured what “nobody is watching” actually costs

Researchers Sam Martin and Fabien Roger built MonitorBench to answer a narrow, practical question: when you ask a language model to act as a monitor, classifying whether a transcript contains a dangerous or misaligned action, does it still work once the transcript is realistically long? Prior agent-monitoring benchmarks rarely tested transcripts past 100K tokens. Real coding-agent transcripts routinely exceed 500K.

The setup is a clean natural experiment. Take the same dangerous action. Present it to a monitor model on its own, and separately, present the identical action embedded after 800,000 tokens of ordinary, benign agent activity. Everything about the action is unchanged. The only variable is how much history precedes it.

Opus 4.6, GPT-5.4, and Gemini 3.1 all missed the buried action 2 to 30 times more often than the standalone version. The paper's own conclusion is blunt: “monitor evaluations that do not consider long-context degradation are likely overestimating monitor performance.” The authors note periodic reminders and better post-training may help. Neither is standard practice in production agent deployments today.

The death zone: it is not just about running out of room

Chroma Research ran a complementary experiment across 18 models spanning Anthropic, OpenAI, Google, and Alibaba, from Claude Opus 4 and o3 down to Qwen3-8B. The task is simple: hide one true, relevant fact among 20 documents in the context window, then ask a question that requires retrieving it. Vary only where the fact sits in the stack of documents.

When the fact sits in positions 5 through 15 of the 20, what the researchers call the death zone, accuracy falls more than 30 points below the same test with the fact placed at the very start or very end. Nothing about the total context length changed between these runs. The only thing that changed is where the one fact that mattered happened to land.

Your million-token window is a marketing number

The same Chroma study cross-checked results against RULER, an independent long-context benchmark, to estimate what fraction of an advertised context window actually holds up under real, non-trivial tasks.

On hard tasks, a model sold on a one-million-token window holds reliable accuracy for roughly 250,000 to 350,000 of them, 25 to 35% of the number on the pricing page. A 200,000-token model does worse in relative terms: 60,000 to 80,000 tokens, or 30 to 40%. The rest of the window still runs. Tokens go in, tokens come out, latency and cost scale exactly like the vendor promised. The model just stops paying attention the way the label implies.

It is not distraction. It is just length.

The obvious objection to both findings above is that a longer context simply contains more distracting material, and the model is getting confused by clutter, not by length itself. Chroma tested that directly: mask every distractor, leave only the one relevant fact and inert padding, and measure accuracy as a function of input length alone.

Accuracy still fell 7.9% on average from length by itself, with no distractors in the mix at all. The model does not need to be confused by competing information. Making the transcript longer is, on its own, enough to cost accuracy.

Guardrails already break before a session gets long

The first two studies test comprehension and monitoring in isolation. AutomationBench-AA, Artificial Analysis's independent run of Zapier's agentic benchmark, tests whether the same failure shows up in real, tool-using work, and finds it shows up early. 657 tasks across Finance, HR, Marketing, Operations, Sales, and Support, executed across 40 simulated SaaS apps including Slack, Salesforce, Jira, and Zendesk, graded against roughly 12,000 assertions, capped at just 50 conversation turns per task.

Source: Artificial Analysis, “Announcing AutomationBench-AA”, July 6, 2026.

Even the best-behaved model in the field, Claude Opus 4.8, averages more than half a guardrail violation per task, and it gets there in 50 turns, nowhere close to the 800,000-token range where MonitorBench measured its worst results. The failure mode MonitorBench and Chroma describe at the extreme end of long-running sessions is already visible at the scale most production agents run today.

The collision nobody is pricing in

Put the three findings next to each other and a single shape emerges, not three unrelated data points. The industry pitch for agentic AI is: run longer, unattended, with more history, and let the model carry more of the judgment call. Every mechanism that exists to catch that same agent doing something wrong, the model's own fact-retrieval accuracy, a separate classifier monitoring the transcript, guardrail compliance inside a live tool-using session, gets measurably worse, not better, as the session gets longer. The exact conditions that make an agent economically useful, long unattended runs, are the exact conditions under which supervising it gets hardest.

None of the three papers propose a single line where a session becomes unsafe, and that absence is itself the finding. Reliability does not fall off a cliff at some known token count. It decays continuously from the first extra token onward, which means a team cannot simply pick a context-length ceiling and call the problem solved. By the time a session crosses a few hundred thousand tokens on a nominally million-token model, it is already running well past the zone anyone has actually measured as reliable, and nothing about the interface tells you that happened.

What we changed about how BrainClaw runs sessions

This research is the reason BrainClaw, our always-on agent runtime, does not treat context length as a free resource to spend down over a single unbroken session. The practical takeaway from all three studies is the same: do not let an agent's working transcript grow indefinitely and hope the model's own attention holds up. Bound it, checkpoint it, and force a fresh, curated context at defined intervals instead.

# Illustrative shape of a context-budget guard for a long-running agent

CONTEXT_BUDGET_TOKENS = 60_000       # inside Chroma's measured "hard task" reliable range
CHECKPOINT_EVERY_TURNS = 25          # force a state summary well before the death zone

def run_session(agent, task):
    turn = 0
    while not task.done:
        state = agent.step(task)
        turn += 1

        if agent.context_tokens() > CONTEXT_BUDGET_TOKENS or turn % CHECKPOINT_EVERY_TURNS == 0:
            summary = agent.summarize_verified_state()   # ✓ OK: distilled, re-verified facts only
            agent.reset_context(seed=summary)             # fresh window, not an ever-growing one
            turn = 0

        if agent.guardrail_violations():
            task.pause_for_review()                       # ⚠ FIX: stop before 0.55/task becomes 5/task

Two design choices come directly out of the data above. First, the context budget is set well inside the “hard task” reliable range Chroma measured, not the advertised window size, because the advertised number is not the one that predicts accuracy. Second, checkpoints happen on a turn count, not just a token count, because AutomationBench-AA shows guardrail violations accumulating well before any context window fills up. A model that is a few points behind on a leaderboard but genuinely reliable inside a bounded, checkpointed context is a better production choice than a frontier model left to run one unbroken session until it hits a wall nobody measured.

If you are building agents that need to run for hours or days rather than a single chat turn, MegaBrain routes each step to the model that is actually reliable at that step's scope, and BrainClaw applies exactly this kind of bounded-session, checkpointed-context pattern by default rather than leaving it to whichever frontier model you pointed the agent at. The context window arms race is going to keep shipping bigger numbers every quarter. Until monitoring research catches up with it, the safer architecture is the one that never lets a single session get long enough to reach the zone nobody has measured.