Anthropic Quantifies Infrastructure Noise in Agentic Coding Evaluations

Anthropic’s internal study shows that the hardware stack can dominate the signal in agentic coding benchmarks. Running Terminal‑Bench 2.0 on a Google Kubernetes Engine cluster, the team discovered a 6‑point swing in success rates between the most‑ and least‑resourced configurations—a gap that dwarfs the typical 1‑ to 3‑point differences separating state‑of‑the‑art models on leaderboards (Anthropic). The discrepancy emerged only after the engineers noticed a spike in pod‑level failures: roughly 6 % of tasks aborted because containers were killed for exceeding their allocated CPU or RAM, not because the Claude model failed to solve the problem.

The root cause, Anthropic explains, lies in how Kubernetes enforces resource limits. In their “strict” setup the per‑task specifications acted as both a guaranteed floor and an immutable ceiling, meaning any transient memory spike triggered an out‑of‑memory kill. By contrast, the sandboxing provider used by the official Terminal‑Bench leaderboard applies a more permissive policy that allows temporary overallocation, preventing premature container termination. When Anthropic relaxed the enforcement—first to a three‑times headroom and eventually to an uncapped configuration—the infra error rate collapsed from 5.8 % to just 0.5 %, and overall success scores rose accordingly (Anthropic).

Across six graduated configurations, the study found that success rates rose monotonically with added headroom, but the incremental gains between the 1× and 3× settings were statistically insignificant (p = 0.40). The sharp drop in infra‑related failures between the strict and 3× setups (5.8 % to 2.1 %) was highly significant (p < 0.001), underscoring that most of the performance swing is attributable to resource enforcement rather than model capability. Anthropic’s conclusion is clear: without consistent, well‑documented infrastructure constraints, agentic coding scores can be misleading, and the “noise” introduced by the runtime environment may eclipse the true differences between competing models.

The findings have immediate implications for the broader AI community, which increasingly relies on agentic benchmarks to gauge readiness for real‑world deployment. As Anthropic notes, static benchmarks such as SWE‑bench evaluate only the model’s output, keeping the execution environment out of the loop. Agentic tests, however, embed the runtime as a core component of the problem‑solving process, meaning that two agents with identical prompts but divergent hardware budgets are effectively taking different exams. This nuance has already prompted benchmark designers to publish recommended CPU and RAM allocations per task in Terminal‑Bench 2.0, but Anthropic warns that specification alone does not guarantee uniform enforcement.

Industry observers have taken note. Ars Technica recently highlighted Claude 3.7 Sonnet’s “extended thinking” capabilities, but did not address the underlying infrastructure variables that could affect its reported performance (Ars Technica). Meanwhile, The Verge’s coverage of Anthropic’s cautious stance toward Pentagon contracts emphasizes the company’s broader focus on reliability and safety, themes that echo the new benchmark study (The Verge). Wired’s reporting on Anthropic’s response to a U.S. military supply‑chain warning also underscores the firm’s sensitivity to external risk factors, now extending to the technical risk of evaluation pipelines (Wired). Together, these pieces suggest that Anthropic is positioning its research not just as a performance showcase but as a call for methodological rigor across the field.

In practice, the study advises benchmark operators to adopt a unified sandboxing layer that decouples resource limits from hard kills, or at least to publish the exact enforcement policy alongside the results. Until such standards are in place, Anthropic cautions that “the performance swing” observed—6 % between the most and least generous setups—should be treated as a baseline of infrastructural variance rather than a definitive measure of model superiority. For developers and enterprises that rely on these leaderboards to select production‑grade agents, the message is clear: the hardware you run on matters at least as much as the model you run.