OpenAI Announces Retirement of Popular AI Coding Benchmark, Ending Ongoing Competition

OpenAI’s decision to retire SWE‑bench Verified stems from a systematic audit that uncovered fundamental flaws in more than half of the benchmark’s 1,200 tasks. The company’s internal review found that 59.4 % of the problems reject correct solutions because they impose undocumented implementation details or validate auxiliary functions that are never mentioned in the prompt, according to OpenAI’s statement to The‑Decoder. This “over‑specification” means that a model can produce a functionally correct answer yet be marked wrong simply for deviating from an arbitrary coding style, undermining the benchmark’s claim to measure genuine programming ability.

Beyond the specification issue, OpenAI highlighted a second, more insidious problem: data leakage. The firm traced a substantial portion of SWE‑bench’s test cases and reference solutions back to publicly available code repositories that have been incorporated into the training sets of leading large‑language models. In practice, GPT‑5.2, Claude Opus 4.5, and Gemini 3 Flash Preview were all able to reproduce original fixes from memory, a finding OpenAI says demonstrates that progress on the benchmark increasingly reflects memorization rather than reasoning. The company warned that this “contamination” skews rankings, allowing models that have simply seen the benchmark data to appear more capable than they would be on truly novel coding tasks.

OpenAI’s announcement also signals a strategic shift in how the industry will evaluate AI‑assisted development. The firm is recommending SWE‑bench Pro—a newer, subscription‑based version that tightens task definitions and isolates test data—as the interim standard, while it builds a proprietary, non‑public suite of coding challenges. By moving away from an open, widely‑scrutinized benchmark, OpenAI can control the quality of evaluation data and reduce the risk that competitors, especially open‑source projects, benefit from “contaminated” metrics that inflate their apparent performance. The move could also pressure other benchmark providers to adopt stricter data hygiene practices, a point noted by analysts who track benchmark integrity across the AI sector.

The retirement of SWE‑bench Verified does not erase its historical impact. Since its launch in 2023, the benchmark served as the de‑facto yardstick for AI coding prowess, prompting a “gold‑rush” of incremental improvements from OpenAI, Anthropic, Google, and a host of Chinese open‑weight models. Weekly leaderboards drove research agendas, with teams fine‑tuning model prompts and sampling strategies to eke out marginal gains. However, the same competitive pressure may have accelerated the very leakage problem OpenAI now cites, as participants routinely scraped benchmark data to train and validate their models.

Industry observers note that the episode underscores a broader limitation of AI benchmarks: they can provide a useful signal but rarely capture the full spectrum of real‑world software development. As The‑Decoder points out, even a “gold standard” like SWE‑bench Verified can become obsolete when its test suite no longer reflects novel problem‑solving. The retirement therefore invites a reassessment of how progress is measured, pushing developers and researchers toward more dynamic, task‑agnostic evaluations that better mimic the open‑ended nature of production code.