Separating signal from noise in coding evaluations
OpenAI Blog · 2026-07-08
OpenAI audits SWE-Bench Pro and finds approximately 30% of its 731 tasks are broken due to overly strict tests, underspecified prompts, and low-coverage tests, retracting its earlier recommendation to adopt the benchmark.
Extraction
Topics: coding-benchmarksbenchmark-evaluationai-evaluationsoftware-engineering-agents
Claims
- Approximately 30% of SWE-Bench Pro tasks are broken, based on a combined human-and-agent audit pipeline.
- Four primary failure categories were identified: overly strict tests, underspecified prompts, low-coverage tests, and misleading prompts.
- Human reviewers marked more tasks as broken than the investigator-agent pipeline, with humans selecting 'low-coverage tests' as the most common issue at 9.4% versus 4.1% for agents.
- Frontier models improved from a 23.3% to 80.3% pass rate on SWE-Bench Pro's public split in eight months, suggesting rapid saturation of a flawed benchmark.
- OpenAI retracts its earlier recommendation to adopt SWE-Bench Pro and calls for new benchmarks built specifically to test model capabilities by experienced software developers.
Key quotes
We estimate that ~30% of SWE-bench Pro tasks are broken, and advise that model developers carefully examine results.
Issues and pull requests from open-source repositories were originally created for human collaboration, often through long back-and-forths between maintainers and contributors. As a result, problem descriptions, merged code, and unit tests do not always line up to form clean, isolated tasks for evaluating models reliably.
Given the issues uncovered in this analysis, we retract our earlier recommendation to adopt SWE-Bench Pro.