SWE-Bench Pro was meant to be a more challenging and contamination-resistant successor to SWE-Bench Verified. However, audits of SWE-Bench Pro have estimated that 30% or more of tasks are broken, leading us to designate this benchmark as Flawed.
Methodology
SWE-Bench Pro consists of 1,865 tasks: 731 are public, 276 are private, and 858 are held-out. The tasks were sourced from actively maintained repos and are meant to simulate long-horizon professional software engineering workflows.
Multiple audits have been performed on the public set:
- On February 24th, 2026, Jonathan Gabor audited 100 random SWE-Bench Pro problems and found 83 had issues.1 The main error type was “requirements inflation” - implementation details specified in the requirements section were not tested.
- On May 26, 2026, Datacurve performed an LLM-assisted audit and found 24% false negatives and 8.5% false positives.2
- On July 8th, 2026, OpenAI published an audit estimating 30% of tasks were broken.3 After an initial datapoint analysis pipeline, each flagged task was independently reviewed by five experienced software engineers. Errors mainly fell into the following four categories: misleading prompts, overly strict tests, underspecified prompts, and low-coverage tests.
Given these results, it seems very likely that more than 20% of questions contain scoring defects, thus we designate this benchmark as Flawed.
In addition to these scoring defects, Poolside documented numerous reward hacking behaviors in the task environment.4
Acknowledgements
This review relies on published audits by OpenAI, Jonathan Gabor, Datacurve, and Poolside.
Rubric
1. Reviewability
| Level | Meaning | Status |
|---|---|---|
| Full | All tasks and scoring logic inspectable*, and harness/API settings used for each model (reasoning effort, token/time limits, tool access, system prompts) are fully disclosed | [proceed to 2] |
| Partial | A representative sample of tasks and scoring logic inspectable, with full harness/API settings disclosed | [proceed to 2] |
| Inadequate | Limited, biased, or no inspectable tasks or scoring logic, harness/API settings undisclosed | [stop → NEI] |
* Either publicly or privately to reviewers.
2. Scoring
This is the minimum standard required to be Verified; failing any of these items results in a Flawed verdict.
| Scoring | |
|---|---|
| Examples |
|
| Default threshold for Flawed | ≥20% of inspected sample† contains errors or there is an issue that corrupts grading at scale |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Benchmark Consistency | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Leaderboard has incomparable results from different versions |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Elicitation | |
|---|---|
| Examples | Model elicitation is extremely constraining and is not the focus of the benchmark:
|
| Default threshold for Flawed | Substantially reduced performance compared to reasonable alternatives for the tasks |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Bias in Evaluation Setup | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Material model-specific advantage found |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | The leaderboard combines runs with a capped cost limit and turn limit of 50 (greyed out) and runs with an uncapped cost and with a turn limit of 250. |
† For benchmarks with >50 available tasks, we will sample a random set of 50 (stratified by category, when present). If the observed error rate is 15–25%, we will expand the sample to 100. For benchmarks with ≤50 available tasks, all available tasks will be assessed.
3. Evaluation Quality
This is the standard we would like all benchmarks to meet, but it is not necessarily disqualifying to omit or fail these items.
| Question | Status | Notes |
|---|---|---|
| Elicitation and resource adequacy: Are the resources given to models (reasoning token/turn budget, tool access, etc.) sufficient for them to perform near their ceiling? | SufficientConstrainingUnreasonably constrainingUnknownNot reviewed | — |
| Scaffold fairness: What scaffold does the leaderboard report? | Shared common scaffoldMix of model-specific and common scaffoldsModel-specific scaffoldsNot reviewed | — |
| Is there evidence/risk of contamination? | Not reviewed | — |
| Has human completability been assessed? | All tasksRepresentative set of tasksPoor implementation (Unrepresentative set of tasks, unreasonable set of participants)Not establishedNot reviewed | — |
| Score range (if possible to estimate) | — floor, — ceilingNot reviewed | — |
| Statistical adequacy: how many runs/model (≥ 5 recommended for error bars) | — runs/modelUnknownNot reviewed | — |
| Construct Validity | Measures stated capabilitiesPartially measures stated capabilitiesDoes not measure stated capabilitiesNot reviewed | — |
Disclaimer
This is a partial review. We stopped inspecting once we discovered errors that exceeded our published threshold for a flawed benchmark.
We attempted to reach out to the benchmark developer prior to launch, and if they write a response to this review we will link it here. If you’re the benchmark developer, please email us at reviews@epoch.ai with a link to your response if you would like us to include it here.
About Benchmark Reviews
Epoch AI’s Benchmark Reviews are independent reviews of external AI benchmarks. Our documentation describes the rubric behind this verdict, how we choose which benchmarks to review, and answers frequently asked questions.