DeepSWE is a software engineering benchmark. Its tasks were written from scratch and span 91 repositories. We found issues in at least 23 out of 113 tasks (over 20.3%), leading us to designate the benchmark as Flawed.
Methodology
We ran a sweep using Opus 5 and Sol 5.6 to analyze all 113 tasks and a subset of trajectories to detect errors. We audited and confirmed each error by hand. We started by looking only at false negatives, and after identifying 23 instances we had surpassed the threshold of evidence for marking the benchmark as Flawed, and therefore did not review other error types.
Of the 23 observed instances of false negatives, 18 (78%) were due to an error that we believe could affect any task. In these cases, the models edited the existing tests. Editing the existing tests and adding new tests can break the grader, but the models have no way to know this and are not instructed to avoid doing so.1
Representative Errors
To grade a submission, the verifier applies the agent’s submitted changes in a fresh container. It also adds the hidden tests and updates any existing tests that would be rendered obsolete by the task. Before doing so, the verifier discards any changes the agent made to these files. This verification system can cause the grading process to fail, mainly in the following ways:
- Duplicated symbol definitions: The agent defines a symbol, usually a test function, that the verifier’s tests also define in the same package. Example: While solving the onedump-dump-encryption-pipeline task, Fable 5 modifies
encryption/config_test.go, definingTestLoadKeyFromEnv,TestLoadKeyFromFile, andTestLoadKeyFromLiteralinside theencryptionpackage. The verifier defines hidden tests with the same names in the same package, producing redeclaration errors. - Agent tests depend on symbols that the verifier removes: The agent defines a symbol in a file that the verifier restores, but references it from another file the verifier leaves unchanged. Example: While solving the actionlint-action-pinning-lint task, Sol 5.6 max defines a helper function,
actionPinningWorkflow, insiderule_action_pinning_test.go, and calls it from other test files. The verifier adds a test file with the same name, removing the agent’s function definition and causing undefined errors.
Errors
| Task | Error Type | Notes | Affected Datacurve trials |
|---|---|---|---|
| actionlint-action-pinning-lint | False negative | The verifier deletes rule_action_pinning_test.go, but command_test.go and linter_test.go still call its actionPinningWorkflow helper. The test package then fails to compile. | Sol 5.6 max |
| anko-typed-variable-bindings | False negative | The model’s vm/vmTypedBindings_test.go and hidden vm/typed_bindings_test.go both define TestTypedBindingsDeclarations, so the test package fails to compile. | Opus 5 max |
| eicrud-keyset-pagination-cursor | False negative | The model adds the CURSOR_* errors to shared/CrudErrors.ts, but the verifier runs Jest against the old generated shared/CrudErrors.js without rebuilding it. The new errors are therefore undefined. | Opus 5 max Kimi K3 max |
| etree-xml-diff-patch | False negative | The verifier deletes diff_test.go, but diff_edge_test.go still calls its mustDoc and docWrite helpers. The P2P test package then fails to compile. | Qwen3 8 Max xhigh |
| geo-shapeindex-serialization | False negative | Model shapeindex_coder_test.go, shapeindex_coding_test.go, or shapeindex_encode_decode_test.go and hidden shapeindex_encode_test.go define the same encodeDecodeIndex helper. The test package then fails to compile. | Opus 4.8 max Opus 4.8 max Qwen3 8 Max xhigh Opus 4.8 xhigh |
| go-critic-doc-link-checker | False negative | The hidden patch changes linttest.go’s stripDirectives to remove every /*! marker, invalidating the surviving position_tests.go and uppercase_alias_tests.go fixtures. Those fixtures make TestCheckers fail. | Sol 5.6 max |
| ink-grid-box-layout | False negative | The placement algorithm is not specified in the instructions. Some models pick a “sparse” algorithm (the CSS default), while the hidden tests require “dense” placement. | Opus 5 max Opus 5 high Fable 5 max Fable 5 max Fable 5 max Fable 5 high Fable 5 high Gemini 3 7 Flash medium Gemini 3 7 Flash medium Fable 5 medium Luna 5.6 xhigh Sol 5.6 low |
| kcp-go-multiplexed-kcp-streams | False negative | The verifier deletes mux_test.go, but mux_edge_test.go, mux_extra_test.go, or mux_kcp_test.go still call its muxPair, pipeSessions, or waitFor helpers. The test package then fails to compile. | Deepseek V4 Pro max Opus 4.8 xhigh Opus 4.8 high Opus 4.8 medium |
| obsidian-linter-auto-table-of-contents | False negative | The test IDs are generated from the English name of the new plugin. Some models name it “Auto TOC”, while the verifier expects “Auto Table of Contents”. | Sol 5.6 max Fable 5 max Sol 5.6 high |
| onedump-dump-encryption-pipeline | False negative | Model-authored test files and hidden test files both define one or more of the same package-level test functions: TestConfigValidateCaseInsensitiveSource, TestDecryptReaderLazyInit, or TestJobEncrypted. The affected package then fails to compile. | Opus 5 xhigh Opus 5 high Sol 5.6 max Fable 5 xhigh Fable 5 max Terra 5.6 max Kimi K3 max Grok 4 6 medium Grok 4 6 xhigh Gemini 3 7 Flash high Grok 4 6 high GPT 5.5 high Deepseek V4 Pro max Sol 5.6 medium Opus 4.8 max Qwen3 8 Max xhigh Sonnet 5 max Glm 5 2 max Sonnet 5 medium Sonnet 4.6 high |
| opa-template-string-reconstruction | False negative | Model cmd/eval_test.go and hidden cmd/eval_partial_template_string_test.go both define TestEvalPartialSourceReconstructsTemplateStrings, so the CLI test package fails to compile. | Sol 5.6 high Grok 4 6 medium Grok 4 6 xhigh Grok 4 6 xhigh Grok 4 6 high |
| participle-grammar-conflict-analysis | False negative | Model analysis_test.go and hidden analyze_test.go both define TestAnalyzeFirstFirstConflict, TestAnalyzeFirstFollowThroughEmbedding, and TestAnalyzeUnreachableAlternative. The test package then fails to compile. | Fable 5 xhigh |
| prometheus-transactional-reload-status | False negative | Model transactional_reload_integration_test.go and hidden reload_state_test.go define different getReloadStatus helpers with the same package-level name, so the test package fails to compile. | Deepseek V4 Pro max |
| returns-validated-error-accumulation | False negative | The hidden patch adds tests/test_validated/__init__.py, causing pytest to import it and tests/test_validated/test_validated_functions/test_validated.py under the same test_validated name. Test collection then aborts. | Grok 4 6 xhigh |
| skrub-duration-encoding | False negative | The model adds _get_durations_dataframe in the middle of the test_auto_cast and test_cleaner_dtypes parameter lists in skrub/tests/test_table_vectorizer.py. Pytest’s positional expected_types IDs shift, so the grader marks the old IDs missing. | GPT 5.4 xhigh |
| sqlfmt-create-table-ddl-formatting | False negative | create table name( satisfies the stated requirement that the opening parenthesis follow the table name on the same line and is consistent with the prompt’s other no-space parenthesis rules. The hidden fixtures require instead a space between the table name and the opening parenthesis create table name (. | Sol 5.6 max Sol 5.6 xhigh Sol 5.6 high |
| task-task-graph-export | False negative | In one run, graph_feature_test.go and hidden graph_test.go both define tests named TestGraphCycle, TestGraphReverse, TestGraphDefaultTask, and TestGraphWildcard. In another, replacing graph_test.go removes GraphTest and GraphTestOption still used by task_test.go. Both test packages fail to compile. | Opus 5 high Glm 5 2 max |
| tengo-destructuring-bindings | False negative | Model destructuring_ext_test.go and hidden destructuring_test.go both define runDestructuring and expectDestructuringError, so the test package fails to compile. | Opus 5 high |
| testem-bail-on-test-failure | False negative | The hidden test accepts the warning only if its text contains the word “invalid”, a requirement not stated in the instructions. | Qwen3 8 Max xhigh |
| updo-policy-alerting | False negative | Model-authored tests and hidden tests independently define the same package-level test names: TestLoadConfigAlertPolicyDefaults, TestHandleWebhookDecisionWithHeaders, TestTrackerSSLAndCooldown, or TestHandleWebhookDecision. The affected package fails to compile, but the verifier reports the tests as missing. | Kimi K3 max |
| vulture-persistent-analysis-cache | False negative | Adding the requested cache settings to DEFAULTS changes auto-generated pytest IDs. The verifier thinks those tests are missing and fails the submission. | Sol 5.6 max Sol 5.6 max Sol 5.6 max Sol 5.6 xhigh Sol 5.6 xhigh Sol 5.6 xhigh Fable 5 xhigh Fable 5 xhigh Fable 5 xhigh Fable 5 xhigh Fable 5 max Fable 5 max Fable 5 max Sol 5.6 high Sol 5.6 high Fable 5 high Fable 5 high Fable 5 high Fable 5 high |
| yaegi-go-embed-directives | False negative | Model interp/interp_embed_test.go and hidden interp/embed_test.go both define tests named TestEmbedString, TestEmbedBytes, and other TestEmbed* names. The test package then fails to compile. | Opus 5 xhigh Qwen3 8 Max xhigh |
| ytt-jsonpath-query-api | False negative | Model pkg/orderedmap/query_test.go and hidden jsonpath_test.go both define m; model pkg/yttlibrary/jsonpath_test.go and hidden jsonpath_starlark_test.go both define TestJSONPathStarlarkQueryOne. The affected package then fails to compile. | Gemini 3 7 Flash medium Grok 4 6 low |
Rubric
1. Reviewability
| Level | Meaning | Status |
|---|---|---|
| Full | All tasks and scoring logic inspectable*, and harness/API settings used for each model (reasoning effort, token/time limits, tool access, system prompts) are fully disclosed | [proceed to 2] |
| Partial | A representative sample of tasks and scoring logic inspectable, with full harness/API settings disclosed | [proceed to 2] |
| Inadequate | Limited, biased, or no inspectable tasks or scoring logic, harness/API settings undisclosed | [stop → NEI] |
* Either publicly or privately to reviewers.
2. Scoring
This is the minimum standard required to be Verified; failing any of these items results in a Flawed verdict.
| Scoring | |
|---|---|
| Examples |
|
| Default threshold for Flawed | ≥20% of inspected sample† contains errors or there is an issue that corrupts grading at scale |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Benchmark Consistency | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Leaderboard has incomparable results from different versions |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Elicitation | |
|---|---|
| Examples | Model elicitation is extremely constraining and is not the focus of the benchmark:
|
| Default threshold for Flawed | Substantially reduced performance compared to reasonable alternatives for the tasks |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Bias in Evaluation Setup | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Material model-specific advantage found |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
† For benchmarks with >50 available tasks, we will sample a random set of 50 (stratified by category, when present). If the observed error rate is 15–25%, we will expand the sample to 100. For benchmarks with ≤50 available tasks, all available tasks will be assessed.
3. Evaluation Quality
This is the standard we would like all benchmarks to meet, but it is not necessarily disqualifying to omit or fail these items.
| Question | Status | Notes |
|---|---|---|
| Elicitation and resource adequacy: Are the resources given to models (reasoning token/turn budget, tool access, etc.) sufficient for them to perform near their ceiling? | SufficientConstrainingUnreasonably constrainingUnknownNot reviewed | — |
| Scaffold fairness: What scaffold does the leaderboard report? | Shared common scaffoldMix of model-specific and common scaffoldsModel-specific scaffoldsNot reviewed | — |
| Is there evidence/risk of contamination? | Not reviewed | — |
| Has human completability been assessed? | All tasksRepresentative set of tasksPoor implementation (Unrepresentative set of tasks, unreasonable set of participants)Not establishedNot reviewed | — |
| Score range (if possible to estimate) | — floor, — ceilingNot reviewed | — |
| Statistical adequacy: how many runs/model (≥ 5 recommended for error bars) | — runs/modelUnknownNot reviewed | — |
| Construct Validity | Measures stated capabilitiesPartially measures stated capabilitiesDoes not measure stated capabilitiesNot reviewed | — |
Disclaimer
This is a partial review. We stopped inspecting once we discovered errors that exceeded our published threshold for a flawed benchmark.
We attempted to reach out to the benchmark developer prior to launch, and if they write a response to this review we will link it here. If you’re the benchmark developer, please email us at reviews@epoch.ai with a link to your response if you would like us to include it here.
Notes
About Benchmark Reviews
Epoch AI’s Benchmark Reviews are independent reviews of external AI benchmarks. Our documentation describes the rubric behind this verdict, how we choose which benchmarks to review, and answers frequently asked questions.