DeepSWE is a benchmark from Datacurve that evaluates coding agents on 113 original, long-horizon software engineering tasks spanning 91 active open-source repositories in five languages. Tasks are written from scratch and never upstreamed, which limits training contamination, and each task is graded by a hand-written program-based verifier that checks observable behavior rather than a specific implementation. Agents run autonomously in sandboxed containers without internet access.

We source results from the public DeepSWE leaderboard data. Our chart reports pass@1: the share of scored rollout attempts that pass, with each configuration running the full task suite four times. The leaderboard’s pass@4 rate and run-to-run confidence interval are kept alongside it in the data export.

All current configurations run through the mini-SWE-agent harness at a specified reasoning effort, and each model appears once per reasoning effort as a separate entry. Context-window failures and agent timeouts count as failures, while provider and infrastructure errors are excluded from scoring. We also keep each configuration’s mean cost, output tokens, and agent steps in the data export.