ExploitBench does a good job of measuring how far a model can take a known V81 vulnerability toward a working exploit. The tasks are 41 real public Common Vulnerabilities and Exposures (CVEs), but only 2 of the exploits are published. This approach limits contamination, and is the benchmark’s main weakness. Progress is scored on a ladder of 16 verifiable capabilities in 5 tiers ranging from simple coverage to full Arbitrary Code Execution (ACE).2 This is a useful and informative evolution compared to benchmarks that only report binary pass/fail results in unrealistic scenarios. Models are evaluated in an agentic harness with satisfactory scaffolding, but are relatively constrained on token expenditure.3
Interpretation
ExploitBench measures how much progress a model can make toward developing a working exploit given a patched but publicly disclosed V8 bug. A high score should be read as a model being able to take a known vulnerability and independently rediscover the exploitation path, not necessarily as a generalization of cyber-capabilities.4
Progress is scored on a ladder of 16 capabilities grouped in 5 cumulative tiers; given this design, a mid-range score should be interpreted as a model reliably clearing lower steps (such as causing a crash or building engine primitives) and then stalling before converting those primitives into ACE. In the current results,5 the jump from T3 to T2 and T1 is where almost every model stops: only GPT-5.5 and Mythos Preview are able to cross T3 without the custom scaffold.
This also means the “cap %” number reported as the main score metric is inflated by union best-of-N-seeds aggregation: a model that reached a high-tier capability once gets the same credit as one that does so reliably. This may be useful in a domain such as cyber security, where a model’s ceiling matters significantly, but it provides little information about reliability. More informative metrics could include the percentage of ACEs reached or the average number of capabilities achieved across seeds.
As of this review, OpenAI reported that GPT-6 Astra reached ACE on all 41 bugs, scoring 100%. However this is not reported on the official leaderboard, and seems to have been influenced by contamination.6 Assuming GPT-6 Astra did not cheat, this confirms that all 41 tasks can be solved to the ACE threshold (although not necessarily through the given vulnerability).
Task Analysis
The 41 tasks are real V8 vulnerabilities, all disclosed in 2024 or later, each of which carried a $10,000 bounty under Google’s v8CTF for the first working ACE exploit. Having real-world tasks instead of synthetic ones is a notable strength of this benchmark, though it also exposes it to contamination risks. The models have no browsing access during evaluation. Difficulty varies by bug class, but there is no comparable human baseline.7 As a result, the only way to gauge difficulty is by looking at how many models reach each stage of the exploit chain.
Each capability is weighted equally across bugs and organized as follows.
- T5-Coverage: the exploit reaches the vulnerable function and lines
- T4-Bug triggering: the vulnerable build behaves differently than the fixed build
- T3-Engine primitives: V8-specific primitives
- T2-General primitives: information leaks that defeat ASLR, and arbitrary read/write across the address space
- T1-Control: hijacking the program counter, then ACE
The grading process is well implemented. High-tier submissions are verified by running the agent’s attack against both flawed and repaired versions of the software and confirming that the attack succeeds on the flawed version but fails on the repaired version. Common reward-hacking methods are also checked, such as blocking an agent from reading files that might reveal the answer or randomizing the secret targets on every run. This also makes the top flags very hard to satisfy with a hard-coded or lucky answer. However, the grader is fully public, which makes it a potential target for more sophisticated reward-hacking by stronger models, although we don’t currently have evidence of this behavior occurring in practice.
Elicitation and Scaffolding
Every model runs in the same agentic loop with a prompt, 6 tools, and a 300-turn budget. There is no token or wall-clock limit, and individual commands time out after 60 seconds.
Models are evaluated in a primary arm (bare model, uniform runner, no context management), an AutoNudge arm that adds three scripted mid-episode prompts,8 and a vendor-CLI arm in which the model’s own agent harness (e.g. Codex) replaces the runner. The effects of coaching are strongly model-dependent: GPT-5.5 receives a T3 reach uplift of 9 (13 to 22 bugs), Gemini 3.1 Pro loses 8 of them (from 16 to 8), and for Mythos Preview it raises the cap percentage from 72% to 78%, but lowers ACE reach from 18 to 16 bugs.
For more accurate comparisons, the homepage leaderboard should be viewed with the “control” filter selected. This filter compares models only by harness, holding other configurations—such as the number of turns or seeds—constant.
The turn limit is the strongest constraint. In an experiment, GPT-5.5 (with the Codex harness) reached ACE 12 times with 3,000 turns compared to 0 or 1 time (with/without Codex respectively) for the 300-turn variant.9
Even with the 300-turn limit, costs can be substantial for frontier models. Mythos Preview’s best run cost almost $60,000, and without AutoNudge it cost around $42,000. The best GPT-5.5 3,000-turn run cost around $31,000, while its comparable 300-turn variant cost $8,000.
Limitations
While none of these limitations cross our threshold for a Flawed verdict, they do inform how a reader should characterize this benchmark.
- Contamination likely inflates scores on the public set, but its effect is difficult to quantify. After GPT-6 Astra scored 100% on the original dataset,10 OpenAI evaluated it on an internal set of 20 tasks containing only vulnerabilities disclosed after the model’s knowledge cutoff, between June and August 2026. Astra scored just 39% on this set. We cannot inspect these tasks, and some of the newer, high-severity vulnerabilities may be more difficult for reasons unrelated to contamination. Nevertheless, the large performance gap suggests that contamination contributes substantially to Astra’s performance on the public set. Browsing is disabled during the evaluation, but this only prevents live lookup and does not prevent pretraining exposure to the vulnerabilities.11
- Elicitation is limited by turns both in capacity and comparisons. The capacity limitation is demonstrated by the GPT 5.5 experiment,12 which shows that allowing models an order of magnitude more turns improves their performance by up to 30% and improves ACE reach by a factor of 12. Token expenditure can vary threefold across models within a single turn, making cost analysis and direct model comparison on this axis difficult.
- While the benchmark’s scoring design is a strength, the headline “cap %” metric rewards breadth and lucky runs over reliable exploits. Each of the 16 capabilities contributes equally to the score. Reaching T1 earns more points than stopping at T3 only because T1 requires passing through more scoring thresholds, not because ACE—the most important capability from a security standpoint—is weighted more heavily. Second, each entry is a best-of-three seeds union: a bug exploited in one run out of three counts the same as one exploited every time. A leaderboard that reported the number of tasks reaching ACE alongside pass@1 with confidence intervals would provide a more informative picture of model performance.
- The benchmark measures development against a single target: a restricted version of V8 bugs. It is therefore unclear how well its results generalize to other cybersecurity capabilities, just as a single-repository coding benchmark may not generalize to software engineering more broadly.
Rubric
1. Reviewability
| Level | Meaning | Status |
|---|---|---|
| Full | All tasks and scoring logic inspectable*, and harness/API settings used for each model (reasoning effort, token/time limits, tool access, system prompts) are fully disclosed | [proceed to 2] |
| Partial | A representative sample of tasks and scoring logic inspectable, with full harness/API settings disclosed | [proceed to 2] |
| Inadequate | Limited, biased, or no inspectable tasks or scoring logic, harness/API settings undisclosed | [stop → NEI] |
* Either publicly or privately to reviewers.
2. Scoring
This is the minimum standard required to be Verified; failing any of these items results in a Flawed verdict.
| Scoring | |
|---|---|
| Examples |
|
| Default threshold for Flawed | ≥20% of inspected sample† contains errors or there is an issue that corrupts grading at scale |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Benchmark Consistency | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Leaderboard has incomparable results from different versions |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Elicitation | |
|---|---|
| Examples | Model elicitation is extremely constraining and is not the focus of the benchmark:
|
| Default threshold for Flawed | Substantially reduced performance compared to reasonable alternatives for the tasks |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | Model elicitation is sufficient for a Pass Verdict, however important limitations are discussed throughout the review. |
| Bias in Evaluation Setup | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Material model-specific advantage found |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
† For benchmarks with >50 available tasks, we will sample a random set of 50 (stratified by category, when present). If the observed error rate is 15–25%, we will expand the sample to 100. For benchmarks with ≤50 available tasks, all available tasks will be assessed.
3. Evaluation Quality
This is the standard we would like all benchmarks to meet, but it is not necessarily disqualifying to omit or fail these items.
| Question | Status | Notes |
|---|---|---|
| Elicitation and resource adequacy: Are the resources given to models (reasoning token/turn budget, tool access, etc.) sufficient for them to perform near their ceiling? | SufficientConstrainingUnreasonably constrainingUnknownNot reviewed | Models use a 300-turn budget: while this is a high cap, agent’s performance still improves in experiments with 3,000 turns. |
| Scaffold fairness: What scaffold does the leaderboard report? | Shared common scaffoldMix of model-specific and common scaffoldsModel-specific scaffoldsNot reviewed | In the Explore section of the website, the eval can be explored in more detail. The Leaderboard should be consulted within the “control” filter for comparisons. |
| Is there evidence/risk of contamination? | As of Sep. 12, 2026: 100% of tasks public, ~5% of solutions publicNot reviewed | All tasks are real publicly disclosed CVEs with description, patch, and any tracker discussion. For two environments the authors have released transcripts of GPT-5.5 reaching ACE and other models reaching lower tiers. |
| Has human completability been assessed? | All tasksRepresentative set of tasksPoor implementation (Unrepresentative set of tasks, unreasonable set of participants)Not establishedNot reviewed | The authors believe at least 40/41 of the environments permit ACE using the supplied exploit. |
| Score range (if possible to estimate) | 0% floor, 100% ceilingNot reviewed | — |
| Statistical adequacy: how many runs/model (≥ 5 recommended for error bars) | 3 runs/modelUnknownNot reviewed | At least 3 seeds per model. When higher, runs are clearly labeled on the Leaderboard. |
| Construct Validity | Measures stated capabilitiesPartially measures stated capabilitiesDoes not measure stated capabilitiesNot reviewed | See “Limitations”. |
Disclaimer
We attempted to reach out to the benchmark developer prior to launch, and if they write a response to this review we will link it here. If you’re the benchmark developer, please email us at reviews@epoch.ai with a link to your response if you would like us to include it here.
-
V8 is the JavaScript engine behind Chromium-based browsers and Node.js.
-
The tiers follow the actual stages of building a browser exploit, so “highest tier reached” is a meaningful description of how far a model got.
-
Models are given 300 turns, defined as one round of model thinking plus one tool call, in a normal run. Experiments performed on GPT-5.5 raising the limit to 3000 turns, show overall performance increasing from 45% to 72%.
-
This point is further discussed in Limitations.
-
The leaderboard is not updated with more recent models, such as Mythos 5.1, GPT-5.6 Sol, and GPT-6 Astra, however their score is reported in their respective model cards. These scores are unverified, and may not be run in comparable settings to the standard ExploitBench leaderboard. In the case of GPT-6 Astra the result may also be the result of reward hacking with a memorized zero day.
-
OpenAI reported that Astra solved one task through memorization of a different vulnerability in their release model card. Since we can’t inspect Astra traces, we also can’t exclude potential cheating instances or investigate configurations.
-
Although the authors believe that at least 40 out of 41 of the vulnerabilities are exploitable to ACE.
-
Section 2.4 of the paper explains the adaptive coaching harness as follows: The “Stuck” message fires after 50 consecutive turns without a grade() call, “Wrap-up” at 75% of the budget, and “Voluntary” when the model stops issuing tool calls before its budget is spent.
-
This is not a clean ablation. GPT-5.5 (Codex) was also given AutoNudge in this experiment, so the observed difference cannot be attributed conclusively to the different limits alone.
-
It is unclear what settings were used for this run, and whether it is comparable to the results on the official leaderboard.
-
Most models on the leaderboard have knowledge cutoffs after most of these disclosures.
-
Discussed in Elicitation and Scaffolding.
About Benchmark Reviews
Epoch AI’s Benchmark Reviews are independent reviews of external AI benchmarks. Our documentation describes the rubric behind this verdict, how we choose which benchmarks to review, and answers frequently asked questions.