PostTrainBench is a benchmark that aims to measure how well LLM agents can autonomously perform post-training, the stage of AI R&D in which a pretrained base model is fine-tuned to follow instructions and perform well on specific tasks. The benchmark uses four base models: Qwen3-1.7B, Qwen3-4B, SmolLM3-3B, and Gemma-3-4B. In each run, an agent post-trains one base model to maximize its score on one of seven target benchmarks.
Agents choose their own post-training methods and data, so the results can be reasonably interpreted as a signal of their ability to post-train a base model to score higher on these target benchmarks; the benchmark does not evaluate other aspects of post-training, such as alignment and safety performance.
The benchmark maintains a leaderboard that ranks agents by their weighted average score across the seven target benchmarks, overall and per base model. As a reference point, the leaderboard also includes scores for the official Instruct models;1 these scores come with caveats discussed in Interpretation.
Interpretation
The headline leaderboard number is a weighted average2 of the post-trained models’ scores across the seven benchmarks. The leaderboard also shows scores by base model and benchmark. Before scoring, LLM judges read each agent’s trace and flag runs that show any of three banned behaviors: data contamination, distillation from disallowed models served through external APIs3, and looking up prior results on the PostTrainBench website. A flagged run gets assigned the base model’s score, rather than 0%, regardless of which rule the model violated. This means that the impact of a cheating penalty varies across benchmarks, depending on how high the base model scores on each.4
Scores are a reasonable estimation of an agent’s capability, but three things can make them misleading:
- Each run targets a single benchmark. Agents can achieve high scores by overfitting one benchmark, not by improving the base model broadly. The leaderboard gives no information about transfer to unseen evals or preservation of other capabilities.
- A mid-range average can mix strong, legitimate runs with flagged ones. The LLM judges decide which runs get flagged, so the validity of the leaderboard depends on their accuracy.5 High variance can come from either inconsistent agents or a mixture of strong runs and flagged runs.
- Gains over the base model are uneven across benchmarks. As of the review date, agents gain an average of 48 points on BFCL, 47 on GSM8K, but just 6 on AIME25. While scoring gives more weight to the benchmarks whose gains are smallest (AIME25, HealthBench, GPQA), an average score cannot show whether an agent made large gains on the easiest tasks and small gains on the hardest ones, or improved moderately across the suite.
There is no human baseline performing the same tasks under matched constraints. Official Instruct models are useful reference points, but benefit from much greater resources (teams of researchers, larger compute budgets, availability of all post-training techniques) while pursuing broader objectives (an Instruct model is generally well-rounded, without targeting any specific benchmark). Some Instruct models were also built before the development of certain post-training techniques that are available to agents. If an agent outperforms or trails an Instruct model, that doesn’t mean the agent is superior or inferior to an expert with the same budget. Transfer to unseen evaluations, preservation of other capabilities, and larger-scale training are not measured.
Task Analysis
The eval suite is broad and draws from different domains. Four benchmarks are included in full:
Three are included as subsets:
- BFCL for function calling, using the exec_simple split;
- HealthBench for healthcare-related questions, using a subset of 245 easier multi-turn questions;
- ArenaHard for writing, including only the creative writing split.
We classify the full BFCL as Flawed due to a 48% observed error rate, although the selected subset used in PostTrainBench may have a different error profile. This is notable, as it is one of the few instances where a model (GPT-5.6 Sol Max, using Codex CLI) beats the Instruct baseline, 94.38% vs 85.00%, averaged across the four base models.
Because all seven benchmarks are public, their train/test splits and paraphrased variants are easy to find online. It is thus a serious concern that models could cheat by looking up answers. PostTrainBench v1.1 mitigates this by using an LLM judge to assess whether the agent used this type of information to artificially boost its score. The judge and agents are given the same deterministic n-gram overlap tool.7 Finally, a programmatic model-identity check compares configuration fields (not weights) to confirm that agents submit the correct model for scoring.
We analyzed8 the traces produced by Opus 5, the only agent with traces for all 56 v1.1 runs.9 Of its runs, nine were flagged by the contamination judge. All eight runs for BFCL were flagged. Cheating attempts ranged from near-duplicate contamination to edge cases. We believe two of these nine (1, 2) are likely accidental, and although correctly detected as cheating by the stated rules, we don’t think they represent malicious intent.10 Nevertheless, agents were given the full rules, so we agree with the judge on all nine verdicts. We also verified all three checks (contamination, API-based distillation, PTB lookup) on the 47 unflagged runs and found no evidence of cheating in any of them.
One Opus 5 run terminated prematurely. Since it scored less than the completed attempt when it stopped, it is not reported on the leaderboard. A full run might have scored higher, though, so the reported score could be an underestimate.
Elicitation & Scaffolding
Agents run through their native CLI scaffold (Claude Code, Codex CLI, Gemini CLI), or through OpenCode where none exists, with a fixed prompt describing the goal, the environment, and the rules. The scaffold manages the agent loop, permissions, tool use, and context. The number of samples per model is limited by cost: two or three runs11 for frontier agents on native CLIs and single runs for other configurations. Agents have a fixed time/compute budget per run of 10 hours on one H100 GPU, with no token limit.
There is a subtle prompting inconsistency: the agent’s prompt says “Internet access is unrestricted”, yet there is a network-level block on PostTrainBench and related sites to prevent the agent from looking up previous research. Opus 5 did not try to access the PostTrainBench site. We did not find any other issues with model elicitation.
The authors also ran three ablation studies.12
- On reasoning effort, results are mixed: GPT-5.1 Codex Max scores best at Medium effort, while High burns almost double the tokens without improving performance; GPT-5.3 Codex High does better than Medium, but at 2.8x the tokens.
- On time limit, agents improve steadily, with some (such as Opus 4.5) plateauing around the five-hour mark, while others keep gaining until the end. PostTrainBench abandoned 20-hour runs because agents tended to stop well before the limit.
- On scaffolding, GPT-5.1 Codex Max got a 12-point uplift when switched from OpenCode to its native Codex CLI.13
Strengths
The benchmark is well engineered, documented, and transparent. v1.1 is a clear improvement over v1.0, addressing many issues, especially around contamination and reward hacking. The published traces allow for an in-depth review. The LLM judges are thoughtfully implemented, and the authors run manual checks as guardrails. An interesting future direction would be to let agents compete with Instruct releases on even terms, with larger training budgets, and access to the full range of techniques.
Limitations
Because of the constraints the models operate under, external validity is limited.
- Distilling from a stronger model served through an API is a legitimate and widely used post-training technique, but banned here.14
- Reuse of previous results is also banned. This is by design, since the authors want to measure an agent’s skill at conducting independent research, but it departs from practice where building on earlier results is the norm.
- Each run optimizes a single eval from a narrow set of tasks, which limits how far results generalize to post-training as practiced.
LLM judges are used to detect cheating, however they have documented biases and can be unreliable, especially when grading non-discrete behaviors. We agree with the contamination judge’s verdicts on all counts we reviewed, but we believe this mechanism could become a failure point as more capable agents are released. The authors address this weakness by manually checking some of the traces.
Rubric
1. Reviewability
| Level | Meaning | Status |
|---|---|---|
| Full | All tasks and scoring logic inspectable,15 and harness/API settings used for each model (reasoning effort, token/time limits, tool access, system prompts) are fully disclosed | [proceed to 2] |
| Partial | A representative sample of tasks and scoring logic inspectable, with full harness/API settings disclosed | [proceed to 2] |
| Inadequate | Limited, biased, or no inspectable tasks or scoring logic, harness/API settings undisclosed | [stop → NEI] |
2. Scoring
This is the minimum standard required to be Verified; failing any of these items results in a Flawed verdict.
| Scoring | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Either errors affect ≥20%16 of the total score in the inspected sample,17 or there is an issue that corrupts grading at scale |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Benchmark Consistency | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Leaderboard has incomparable results from different versions |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Elicitation | |
|---|---|
| Examples | Model elicitation is extremely constraining and is not the focus of the benchmark:
|
| Default threshold for Flawed | Substantially reduced performance compared to reasonable alternatives for the tasks |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Bias in Evaluation Setup | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Material model-specific advantage found |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
3. Evaluation Quality
This is the standard we would like all benchmarks to meet, but it is not necessarily disqualifying to omit or fail these items.
| Question | Status | Notes |
|---|---|---|
| Elicitation and resource adequacy: Are the resources given to models (reasoning token/turn budget, tool access, etc.) sufficient for them to perform near their ceiling? | SufficientConstrainingUnreasonably constrainingUnknownNot reviewed | — |
| Scaffold fairness: What scaffold does the leaderboard report? | Shared common scaffoldMix of model-specific and common scaffoldsModel-specific scaffoldsNot reviewed | — |
| Is there evidence/risk of contamination? | As of Sep. 9, 2026: 100% of tasks public 0% of solutions publicNot reviewed | The task is broadly defined as improving a benchmark score for a base model, so the solution is a method used, not the specific answers to the benchmark tasks. The benchmarks do have public solutions and an LLM judge is used to assess contamination. |
| Has human completability been assessed? | All tasksRepresentative set of tasksPoor implementation (Unrepresentative set of tasks, unreasonable set of participants)Not establishedNot reviewed | — |
| Score range (if possible to estimate) | 0% floor, — ceilingNot reviewed | The ceiling is unknown given the current eval suite. Instruct’s total average of 51.14% provides a useful reference. |
| Statistical adequacy: how many runs/model (≥ 5 recommended for error bars) | 2 runs/modelUnknownNot reviewed | 2 runs for each of the 4 models x 7 benchmarks (56 total). |
| Construct validity | Measures stated capabilitiesPartially measures stated capabilitiesDoes not measure stated capabilitiesNot reviewed | Models don’t get access to all post-training techniques and are only focused on a single benchmark per run. |
Developer Response
Link to developer response: posttrainbench.com/blog/epoch-ai-review-response
-
An Instruct release is an official post-trained model: a well-rounded, useful assistant that can perform well on a broad variety of tasks thanks to techniques such as supervised fine-tuning.
-
Eval selection and weighting details are in Section 2.2 and Table 5 of the paper.
-
Agents may not distill from a model served through an external API. Local distillation is allowed.
-
For example, base models average ~20% on GSM8K but only ~1.5% on BFCL, ArenaHard, and AIME25, so a flagged GSM8K run scores higher than a flagged BFCL run.
-
Based on our review of all 163 available v1.1 traces, we think the judges interpret the given rules correctly in the vast majority of cases (≥80%). A few runs (low single digits) flagged incidental contamination or edge cases, where another judge (human or LLM) may rule differently. We believe the contamination judge is appropriately calibrated in strictness. The current contamination judge is GPT-5.4 running in Codex CLI.
-
AIME25 has no release paper since it is not a standardized benchmark, but a mathematical competition. The dataset can be found here.
-
The tool is a script that checks the training data for exact snippets of the benchmark’s test questions; paraphrased items are intentionally not flagged by the tool.
-
This was done with the help of one subagent per run, all orchestrated by Fable 5.1.
-
Other agents have complete traces, but only their flagged runs were re-run under the v1.1 rules, so their traces mix v1.0 and v1.1. The number of v1.1 traces ranges from 35 for Grok 4.5 to 4 for GPT-5.6 Sol; four agents have none.
-
These were not authored by the agent but were present in the public dataset it used. The 3-row limit was exceeded by, respectively, 1 and 3 rows.
-
The original paper reports three runs per agent. For v1.1, eight of the most expensive agents were run twice, the remaining were run three times, and other test configurations just once.
-
The authors run four studies in total, but we don’t count the one on model size, since it is not technically an ablation. All were run on v1.0.
-
All leaderboard results were eventually reported with models within their native harness, where one exists.
-
Some model providers prohibit third parties from distilling their models in this way. Labs nevertheless use it internally to produce smaller and cheaper but capable models.
-
Either publicly or privately to reviewers under an NDA.
-
A benchmark is rated Flawed if scoring defects could affect ≥20% of the benchmark’s full score. For benchmarks with equally weighted all-or-nothing tasks, our threshold corresponds to ≥20% of tasks having defects. For benchmarks with partial scoring or unequal task weights, our threshold corresponds to ≥20% of the total possible score affected by defects. For single task benchmarks, our threshold is the reported score having defects that could affect it by ≥20%.
-
For benchmarks with >50 available tasks, we will sample a random set of 50 (stratified by category, when present). If the observed error rate is 15–25%, we will expand the sample to 100. For benchmarks with ≤50 available tasks, all available tasks will be assessed.
About Benchmark Reviews
Epoch AI’s Benchmark Reviews are independent reviews of external AI benchmarks. Our documentation describes the rubric behind this verdict, how we choose which benchmarks to review, and answers frequently asked questions.