PostTrainBench is a benchmark that aims to measure how well LLMs can autonomously perform post-training, the stage of AI R&D in which a pretrained base model is fine-tuned to follow instructions and perform well on specific tasks. Each agent is given four base models and tasked with maximizing the resulting model’s score on each of seven target benchmarks. PostTrainBench provides useful signals of models’ autonomous post-training capabilities in a lightly constraining setting. However, restrictions on permitted techniques and narrow scope, covering only part of post-training, limit how broadly its results generalize. Comparison with Instruct scores, while a useful reference point, don’t represent a ceiling and there is no clear saturation point.
Interpretation
The leaderboard headline number is a weighted average of the post-trained model’s scores across the evaluation suite, but per-model and per-benchmark breakdowns are also available. Before scoring, LLM judges read the agent’s trace and flag runs that show any of three banned behaviors: contamination, use of external APIs or distillation, and looking up prior results on the PostTrainBench website. A flagged run is scored as the base model’s performance rather than at 0%.1
Scores may not represent a model’s general ability to post-train for three reasons:
- Models can achieve high scores by narrowly (over)fitting to the single eval rather than teaching general, transferable skills to the base model.
- A middling average can mix strong legitimate runs with disqualified ones (the judge is the only thing separating those cases, hence leaderboard validity is bounded by judge accuracy in both directions2). High variance can be created either by inconsistent models or a mixture of strong runs and runs flagged as contaminated.
- Gains are generally uneven across benchmarks, spanning roughly from 48 points on BFCL and 47 in GSM8K to just 6 on AIME25 for the average model on the leaderboard as of the review date. Because the weighting is larger where gains are the smallest3 (AIME25, HealthBench, GPQA main), an average score can reflect strong performance on easier tasks and limited progress on the hardest ones rather than moderate improvement across the board.
Task Analysis
The eval suite is broad and draws from different domains. The following benchmarks have been selected in full capacity:
These benchmarks only included selected subsets5:
- BFCL for function calling used the exec_simple split;
- HealthBench for healthcare-related questions include a subset of easier multi-turn 245 questions;
- ArenaHard for writing where only the creative writing split was included.
We classify one of the included benchmarks, BFCL, as Flawed due to an observed error rate of 48%, but the selected subset used in PostTrainBench may have a different error profile (see our BFCL review). This is notable, as it is one of the few instances where a model (GPT 5.6 Sol Max, using Codex CLI) beats the Instruct baseline on the aggregate average (94.38% vs 85.00%).
Because all seven benchmarks are public, their train/test splits and paraphrased variants are easy to find online. This means models cheating by looking up the answers to use for training is a serious concern. PostTrainBench 1.1 mitigates this by using an LLM judge to assess if the agent used this type of information to artificially boost its score without completing the task the intended way. Judges and agents are given the same deterministic n-gram overlap tool.6 Finally, a programmatic model-identity check that compares configuration fields (not weights) is used to ensure that agents submit the correct model for scoring.
We analyzed7 the traces produced by Opus 5, the only agent with a complete set of v1.1 traces.8 Out of 56 total runs, 9 were flagged by the contamination judge: notably, all 8 runs for BFCL were flagged. Cheating attempts ranged from near-duplicate contamination to edge cases. We believe two of these nine (1, 2) are likely accidental, and although correctly detected as cheating by the stated rules, don’t represent actual malicious intent.9 Nevertheless, agents were given the full rules, so we agree with the judge on all nine verdicts.
We also found one anomalous run that was terminated prematurely; while it had no impact on the final performance due to scoring lower than the other attempt, this was a lucky instance and these errors can have an impact on the scores.
The public viewer only shows traces and verdicts for contamination and model identity, not API-distillation and PostTrainBench lookup. We also checked the other 47 unflagged runs by Opus 5 and found that they were correct: in none of them did we see evidence of cheating any of the checks.
There is no reported human baseline performing the same tasks under matched constraints. Official Instruct models are useful reference points, but benefit from much greater resources (teams of researchers, larger compute budgets, availability of all post-training techniques) while pursuing broader objectives (an Instruct model is generally well-rounded, without targeting any specific benchmark). However, they may suffer from using older techniques. Neither outperforming nor trailing them establishes superiority or inferiority to an expert with the same budget. Transfer to unseen evaluations, preservation of other capabilities, and larger-scale training are not measured.
Elicitation & Scaffolding
Agents run through their native CLI scaffold (Claude Code, Codex CLI, Gemini CLI), with models that lack a scaffold using OpenCode, with a fixed prompt describing the goal, the environment, and the rules. The scaffold manages the agent loop, permissions, tool use, and context. The number of samples per model is limited by cost: two runs10 for frontier agents on native CLIs and single runs for other configurations. Agents have a fixed time/compute budget per run: 10 hours on one H100 GPU, with no token limit.
There is a subtle prompting issue: the agent’s prompt says “Internet access is unrestricted”, yet visiting the PostTrainBench website is a disqualifying offense. Wasteful potential attempts by agents are avoided by a network-level block for PostTrainBench and related sites. Opus 5, the only model with complete traces after this rule was introduced, didn’t attempt this behavior, which might impact the results of future runs. We did not find any other issues with model elicitation.
Finally, the authors ran three ablation studies.11
- On reasoning effort, results are mixed: GPT 5.1 Codex Max scores best at Medium effort, while High burns almost double the tokens without improving performance; GPT 5.3 Codex High does better than Medium, but at 2.8x the tokens.
- On time limit, agents improve steadily, with some (such as Opus 4.5) plateauing around the five-hour mark, while others keep gaining until the end. 20-hour runs were abandoned because agents tended to stop well before the limit.
- On scaffolding, they show that GPT-5.1 Codex Max got a 12-point uplift when switched from OpenCode to its native Codex CLI.12
Strengths
The benchmark is well engineered, documented, and transparent. v1.1 is a clear improvement over its predecessor, v1.0, addressing many issues, especially around contamination and reward-hacking behavior. The authors published some model traces, which are very helpful for a more in-depth review.
Limitations
- Due to the constraints the models operate under, external validity is currently narrower than one might hope: distillation and reuse of previous results are legitimate, widely used post-training techniques, but are banned here. Generalizability is further limited by the narrowed scope of post-training tasks chosen for this benchmark, especially since models can focus on maximizing the score of a single eval per run.
- LLM judges are used to check for cheating behaviors: these have documented biases and can be unreliable, especially when grading non-discrete behaviors. We agreed with the contamination judge’s verdicts on all reviewed counts, but we believe this mechanic might be a failure point with future model releases that are more capable.
- Judge traces for the other two checks (API-distillation and PostTrainBench lookup) aren’t provided, and thus were not evaluated.
Rubric
1. Reviewability
| Level | Meaning | Status |
|---|---|---|
| Full | All tasks and scoring logic inspectable*, and harness/API settings used for each model (reasoning effort, token/time limits, tool access, system prompts) are fully disclosed | [proceed to 2] |
| Partial | A representative sample of tasks and scoring logic inspectable, with full harness/API settings disclosed | [proceed to 2] |
| Inadequate | Limited, biased, or no inspectable tasks or scoring logic, harness/API settings undisclosed | [stop → NEI] |
* Either publicly or privately to reviewers.
2. Scoring
This is the minimum standard required to be Verified; failing any of these items results in a Flawed verdict.
| Scoring | |
|---|---|
| Examples |
|
| Default threshold for Flawed | ≥20% of inspected sample† contains errors or there is an issue that corrupts grading at scale |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Benchmark Consistency | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Leaderboard has incomparable results from different versions |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Elicitation | |
|---|---|
| Examples | Model elicitation is extremely constraining and is not the focus of the benchmark:
|
| Default threshold for Flawed | Substantially reduced performance compared to reasonable alternatives for the tasks |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Bias in Evaluation Setup | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Material model-specific advantage found |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
† For benchmarks with >50 available tasks, we will sample a random set of 50 (stratified by category, when present). If the observed error rate is 15–25%, we will expand the sample to 100. For benchmarks with ≤50 available tasks, all available tasks will be assessed.
3. Evaluation Quality
This is the standard we would like all benchmarks to meet, but it is not necessarily disqualifying to omit or fail these items.
| Question | Status | Notes |
|---|---|---|
| Elicitation and resource adequacy: Are the resources given to models (reasoning token/turn budget, tool access, etc.) sufficient for them to perform near their ceiling? | SufficientConstrainingUnreasonably constrainingUnknownNot reviewed | — |
| Scaffold fairness: What scaffold does the leaderboard report? | Shared common scaffoldMix of model-specific and common scaffoldsModel-specific scaffoldsNot reviewed | — |
| Is there evidence/risk of contamination? | As of Sep. 9, 2026: 100% of tasks public, 0% of solutions publicNot reviewed | The task is broadly defined as improving a benchmark score for a base model, so the solution is a method used, not the specific answers to the benchmark tasks. The benchmarks do have public solutions and an LLM judge is used to assess contamination. |
| Has human completability been assessed? | All tasksRepresentative set of tasksPoor implementation (Unrepresentative set of tasks, unreasonable set of participants)Not establishedNot reviewed | — |
| Score range (if possible to estimate) | 0% floor, — ceilingNot reviewed | The ceiling is unknown given the current eval suite. Instruct’s total average of 51.14% provides a useful reference. |
| Statistical adequacy: how many runs/model (≥ 5 recommended for error bars) | 2 runs/modelUnknownNot reviewed | 2 runs for each of the 4 models x 7 benchmarks (56 total). |
| Construct Validity | Measures stated capabilitiesPartially measures stated capabilitiesDoes not measure stated capabilitiesNot reviewed | Models don’t get access to all post-training techniques and are only focused on a single benchmark per run. |
Disclaimer
We attempted to reach out to the benchmark developer prior to launch, and if they write a response to this review we will link it here. If you’re the benchmark developer, please email us at reviews@epoch.ai with a link to your response if you would like us to include it here.
-
The base models score only ~1.5% on AIME25 and Arena-Hard, but ~20% for GSM8K, so the impact of any penalty for cheating differs.
-
Overall, based on our review of all 163 v1.1 available traces, we think the judge does interpret the given rules correctly in the vast majority of cases (≥80%). A few runs (single low digit) flagged incidental contamination or edge cases, where another judge (human or LLM) would rule differently. This is largely unavoidable due to contamination being a matter of degree rather than binary. We don’t think the judge should be pushed to be stricter or more lenient on that basis. The current contamination judge is GPT-5.4 running in Codex CLI.
-
This is done intentionally, as described in the “Scoring” paragraph in Section 2.1 of the paper.
-
AIME25 doesn’t have a release paper since it is not a standardized benchmark, but a mathematical competition. The dataset can be found here.
-
Further details on selection can be found in Section 2.2 of the benchmark’s paper.
-
The tool is a script that checks the training data for exact snippets of the benchmark’s test questions; paraphrased items are intentionally not flagged by the tool.
-
This was done with the help of one subagent per run, all orchestrated by Fable 5.1.
-
The rest are partial at best: from 35 for Grok 4.5 to 4 for GPT 5.6 Sol. Four of the agents have no v1.1 traces at all.
-
These were not authored by the agent but were present in the public dataset it used. The 3-row limit was exceeded by, respectively, 1 and 3 rows.
-
The original paper mentions three runs, but we limit our review to v1.1, which ran just twice.
-
The authors run four total studies, but we didn’t include the one on model size since it’s not technically an ablation. The studies were run on v1.0.
-
All results were eventually reported with models within their native harness, if available.
About Benchmark Reviews
Epoch AI’s Benchmark Reviews are independent reviews of external AI benchmarks. Our documentation describes the rubric behind this verdict, how we choose which benchmarks to review, and answers frequently asked questions.