CritPt v1 (Complex Research using Integrated Thinking - Physics Test)1 is meant to test research-level reasoning on modern physics research areas. It consists of 71 research challenges, split into 190 checkpoint tasks that all have open-ended yet machine-verifiable answers. The challenges were created by more than 50 physics researchers.2 We designated the benchmark Flawed because two separate audits (one by Ansari et al. 2026 and one by Anthropic) found scoring defects in at least 20% of the challenges, which is our threshold for a Flawed verdict.
Methodology and Findings
Ansari et al. reviewed 56 challenges and found errors in 21 of them, meaning at least 21 of the total 71 (29.6%) challenges were flawed.3 These errors were in the challenge statements; the reference solutions are not public, so the audit could not check them. The majority of errors were due to underspecified questions, missing assumptions, and incorrect or incomplete mathematical functions. Ansari et al. repaired 19 of the flawed challenges and discarded the remaining 2, due to underspecification. The table below compares scores on the original benchmark of 70 questions4 with scores on the repaired subset of 54 challenges.
| Original mean@55 | Repaired mean@46 | |
|---|---|---|
| Fable 5 | 28.57% (Max) | 78.24% (High) |
| GPT-5.6 Sol Max | 32.29% | 87.50% |
| Gemini 3.1 Pro High (no tools) | 17.71% | 54.63% |
- Repaired challenges: 01, 07, 08, 12, 13, 22, 28, 31, 32, 40, 42, 44, 48, 56, 57, 61, 62, 67, 68
- Excluded challenges: 41, 46
- Validated challenges: 02, 03, 04, 05, 06, 09, 10, 14, 16, 17, 18, 19, 20, 23, 26, 29, 30, 33, 35, 39, 45, 47, 49, 50, 51, 53, 55, 58, 59, 60, 64, 65, 66, 69, 70
Separately, Anthropic reported in its system card for Fable 5.1 and Mythos 5.1 that it tested its models on an internal version of the benchmark, called CritPT-Corrected. In this version, experts revised 31 of the original benchmark’s 71 challenges (43.7%) because those 31 were flagged for “underspecification, ambiguity, or typos”.7 Reported scores for Opus 4.8 improved 55.6 percentage points, from 20.9% on CritPt v1 (using an independent evaluation run by Artificial Analysis) to 76.5% on CritPT-Corrected (with tools, using Claude Opus 4.8 as a judge).8 These gains cannot come from the fixes alone, because the fixes affected only 43.7% of challenges. It is unknown how much of this improvement was due to tool usage.
The CritPt website includes an update from September 2026 that states: “We are actively maintaining the benchmark and incorporating community feedback. All benchmark problems are currently under expert review.”
Example Repaired Problem
Red: removed or replaced text in the original. Blue: added or replacement text in the repair.
| Original challenge 44 | Repaired challenge 44 |
|---|---|
| Problem setup Consider the Kitaev honeycomb model at the isotropic limit (assuming \(J_x=J_y=J_z=1\)) on a 3x2 Bravais lattice with periodic boundary conditions. Main problem How many degenerate ground states are there? How many of them are in the flux-free sector? Compute the energy of the ground states with three decimal precision. | Consider the Kitaev honeycomb model at the isotropic limit (assuming \(J_x=J_y=J_z=1\)) on a 3x2 Bravais lattice with periodic boundary conditions. To fix the normalization and sign convention unambiguously, take \(\displaystyle H=-\sum_{\langle ij\rangle_x}\sigma_i^x\sigma_j^x-\sum_{\langle ij\rangle_y}\sigma_i^y\sigma_j^y-\sum_{\langle ij\rangle_z}\sigma_i^z\sigma_j^z,\) where \(\sigma_i^\alpha\) denotes the standard \(2\times 2\) Pauli matrix acting on the local spin-\(\frac{1}{2}\) Hilbert space at site \(i\). Main problem: How many degenerate ground states are there? How many of them are in the flux-free sector? Compute the energy of the ground states with three decimal precision. |
As originally stated in the benchmark, the problem does not write the Hamiltonian or distinguish Pauli matrices \(\sigma^\alpha\) from spin operators \(S^\alpha=\sigma^\alpha/2\). Both of these are common conventions, and because of these ambiguities, ground-state energies may differ by a factor of four.11
Rubric
1. Reviewability
| Level | Meaning | Status |
|---|---|---|
| Full | All tasks and scoring logic inspectable,12 and harness/API settings used for each model (reasoning effort, token/time limits, tool access, system prompts) are fully disclosed | [proceed to 2] |
| Partial | A representative sample of tasks and scoring logic inspectable, with full harness/API settings disclosed | [proceed to 2] |
| Inadequate | Limited, biased, or no inspectable tasks or scoring logic, harness/API settings undisclosed | [stop → NEI] |
2. Scoring
This is the minimum standard required to be Verified; failing any of these items results in a Flawed verdict.
| Scoring | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Either errors affect ≥20%13 of the total score in the inspected sample,14 or there is an issue that corrupts grading at scale |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Benchmark Consistency | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Leaderboard has incomparable results from different versions |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Elicitation | |
|---|---|
| Examples | Model elicitation is extremely constraining and is not the focus of the benchmark:
|
| Default threshold for Flawed | Substantially reduced performance compared to reasonable alternatives for the tasks |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Bias in Evaluation Setup | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Material model-specific advantage found |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
3. Evaluation Quality
This is the standard we would like all benchmarks to meet, but it is not necessarily disqualifying to omit or fail these items.
| Question | Status | Notes |
|---|---|---|
| Elicitation and resource adequacy: Are the resources given to models (reasoning token/turn budget, tool access, etc.) sufficient for them to perform near their ceiling? | SufficientConstrainingUnreasonably constrainingUnknownNot reviewed | — |
| Scaffold fairness: What scaffold does the leaderboard report? | Shared common scaffoldMix of model-specific and common scaffoldsModel-specific scaffoldsNot reviewed | — |
| Is there evidence/risk of contamination? | Not reviewed | — |
| Has human completability been assessed? | All tasksRepresentative set of tasksPoor implementation (Unrepresentative set of tasks, unreasonable set of participants)Not establishedNot reviewed | — |
| Score range (if possible to estimate) | — floor, — ceilingNot reviewed | — |
| Statistical adequacy: how many runs/model (≥ 5 recommended for error bars) | — runs/modelUnknownNot reviewed | — |
| Construct validity | Measures stated capabilitiesPartially measures stated capabilitiesDoes not measure stated capabilitiesNot reviewed | — |
Disclaimer
We attempted to reach out to the benchmark developer prior to launch, and if they write a response to this review we will link it here. If you’re the benchmark developer, please email us at reviews@epoch.ai with a link to your response if you would like us to include it here.
-
We reviewed v1. As of October 22, 2026, the leaderboard reports v2. v2 revised 47 challenges, full details are available at the latest commit.
-
Experience levels ranged from senior Ph.D. students to professors.
-
Due to limited reviewer capacity, only 56 of 71 questions were reviewed. Ansari et al. reviewed challenges within their area of expertise that they were confident they could solve.
-
Leaderboards exclude the example problem and report scores on the remaining 70 questions.
-
Scores are from Artificial Analysis, run without an agent harness/tools, against the official grading server. Mean@k is the average accuracy over k attempts.
-
Scored using an LLM judge against the Ansari et al.s’ answers, not the grading server. Tool-use was enabled for Fable and Sol.
-
Claude Fable 5.1 & Claude Mythos 5.1 system card, section 8.9
-
Pass@1 scores at max effort without tool use, averaged over 5 attempts per problem.
-
Pass@1 scores at max effort with tools, averaged over 16 attempts per problem.
-
Ansari et al. Figure 4
-
Either publicly or privately to reviewers under an NDA.
-
A benchmark is rated Flawed if scoring defects could affect ≥20% of the benchmark’s full score. For benchmarks with equally weighted all-or-nothing tasks, our threshold corresponds to ≥20% of tasks having defects. For benchmarks with partial scoring or unequal task weights, our threshold corresponds to ≥20% of the total possible score affected by defects. For single task benchmarks, our threshold is the reported score having defects that could affect it by ≥20%.
-
For benchmarks with >50 available tasks, we will sample a random set of 50 (stratified by category, when present). If the observed error rate is 15–25%, we will expand the sample to 100. For benchmarks with ≤50 available tasks, all available tasks will be assessed.
About Benchmark Reviews
Epoch AI’s Benchmark Reviews are independent reviews of external AI benchmarks. Our documentation describes the rubric behind this verdict, how we choose which benchmarks to review, and answers frequently asked questions.