CMT-Benchmark v1.0.0 (Condensed Matter Theory)1 is designed to test AI models’ understanding of condensed matter theory at the level of an expert researcher; it consists of 50 original problems across 8 problem types.2 We designated the benchmark Flawed because an audit by physics researchers (Ansari et al. 2026) found scoring defects in 60% of the problems; our threshold for a Flawed verdict is 20% or more.
Methodology
Ansari et al. reviewed all 50 problems and found errors in 30 (60%) of them. The majority of errors were due to missing assumptions, incorrect mathematical expressions, incorrect or incomplete reference answers, or significant typos/formatting errors. Ansari et al. repaired 29 of the flawed problems and discarded the remaining one, due to ambiguity. Scores on the original 50 questions and the repaired subset of 49 questions are shown in the table below. Ansari et al. graded both the original set and repaired subset with an LLM judge, since they report that the official grading code was unavailable to them. They did have access to the reference solutions, which the LLM judge used to grade the original set. The repaired subset was graded using reviewer-created solutions for problems with errors in the reference solutions. The table below compares scores on the original benchmark with scores on the repaired subset of 49 questions.
| Original mean@43 | Repaired mean@4 | Original pass@44 | Repaired pass@4 | |
|---|---|---|---|---|
| Fable 5 High | 53.5% | 85.2% | 60% | 93.88% |
| GPT-5.6 Sol High | 61% | 87.24% | 72% | 97.96% |
| Gemini 3.1 Pro High (no tools) | 50.5% | 78.06% | 58% | 87.76% |
- Repaired problems: 0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 18, 23, 26, 27, 31, 35, 36, 38, 39, 40, 41, 42, 43, 44, 46
- Excluded problem: 14
- Validated problems: 4, 15, 17, 19, 20, 21, 22, 24, 25, 28, 29, 30, 32, 33, 34, 37, 45, 47, 48, 49
Example Repaired Problem
Red: removed or replaced text in the original. Blue: added or replacement text in the repair.
| Original problem 31 | Repaired problem 315 |
|---|---|
| Consider a quantum Ising a) At \(h=1\) and \(g=0\), the excitation energy vanishes. b) c) At \(h=10\) and \(g=1\), d) At \(h=0.1\) and \(g=1\), \(\langle\sigma_i^x\sigma_j^x\rangle\) vanishes as a power-law for a large \(\lvert i-j\rvert\). Solution: a; c | Consider a one-dimensional quantum Ising chain at zero temperature, with each nearest-neighbor bond counted once, with the following Hamiltonian: \(H=-\sum_{i,j}\sigma_i^z\sigma_j^z+h\sum_i\sigma_i^x+g\sum_i\sigma_i^z\). Unless explicitly defined as connected, the correlation functions below are ordinary correlators. Which of the following statements are correct? Indicate all that apply. a) At \(h=1\) and \(g=0\), the excitation energy vanishes. b) At \(g=0\), there is a symmetry breaking transition at a finite value of \(h\). c) At \(h=10\) and \(g=1\), the connected correlation function \(\langle\sigma_i^z\sigma_j^z\rangle-\langle\sigma_i^z\rangle\langle\sigma_j^z\rangle\) vanishes exponentially as a function of \(\lvert i-j\rvert\). d) At \(h=0.1\) and \(g=1\), \(\langle\sigma_i^x\sigma_j^x\rangle\) vanishes as a power-law for a large \(\lvert i-j\rvert\). Solution: a; b; c |
Rubric
1. Reviewability
| Level | Meaning | Status |
|---|---|---|
| Full | All tasks and scoring logic inspectable,6 and harness/API settings used for each model (reasoning effort, token/time limits, tool access, system prompts) are fully disclosed | [proceed to 2] |
| Partial | A representative sample of tasks and scoring logic inspectable, with full harness/API settings disclosed | [proceed to 2] |
| Inadequate | Limited, biased, or no inspectable tasks or scoring logic, harness/API settings undisclosed | [stop → NEI] |
2. Scoring
This is the minimum standard required to be Verified; failing any of these items results in a Flawed verdict.
| Scoring | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Either errors affect ≥20%7 of the total score in the inspected sample,8 or there is an issue that corrupts grading at scale |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Benchmark Consistency | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Leaderboard has incomparable results from different versions |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Elicitation | |
|---|---|
| Examples | Model elicitation is extremely constraining and is not the focus of the benchmark:
|
| Default threshold for Flawed | Substantially reduced performance compared to reasonable alternatives for the tasks |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Bias in Evaluation Setup | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Material model-specific advantage found |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
3. Evaluation Quality
This is the standard we would like all benchmarks to meet, but it is not necessarily disqualifying to omit or fail these items.
| Question | Status | Notes |
|---|---|---|
| Elicitation and resource adequacy: Are the resources given to models (reasoning token/turn budget, tool access, etc.) sufficient for them to perform near their ceiling? | SufficientConstrainingUnreasonably constrainingUnknownNot reviewed | — |
| Scaffold fairness: What scaffold does the leaderboard report? | Shared common scaffoldMix of model-specific and common scaffoldsModel-specific scaffoldsNot reviewed | — |
| Is there evidence/risk of contamination? | Not reviewed | — |
| Has human completability been assessed? | All tasksRepresentative set of tasksPoor implementation (Unrepresentative set of tasks, unreasonable set of participants)Not establishedNot reviewed | — |
| Score range (if possible to estimate) | — floor, — ceilingNot reviewed | — |
| Statistical adequacy: how many runs/model (≥ 5 recommended for error bars) | — runs/modelUnknownNot reviewed | — |
| Construct validity | Measures stated capabilitiesPartially measures stated capabilitiesDoes not measure stated capabilitiesNot reviewed | — |
Disclaimer
We attempted to reach out to the benchmark developer prior to launch, and if they write a response to this review we will link it here. If you’re the benchmark developer, please email us at reviews@epoch.ai with a link to your response if you would like us to include it here.
-
We reviewed v1.0.0. The version and changelog provides up to date information on the current version.
-
Problem types include Hartree-Fock, Exact Diagonalization, Density Matrix Renormalization Group, Quantum Monte Carlo, Variational Monte Carlo, Projected Entangled Pair States, Statistical Mechanics, and Other (model-building and application of fundamental principles).
-
mean@k is the average accuracy over k attempts
-
pass@k is the percent of problems solved in at least one of k attempts
-
“The value \(h=1\) assumes an unstated one-dimensional normalization. For \(g\neq 0\), the ordinary correlator in (c) approaches a nonzero product of one-point functions; exponential decay applies to the connected correlator. The reference also omits the \(g=0\) symmetry-breaking transition in (b). The documented repair specifies these assumptions and changes the answer from (a; c) to (a; b; c).” (Ansari et al.)
-
Either publicly or privately to reviewers under an NDA.
-
A benchmark is rated Flawed if scoring defects could affect ≥20% of the benchmark’s full score. For benchmarks with equally weighted all-or-nothing tasks, our threshold corresponds to ≥20% of tasks having defects. For benchmarks with partial scoring or unequal task weights, our threshold corresponds to ≥20% of the total possible score affected by defects. For single task benchmarks, our threshold is the reported score having defects that could affect it by ≥20%.
-
For benchmarks with >50 available tasks, we will sample a random set of 50 (stratified by category, when present). If the observed error rate is 15–25%, we will expand the sample to 100. For benchmarks with ≤50 available tasks, all available tasks will be assessed.
About Benchmark Reviews
Epoch AI’s Benchmark Reviews are independent reviews of external AI benchmarks. Our documentation describes the rubric behind this verdict, how we choose which benchmarks to review, and answers frequently asked questions.