Humanity’s Last Exam is a knowledge-based benchmark compiled through submissions from the public. After auditing a random sample of 48 questions, we found that 22 (46%) questions had substantial accuracy-altering errors, leading us to designate the benchmark as Flawed.
Methodology
We reviewed the original benchmark, Humanity’s Last Exam, not Humanity’s Last Exam-Rolling or Humanity’s Last Exam-Verified. We randomly sampled 48 questions, 6 from each of 8 categories (Biology/Medicine, Chemistry, Computer Science/AI, Engineering, Humanities/Social Science, Mathematics, Physics, Other), then prompted Fable 5 to surface errors, primarily of three kinds: questions that are essentially impossible to answer correctly as written (impossible), questions that could mark an objectively correct answer incorrect (false negatives), and questions that could mark an incorrect answer correct (false positives). We audited and confirmed each error by hand, filtering out identified errors that did not meet our evidence bar.
Of those 48 questions, we found that 12 were impossible to answer correctly as written, for reasons like missing assumptions never given to the LLM, contradictory premises, multiple valid interpretations, or questions that demanded objective answers to subjective questions. An additional 10 well-posed questions could produce false negatives, of which 5 could also produce false positives. In total, 22 of 48 questions, or 46%, were defective.
Representative Errors
- Computer Science/AI: Impossible. The question asks for a “4-point DFT”, a discrete Fourier transform over 4 samples, but the sequence has 8 entries.
- Engineering: False negative, false positive. The author’s answer key ends with the formula printed in option E, but the answer key says D.
- Humanities/Social Science: False negative. The question asks for a three-letter abbreviation for 1 Kings. Both “1kin” and “1kgs” are commonly used abbreviations, but the answer key matches only “1kin”. “1kgs” is graded as incorrect.
- Mathematics: False negative, false positive. The author’s rationale calculates the normalization constant as 1/21.3535, but the answer key is 21.35.
Errors
| Task | Category | Error Type | Notes |
|---|---|---|---|
| 6736c2e82db9d8cc7cd8e607 | Biology/Medicine | Impossible | Medical case study already states the confirmed diagnosis, but asks for what the next diagnostic step is |
| 66ff35926045e9e64bbc5720 | Chemistry | Impossible | Asks for density as a function of height, but answer key is a single number |
| 66f275c6e2a2b5d3594eae87 | Computer Science/AI | Impossible | Asks for the smallest possible memory, but answer key’s program is provably not the smallest |
| 672f6cfd609b665328565e5c | Computer Science/AI | Impossible | The most memory-efficient program has no defined minimum, answer key’s 66 bytes can be beaten |
| 66f1384dba7cd382e52dc0c6 | Computer Science/AI | Impossible | Question asks for a 4-point DFT but the sequence has 8 entries |
| 66f2298b868e51658baf900a | Engineering | Impossible | Data supports several defensible values, key silently ignores one listed input |
| 67200b2f79e871b075407673 | Engineering | Impossible | Problem says the circuit is matched, then deducts a quantity that doesn’t need to be deducted if the circuit is matched |
| 67aacfd513ec9e1a16359d51 | Engineering | Impossible | CPU-cycle count depends on how you measure |
| 671fadb2272218e5d336a33f | Engineering | Impossible | No answer option is fully correct |
| 6737224020f2fc6d8d08075c | Humanities/Social Science | Impossible | Asks about the themes of the author’s own unpublished artwork |
| 671683479e5d26a2905120dc | Other | Impossible | Intended solution requires a mob that cannot spawn close to the player under normal conditions, and an alternate block also works under abnormal conditions of the intended solution |
| 672333955d82e15ca8e37afb | Physics | Impossible | Answer changes depending on which uncertainty-principle convention is used, and none is specified |
| 67371496e04511118435d5a4 | Computer Science/AI | False negative, false positive | Key’s time-dilation factor has a misplaced decimal (0.9963 vs 0.963) |
| 671fee7b3fdf9b50dcb7ca58 | Engineering | False negative, false positive | Mislabeled answer: the author’s own rationale matches option E, but key is D |
| 6711e5e05e64a53ed09449fd | Humanities/Social Science | False negative, false positive | Answer key (0.218) comes from rounding mid-calculation, exact answer is 0.220 |
| 67381ce26a5242a22fe4681f | Mathematics | False negative, false positive | Answer key gives the reciprocal of the answer found in its own rationale |
| 67361730dba36cc0d595f422 | Physics | False negative, false positive | Answer key cites some interference that doesn’t occur |
| 6734b98afaf4e6eb454c1c9d | Biology/Medicine | False negative | Answer key misspells the organism’s genus name and LLM judge may thus mark it wrong |
| 67253e40cff9fdccf85f3f08 | Chemistry | False negative | Multiple molecules also satisfy every listed constraint, but only one is in the answer key |
| 6730f3c006cd33fe46ca2dfe | Chemistry | False negative | Answer key’s mass value is slightly off at the demanded precision |
| 67190f1eee94d36fc7734311 | Humanities/Social Science | False negative | Key uses a nonstandard Bible-book abbreviation that could easily be graded incorrectly due to difference in abbreviation |
| 6724f79b262cac26e3c38360 | Humanities/Social Science | False negative | Only one Unicode rendering of the same word is accepted in the answer key |
The answer keys of the other 26 sampled questions were found to be correct, with id values as follows: 668825f80a642802bdfeadfa, 66e4cdec11c64a7e4051b2d9, 66e883265ab37f0a7da089be, 66eaa5ddc7a3252f0f3fe53f, 66eaf9bbb082c5e6a76a49b5, 66eb3aab27b19dd593af7ebe, 66ecfd03e7478ca5fcbb7495, 66f02cc56ea864e80ba2d3d7, 66f042e57cd17eb270eb9ccb, 66f864c8f7c10f2d2ba396df, 66fea0c4cb66b0e85c55ee52, 67029d8de7ea2ce5def2f39f, 6705b23880154ff607a205b5, 670be48d7038d6936230870a, 6713a7d01e0e03ffe2253dd0, 67142e7e7da71e9cbf55a7f9, 67178b22b7f6dd9be2a4dd68, 671ad72c9fdc33c08a784b3a, 671ee4d26edc2afd69958953, 67216f0abddbccdce9955e93, 6722b1f9f3a489ee08117730, 672455ce7a14971607e3d9f0, 672588e1f71812e186947615, 672fadd93c2722c42adabef3, 67378905b73b79a4008926af and 674982572b1759933c70c4eb.
Rubric
1. Reviewability
| Level | Meaning | Status |
|---|---|---|
| Full | All tasks and scoring logic inspectable*, and harness/API settings used for each model (reasoning effort, token/time limits, tool access, system prompts) are fully disclosed | [proceed to 2] |
| Partial | A representative sample of tasks and scoring logic inspectable, with full harness/API settings disclosed | [proceed to 2] |
| Inadequate | Limited, biased, or no inspectable tasks or scoring logic, harness/API settings undisclosed | [stop → NEI] |
* Either publicly or privately to reviewers.
2. Scoring
This is the minimum standard required to be Verified; failing any of these items results in a Flawed verdict.
| Scoring | |
|---|---|
| Examples |
|
| Default threshold for Flawed | ≥20% of inspected sample† contains errors or there is an issue that corrupts grading at scale |
| Status |
Pass
Flag [stop → Flawed]
Not reviewed |
| Notes | — |
| Benchmark Consistency | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Leaderboard has incomparable results from different versions |
| Status |
Pass
Flag [stop → Flawed]
Not reviewed |
| Notes | — |
| Elicitation | |
|---|---|
| Examples | Model elicitation is extremely constraining and is not the focus of the benchmark:
|
| Default threshold for Flawed | Substantially reduced performance compared to reasonable alternatives for the tasks |
| Status |
Pass
Flag [stop → Flawed]
Not reviewed |
| Notes | — |
| Bias in Evaluation Setup | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Material model-specific advantage found |
| Status |
Pass
Flag [stop → Flawed]
Not reviewed |
| Notes | — |
† For benchmarks with >50 available tasks, we will sample a random set of 50 (stratified by category, when present). If the observed error rate is 15–25%, we will expand the sample to 100. For benchmarks with ≤50 available tasks, all available tasks will be assessed.
3. Evaluation Quality
This is the standard we would like all benchmarks to meet, but it is not necessarily disqualifying to omit or fail these items.
| Question | Status | Notes |
|---|---|---|
| Elicitation and resource adequacy: Are the resources given to models (reasoning token/turn budget, tool access, etc.) sufficient for them to perform near their ceiling? |
Sufficient
Constraining
Unreasonably constraining
Unknown
Not reviewed | — |
| Scaffold fairness: What scaffold does the leaderboard report? |
Shared common scaffold
Mix of model-specific and common scaffolds
Model-specific scaffolds
Not reviewed | — |
| Is there evidence/risk of contamination? |
Not reviewed | — |
| Has human completability been assessed? |
All tasks
Representative set of tasks
Poor implementation (Unrepresentative set of tasks, unreasonable set of participants)
Not established
Not reviewed | — |
| Score range (if possible to estimate) | — floor, — ceiling
Not reviewed | — |
| Statistical adequacy: how many runs/model (≥ 5 recommended for error bars) | — runs/model
Unknown
Not reviewed | — |
| Construct Validity |
Measures stated capabilities
Partially measures stated capabilities
Does not measure stated capabilities
Not reviewed | — |
Disclaimer
This is a partial review. We stopped inspecting once we discovered errors that exceeded our published threshold for a flawed benchmark.
We attempted to reach out to the benchmark developer prior to launch, and if they write a response to this review we will link it here. If you’re the benchmark developer, please email us at reviews@epoch.ai with a link to your response if you would like us to include it here.
About Benchmark Reviews
Epoch AI’s Benchmark Reviews are independent reviews of external AI benchmarks. Our documentation describes the rubric behind this verdict, how we choose which benchmarks to review, and answers frequently asked questions.