Benchmark Reviews documentation

OverviewMethodologyIncluded benchmarksFAQ
Show sidebarMethodology

Methodology

We use the following rubric to evaluate each benchmark.

Rubric v1. We expect future revisions to account for new types of benchmark flaws.

Verdict Categories

VerdictDefinition
VerifiedReviewability (Section 1) was judged as Full or Partial, and no score-affecting defect (Section 2) was severe enough to cross our quality threshold. All noted issues are logged as caveats in the review's Limitations section.
FlawedReviewability (Section 1) was judged as Full or Partial, but at least one score-affecting defect (Section 2) was severe enough to cross our quality threshold.
Not enough infoRequired information was unavailable to review, so a Verified/Flawed label cannot be supported.

1. Reviewability

LevelMeaningStatus
FullAll tasks and scoring logic inspectable,1 and harness/API settings used for each model (reasoning effort, token/time limits, tool access, system prompts) are fully disclosed [proceed to 2]
PartialA representative sample of tasks and scoring logic inspectable, with full harness/API settings disclosed [proceed to 2]
InadequateLimited, biased, or no inspectable tasks or scoring logic, harness/API settings undisclosed [stop → NEI]

2. Scoring

This is the minimum standard required to be Verified; failing any of these items results in a Flawed verdict.

Scoring
Examples
  • Essentially impossible to correctly answer as written (e.g., underspecified task, hidden requirement, missing file/tool)
  • False Negative (e.g., overly strict scorer, stale/incorrect ground truth, dependent on live external state that can drift, sandbox failure independent of the agent)
  • False Positive (e.g., lax scorer, reward-hackable environment, stated skill can be bypassed via a shortcut such as exploiting an error in the scoring logic or retrieving the answer from the harness/web)
  • Task egregiously doesn’t measure the claimed capability
Default threshold for FlawedEither errors affect ≥20%2 of the total score in the inspected sample,3 or there is an issue that corrupts grading at scale
Status
Pass Flag [stop → Flawed] Not reviewed
Notes—
Benchmark Consistency
Examples
  • Scorer, instructions, or ground truth changed without a version bump
Default threshold for FlawedLeaderboard has incomparable results from different versions
Status
Pass Flag [stop → Flawed] Not reviewed
Notes—
Elicitation
Examples

Model elicitation is extremely constraining and is not the focus of the benchmark:

  • Under-resourced relative to task size (token/turn/time limit, sandbox resources)
  • Poor context management
  • Lack of agentic environment where one would be natural to provide
  • Excessive non-voluntary termination for agentic benchmarks
Default threshold for FlawedSubstantially reduced performance compared to reasonable alternatives for the tasks
Status
Pass Flag [stop → Flawed] Not reviewed
Notes—
Bias in Evaluation Setup
Examples
  • Uneven compute/token budgets
  • Unfair scaffold choice (e.g., only a subset of models optimized)
Default threshold for FlawedMaterial model-specific advantage found
Status
Pass Flag [stop → Flawed] Not reviewed
Notes—

3. Evaluation Quality

This is the standard we would like all benchmarks to meet, but it is not necessarily disqualifying to omit or fail these items.

QuestionStatusNotes
Elicitation and resource adequacy: Are the resources given to models (reasoning token/turn budget, tool access, etc.) sufficient for them to perform near their ceiling?
Sufficient Constraining Unreasonably constraining Unknown Not reviewed
—
Scaffold fairness: What scaffold does the leaderboard report?
Shared common scaffold Mix of model-specific and common scaffolds Model-specific scaffolds Not reviewed
—
Is there evidence/risk of contamination?
As of review date:
—% of tasks public
—% of solutions public Not reviewed
—
Has human completability been assessed?
All tasks Representative set of tasks Poor implementation (Unrepresentative set of tasks, unreasonable set of participants) Not established Not reviewed
—
Score range (if possible to estimate)
— floor, — ceiling Not reviewed
—
Statistical adequacy: how many runs/model (≥ 5 recommended for error bars)
— runs/model Unknown Not reviewed
—
Construct validity
Measures stated capabilities Partially measures stated capabilities Does not measure stated capabilities Not reviewed
—

For benchmarks with existing credible reviews, we will cite the external review rather than replicate the findings.
Notes
  1. Either publicly or privately to reviewers under an NDA. Return
  2. A benchmark is rated Flawed if scoring defects could affect ≥20% of the benchmark’s full score. For benchmarks with equally weighted all-or-nothing tasks, our threshold corresponds to ≥20% of tasks having defects. For benchmarks with partial scoring or unequal task weights, our threshold corresponds to ≥20% of the total possible score affected by defects. For single task benchmarks, our threshold is the reported score having defects that could affect it by ≥20%. Return
  3. For benchmarks with >50 available tasks, we will sample a random set of 50 (stratified by category, when present). If the observed error rate is 15–25%, we will expand the sample to 100. For benchmarks with ≤50 available tasks, all available tasks will be assessed. Return
Overview
Included benchmarks