Benchmark Review

ARC-AGI-3

Reviewed: Oct. 30, 2026

Benchmark creator: ARC Prize Foundation

Verdict: Flawed

ARC-AGI-3 (Abstraction and Reasoning Corpus for Artificial General Intelligence 3) is an interactive reasoning benchmark designed to measure AI agents’ ability to explore new environments and adapt their strategy across increasingly difficult levels of multiple games. Each environment consists of a 64×64 grid and requires the player to infer the rules and the objective of the game. Harder levels often combine mechanics introduced in earlier levels. ARC-AGI-3 is scored via action efficiency1 relative to a variable human baseline. However, models are never told to be action efficient; they are simply told, “Your goal is to win.” This means it measures a mixture of capability and propensity towards efficiency. Efficiency gains are capped at 7% above the human baseline, making scores harder to interpret. In addition, the benchmark has numerous problems with version consistency, model elicitation in the official ‘Standard’ harness, and bias in evaluation setup. Each of these, along with the scoring issues, meets a threshold in our rubric, so we designate ARC-AGI-3 as Flawed.

Methodology

We reviewed ARC-AGI-3 as of October 30th, 2026, using only publicly available information: the 25 public games, ARC Prize’s documentation and harness code, the verified leaderboard, public human gameplay logs, and published analyses by OpenAI and ARC Prize.

ARC-AGI-3 consists of 25 public, 55 semi-private, and 55 fully private grid-based games.2 Each game is an interactive turn-based environment rather than a static input/output pair as in ARC-AGI-1 and ARC-AGI-2. Models are not told the objective or given instructions; instead, they must infer the rules.

The leaderboard reports Relative Human Action Efficiency (RHAE), which is constructed as follows:3

  1. Human baseline: A human baseline for each level is created by taking the upper median4 of multiple first-time players, ranked by actions.
  2. Level score: The score on each level is \(\left(\frac{\text{human\_baseline\_actions}}{\text{AI\_actions}}\right)^2\), capped at 1.15.5 Runs are stopped after 5 x human_baseline_actions. Failing to complete a level ends the game and results in a score of 0 for any remaining levels.
  3. Game score: The game score is the weighted average of all per-level scores using the 1-indexed level number as the weight. Levels increase in difficulty, so harder levels carry more weight. The game score is capped at the weighted fraction of completed levels. For example, if all levels are completed, the cap is 1; if the first 3 of 5 levels are completed, the cap is 6/15.
  4. Total score: The total score is the average of all game scores.

The verified leaderboard reports both the ‘Standard’ and the ‘Provider Adapter’ harness.6 The ‘Provider Adapter’ harness preserves reasoning and uses compaction. In addition, an unverified community leaderboard allows self-reported scores.

Findings

Scoring:
The goal of the benchmark is to exceed human-level action efficiency, but models are told only “You are playing a game. Your goal is to win.” Models are given an action budget of five times the human baseline on each level. Failing a level ends overall progress in the game, so future levels score 0. While the impact is negligible on that level (at most 4%), stopping progress on higher levels can have a significant impact.

There are numerous issues with the headline score reported on the leaderboard:

  • It is not a percentage, even though it is reported as one. If the human baseline for a level is 5 actions and a model takes 25, the score for that level will be \((5/25)^2\) = 4%. Reporting this as 4% suggests the model is 4% as efficient as a human, even though it uses five times as many actions.
  • The score conflates completion and efficiency. For example, in a 6-level game, finishing every level in twice as many actions as the human baseline results in a score of 25%. Finishing only the first 3 levels at the human baseline results in a score of 28.6%.
  • The human baseline is an input to the leaderboard score, and it varies based on level difficulty. Each game has at least ten participants, but the human baseline is the upper median of those who completed the level. Thus, the human baseline is more challenging to beat on harder levels. This effect is compounded because harder levels carry more weight.
  • The human baseline is hard even for humans. All levels were fully solved by at least 2 humans, but only 145/342 of the human attempts on the public set (which is easier than the semi-private set used in the leaderboard) resulted in completion. Because of these effects, the average human tester would only score 48% on the public set.7
  • The game level cap may prevent models with efficiency above the baseline from being distinguished. This is particularly concerning for differentiating frontier models. As scores approach 100%, it would be useful to distinguish performance above the human baseline, not just up to it. On the semi-private set, GPT-6 Astra (max) uses fewer actions than the human baseline on 96% of levels and, on average, 51.7% fewer actions per level, scoring 98.6% with the ‘Provider Adapter’ harness.8

It would be better practice to score models and humans against the known optimal playthroughs, which would provide a meaningful ceiling.

Elicitation: The ‘Standard’ harness substantially under-elicits models. Reasoning is discarded after every action, and history is trimmed at 175,000 characters, roughly 10-14 frames of memory.9 These effects are large. OpenAI found that “retaining reasoning and compaction allow GPT‑5.6 Sol (max) to achieve roughly 3x the score with 6x fewer output tokens”.10 As of October 30th, 2026, every model evaluated at the same effort level under both harnesses scored substantially higher with the ‘Provider Adapter’ harness. For example, GPT-6 Astra (max) rose from 62.7% to 98.6%. Neither harness sends frames as images: both send text grids.11

Benchmark consistency: The games and the toolkit software are versioned; however, the harness, prompt, and leaderboard entries are not. On May 20th, 2026, the system prompt was changed to explicitly allow models to include context they want to carry forward,12 with no version update. Older results were not re-run, so the leaderboard mixes results from different setups, although it is not clear what (if any) impact this has on results.

Bias in evaluation setup: The verified leaderboard claims a $10,000 cap per run, but 19 entries as of October 30th, 2026 cost more than this, up to $49,800. The verified leaderboard notes that “for models that were not able to produce full test outputs, remaining tasks were marked as incorrect.” The reason for incomplete test output is not provided, and these cases are not designated on the leaderboard.

Rubric

1. Reviewability

LevelMeaningStatus
FullAll tasks and scoring logic inspectable,13 and harness/API settings used for each model (reasoning effort, token/time limits, tool access, system prompts) are fully disclosed [proceed to 2]
PartialA representative sample of tasks and scoring logic inspectable, with full harness/API settings disclosed [proceed to 2]
InadequateLimited, biased, or no inspectable tasks or scoring logic, harness/API settings undisclosed [stop → NEI]

2. Scoring

This is the minimum standard required to be Verified; failing any of these items results in a Flawed verdict.

Scoring
Examples
  • Essentially impossible to correctly answer as written (e.g., underspecified task, hidden requirement, missing file/tool)
  • False Negative (e.g., overly strict scorer, stale/incorrect ground truth, dependent on live external state that can drift, sandbox failure independent of the agent)
  • False Positive (e.g., lax scorer, reward-hackable environment, stated skill can be bypassed via a shortcut such as exploiting an error in the scoring logic or retrieving the answer from the harness/web)
  • Task egregiously doesn’t measure the claimed capability
Default threshold for FlawedEither errors affect ≥20%14 of the total score in the inspected sample,15 or there is an issue that corrupts grading at scale
Status
Pass Flag [stop → Flawed] Not reviewed
Notes—
Benchmark Consistency
Examples
  • Scorer, instructions, or ground truth changed without a version bump
Default threshold for FlawedLeaderboard has incomparable results from different versions
Status
Pass Flag [stop → Flawed] Not reviewed
Notes—
Elicitation
Examples

Model elicitation is extremely constraining and is not the focus of the benchmark:

  • Under-resourced relative to task size (token/turn/time limit, sandbox resources)
  • Poor context management
  • Lack of agentic environment where one would be natural to provide
  • Excessive non-voluntary termination for agentic benchmarks
Default threshold for FlawedSubstantially reduced performance compared to reasonable alternatives for the tasks
Status
Pass Flag [stop → Flawed] Not reviewed
Notes—
Bias in Evaluation Setup
Examples
  • Uneven compute/token budgets
  • Unfair scaffold choice (e.g., only a subset of models optimized)
Default threshold for FlawedMaterial model-specific advantage found
Status
Pass Flag [stop → Flawed] Not reviewed
Notes—

3. Evaluation Quality

This is the standard we would like all benchmarks to meet, but it is not necessarily disqualifying to omit or fail these items.

QuestionStatusNotes
Elicitation and resource adequacy: Are the resources given to models (reasoning token/turn budget, tool access, etc.) sufficient for them to perform near their ceiling?
Sufficient Constraining Unreasonably constraining Unknown Not reviewed
—
Scaffold fairness: What scaffold does the leaderboard report?
Shared common scaffold Mix of model-specific and common scaffolds Model-specific scaffolds Not reviewed
—
Is there evidence/risk of contamination?
Not reviewed
—
Has human completability been assessed?
All tasks Representative set of tasks Poor implementation (Unrepresentative set of tasks, unreasonable set of participants) Not established Not reviewed
—
Score range (if possible to estimate)
— floor, — ceiling Not reviewed
—
Statistical adequacy: how many runs/model (≥ 5 recommended for error bars)
— runs/model Unknown Not reviewed
—
Construct validity
Measures stated capabilities Partially measures stated capabilities Does not measure stated capabilities Not reviewed
—

Disclaimer

We attempted to reach out to the benchmark developer prior to launch, and if they write a response to this review we will link it here. If you’re the benchmark developer, please email us at reviews@epoch.ai with a link to your response if you would like us to include it here.

Notes
  1. ARC-AGI-3 defines intelligence as efficiency across exploration, modeling, goal-setting, and planning & execution, and uses action efficiency, or the number of actions required to solve a new environment, as the single measure. Return

  2. The public set is intentionally easier and is not reported in the leaderboard. The semi-private set is accessed via an external API and used to test frontier models. The fully private set is used for the ARC Prize competition. Return

  3. This reflects the scoring function as of October 30th, 2026. Numerous changes to the scoring function were made after the benchmark was released. These changes were documented in the changelog and older leaderboard scores were re-computed using the new methodology. Return

  4. For odd numbers of players, this is the median. For an even number of players n, this is the middle entry with more actions. Return

  5. The cap of 1.15 was added to prevent positive outliers from having an outsize effect on the total score. Since the level score is a squared ratio, efficiency gains scale quickly. In practice, this prevents efficiency gains beyond ~7% from being distinguishable. Return

  6. The ‘Provider Adapter’ harness was added to the verified leaderboard on September 3rd, 2026. More details can be found here. Return

  7. Based on official gameplay logs, calculated by OpenAI. Return

  8. https://arcprize.org/blog/astra Return

  9. Each action returns at least 1 frame. Animation, such as a sliding piece, is represented with multiple frames. If more than 7 frames are returned, the model sees only 7 evenly spaced frames, including the first and last. The human baseline ranges from 6 to 578 actions. Return

  10. https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores/ Return

  11. The text grid consists of 64 rows of 64 numbers. Each number ranges from 0-15 and is meant to represent a color. The frame is sent as a text grid, rather than the image, in both the ‘Standard’ and ‘Provider Adapter’ harness. Return

  12. https://github.com/arcprize/arc-agi-3-benchmarking/commit/0138a6aed609326a482783189ade4dda15b6a83a Return

  13. Either publicly or privately to reviewers under an NDA. Return
  14. A benchmark is rated Flawed if scoring defects could affect ≥20% of the benchmark’s full score. For benchmarks with equally weighted all-or-nothing tasks, our threshold corresponds to ≥20% of tasks having defects. For benchmarks with partial scoring or unequal task weights, our threshold corresponds to ≥20% of the total possible score affected by defects. For single task benchmarks, our threshold is the reported score having defects that could affect it by ≥20%. Return
  15. For benchmarks with >50 available tasks, we will sample a random set of 50 (stratified by category, when present). If the observed error rate is 15–25%, we will expand the sample to 100. For benchmarks with ≤50 available tasks, all available tasks will be assessed. Return

About Benchmark Reviews

Epoch AI’s Benchmark Reviews are independent reviews of external AI benchmarks. Our documentation describes the rubric behind this verdict, how we choose which benchmarks to review, and answers frequently asked questions.

Go to sourceRead the documentation and FAQs