ARC-AGI-3 (Abstraction and Reasoning Corpus for Artificial General Intelligence 3) is an interactive reasoning benchmark designed to measure AI agents’ ability to explore new environments and adapt their strategy across increasingly difficult levels of multiple games. Each environment consists of a 64×64 grid and requires the player to infer the rules and the objective of the game. Harder levels often combine mechanics introduced in earlier levels. ARC-AGI-3 is scored via action efficiency1 relative to a variable human baseline. However, models are never told to be action efficient; they are simply told, “Your goal is to win.” This means it measures a mixture of capability and propensity towards efficiency. Efficiency gains are capped at 7% above the human baseline, making scores harder to interpret. In addition, the benchmark has numerous problems with version consistency, model elicitation in the official ‘Standard’ harness, and bias in evaluation setup. Each of these, along with the scoring issues, meets a threshold in our rubric, so we designate ARC-AGI-3 as Flawed.
Methodology
We reviewed ARC-AGI-3 as of October 30th, 2026, using only publicly available information: the 25 public games, ARC Prize’s documentation and harness code, the verified leaderboard, public human gameplay logs, and published analyses by OpenAI and ARC Prize.
ARC-AGI-3 consists of 25 public, 55 semi-private, and 55 fully private grid-based games.2 Each game is an interactive turn-based environment rather than a static input/output pair as in ARC-AGI-1 and ARC-AGI-2. Models are not told the objective or given instructions; instead, they must infer the rules.
The leaderboard reports Relative Human Action Efficiency (RHAE), which is constructed as follows:3
- Human baseline: A human baseline for each level is created by taking the upper median4 of multiple first-time players, ranked by actions.
- Level score: The score on each level is \(\left(\frac{\text{human\_baseline\_actions}}{\text{AI\_actions}}\right)^2\), capped at 1.15.5 Runs are stopped after 5 x human_baseline_actions. Failing to complete a level ends the game and results in a score of 0 for any remaining levels.
- Game score: The game score is the weighted average of all per-level scores using the 1-indexed level number as the weight. Levels increase in difficulty, so harder levels carry more weight. The game score is capped at the weighted fraction of completed levels. For example, if all levels are completed, the cap is 1; if the first 3 of 5 levels are completed, the cap is 6/15.
- Total score: The total score is the average of all game scores.
The verified leaderboard reports both the ‘Standard’ and the ‘Provider Adapter’ harness.6 The ‘Provider Adapter’ harness preserves reasoning and uses compaction. In addition, an unverified community leaderboard allows self-reported scores.
Findings
Scoring:
The goal of the benchmark is to exceed human-level action efficiency, but models are told only “You are playing a game. Your goal is to win.”
Models are given an action budget of five times the human baseline on each level.
Failing a level ends overall progress in the game, so future levels score 0.
While the impact is negligible on that level (at most 4%), stopping progress on higher levels can have a significant impact.
There are numerous issues with the headline score reported on the leaderboard:
- It is not a percentage, even though it is reported as one. If the human baseline for a level is 5 actions and a model takes 25, the score for that level will be \((5/25)^2\) = 4%. Reporting this as 4% suggests the model is 4% as efficient as a human, even though it uses five times as many actions.
- The score conflates completion and efficiency. For example, in a 6-level game, finishing every level in twice as many actions as the human baseline results in a score of 25%. Finishing only the first 3 levels at the human baseline results in a score of 28.6%.
- The human baseline is an input to the leaderboard score, and it varies based on level difficulty. Each game has at least ten participants, but the human baseline is the upper median of those who completed the level. Thus, the human baseline is more challenging to beat on harder levels. This effect is compounded because harder levels carry more weight.
- The human baseline is hard even for humans. All levels were fully solved by at least 2 humans, but only 145/342 of the human attempts on the public set (which is easier than the semi-private set used in the leaderboard) resulted in completion. Because of these effects, the average human tester would only score 48% on the public set.7
- The game level cap may prevent models with efficiency above the baseline from being distinguished. This is particularly concerning for differentiating frontier models. As scores approach 100%, it would be useful to distinguish performance above the human baseline, not just up to it. On the semi-private set, GPT-6 Astra (max) uses fewer actions than the human baseline on 96% of levels and, on average, 51.7% fewer actions per level, scoring 98.6% with the ‘Provider Adapter’ harness.8
It would be better practice to score models and humans against the known optimal playthroughs, which would provide a meaningful ceiling.
Elicitation: The ‘Standard’ harness substantially under-elicits models. Reasoning is discarded after every action, and history is trimmed at 175,000 characters, roughly 10-14 frames of memory.9 These effects are large. OpenAI found that “retaining reasoning and compaction allow GPT‑5.6 Sol (max) to achieve roughly 3x the score with 6x fewer output tokens”.10 As of October 30th, 2026, every model evaluated at the same effort level under both harnesses scored substantially higher with the ‘Provider Adapter’ harness. For example, GPT-6 Astra (max) rose from 62.7% to 98.6%. Neither harness sends frames as images: both send text grids.11
Benchmark consistency: The games and the toolkit software are versioned; however, the harness, prompt, and leaderboard entries are not. On May 20th, 2026, the system prompt was changed to explicitly allow models to include context they want to carry forward,12 with no version update. Older results were not re-run, so the leaderboard mixes results from different setups, although it is not clear what (if any) impact this has on results.
Bias in evaluation setup: The verified leaderboard claims a $10,000 cap per run, but 19 entries as of October 30th, 2026 cost more than this, up to $49,800. The verified leaderboard notes that “for models that were not able to produce full test outputs, remaining tasks were marked as incorrect.” The reason for incomplete test output is not provided, and these cases are not designated on the leaderboard.
Rubric
1. Reviewability
| Level | Meaning | Status |
|---|---|---|
| Full | All tasks and scoring logic inspectable,13 and harness/API settings used for each model (reasoning effort, token/time limits, tool access, system prompts) are fully disclosed | [proceed to 2] |
| Partial | A representative sample of tasks and scoring logic inspectable, with full harness/API settings disclosed | [proceed to 2] |
| Inadequate | Limited, biased, or no inspectable tasks or scoring logic, harness/API settings undisclosed | [stop → NEI] |
2. Scoring
This is the minimum standard required to be Verified; failing any of these items results in a Flawed verdict.
| Scoring | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Either errors affect ≥20%14 of the total score in the inspected sample,15 or there is an issue that corrupts grading at scale |
| Status |
Pass
Flag [stop → Flawed]
Not reviewed |
| Notes | — |
| Benchmark Consistency | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Leaderboard has incomparable results from different versions |
| Status |
Pass
Flag [stop → Flawed]
Not reviewed |
| Notes | — |
| Elicitation | |
|---|---|
| Examples | Model elicitation is extremely constraining and is not the focus of the benchmark:
|
| Default threshold for Flawed | Substantially reduced performance compared to reasonable alternatives for the tasks |
| Status |
Pass
Flag [stop → Flawed]
Not reviewed |
| Notes | — |
| Bias in Evaluation Setup | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Material model-specific advantage found |
| Status |
Pass
Flag [stop → Flawed]
Not reviewed |
| Notes | — |
3. Evaluation Quality
This is the standard we would like all benchmarks to meet, but it is not necessarily disqualifying to omit or fail these items.
| Question | Status | Notes |
|---|---|---|
| Elicitation and resource adequacy: Are the resources given to models (reasoning token/turn budget, tool access, etc.) sufficient for them to perform near their ceiling? |
Sufficient
Constraining
Unreasonably constraining
Unknown
Not reviewed | — |
| Scaffold fairness: What scaffold does the leaderboard report? |
Shared common scaffold
Mix of model-specific and common scaffolds
Model-specific scaffolds
Not reviewed | — |
| Is there evidence/risk of contamination? |
Not reviewed | — |
| Has human completability been assessed? |
All tasks
Representative set of tasks
Poor implementation (Unrepresentative set of tasks, unreasonable set of participants)
Not established
Not reviewed | — |
| Score range (if possible to estimate) | — floor, — ceiling
Not reviewed | — |
| Statistical adequacy: how many runs/model (≥ 5 recommended for error bars) | — runs/model
Unknown
Not reviewed | — |
| Construct validity |
Measures stated capabilities
Partially measures stated capabilities
Does not measure stated capabilities
Not reviewed | — |
Disclaimer
We attempted to reach out to the benchmark developer prior to launch, and if they write a response to this review we will link it here. If you’re the benchmark developer, please email us at reviews@epoch.ai with a link to your response if you would like us to include it here.
-
ARC-AGI-3 defines intelligence as efficiency across exploration, modeling, goal-setting, and planning & execution, and uses action efficiency, or the number of actions required to solve a new environment, as the single measure.
-
The public set is intentionally easier and is not reported in the leaderboard. The semi-private set is accessed via an external API and used to test frontier models. The fully private set is used for the ARC Prize competition.
-
This reflects the scoring function as of October 30th, 2026. Numerous changes to the scoring function were made after the benchmark was released. These changes were documented in the changelog and older leaderboard scores were re-computed using the new methodology.
-
For odd numbers of players, this is the median. For an even number of players n, this is the middle entry with more actions.
-
The cap of 1.15 was added to prevent positive outliers from having an outsize effect on the total score. Since the level score is a squared ratio, efficiency gains scale quickly. In practice, this prevents efficiency gains beyond ~7% from being distinguishable.
-
The ‘Provider Adapter’ harness was added to the verified leaderboard on September 3rd, 2026. More details can be found here.
-
Based on official gameplay logs, calculated by OpenAI.
-
Each action returns at least 1 frame. Animation, such as a sliding piece, is represented with multiple frames. If more than 7 frames are returned, the model sees only 7 evenly spaced frames, including the first and last. The human baseline ranges from 6 to 578 actions.
-
https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores/
-
The text grid consists of 64 rows of 64 numbers. Each number ranges from 0-15 and is meant to represent a color. The frame is sent as a text grid, rather than the image, in both the ‘Standard’ and ‘Provider Adapter’ harness.
-
https://github.com/arcprize/arc-agi-3-benchmarking/commit/0138a6aed609326a482783189ade4dda15b6a83a
-
Either publicly or privately to reviewers under an NDA.
-
A benchmark is rated Flawed if scoring defects could affect ≥20% of the benchmark’s full score. For benchmarks with equally weighted all-or-nothing tasks, our threshold corresponds to ≥20% of tasks having defects. For benchmarks with partial scoring or unequal task weights, our threshold corresponds to ≥20% of the total possible score affected by defects. For single task benchmarks, our threshold is the reported score having defects that could affect it by ≥20%.
-
For benchmarks with >50 available tasks, we will sample a random set of 50 (stratified by category, when present). If the observed error rate is 15–25%, we will expand the sample to 100. For benchmarks with ≤50 available tasks, all available tasks will be assessed.
About Benchmark Reviews
Epoch AI’s Benchmark Reviews are independent reviews of external AI benchmarks. Our documentation describes the rubric behind this verdict, how we choose which benchmarks to review, and answers frequently asked questions.