Back to benchmarks

Chess Puzzles

100 novel puzzles, generated programmatically with a chess engine. Each puzzle has a single best next move.

Chess Puzzles

The benchmark consists of chess positions where models must select the best next move. All positions were generated programmatically by Epoch and do not appear in any other source. Each position has a single best move for the player whose turn it is, as judged by the Stockfish chess engine. Models are graded based on whether they select the correct move.

While the broader significance of chess is low, we find this benchmark useful for several reasons.

  • It plausibly measures important aspects of reasoning, including spatial reasoning and planning. In this sense, it is a “lite” version of more involved video game benchmarks.
  • It is cheap to create and run: samples are generated programmatically and evaluating a model on a sample usually requires only a single API call.
  • It is not saturated: as of publication (2025-12-10), the highest score by a frontier model is less than 40%. What’s more, we can generate more difficult samples as model capabilities advance.

Methodology

Models are given the board state via Forsyth–Edwards Notation (FEN); they are not asked to process an image. You can see model solutions using our log viewer (versions 1.x only so far, version 2 incoming), for instance here are the logs for GPT-5. Several websites, e.g. Chess.com, have tools to load FEN strings, display them as board states, and analyze positions.

Models are given the following prompt, with the token budget filled in (1,000,000 output tokens in our runs).

Please answer the question below. Work through the problem carefully and then submit your final answer using the `submit_answer` tool.

You have a maximum of {budget} output tokens. I will remind you of how many tokens you have remaining.

**Here is the question:**

Below is a FEN string representing the state of a chess game. Analyze the position and determine the best next move for the player whose turn it is.

{FEN}

Models answer by calling a submit_answer tool, described to them as follows: “Submit the best next move as a four-character string stating the starting and ending squares of the piece to be moved, e.g. b1c3. The conversation will end immediately.” The submitted move is compared for an exact match to the answer key.

If a response is cut off at the model’s own output-token limit before it submits, the model is asked to continue. If the token budget runs out first, the model is asked to submit its best move immediately, at its lowest reasoning effort.

Board states are generated by a heuristic search and filtering program. The first six full moves are played randomly. Subsequently, moves are played by Stockfish set to a randomly-drawn Elo rating. After a game concludes, full-powered Stockfish is used to find positions where there is a single best move that the player model did not select. Positions are filtered out if the best move is capturing an undefended piece or a higher-value piece with a lower-value piece as these usually constitute obvious blunders. We found that one desired position was generated roughly every five simulated games.

The implementation of the puzzle generator can be found here. The dataset was generated in a single run, with “chaos” mode enabled. The Inspect task definition can be found in our chess-puzzles-mirror repository.

Our subjective assessment of the positions is that they range in difficulty from somewhat straightforward to somewhat difficult. We show one example of each below.

Chess puzzle position 1 Chess puzzle position 2

Two positions featured in the benchmark. White’s only good move in the position on the left is relatively straightforward, even if the reasons it is the only good move are less clear. White’s only good move in the position on the right is far less obvious.

Human baseline

To put model scores in context, we evaluated several human players on a sample of puzzles from the benchmark.

Five players each answered a sample of 20 randomly chosen puzzles. Players were asked to work alone, without use of AI or a chess engine. Each player was asked to identify the single best move in each position and to record the time taken to answer.

Chess ability varied across players, with US Chess Federation (USCF) ratings ranging from 1700 to 2400 (one player instead reported a rating of 2000 from a popular online chess website). One player was accidentally shown a chess engine’s solution to one puzzle; that puzzle was excluded when calculating their score.

Results

The two highest-scoring players both answered 75% of puzzles correctly. This shows that these puzzles are very challenging for humans, at least when spending on the order of minutes per puzzle. The sampled puzzles are also harder than the full puzzle set on average: the mean AI score is 5.4 percentage points lower on the sample than on the full set, as of October 16, 2026.

Anonymized results for both human players and models are available in this spreadsheet (its model results are from version 1 of the benchmark).

PlayerRatingCorrectCorrect (%)Time per puzzle, mean (mins)
A24001575%2.0
B20351575%Not reported
C22651050%3.2
D17009147%6.5
E20002840%6.8

Changelog

2026-10-14. We released version 2 of the benchmark and re-ran every model we could. Scores are broadly similar but not exactly comparable with version 1, so version 1 scores are no longer shown. The changes are described in the task’s changelog.
Notes

  1. Out of 19 puzzles; one puzzle was excluded after this player was accidentally shown an engine solution. Return

  2. This player provided an online (Chess.com) rating rather than an over-the-board rating. Return

External review

Generality Labs, an independent benchmark auditing organization, audited Chess Puzzles in October 2026.

Epoch’s response

Generality Labs reviewed Chess Puzzles and gave it a Verified verdict. We appreciate the thoughtful analysis and the errors they pointed out. Though we had already identified some of these issues internally, others were new to us. Version 2.0.0 fixes the major issues.

Version 2.0.0 uses qa_agent, which is a scaffold that nudges the model to continue if it stops before submitting a move, tells it how many tokens it has left, and gives it a tool to submit its move. This fixes the following errors:

  • Note from the audit: “[Models] are instructed to output their best move in a specific format, with the answer extracted by an LLM judge and compared to the gold answer. We identified a negligible number of issues with the judge extracting the model submitted answer.”
    • Our response: Scoring no longer uses an LLM judge. It is now deterministic exact-match: the model submits its answer, the answer is processed to normalize case, punctuation, and whitespace, then the answer is compared to the answer key.
  • Note from the audit: “Several Claude configurations frequently exhaust their token budgets without submitting an answer. We recommend reporting token-limit and content-filter failures separately from incorrect answers, and ensuring such failures are appropriately and robustly handled.”
    • Our response: With qa_agent, we no longer see token-limit failures as frequently. When a model’s response gets cut off by its built-in output-token limit (128k for most current models), qa_agent now prompts the model to continue rather than scoring the response as no answer. If a model uses up its per-puzzle token budget before submitting a move, qa_agent now asks it for an immediate answer at its lowest reasoning effort. This is recorded in the sample metadata as “forced_final_answer”. As of October 16, 2026, the 14,800 samples from version 2.0.0 runs had only 25 content-filter failures (0.17%) and the 400 samples from version 2.0.1 runs had no content-filter failures (0%).

We have also open-sourced the benchmark and added a changelog to address the following feedback: “Differences between benchmark versions are not publicly documented. Although some evaluation settings can be determined from logs, token budgets and judge models are found to vary within a single recorded version.” Separately, due to insufficient recordkeeping, we could not verify which Stockfish version we used to create the puzzles.

Citations

Citation

Epoch AI, 'Chess Puzzles'. Published online at epoch.ai. Retrieved from 'https://epoch.ai/benchmarks/chess-puzzles' [online resource]. Accessed 26 Oct 2026.

BibTeX Citation

@misc{EpochChessPuzzles2025, title = {{Chess Puzzles}}, author = {{Epoch AI}}, year = {2025}, month = {12}, url = {https://epoch.ai/benchmarks/chess-puzzles}, note = {Accessed: 26 Oct 2026} }