The benchmark consists of puzzles based on a well-known game, where models must select the single best next move in the game. It is similar in setup to our Chess Puzzles benchmark, except that we keep the game’s identity secret to reduce the risk of benchmark-specific preparation.

This benchmark is highly analogous to our Chess Puzzles benchmark. Puzzles were generated programmatically by sampling random games, rolling forward to mid-game states, and filtering to states where there is a single best next move.

Models are given the game state as text, in a minimal agent scaffold, and conclude by submitting their chosen move through a submission tool. Submissions are normalized into the game’s standard notation and compared to the answer key. Responses from which no valid move can be extracted receive no credit and are recorded as unanswered; responses that name an illegal move are marked incorrect.

To keep the game’s identity undisclosed, we are not currently publishing the prompt, example positions, or model transcripts.