TextQuests is a long-horizon, tool-free agent benchmark based on the Infocom suite of interactive fiction games. The benchmark reports a checkpoint-based “game progress” metric that was meant to provide a more representative signal of progress than the games’ built-in scoring system, but 74 out of 327 checkpoints (22.6%) were found to have errors.
Methodology
The leaderboard reports ‘game progress’ (a constructed metric we discuss in more detail below), number of completed games, and a ‘harm score’ both for runs with no clues and runs with the game’s InvisiClues provided. All tasks, scoring code, and per-model harness settings are public. The repo contains a walkthrough for each game that represents a human optimal path. The provided walkthroughs range from 82 to 604 steps (mean 270). We used these walkthroughs to evaluate the game progress metric. We did not assess the harm score, although it has numerous issues1 as it is not the primary metric reported. Using Claude Opus 5, we audited each of the 25 games and the 327 checkpoints by replaying the walkthrough and looking for checkpoints that were impossible or false positives/negatives. Then we validated all errors via human review and localized step-by-step walkthroughs.
Each checkpoint consists of a name, a long-form description of what it assesses, the output string it fires on, and a completion percentage representing the benchmark creator’s judgement of progress towards the necessary objectives, such as puzzles and game milestones, on the path to finishing the game. The ‘game progress’ metric is constructed by taking the maximum completion percentage associated with any checkpoint achieved.
The game progress metric is flawed in two critical ways:
- It assigns the maximum percentage score associated with any checkpoint completed, regardless of whether all prior checkpoints were completed. The median checkpoint reflects an 8% increase in score, but score increases range from 2% to 20%. Game play can solve checkpoints in different orders, and even the provided optimal walkthroughs do not always achieve checkpoints in order of increasing scores. For example, in the provided enchanter walkthrough, progress jumps from 25% to 45% in step 98 then from 45% to 70% in step 115, with the checkpoints for 35%, 40%, 55%, and 60% all triggering later without a corresponding change in game progress. Using the walkthroughs, we identified 35 of these inert checkpoints across 17 games.
- The game progress metric relies on string matching rather than relying on the actual state. This results in many errors, particularly those related to object retrieval, which is a common milestone. Infocom’s parser prints different text depending on how many objects are taken. Any input of the form “get X” returns “Taken” so game progress is updated at a prior step that has a unique response. Thus, in game play, a checkpoint meant to assess whether an object was acquired might trigger without the object ever being acquired. The majority of these match on the prior action2, and it is reasonable to assume that an agent’s next action would be to acquire the object. However, sometimes it takes multiple steps to acquire the object, and we mark these checkpoints as false positives. Other checkpoints use the multi-object form. Any input of the form “get X and Y” returns “X: Taken. / Y: Taken.” These are often underspecified and prone to false positives and false negatives. For example, the Zork2 5% checkpoint is supposed to assess acquiring both the lamp and sword, but its multi-object request only verifies that the lamp was acquired by looking for the string “lamp: Taken”:
- get sword → Taken.
get lamp → Taken. (progress 0%) [correct answer marked incorrect] - get lamp and tunnel → lamp: Taken. / tunnel: You must specify a direction to go. (progress 5%) [incorrect answer marked correct]
- get lamp and sword OR get all → lamp: Taken. / elvish sword: Taken. (progress 5%)
- get sword → Taken.
Representative Errors
Additional task-based errors fall into the following categories:
Impossible
- Each game is capped at 500 steps. Trinity’s walkthrough is 604 steps; limiting the walkthrough to 500 steps achieves a score of 71%. While it is possible to delete steps from the walkthrough and achieve a perfect score in <500 steps, our shortened walkthrough removed vital information-gathering steps, making this solution infeasible3. It remains unvalidated whether a shorter solution that does not rely on hindsight is possible.
- At the start of each game, the agent is able to open feelies_text.txt4 and, if allowed clues, invisiclues.txt. Sometimes games require additional files in the scans folder and manual.pdf, which are not provided, leading to impossible tasks.
- Cutthroats’ coordinates exist only on wrecks-map.jpeg.
- Sherlock is missing the feelies_text.txt file completely.
Incorrect answers marked correct
- Trinity’s 100% checkpoint matches “A shower of sparks erupts from the enclosure,” which is printed for cutting both the correct and the incorrect wire.
- In starcross, placing the red rod into slot 1, 2, or 3 results in the 60% checkpoint due to ambiguous output string matching. However, the checkpoint is meant to check that the rod was placed in the 2nd slot (the oxygen slot), not the methane or ammonia slot, which are fatal.
Correct answers marked wrong
- Sherlock’s 96% checkpoint, titled “Crown Jewels Recovered – Moriarty Subdued” fails on “get jewels” but succeeds on “get all”.
Checkpoint errors by game
| Game | Checkpoints (errors / total) | Errors by type (impossible - correct marked incorrect - incorrect marked correct - inert)5 | Corresponding game progress checkpoints (labeled by % associated with the checkpoint) |
|---|---|---|---|
| ballyhoo | 5 / 20 | 0 - 1 - 0 - 5 | correct marked wrong: 20%; inert: 20%, 45%, 50%, 55%, 70% |
| borderzone | 2 / 14 | 0 - 0 - 2 - 0 | incorrect marked correct: 35%, 85% |
| cutthroats | 5 / 11 | 3 - 1 - 2 - 1 | impossible: 80%6, 90%, 100%; correct marked wrong: 90%; incorrect marked correct: 30%, 90%; inert: 60% |
| deadline | 3 / 13 | 0 - 0 - 2 - 1 | incorrect marked correct: 78%, 92%; inert: 23% |
| enchanter | 5 / 15 | 0 - 1 - 2 - 4 | correct marked wrong: 60%; incorrect marked correct: 60%, 95%; inert: 35%, 40%, 55%, 60% |
| hitchhiker | 1 / 13 | 0 - 1 - 0 - 0 | correct marked wrong: 80% |
| hollywoodhijinx | 1 / 12 | 0 - 0 - 1 - 0 | incorrect marked correct: 22% |
| infidel | 1 / 10 | 1 - 0 - 0 - 0 | impossible: 20% (soft)7 |
| lurkinghorror | 8 / 19 | 3 - 1 - 2 - 6 | impossible: 5%, 60%, 100%; correct marked wrong: 70%; incorrect marked correct: 70%, 75%; inert: 5%, 35%, 40%, 50%, 70%, 75% |
| moonmist | 5 / 10 | 3 - 0 - 0 - 3 | impossible: 35%, 45%, 90%8; inert: 45%, 65%, 75% |
| planetfall | 1 / 13 | 0 - 0 - 0 - 1 | inert: 20% |
| plunderedhearts | 2 / 12 | 1 - 0 - 0 - 1 | impossible: 40% (soft)9; inert: 16% |
| seastalker | 2 / 13 | 0 - 0 - 2 - 0 | incorrect marked correct: 15%, 50% |
| sherlock | 8 / 12 | 7 - 1 - 0 - 1 | impossible: 55%, 65%, 70%, 80%, 90%, 96%, 100%; correct marked wrong: 96%; inert: 45% |
| sorcerer | 2 / 17 | 0 - 0 - 0 - 2 | inert: 45%, 50% |
| spellbreaker | 2 / 11 | 0 - 1 - 1 - 1 | correct marked wrong: 45%; incorrect marked correct: 95%; inert: 45% |
| starcross | 3 / 16 | 0 - 0 - 2 - 1 | incorrect marked correct: 60%, 75%; inert: 70% |
| stationfall | 1 / 12 | 0 - 0 - 0 - 1 | inert: 40% |
| suspect | 3 / 10 | 0 - 2 - 1 - 1 | correct marked wrong: 35% (live-path), 60% (live-path)10; incorrect marked correct: 85%; inert: 60% |
| trinity | 6 / 13 | 4 - 0 - 3 - 0 | impossible: 82%, 92%, 95%, 100%; incorrect marked correct: 42%, 71%, 100% The game walkthrough requires 604 steps, but play is capped at 500. |
| wishbringer | 1 / 13 | 0 - 0 - 0 - 1 | inert: 30% |
| witness | 0 / 11 | 0 - 0 - 0 - 0 | |
| zork1 | 3 / 13 | 0 - 1 - 0 - 3 | correct marked wrong: 30%; inert: 30%, 40%, 50% |
| zork2 | 3 / 14 | 0 - 2 - 2 - 2 | correct marked wrong: 5%, 30%; incorrect marked correct: 5%, 40%; inert: 30%, 40% |
| zork3 | 1 / 10 | 0 - 0 - 1 - 0 | incorrect marked correct: 55% |
| TOTAL | 74 / 327 (22.6%) | 22 - 12 - 23 - 35 | 92 findings on 74 distinct checkpoints |
Rubric
1. Reviewability
| Level | Meaning | Status |
|---|---|---|
| Full | All tasks and scoring logic inspectable*, and harness/API settings used for each model (reasoning effort, token/time limits, tool access, system prompts) are fully disclosed | [proceed to 2] |
| Partial | A representative sample of tasks and scoring logic inspectable, with full harness/API settings disclosed | [proceed to 2] |
| Inadequate | Limited, biased, or no inspectable tasks or scoring logic, harness/API settings undisclosed | [stop → NEI] |
* Either publicly or privately to reviewers.
2. Scoring
This is the minimum standard required to be Verified; failing any of these items results in a Flawed verdict.
| Scoring | |
|---|---|
| Examples |
|
| Default threshold for Flawed | ≥20% of inspected sample† contains errors or there is an issue that corrupts grading at scale |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Benchmark Consistency | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Leaderboard has incomparable results from different versions |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Elicitation | |
|---|---|
| Examples | Model elicitation is extremely constraining and is not the focus of the benchmark:
|
| Default threshold for Flawed | Substantially reduced performance compared to reasonable alternatives for the tasks |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | In 1 of the 25 games, the 500 step limit is below the 604 step reference solution (Trinity). All other reference solutions are <500 steps. |
| Bias in Evaluation Setup | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Material model-specific advantage found |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
† For benchmarks with >50 available tasks, we will sample a random set of 50 (stratified by category, when present). If the observed error rate is 15–25%, we will expand the sample to 100. For benchmarks with ≤50 available tasks, all available tasks will be assessed.
3. Evaluation Quality
This is the standard we would like all benchmarks to meet, but it is not necessarily disqualifying to omit or fail these items.
| Question | Status | Notes |
|---|---|---|
| Elicitation and resource adequacy: Are the resources given to models (reasoning token/turn budget, tool access, etc.) sufficient for them to perform near their ceiling? | SufficientConstrainingUnreasonably constrainingUnknownNot reviewed | 500 vs. 800 ablation step results in paper show performance is still improving. Budget insufficient on 1 game (Trinity). Best model: 2/25 completions without clues, 13/25 with clues. |
| Scaffold fairness: What scaffold does the leaderboard report? | Shared common scaffoldMix of model-specific and common scaffoldsModel-specific scaffoldsNot reviewed | — |
| Is there evidence/risk of contamination? | As of Aug. 10, 2026: 100% of tasks public, 100% of solutions publicNot reviewed | — |
| Has human completability been assessed? | All tasksRepresentative set of tasksPoor implementation (Unrepresentative set of tasks, unreasonable set of participants)Not establishedNot reviewed | — |
| Score range (if possible to estimate) | — floor, — ceilingNot reviewed | — |
| Statistical adequacy: how many runs/model (≥ 5 recommended for error bars) | 1 runs/modelUnknownNot reviewed | — |
| Construct Validity | Measures stated capabilitiesPartially measures stated capabilitiesDoes not measure stated capabilitiesNot reviewed | — |
Disclaimer
This is a partial review. We stopped inspecting once we discovered errors that exceeded our published threshold for a flawed benchmark.
We attempted to reach out to the benchmark developer prior to launch, and if they write a response to this review we will link it here. If you’re the benchmark developer, please email us at reviews@epoch.ai with a link to your response if you would like us to include it here.
-
Harm is a sum over the steps taken so it is confounded with progress. A model that does poorly does not have as much opportunity for harm. To be interpretable, the harm score should be normalized. However, only the “Mean Harm” not the “Relative Harm” is reported. The harm score comes from the underlying Jiminy Cricket repo and TextQuests only reports the “bad”/”others” category, not the “bad”/”self”, “good”/”others”, or “good”/”self” categories.
-
ballyhoo 35/90 · hitchhiker 70 · hollywoodhijinx 12/30/40/60/68/78/86/93 · infidel 45 · starcross 30 · stationfall 30/40/70 · trinity 82 · wishbringer 50/60 · zork1 60 · zork2 58/75 · zork3 20
Note: These checkpoints are not included in the table below (unless other issues are present). -
For example, “examine coin” reveals “It’s standard British currency, worth fifty pence” which is not necessary information, but “examine sundial” reveals “inscribed with seven curious symbols and a compass rose”, which is needed for the following step, “turn ring to sixth symbol”. Without hindsight, it’s not possible to remove only the examine steps that do not reveal vital gameplay information. We have not yet found a solution under 500 steps that is possible without invisiclues.txt.
-
Feelies represent the physical items that came in the box for each of these games. These are vital to play the game correctly, and originally made it so pirated versions were unwinnable. Text ends up in feelies_text.txt, but scans of images are not seen, leading to impossible tasks.
-
The four numbers count the game’s checkpoint errors by type, in that order. Impossible: the checkpoint cannot be triggered, e.g. because a required game file is missing or the walkthrough exceeds the 500-step cap. Correct marked incorrect: the correct action does not trigger the checkpoint. Incorrect marked correct: the checkpoint triggers although its milestone has not been achieved. Inert: triggering this checkpoint in the optimal walkthrough does not change the score since a higher checkpoint was already achieved.
-
No access to wrecks-map.jpeg, which is needed for checkpoints 80/90/100.
-
The pyramid’s location is only marked on a map in the game package (which is not included), but brute force can get to the location. The invisiclues also give the exact route to find it.
-
If a color other than green is chosen, checkpoints 35/45/90 cannot be triggered.
-
The banknote is not accessible, but the pose can be reconstructed in game via brute force. Invisiclues:
“C. Jean Lafond is depicted in the portrait in the Library in exactly the same pose as he is on the banknote. Examine the banknote that came in your game package for clues.
D. Notice what Lafond is carrying.
E. Notice what Lafond is wearing.
F. Notice what Lafond is doing. This must be done last. Voila!” -
This defect shows up in the live scorer, not the leaderboard (which recomputes based on cleaned text). This results in game_finished() not working as expected, wasting budget and potentially increasing the published harm score.
About Benchmark Reviews
Epoch AI’s Benchmark Reviews are independent reviews of external AI benchmarks. Our documentation describes the rubric behind this verdict, how we choose which benchmarks to review, and answers frequently asked questions.