EBR-bench evaluates whether AI systems can improve at an unfamiliar, complex task through repeated attempts and note-taking — a proxy for the broader ability to “learn on the job.” Models repeatedly play Earthborne Rangers, a campaign-style board game in which a player explores a wilderness landscape, overcoming obstacles and pursuing objectives.
We chose this game because it is relatively obscure, so models are unlikely to have memorized it during training; it is almost entirely card-based with very little spatial reasoning, so weak multimodal reasoning is not the bottleneck; and it requires a layered mix of strategy and tactics — deck-building plus turn-by-turn play — over a long time horizon. A single playthrough of the segment we evaluate takes an experienced human 2–4 hours, and mastering the game can require dozens of playthroughs.
In our early experiments, we saw little evidence of learning from experience: scores stay roughly flat over repeated play, well below an expert human baseline. For the full launch analysis, see our publication, AI doesn’t get better at this board game with practice.
Epoch has used Earthborne Rangers content solely for the purposes of research and commentary. This work is conducted independently of Earthborne Games. Epoch does not distribute or sell access to the underlying copyrighted material from the game.
The explorer contains nine charts of two kinds.
The first kind are our standard benchmarking hub charts (“Score vs. Release Date”, “Score vs. Training Compute (FLOP)”, “Score vs. ECI”, and “Leaderboard”), which rank each model based on their topline score on default settings. Default settings consist of ten playthroughs in a sample, with each sample consisting of eight learning playthroughs and two scored playthroughs; the topline score for each model is the better of their two scored playthroughs. Start here to see how competent each model is at EBR-bench, and how that has changed over time.
The second kind are our EBR-specific learning experiments: “Learning trajectories”, “Topline scores”, “Learning gain”, “Strategy guide effect”, and “Deck diversity”. These charts help analyze whether LLM agents can improve their tactics and strategy over time through repeated play and note-taking. They cover a smaller set of models across a wider range of experimental conditions, such as eighteen-playthrough samples rather than ten-playthrough samples. We plan to expand the model coverage in these charts over time.
The guidance below applies to the five learning experiments.
To examine agents’ tactical play quality, select the “Fatigue / turn” or “Injuries / turn” metric under various charts. These are in-game metrics representing harmful consequences from poor tactical decision-making; as such, a skilled player is expected to minimize these values, and a learning player should be able to reduce them across playthroughs.
To examine agents’ strategic play quality, navigate to charts showing “Total travels” or “Deck diversity”. Travels are an in-game metric measuring the distance covered by the player throughout their playthrough; more skilled players aiming for more objectives will tend to travel more, and a learning player should be able to increase their amount of travels over time. Deck diversity measures the distribution of deck types a player explores: a skilled player determined to find the best possible deck will tend to explore different deck archetypes across their learning playthroughs.
Finally, to examine whether agents learn more given more time to explore and experiment, look for comparisons between “10 playthrough” and “18 playthrough” results. Agents are always scored on the better of their final two playthroughs, but “10 playthrough” results give them eight unscored playthroughs used for learning and “18 playthrough” results give them sixteen unscored playthroughs used for learning. A learning player should score substantially higher, and have improved fatigue/injuries/travel metrics, given twice as many unscored learning playthroughs.
Cross-model comparisons
These four views report each model’s topline score as a percentage of the 21 available objectives, for every model we have evaluated, at the default 10-playthrough setting.
Learning experiments
The five charts below also include a toggle to switch between data from lower compaction threshold settings (~250k tokens) and higher compaction settings (880k-950k tokens). Broadly, GPT-series models show a performance penalty on higher compaction settings.
We primarily report each model’s “topline score”, their top score over the final 2 playthroughs of a run. Each model’s topline score is sampled 10 times. Agents are informed in their system prompt that only their final 2 playthroughs are scored, and that the other playthroughs are free to be used for exploration and learning. By default, a sample or “run” consists of 10 playthroughs. Agents are encouraged to take notes as their only means of retaining specific information across compactions.
Each model is also run at two compaction threshold settings (the amount of input tokens beyond which the current context is summarized and compressed): 250,000 tokens, and a higher value around 90% of the model’s reported maximum context window (between 880,000 and 950,000 tokens). We then report the better topline score between the two compaction settings as the model’s canonical topline score in the leaderboard.
Each playthrough consists of the first five in-game days of the game’s campaign. Across these days, and within the subset of the full game provided in our digital implementation, players can score up to 21 distinct objectives, consisting of a mix of missions completed, rewards unlocked, and nontrivial in-game “notable events” recorded. The top score from any human so far is 21 out of 21.
The agent’s system prompt contains basic information about the nature of the experiment, the scoring methodology, a tool-calling guide, and a compressed version of the game’s rulebook. The agent’s tools consist of a card lookup utility, an image of the game’s map, and a text-based pathfinding utility for models that struggle with vision. The agent can also read and edit a notes directory, and can read a set of whitelisted reference files, consisting of the game’s full rulebook, its rules glossary, and a digital interface guide that notes differences between the actual game and the provided digital implementation.
Models act through a simple ReAct agent harness. We plan to explore more sophisticated harnesses over time.
Beyond the topline score, we measure learning directly with an “ablated learning score”: alongside the standard 10-playthrough samples, we run matched samples where the agent gets only 2 playthroughs and is prompted to attempt to maximize their score immediately with no learning runway. The difference between the two setups’ mean toplines isolates the contribution of the 8 practice playthroughs, and we report it with a 90% confidence interval.
As mentioned, we currently evaluate only the game’s initial five-day introductory segment; full campaigns span 20–25 days.
Human Baselines
From July 15th to August 15th, 2026, we ran a human baseline experiment to measure the performance of humans with a variety of pre-existing experience levels at playing and learning this version of Earthborne Rangers. A total of 15 participants were recruited from among personal and professional networks of the benchmark creators, and participants were compensated at a rate of $30/hr, with a maximum participation time of 60 hours. A +25% bonus was provided for a topline score of 10 or above (“better than AI”); an additional +25% bonus (for a total of +50%) was provided for a topline score of 15 or above (“expert performance”).
Participants played in conditions as similar to the AI agents as possible. They were informed they had 10 playthroughs to use to learn the game, and their final two playthroughs would be scored; due to real-life considerations and the time-consuming nature of the experiment, participants were also given the option to declare they would end their participation early and pre-register their next two playthroughs (instead of their 9th and 10th) as their scored playthroughs.
Participants played the game through a private website created by Epoch, through a text-based interface built on the same game engine provided to the agents (with some affordances to make the UI more tolerable for humans, such as organizing card information in boxes and color-coding some keywords). To the greatest extent possible, humans and AI were provided with the exact same information about the game, and humans were strongly encouraged not to use the internet to try to help themselves (even though the internet broadly lacks useful information on performing well at this version of the game anyway).
Of the 15 initial participants, 2 dropped out early and have been excluded from the published dataset. We publish aggregate statistics for the 13 remaining participants, along with the data of the top human participant (included in the aggregates), who achieved the maximum score of 21/21 on their 5th playthrough. One exception is in the “learning trajectories” chart, where instead of an aggregate, we publish the trajectories for the median participant by topline score, to show a more coherent and broadly representative learning trajectory as compared to the strange lines that would have resulted from aggregating lines across participants who ended their participation at wildly different points.
We also calculated the “learning gain” of human participants differently than we did for AI agents. Recall that AI agents’ learning scores are the difference between their topline (best-of-2) performance in a 10 or 18-playthrough run and their topline performance in an “ablated” 2-playthrough run where they are given no runway to learn and informed they must score as high as possible immediately. Since this type of ablation is not possible with humans, we instead simply take the difference between humans’ topline score from their terminal scored playthroughs and their initial two playthroughs. We do the same for the fatigue, injury, and travel metrics from playthrough associated with their topline score in each window. We call these “within-run learning scores” for humans, as opposed to “ablated learning scores” for AI.
This may inflate human learning scores, as humans were informed their initial playthroughs would not be scored, making it possible they would treat the playthroughs as exploratory or throwaway and not pursue objectives. However, Earthborne Rangers is generally considered a difficult game to learn, so we expect the initial playthrough scores and metrics would be relatively low anyways even if humans were instructed to try their best immediately. Additionally, based on a review of human gameplay transcripts, we did not observe anyone treating their initial playthroughs as “throwaway”. Participants made an earnest effort on their early playthroughs, and generally did not want to waste playthroughs not learning about any objectives.
Methodology updates
2026-08-19. We added data from our human baseline experiment participants.
We recruited 15 participants who brought a variety of pre-existing experience with gaming in general and Earthborne Rangers specifically, playing under conditions as similar to the agents as possible. See the “Human Baselines” subsection of Methodology above for more details.
2026-08-11. We updated the benchmark to v3 methodology.
We found in further experiments that compaction threshold settings can have a significant impact (on the order of 1 to 3 objectives) on certain models with certain task parameters. As such, we now run every model at two compaction settings: 250,000 tokens, and a higher value around 90% of the model’s reported maximum context window (usually 880,000 for GPT models, since anything higher can cause issues with the compaction endpoint in rare cases, and around 950,000 for other models). We then report the better topline score between the two compaction settings as the model’s canonical topline score in the leaderboard.
Of the five models we’ve actively run experiments on (Claude Opus 4.8, Claude Opus 5, Claude Fable 5, GPT-5.5, and GPT-5.6 Sol), this change in methodology affected the topline scores for each as follows:
2026-07-27. We updated the benchmark to v2 methodology.
We changed the topline scoring from the mean of a sample’s final 20% of playthroughs to the best of its final two playthroughs, matching the setup of our ongoing human baseline studies. Each model is now evaluated with at least ten independent samples rather than one to three, considerably improving confidence intervals. We also introduced the ablated learning score described above.
We increased the context compaction threshold, from 250k tokens to approximately 90% of each model’s full context window. We are still conducting experiments to find a compaction policy that does not under-elicit model performance.
Scores for models not re-evaluated under v2 continue to reflect the v1 methodology; the two are not directly comparable. v1 datapoints can be identified visually by their lack of error bars.
We also changed the “strategy guide” used to maximize elicitation to a more detailed one with a specific deck recommendation and an explicit series of steps to follow in-game for a high score. This more accurately reflects our claim that the strategy guide is an “answer sheet” that reflects the expert human approach to maximizing score.
Human baseline experiments began during this period, so we also updated the agent-facing UI to be closer to the interface presented to the human baseliners. Mainly, agents now have a separate tool call that lets them view the contents of the game’s discard piles, rather than those discard piles’ contents always being in-context.
The agent-facing system prompt also now informs them of the human expert performance of 20/21 objectives.
Have a question? Noticed something wrong? Let us know.
A test of AI systems' learning capabilities, measuring whether their scores improve across repeated playthroughs of the relatively obscure game Earthborne Rangers.