Overview

EBR-bench evaluates whether AI systems can improve at an unfamiliar, complex task through repeated attempts and note-taking — a proxy for the broader ability to “learn on the job.” Models repeatedly play Earthborne Rangers, a campaign-style board game in which a player explores a wilderness landscape, overcoming obstacles and pursuing objectives.

We chose this game because it is relatively obscure, so models are unlikely to have memorized it during training; it is almost entirely card-based with very little spatial reasoning, so weak multimodal reasoning is not the bottleneck; and it requires a layered mix of strategy and tactics — deck-building plus turn-by-turn play — over a long time horizon. A single playthrough of the segment we evaluate takes an experienced human 2–4 hours, and mastering the game can require dozens of playthroughs.

In our early experiments, we saw little evidence of learning from experience: scores stay roughly flat over repeated play, well below an expert human baseline. For the full launch analysis, see our publication, AI doesn’t get better at this board game with practice.

Epoch has used Earthborne Rangers content solely for the purposes of research and commentary. This work is conducted independently of Earthborne Games. Epoch does not distribute or sell access to the underlying copyrighted material from the game.

Reading this explorer

The explorer contains nine charts of two kinds.

The first kind are our standard benchmarking hub charts (“Score vs. Release Date”, “Score vs. Training Compute (FLOP)”, “Score vs. ECI”, and “Leaderboard”), which rank each model based on their topline score on default settings. Default settings consist of ten playthroughs in a sample, with each sample consisting of eight learning playthroughs and two scored playthroughs; the topline score for each model is the better of their two scored playthroughs. Start here to see how competent each model is at EBR-bench, and how that has changed over time.

The second kind are our EBR-specific learning experiments: “Learning trajectories”, “Topline scores”, “Learning gain”, “Strategy guide effect”, and “Deck diversity”. These charts help analyze whether LLM agents can improve their tactics and strategy over time through repeated play and note-taking. They cover a smaller set of models across a wider range of experimental conditions, such as eighteen-playthrough samples rather than ten-playthrough samples. We plan to expand the model coverage in these charts over time.

The guidance below applies to the five learning experiments.

To examine agents’ tactical play quality, select the “Fatigue / turn” or “Injuries / turn” metric under various charts. These are in-game metrics representing harmful consequences from poor tactical decision-making; as such, a skilled player is expected to minimize these values, and a learning player should be able to reduce them across playthroughs.

To examine agents’ strategic play quality, navigate to charts showing “Total travels” or “Deck diversity”. Travels are an in-game metric measuring the distance covered by the player throughout their playthrough; more skilled players aiming for more objectives will tend to travel more, and a learning player should be able to increase their amount of travels over time. Deck diversity measures the distribution of deck types a player explores: a skilled player determined to find the best possible deck will tend to explore different deck archetypes across their learning playthroughs.

Finally, to examine whether agents learn more given more time to explore and experiment, look for comparisons between “10 playthrough” and “18 playthrough” results. Agents are always scored on the better of their final two playthroughs, but “10 playthrough” results give them eight unscored playthroughs used for learning and “18 playthrough” results give them sixteen unscored playthroughs used for learning. A learning player should score substantially higher, and have improved fatigue/injuries/travel metrics, given twice as many unscored learning playthroughs.

The charts

Cross-model comparisons

These four views report each model’s topline score as a percentage of the 21 available objectives, for every model we have evaluated, at the default 10-playthrough setting.

• Score vs Release Date plots each model’s score against its release date, with the expert human level marked for reference. “Frontier trend only” restricts the view to models that were state of the art on this benchmark when released.
• Score vs Training Compute (FLOP) plots score against the compute used to train each model. Coverage is limited to models with a published training compute estimate.
• Score vs ECI plots score against the Epoch Capabilities Index, together with the score the index would predict. Models above the trend line perform better at Earthborne Rangers than their general capability level implies; models below it perform worse.
• Leaderboard ranks every evaluated model by score, with 90% confidence intervals where a model was run multiple times.

Learning experiments

• Learning trajectories shows how agents’ metrics vary across the playthroughs in their run, on either 10-playthrough or 18-playthrough settings. Their final two scored playthroughs are highlighted. Note that the trends here do not necessarily indicate a strong learning signal, as some agents aggressively skip their learning playthroughs, which depresses their early scores.
• Topline scores shows how agents’ topline scores vary between models, and also between 10-playthrough and 18-playthrough settings. An agent that can usefully exploit additional playthroughs to learn should show higher topline scores on 18-playthrough than 10-playthrough.
• Learning gain shows how agents’ metrics differ between their normal 18-playthrough or 10-playthrough runs and special “ablated” runs, where they are only given 2 playthroughs and informed they must immediately maximize their scores with no unscored playthroughs allocated for learning. In other words, this chart represents the true controlled experiment for whether learning playthroughs help at all. If a model’s learning score has a 90% confidence interval that doesn’t overlap with zero, that’s a statistically significant signal that it actually learns (or gets worse).
• Strategy guide effect shows the effect of giving the agents a human-expert-written strategy guide that details the deck and route used by the human expert to score 20/21 objectives, along with a variety of tactical tips. It compares the scores of agents given this strategy guide and 2 playthroughs with the “ablated” runs described above, which also only have 2 playthroughs to work with. This sets an upper bound for how well an agent could perform given that they write high-quality notes for themselves through learning.
• Deck diversity shows the extent to which models do or don’t explore different types of deck. Each “deck archetype” is defined as a combination of an in-game “Background” card pool, an in-game “Specialty” card pool, and an in-game “Role card” with a special ability. A skilled player would use their learning playthroughs as an opportunity to explore a wider diversity of decks.