EBR-bench evaluates whether AI systems can improve at an unfamiliar, complex task through repeated attempts and note-taking — a proxy for the broader ability to “learn on the job.” Models repeatedly play Earthborne Rangers, a campaign-style board game in which a player explores a wilderness landscape, overcoming obstacles and pursuing objectives.

We chose this game because it is relatively obscure, so models are unlikely to have memorized it during training; it is almost entirely card-based with very little spatial reasoning, so weak multimodal reasoning is not the bottleneck; and it requires a layered mix of strategy and tactics — deck-building plus turn-by-turn play — over a long time horizon. A single playthrough of the segment we evaluate takes an experienced human 2–4 hours, and mastering the game can require dozens of playthroughs.

In our early experiments, we saw little evidence of learning from experience: scores stay roughly flat over repeated play, well below an expert human baseline. For the full launch analysis, see our publication, AI doesn’t get better at this board game with practice.

Epoch has used Earthborne Rangers content solely for the purposes of research and commentary. This work is conducted independently of Earthborne Games. Epoch does not distribute or sell access to the underlying copyrighted material from the game.

We primarily report each model’s “topline score”, the mean score over the final 20% of a run’s playthroughs. Agents are informed in their system prompt that only their final 20% of playthroughs are scored, and that the other 80% of playthroughs are free to be used for exploration and learning. By default, a sample or “run” consists of 10 playthroughs. Agents are encouraged to take notes as their only means of retaining specific information across compactions.

Each playthrough consists of the first five in-game days of the game’s campaign. Across these days, and within the subset of the full game provided in our digital implementation, players can score up to 21 distinct objectives, consisting of a mix of missions completed, rewards unlocked, and nontrivial in-game “notable events” recorded. The top score from any human so far is 20 out of 21.

The agent’s system prompt contains basic information about the nature of the experiment, the scoring methodology, a tool-calling guide, and a compressed version of the game’s rulebook. The agent’s tools consist of a card lookup utility, an image of the game’s map, and a text-based pathfinding utility for models that struggle with vision. The agent can also read and edit a notes directory, and can read a set of whitelisted reference files, consisting of the game’s full rulebook, its rules glossary, and a digital interface guide that notes differences between the actual game and the provided digital implementation.

Models act through a simple ReAct agent harness. We plan to explore more sophisticated harnesses over time. We have most models compact every 250k tokens, as we found larger compaction thresholds made little difference in performance; older models may compact more frequently if their maximum context window is smaller than 250k tokens.

As mentioned, we currently evaluate only the game’s initial five-day introductory segment; full campaigns span 20–25 days. We plan to expand coverage and run more statistically robust human baselines over time.