The Factorio Learning Environment (FLE) evaluates large language models as agents in the construction and management simulation game, Factorio. It specifically tests agents’ ability to design and execute complex plans, maintain long-term coherence, write accurate code and reason spatially within an open-ended, dynamic environment.

Factorio is a game where players crash-land on an alien planet and must build automated factories from the ground up. Starting with basic resource gathering and manual crafting, players design intricate production lines to automate the creation of increasingly complex items, often culminating in launching a rocket into space.

The Factorio Learning Environment includes two main settings:

• Lab-play: 24 structured tasks requiring agents to build specific factories under resource constraints. Performance in lab-play is measured by the fraction of lab tasks completed by the model.
• Open-play: An unbounded task where agents aim to build the largest possible factory on a procedurally generated map. Performance is measured by the production score, a metric of economic activity that aggregates the value of all the items produced so far.

FLE was introduced by Hopkins et al. in Factorio Learning Environment (2025).

We source Factorio Learning Environment (FLE) evaluation results directly from the official leaderboard, maintained by the benchmark creators.

The benchmark evaluates LLMs acting as agents within the Factorio game. Agents interact with the game by generating Python code using a provided API within a Read-Eval-Print Loop (REPL). Each interaction cycle (code generation, execution, observation) constitutes one step.

For the lab-play setting, agents are tasked to build fully automatic production lines for 24 distinct target entities of increasing complexity, such as “Electronic circuit”, “Engine unit” or “Sulfuric acid”. Agents start with an inventory containing enough items to complete the task. Each task involves a trajectory of 128 API calls. The task counts as completed if the agent built a production line for the target entitity that achieves a target throughput during a 60-second period. The benchmark authors manually verified successful production lines to check that the agents did not cheat. The reported metric is the mean success rate across 8 runs per task.

For the open-play setting, agents operate in a world with unbounded space and resources, and are tasked to “build the largest factory possible.” The task stops after 5000 steps. The reported metric is the median production score across 8 independent runs.

For detailed methodology, including the specific API provided to agents, scoring calculations, environment setup, and agent scaffolding used for the leaderboard results, please refer to the original Factorio Learning Environment paper (arXiv:2503.09617) and code in the Factorio Learning Environment GitHub repository.