MirrorCode is Epoch AI’s benchmark for long-horizon coding, co-developed with METR. In a MirrorCode task, AI models are tasked with reimplementing an entire program end-to-end, without access to the original source code. AI solutions must match the original program’s output exactly on end-to-end tests, including held-out tests. MirrorCode’s 25 target programs span different areas of computing: Unix utilities, data serialization and query tools, bioinformatics, interpreters, static analysis, cryptography, and compression.

The scores shown here use the leaderboard configuration, MirrorCode (ML, +Private, 2L): the 15 target programs from the Medium and Large buckets, each evaluated in two implementation languages (generally Go and Ada), giving 30 tasks. Each task is run three times, with a budget of 10 billion tokens and 7 days per attempt.

For more details, see the MirrorCode page, the paper, or the open-source repository.

In MirrorCode, we test AI agents on their ability to reimplement programs with a command-line interface. Without access to the original program’s source code or the web, a full reimplementation requires devising a structure for the entire program, rather than merely translating the code piece-by-piece. The AI has execute-only access to the original program with arbitrary arguments and can observe its outputs, allowing it to explore the original program’s behavior (a black-box oracle). The AI also has access to documentation describing the program, and example test inputs to further pin down the scope of the reimplementation. The pre-existing program, documentation, and test examples act as a precise specification for how the AI-written software should behave, and the AI is left to determine how to satisfy it.

The AI’s solution is evaluated via end-to-end tests derived from the original program’s test suites, real-world data, and LLM-assisted generation. Each target program has hundreds to thousands of test cases. To pass, the AI’s solution must produce exactly the same output (to the stdout and stderr streams) as the reference program. If all end-to-end tests were exposed, agents could simply create a lookup table instead of genuinely solving MirrorCode tasks; we prevent this by holding out a subset of tests, averaging 34% held-out tests across the benchmark. We use 100% of tests passing as the threshold for considering a MirrorCode task to be solved.

MirrorCode is designed from first principles to render entire classes of cheats ineffective. An AI might try to wrap or otherwise use the existing target program: we prevent this by copying the agent’s code to a separate sandbox during scoring, where the reference binary is absent. It might create a look-up table rather than implementing general-purpose code: hidden test cases mean hard-coded solutions fail, and are never counted as successful reimplementations. It might interfere with the scoring mechanism: we prevent this by separating scoring from the sandboxes used for development, assessing the solution’s logged outputs for string equality against the original program’s outputs. We also tested our environments with a red-teaming evaluation in which the agent is explicitly instructed to circumvent normal evaluation mechanisms to obtain a high score by cheating; no agent successfully cheated using our current scaffold.

We conduct all evaluations using a simple agent scaffold based on the Inspect library’s ReAct agent. This allows shell usage and exposes the text_editor tool for file reading and editing. In addition to the text_editor and submit tools, we give models access to an evaluate_testcases tool, which runs a scoring pass on their code against visible tests, but does not end the session. We use compaction to let trajectories run longer than their maximum supported context. All evaluations are sandboxed within a Docker container, and the agent is not provided with internet access, preventing it from looking up the original program’s source code or using third-party dependencies.

The scores shown here use the leaderboard configuration, MirrorCode (ML, +Private, 2L): we run the 15 target programs from the Medium and Large buckets (including the two private programs in those buckets), and drop the Small bucket. Each target program is evaluated in two implementation languages — one mainstream (generally Go) and one low-resource (Ada) — giving 30 tasks. We run each task three times, with a budget of 10 billion tokens and 7 days per attempt, using high reasoning effort where the API supports it. We report each model’s average rate of 100%-solves across the 15 target programs; error bars show ± standard error. These scores are not directly comparable with the paper, which evaluated all 25 target programs, used all six agent implementation languages on the Small and Medium buckets, and gave attempts a budget of 1 billion tokens except on Large targets, with no time limit.

Because MirrorCode tasks involve reimplementing open-source programs, AI models are likely to have seen the original codebases in pretraining. However, AI successfully reimplemented several target programs that passed our memorization screen, and failed to reimplement programs where the screen showed evidence of memorization. This suggests that the results were not dominated by memorization, but we cannot rule out the possibility that memorization contributes to AI performance. See the paper for details.

We release our scaffold and 22 of the 25 MirrorCode target programs (totaling 132 task instances across the six supported programming languages) as open-source, with the other three targets kept as a private test set.