Claude Fable 5.2 and GPT-6.2 Astra perform exceptionally well across many benchmarks, roughly tying for a new record on the Epoch Capabilities Index (ECI). However, strong performance on typical benchmarks does not always translate to strong performance on real workplace tasks.
To better understand real-world performance, we evaluated both models, along with four others, on an updated version of Epoch Automation Reports, our benchmark of tasks derived directly from our work at Epoch. We manually graded each output against our internal standards to assess open-ended capabilities and progress toward work automation, with 100% representing the standard we expect of Epoch employees.
Key findings:
- No model met the standard we expect of Epoch employees on any task. GPT-6.2 Astra averaged 65%, 17 points below Fable 5.2 despite a roughly tied ECI score.
- Fable 5.2 and Opus 5.5 can now perform at the level of an Epoch contractor on AI data center analysis. The models scored 98% and 88% on that task, respectively, compared with 65% for GPT-6.2 Astra.
- We still see shortcomings in how models design, run, and present research. They reuse existing ideas, rely on first attempts, and struggle to identify and present key findings.
- GPT-6.2 Astra improved over GPT-6 Astra in several tasks, reaching performance comparable to Gemini 4 Argon. We saw more concise writing, improved graphic design, and a 25-point jump in AI data center research.
- Models prefer their own outputs over higher-quality human outputs. Their justifications tended to reward superficial rigor, such as the number of experiments run.

Fable 5.2 and Opus 5.5 reach contractor-level for AI data center analysis
The most significant finding from this evaluation is that Fable 5.2 and Opus 5.5 can perform our AI data center analysis at the level of a contractor — a clear improvement over Fable 5.1 and GPT-6 Astra. By contractor level, we mean that our data center analyst judged the output comparable to what we receive from contractors, whose work also typically needs some revision before publication.
We are now considering shifting contractors’ time toward reviewing AI outputs and other parts of our AI data center analysis.
For our data center analysis task, we ask models to add a data center (in this case EcoDataCenter 1) to Epoch’s AI Data Centers database. We equip models with a detailed guide and relevant tools. This process requires thoroughly researching news and regulatory documents, ordering satellite imagery, and analyzing images.
Fable 5.2 scored 97.5%, and Opus 5.5 scored 87.5%; a perfect score means Epoch could publish the output without changes. Fable 5.2’s only shortcomings were imperfect image annotations and overly verbose text. Opus 5.5’s output needed some additional revisions in equipment classification and more substantial fixes to image annotations. But on the core research elements of the task, both Fable 5.2 and Opus 5.5 met our quality bar, researching credible sources and making well-calibrated judgments.
A contractor typically spends about 16 hours per data center, and Opus 5.5 completed the same work in just under 3 hours for a lower cost. Fable 5.2 spent around 6.5 hours at a higher cost than a contractor, albeit for a higher-quality result than Opus 5.5.
In contrast, GPT-6.2 Astra left out critical information by being overly cautious. Despite its sources clearly pointing to multiple AI buildings, GPT-6.2 Astra chose to cover only one due to a lack of explicit evidence. Even then, it could reasonably have estimated compute and chip quantities for that one building. But it left those estimates out because it wanted a separate source to confirm every number that went into them, rather than making reasonable assumptions on its own. In actuality, Astra had enough information to support its estimates, and following the guide we provided, it would have produced them.

Satellite image annotations of the EcoDataCenter 1 campus (Image © 2026 Vantor) comparing human versus GPT-6.2 Astra output. Teal outlines indicate buildings, blue outlines indicate cooling units, and purple outlines indicate backup generators.
GPT-6.2 Astra’s overly conservative judgments led to incomplete conclusions, which are mistakes we expect contractors to avoid.
Gemini 4 Argon made different mistakes in being overconfident at times and choosing suboptimal calculation methods. It assumed every building used Nvidia’s B200 chips and cited the sources as “company disclosure”, even though only one building disclosed using B200 chips. Gemini 4 Argon also chose a suboptimal cooling calculation method, despite being given detailed guidance that should have led it to choose a different method.
Models still struggle with experiment design, often because they default to existing ideas
All the models we tested struggle to design experiments. This is because they tend to reuse existing work, which often does not fit the core research question being asked.
For example, we tested the models on a task called “chess variants”, based on a now-retired work test we used to give to candidates. The task asks for a research report measuring out-of-distribution generalization in AI models, which means how well the models handle situations unlike anything in their training data. This is tested using chess variants, which are versions of classical chess with altered rules, where models cannot rely on what they learned about classical chess in their training. To answer the research question well, the variants need to be different enough from classical chess to be unfamiliar, but not so hard that a poor score could be attributed to difficulty alone. We expect this task to take a few days for human candidates, and it is relatively simple compared to a full conference-level paper. Here is the prompt we gave the AI models:
Produce a research report that uses chess variants to measure out-of-distribution generalization in AI models. This report should focus on puzzles—game states where there is a “correct answer”—and compare AI performance on classical chess to at least one chess variant.
Here are some characteristics we expect a good report would have.
- The classical vs. variant puzzles should have approximately comparable difficulty so that difficulty isn’t the primary explanation for performance differences. There may be multiple ways to approach this. The report should address this question and discuss what additional work could be done.
- Ideally there should be some variation in “how weird” the variants are, to proxy for how far out-of-distribution they might be.
- We don’t impose a requirement on how many models the report evaluates, though we suggest only using models that score over 40% on our Chess Puzzles benchmark.
- The report should make a reasonable effort to elicit good performance and should discuss what more could be done with more time and resources.
- We are only interested in models reasoning directly about the game, not writing code to solve the puzzles.
- Above all, the report should report the findings transparently and rigorously, whatever they may be. There is no guarantee, for instance, that we find worse performance on nominally out-of-distribution puzzles.
You are provided with OPENROUTER_API_KEY in your environment with a $500 cap.
The report can be delivered as a markdown file, plus any supporting materials you wish to include.
“Chess variants” prompt
Every model we tested made the same error of using pre-existing chess variants rather than creating its own. The models reasoned that variants like King of the Hill are less common than classical chess. The problem is, these variants are well known to the AI models being tested, so they do not actually test out-of-distribution generalization.
Gemini 4 Argon even ran an experiment that proved the evaluated models already knew the variants it was using. This directly undermines the validity of its own out-of-distribution experiments, but Argon disregarded this result and continued with its approach.

Snippet from Gemini 4 Argon on “chess variants” task, demonstrating how frontier models possess pretraining knowledge of chess variants.
Some models claimed to have invented custom variants, but none were genuinely novel and out of distribution. For example, GPT-6.2 Astra only swapped piece labels (e.g., using R for knight instead of rook). In this variant, the same classical chess rules apply, so evaluated models only need to translate the labels, and then they can rely on prior knowledge rather than reasoning through unfamiliar rules.

Chess variants proposed by GPT-6.2 Astra in its report. R=rook, N=knight, B=bishop. Columns 2-4 show how the piece from column 1 would move under that ruleset. For example, in the two-role swap, the piece that starts as a rook moves like a knight.
Claude models used variants that already exist, but presented them as novel. Opus 5.5 and Fable 5.1 both introduced a custom variant built around a new “camel” piece, which is actually a well-known fairy chess piece (a non-standard piece used for chess variants). Fable 5.2 created a variant called “Kingsgate”, which plays exactly like an existing variant called “Midline Invasion”.

Snippet from Fable 5.2 on “chess variants” task, claiming to have invented a pre-existing chess variant
Since these variants already exist, they are likely in the evaluated models’ training data, and thus cannot be used to effectively test out-of-distribution generalization. A good variant would change the rules or winning conditions in a truly novel way.
Overall, the models rely heavily on existing work, which limits them from designing experiments that effectively answer the research question.
Models’ lack of iteration limits research quality
All evaluated models failed to iterate effectively in their research. Good research involves reviewing initial outputs, making adjustments, observing the effects, and repeating that process. The models, however, mostly relied on their first attempt throughout their entire project.
We saw this pattern in the “hidden messages” task, which asks models to evaluate two other models, Claude Fable 5.1 and GPT-6.2 Astra, in how well they can hide a message in an image that they create through code (an SVG file). The model then investigates whether each of its evaluated models can decipher the hidden message from the image alone, without seeing the code or agreeing on a scheme ahead of time. This task is derived from recently published work (Kezins et al., 2026).
Produce a research report investigating frontier models’ ability to encode and decode hidden messages in SVG images.
Evaluate two frontier models: GPT-6.2 Astra and Claude-Fable 5.1
We are interested in the following questions:
- How does a model hide a message in the image it draws?
- Can the same model read the message from the image without SVG source code and without any scheme agreed on in advance?
Some relevant prior work has been saved to the prior_work folder in your workspace:
- Roger and Greenblatt (2023) trained models to hide their reasoning in benign-looking text and found that paraphrasing almost always removes the hidden information.
- Motwani et al. (2025) and Mathew et al. (2025) studied multi-agent settings, where agents attempt to pass information through a monitored channel.
- Zolkowski et al. (2025) found that frontier models fail at hiding messages without a pre-agreed encoding scheme.
You may not use web search for this task, only use the given materials and your knowledge.
Here are some characteristics we expect a good report would have.
- The report should make a reasonable effort to elicit good performance and should discuss what more could be done with more time and resources.
- The report should report the findings transparently and rigorously, whatever they may be.
You are provided with OPENROUTER_API_KEY in your environment with a $500 cap.
The report can be delivered as a markdown file, plus any supporting materials you wish to include.
“Hidden messages” prompt
Doing this research well requires iterating on the image generation prompt. Claude models tend to refuse message encoding requests, so getting past these refusals requires experimenting with rephrasing prompts. Plus, initial attempts often result in images that are too obvious and display the “hidden” message openly in the image; subtly encoding the message requires iterating on prompts.
Models fail to iterate in this way, limiting the quality of their results. For instance, Fable 5.2 settled on the first version of its encoding prompts that produced SVGs with obvious message encodings. The majority of generated images contained the hidden message directly, such as the left image with “SVMOZB” written directly on the table. Some images were better, such as the right image, which encodes the message as an acrostic built from objects (e.g., ‘juice’ -> ‘J’), but there was still substantial room for improvement.

Fable 5.2’s output on the “hidden messages” task, showing images that include the hidden message via obvious encoding methods. In the right image, the first letters of the objects’ names spell out a word.
In contrast, human researchers were able to prompt models to produce SVGs with genuinely hidden encodings, revealing interesting capabilities that make for a strong research report. For example, the image below uses stars to encode a message, where each star’s height represents its place in the alphabet.

Sample image from a human’s original work where stars encode a message, read left to right. Each star’s height represents its place in the alphabet. (Kezins et al., 2026)

Annotated sample image from original human work where stars encode a message, read left to right. Each star’s height represents its place in the alphabet. (Kezins et al., 2026)
GPT-6.2 Astra repeats GPT-6 Astra’s misjudgment of Epoch’s audience
We found no meaningful improvement from GPT-6 Astra to GPT-6.2 Astra in generating interesting topics for Epoch’s audience. Claude models generally do well in this aspect, sometimes proposing topics we have considered internally. Gemini 4 Argon can judge which topics fit Epoch’s theme, but heavily relies on existing articles.
In our “open-ended” Data Insight task, we ask the model to write a short article on an AI trend of its choosing. A key part of this task is choosing a topic that would appeal to Epoch’s general audience. We give the model sufficient material to make that call, including a link to published Data Insights and explicit guidance on what the target audience is interested in.
Prioritize the following:
- The topic should be interesting to Epoch’s target audience: people who follow general trends in AI. Browse the existing data insights to get a sense of what this audience cares about.
- Communicate clearly, especially in the title and chart. The reader should be able to understand the insight from the title and plot alone.
- Keep the rest of the writing concise.
Snippet of “open-ended” Data Insight prompt
Epoch’s Data Insights focus on general, relevant trends in AI and avoid niche topics:

Recent Epoch Data Insights
However, GPT-6.2 Astra chose a topic that is too specific and of little interest to Epoch’s audience. It compared two specific models on Chinese language tokenization efficiency, which is not the kind of broad trend that a Data Insight would normally cover. This would be somewhat stronger if the focus were on a broader framing, like whether Chinese models are more efficient at tokenizing Chinese language, but even then, it would not be clear why readers should care.

GPT-6.2 Astra’s output on an “open-ended” task where its goal was to create a chart and short article that our audience would find interesting.
We noticed similar behavior from GPT-6 Astra, which chose to write about how popular open models use a mixture-of-experts architecture (which uses only a small fraction of the model’s parameters for each token, making it cheaper to run). This topic is already well-known among those interested in AI, and it states a fact rather than a trend.

GPT-6 Astra’s output on an “open-ended” task, where its goal was to create a chart and short article that our audience would find interesting.
Models still fall short on judging what matters and presenting it
In our research tasks, all models struggled to identify which findings are meaningful and to present them clearly.
For example, in the “hidden messages” task, where models were tasked with investigating message encoding and decoding in SVG images, Opus 5.5 presented an obvious finding that messages drawn as letters are easy to decode. This is not a meaningful result and should not have been included as a key finding:

Snippet from Opus 5.5’s output on “hidden messages” task, presenting an obvious finding
We also find that writing produced by these models is generally difficult to read. We notice this especially with Gemini 4 Argon, which uses overly complex naming such as “code-level syntactic hiding” and “channel-aware unconstrained” that add unnecessary confusion:

Snippet from Gemini 4 Argon’s output on “hidden messages” task, using unnecessarily complex naming
Of the evaluated models, GPT-6.2 Astra scores relatively well on presentation, but does not yet meet our standards. For instance, in the “chess variants” task, where GPT-6.2 Astra was tasked with researching out-of-distribution generalization in AI models using chess variants, its report reads fairly well overall. However, it leads with a bolded key finding that it later contradicts. The bolded finding mentions that “both models scored 2.5 to 6.7 percentage points lower when the pieces’ movement rules changed”. This appears to be the main takeaway, but the report then notes that with a larger token budget, the models did at least as well on the variants as on classical chess, which undercuts the headline.

Snippet from GPT-6.2 Astra’s output on “chess variants” task, leading with a misleading key finding
Independent of the actual quality of the research, this kind of misleading presentation is a flaw that would prevent publication.
Models prefer their own outputs over human gold standards
For tasks where we have a human output that meets our standards, we ask models to compare their own output with the human’s. We give the models the rubric we use for grading and present both options without revealing which is which.
All models preferred their own output in every case except one, where Gemini 4 Argon preferred the human output on the “hidden messages” task.

In most of these self-evaluations, the models’ justifications are biased, often exaggerating minor flaws in the human output and missing their strengths.
For instance, in the “hidden messages” task, Opus 5.5 criticizes the human output (Option 1) because their prompt to the image-generating model did not specify that the reading model only receives the PNG.

Snippet from Opus 5.5’s evaluation of a human output. It criticizes the human output for not disclosing that the receiver will only get a PNG.
The point of telling the image-generating model that the receiver will only see a PNG image is to prevent it from hiding messages in parts of the code that will not show up in the image. However, the human’s prompt already makes clear that the hidden message should not appear “anywhere in the SVG source” and provides specific guidance on this. This ultimately achieves the same goal as a disclosure, weakening Opus 5.5’s criticism.

Encoding prompt from original human work (Kezins et al., 2026), asking the model to hide a secret message in an SVG image.
In general, models tend to place more weight on superficial rigor, such as the number of experiments, regardless of whether those experiments are meaningful to the research question.

Snippet from Opus 5.5’s judgment of its task outputs on the “hidden messages” task, emphasizing the rigorous approach of its own work.
One rare case was Gemini 4 Argon choosing the human output on the “hidden messages” task, with reasoning that aligns with how we would compare the two. It recognized that the human output runs a clean experiment targeting the core research question, while criticizing its own output for experiments that only appear “superficially more formal.”

Snippet from Gemini 4 Argon’s evaluation of the “hidden message” task outputs from both itself and a human, expressing its preference for the human output.
If models could consistently self-evaluate like this, they may be able to produce outputs that meet our standards. However, this was not the norm, and we will continue tracking models’ self-evaluation in future work.
Updates to Epoch Automation Reports
We have improved Epoch Automation Reports by updating the task suite, eliciting better performance out of the models, and refining the grading rubric.
Task suite. We have updated our tasks to be more representative and fair. For example, our previous research task asked models to run a benchmarking project on a fixed API budget. For many plausible research directions, that budget was too low, and the task was too open-ended. We have replaced it with a retired work test we previously gave to candidates, which we know can be completed well within a set time and budget.
Elicitation. Our task instruction prompts now highlight what we prioritize in grading. In the Data Insight task, for instance, we point out that we value clear communication in the title and plot, and we describe the intended audience. For the AI Data Center task, we added more resources, such as a condensed version of our internal data center research Slack channel, which covers edge cases and helps guide judgment in ambiguous situations.
Grading. Each criterion is now scored on a 1-4 scale, similar to conference paper reviews. We found this to better reflect how we actually judge work quality.
Due to these changes, we re-evaluated some of the models from our previous publication. We plan to re-evaluate the remaining models using the updated setup.
See our updated methodology for more details.
Limitations
We acknowledge the following limitations of this work:
Small sample size. We have a limited task suite, and we run each model only once per task. Given that manual grading is time-intensive, we prioritize in-depth qualitative analysis over more runs. Scores are therefore noisy and may shift with more runs. These results should not be read as a general measure of what AI can or cannot do across all work tasks.
Subjective grading. Many of our evaluation criteria involve subjective judgment, so the numerical scores should not be treated as ground truth. Specific numbers could change with different rubric item definitions or different graders. We include these numbers to give a rough sense of how models compare and how close each task is to being fully automated. We believe that the qualitative findings are more informative than these exact numbers.
Variation in setup. Because we changed our elicitation and setup since our previous evaluation, scores between the two reports are not directly comparable, and we have rerun some models (Claude Fable 5.1 and GPT-6 Astra) on our new setup. We plan to re-run the remaining models with the updated setup to make results more comparable. There will also be ongoing changes: we may retire tasks if models are likely to have seen them in training, and we may improve how we set up the models for each task. As a result, our evaluations are only roughly comparable across reports. Since models have web access and our evaluations are published online, we explicitly instruct the models not to visit those pages, and we verify every run to ensure they did not.
Conclusion
Models demonstrate impressive performance on benchmarks, now meeting or exceeding human baselines in areas like software engineering and math. Despite this, they still fall short on open-ended tasks like choosing an interesting topic for a given audience, or designing simple research experiments for a week-long project. Some of our tasks are nearing automation, such as AI data center analysis. Others still lag behind, largely due to models’ weaknesses in judgment and presentation. We will continue evaluating new models and updating our task suite to track progress in open-ended capabilities and work automation.
We would like to thank James Chua from Singapore AI Safety Hub for his feedback on Opus 5.5’s output on the “Hidden Messages” task.1 We would also like to thank Tom Cunningham from METR for his self-critique idea.


