Back to benchmarks

Epoch Automation Reports

By Epoch
Nov. 17, 2026

A benchmark of realistic, open-ended tasks drawn from Epoch AI’s own work. Models produce work outputs, which human graders evaluate against internal quality standards.

Agent
Epoch administered

Rubric scores by system and task category

Mean rubric scores across each task category. A score of 100% means the output meets the internal standard we’d expect from an Epoch employee. Hover over a cell to see individual task scores.

System
Graphic design
Data insight generation
AI data center research
Research
Average
Claude Fable 5.2Claude Code
82%
Claude Opus 5.5Claude Code
78%
Gemini 4 ArgonAntigravity
67%
GPT-6.2 AstraCodex
65%
Claude Fable 5.1Claude Code
64%
GPT-6 AstraCodex
56%

Explore the rubric

Pick a system to see how it did on each task

Claude Fable 5.2

Claude Code

Model performance chart

Create an Epoch-style graphic to illustrate how model progress may be understated with constant token budget elicitation relative to max elicitation.

Score
62.5%
Time
3h 57m
Cost
$375.73
Tokens
257.1M
Final output
Claude Fable 5.2's final output for Model performance chart
This is an AI generated output, and should not be cited or shared as Epoch's work.
Full prompt

I’m working on a gradient updates article.

Create a diagram to illustrate this:

Models have been improving over time in both performance and a fixed token budget but also their ability to productively use large token budgets. This means that constant-elicitation understates recent progress relative to max elicitation.

Here is the Figma file we use for graphics generation (treat this as read-only):
[link to Figma file]

You can access only two things: the Gradient Updates section of Epoch’s site and the Figma file above. “Gradient Updates” means https://epoch.ai/gradient-updates and any page whose URL starts with https://epoch.ai/gradient-updates/. Everything else is off-limits, including other epoch.ai pages like https://epoch.ai/publications, all other websites, and web search.

Follow these guidelines when creating your diagram:

  • Effectively communicate the main point in the simplest way possible. It is not necessary to capture every single detail, since the description will be presented to the reader alongside your generated diagram. Could someone glancing at this for five seconds tell you the main point?
  • Every element (box, arrow, etc.) should earn its place. If you can remove something while preserving the main point, remove it.
  • Color selection should encode meaning. Do not simply use colors for styling purposes.
  • Follow graphic design principles (visual balance, spacing, alignment, etc.).
  • Match Epoch’s house style based on the provided Figma file.

Your final output should be a diagram.

Rubric
Communication
60% weighted
5/8
  • Helps communicate the main pointPartially met
  • Intentional design choicesMostly met
General design knowledge
20% weighted
2/4
  • Follows basic design principlesPartially met
Epoch-specific rules
20% weighted
3/4
  • Fits Epoch’s house styleMostly met

Epoch Automation Reports

Epoch Automation Reports is Epoch AI’s benchmark of realistic, open-ended tasks drawn from our own work. For each task, we give a model the relevant context and tools and ask it to produce an output. A human grader then evaluates the output using rubrics that reflect our standards. Tasks span a diverse range, from graphic design to AI data center research, and we plan to add more over time.

Most existing benchmarks focus on well-defined tasks with easily verifiable outputs. In contrast, real work often calls for creativity and judgment in ambiguous situations. Existing benchmarks do not fully capture these open-ended aspects, creating a gap between benchmark scores and real-world performance.

We aim to combine the systematic approach of benchmarks with the holistic impressions that come from testing models on real work. We hope this sheds light on what easily verifiable benchmarks might miss and informs broader questions about AI adoption. At the minimum, it should inform our own adoption: if models do very well, we can automate these tasks within Epoch.

For more details, see our report, Can AI automate Epoch?.

Methodology

Overview

We build our task suite based on the work people actually do within Epoch AI. We run each model on its corresponding agent harness at its highest reasoning setting and provide task-specific resources. As a general principle, we give models as much context as we would give a human contractor. We also disable memory and skill writing so what a model learns on one task does not carry over to future tasks. A human grader then scores each output against rubrics that reflect our internal quality standards.

Our task suite changes over time as we add new tasks. We may also retire tasks that recent models are likely to have seen during training or that we judge to be less representative of our work. As a result, a model’s aggregate score may change over time as tasks are added or retired, or as it is re-evaluated with updated rubrics. The methodology described here reflects our most up-to-date tasks and evaluations.

We also adjust category scores based on task difficulty. Within each category, we average task scores at each difficulty level, then combine these averages using relative weights of 2:3:2 for easy, average, and difficult tasks. If a category does not include tasks at a specific difficulty level, we exclude that level and proportionally redistribute its weight across the remaining levels.

Graphic design

Task. Create an Epoch-style graphic that illustrates a given description.

Purpose. This task tests how well the model learns patterns from provided resources and communicates ideas clearly.

Provided tools/resources.

  • Figma Remote MCP server where supported (e.g., Claude Code), and Figma Desktop MCP server otherwise (e.g., Qwen Code).
  • A link to a Figma file with Epoch graphics from June-August 2026, along with Epoch’s color palette, design guidelines, templates, and core components.
  • A link to Epoch’s Gradient Updates page, which includes graphics that can be used as further reference.

Evaluation. Grading prioritizes communication and general design principles. Communication covers how well the graphic illustrates the description as simply as possible. A high-quality graphic should not simply restate the description; it should help viewers understand the message better than the text alone. General design principles, such as visual balance, spacing, and alignment, serve as the baseline for an effective graphic.

Data insight generation

Task. Create an Epoch-style Data Insight based on a provided direction.

Data Insights are short articles regularly published by Epoch AI, which break down complex AI trends into focused snapshots.

Purpose. This task tests how well the model identifies patterns in the provided resources, judges interesting topics, and communicates ideas clearly.

Provided tools/resources.

  • Google Drive MCP server, so models can read the provided Google Doc. Some agent harnesses configure this automatically (e.g., Claude Code), while others require manual setup through the Google Cloud Console (e.g., Qwen Code).
  • A link to an internal Google Doc containing the Data Insight template and a corresponding checklist. The template outlines key sections, each with a description of what should be contained. The checklist covers what is typically reviewed before publication, such as “Is the title 80–90 characters (or less)?”
  • A link to Epoch’s Data Insights page for reference.
  • An adapted version of Epoch’s internal website repository, scoped to relevant parts for data insight generation.

Evaluation. Grading weighs the title and plot most heavily — since most readers look only at the title and plot, with the text providing secondary context. These two elements alone should communicate the insight clearly.

AI data center research

Task. Research and analyze a given AI data center to add to Epoch’s AI Data Centers database.

Purpose. This task tests how well the model follows instructions, handles computer use, analyzes detailed images, and sustains progress over long horizons.

Provided tools/resources.

  • Airtable MCP server, connected to model-specific Airtable base.
  • SkyWatch MCP server, for searching satellite imagery.
  • Google Drive MCP server, for reading and writing Google Docs/Sheets, and uploading files to folders.
  • QGIS MCP server, for image analysis.
  • Model-specific Google account.
  • Model-specific SkyWatch Explore account, with credit card set up for automatic checkout.
  • Browser access through provider-specific extensions where available (e.g., Claude in Chrome), and through the Chrome DevTools MCP otherwise.
  • Computer use through native tools where available (e.g., Claude Computer Use), and through CUA Driver MCP otherwise.
  • A Google Doc link containing a detailed guide for researching and analyzing a data center. This is the same guide given to contractors, adapted for agents (e.g., allowing them to purchase images without approval).
  • A condensed version of our Data Centers Slack channel, which covers edge cases and helps guide judgment in ambiguous situations.
  • A review checklist.

Evaluation. Each output is evaluated by one of our data center analysts, who assesses whether the model made reasonable judgments and how well it performed its analysis.

Tasks occasionally require human intervention. The most common case involves SkyWatch imagery, which can take up to 48 hours to arrive after ordering. Models can set up an automated check for this to make the task fully autonomous. If the check does not work as intended or is not set up at all, a human has to notify the model when images arrive. Interventions are also needed when a model’s mistake causes API errors (e.g., sending an image too large for the API request). In these cases, we start a new session and point the agent to its previous trace so it can pick up where it left off.

Research

Task. Given a research direction and a fixed budget, design and run experiments, then write up the findings in a research report.

Purpose. This task tests the model’s ability to design experiments, analyze results, and present them.

Provided tools/resources.

  • Prior work (if applicable)
  • An OpenRouter API key with a $500 cap for pilot experiments

Evaluation. Grading prioritizes research. We assess whether the model’s experiments are set up to answer the research question and whether it draws the right conclusions from the results.

Limitations

Small sample size. We have a limited task suite, and we run each model once per task. The scores are therefore noisy and may shift with more runs. These results should not be read as a general measure of what AI can or cannot do across all work tasks.

Subjective grading. Scoring rubrics involve subjective judgment, so numerical scores should not be treated as ground truth. Specific numbers could change with different rubric item definitions or different graders. We include these numbers to give a rough sense of how models compare and how close each task is to being fully automated. We believe that qualitative findings are more informative than these exact numbers, which we publish through our Epoch Automation Reports.

Epoch-specific tasks. Our tasks are scoped to Epoch work, and thus are not representative of all work tasks. We also evaluate these model outputs by Epoch’s standards, which may not align with how other work tasks should be evaluated.

Different agent harnesses. We test each model on its own harness, such as GPT-6 Astra in Codex. Some differences we observe may be influenced by the harness rather than the model itself. For instance, GPT-6 Astra in Codex with its built-in browser extension may help boost performance compared to Qwen 3.8 Max in Qwen Code with the Chrome DevTools MCP. We chose this setup to reflect how people actually use these models in their work, but we acknowledge that it may introduce advantages for some models over others.

Future work

We plan to expand this task suite over time, adding new tasks and retiring those that become saturated. As new models are released, we will also publish updated results along with commentary on their performance.

Changelog

2026-11-17

Report

Changes from the initial release of Epoch Automation Reports:

Task suite. We’ve updated our tasks to be more representative and fair. For example, our previous research task asked models to run a benchmarking project on a fixed API budget. For many plausible research directions, that budget was too low, and the task was too open-ended. We’ve replaced it with a retired work test we previously gave to candidates, which we know can be completed well within a set time and budget.

Graphic design:

  • Removed “Wealth redistribution” task

Data insight generation:

  • Removed “arXiv” and “Polling” tasks
  • Added “Token prices” task

Data explorer generation:

  • Removed due to overlap with Graphic design and prioritization of other task categories

AI data center research:

  • Removed “Tesla Cortex 1” task

Research:

  • Removed “Benchmarking (9 questions)” task
  • Added “Chess variants” and “Hidden messages” tasks

Elicitation. Our prompts now highlight what we prioritize in grading. In the Data Insight task, for instance, we point out that we value clear communication in the title/plot, and describe the intended audience. For the AI Data Center task, we added more resources, such as a condensed version of our internal data center research Slack channel, which covers edge cases and helps guide judgment in ambiguous situations.

Grading. Each criterion is now scored on a 1-4 scale, similar to conference paper reviews. We found this to better reflect how we actually judge work quality.

Due to these changes, we re-evaluated some of the models from our previous publication. We plan to re-evaluate the remaining models using the updated setup.

Citations

Citation

Epoch AI, 'Epoch Automation Reports'. Published online at epoch.ai. Retrieved from 'https://epoch.ai/benchmarks/epoch-automation-reports' [online resource]. Accessed 18 Nov 2026.

BibTeX Citation

@misc{EpochEpochAutomationReports2026, title = {{Epoch Automation Reports}}, author = {{Epoch AI}}, year = {2026}, month = {11}, url = {https://epoch.ai/benchmarks/epoch-automation-reports}, note = {Accessed: 18 Nov 2026} }