Agents’ Last Exam (ALE) is a benchmark designed to measure agents’ ability to perform long-horizon, economically valuable tasks with verifiable outcomes. ALE v1.1 has 122 public tasks out of a reported total of 1,500. We analyzed a stratified sample of 50 public tasks, and severe grading errors affect 22 of them (44%). In each of these 22 tasks, an agent can score 100% without doing the intended work by taking a shortcut. In 9 tasks, the answer is public and easy for agents to find. In 8 tasks, the grader accepts trivial or made-up work. In the other 5, agents can easily tamper with the grader. The count of 22 tasks leaves out tasks where an error lets an agent earn some points but not a full score, tasks that require a more elaborate shortcut, and tasks that don’t give credit for correct work, so we expect the true error rate to be higher. The 44% is above the 20% threshold in our rubric, so we designate ALE as Flawed.
Methodology
We sampled 50 of the 122 public tasks1 in ALE v1.1, stratified by the benchmark’s 13 categories,2 with at least one task from each. The sample covered all three of ALE’s task tiers: Near-Term (workflows agents can already partly complete), Full-Spectrum (tasks chosen for broad domain coverage), and Last-Exam (the hardest workflows).
Our review consisted of three steps. First, we used Claude Opus 5.5, with one subagent per task, to find potential errors. Next, we used separate Claude instances to try to disprove those findings by rereading the grading code and, where it could run on our machines, running it on the shortcut outputs. Finally, we manually reviewed each flagged task. Given the high number of errors, we only report tasks where a simple unintended shortcut can score 100%.3
Findings
An agent can score 100% on 22 of the 50 sampled tasks (44%) without doing the intended work or by skipping a large part of it. These errors fall into three categories:
- Public answer (9 tasks): The answer can be found in the benchmark’s GitHub repository or on a public website. Agents have internet access, with some caveats discussed later, so an agent can look up the answer instead of completing the task.
- Lax grader (8 tasks): The grader gives full credit for incomplete or incorrect work. The agent can skip most of the intended work by submitting trivial outputs or by creating made-up or incorrect outputs.
- Grader tampering (5 tasks): The agent can easily tamper with the grader, often because the grader runs code from a folder the agent can write to. An agent with web access can read the grader code from the public GitHub repository during its run and prepare files that tamper with the grader. This can happen even if the grader is copied to the agent’s machine only when the agent finishes.
In this benchmark, agents are usually allowed to browse the internet. However, 6 of the 22 flagged tasks try to restrict it in different ways: three prompts discourage web search or internet use, one input file states (incorrectly) that there is no internet access, and two task cards (each task’s settings file, which is not shown to the agent) try to turn the network off.4 None of these restrictions are enforced in practice, so an agent that accessed the internet would not be penalized.
Representative Errors
Public answer. Task: arbitration fee calculation (legal_dr_fees_01). The agent receives the fee rules of the Beijing Arbitration Commission and five disputes. For each dispute, it must calculate the institution fee, the arbitrator’s fee, and the total fee. The prompt instructs agents that web search use is disallowed. The grader compares the agent’s output with 15 amounts that are written directly in its own code, which is public on the benchmark’s GitHub repository. Nothing enforces the browsing rule, so an agent can copy these 15 amounts and score 100% without reading the fee rules.
Lax grader. Task: paper replication (atwood_2022_measles_vaccine_reproduction). The agent needs to replicate some findings from an applied economics paper. Using the provided replication package, the agent has to reproduce the six “vaccination effect” coefficients found in Table 2 of the paper, reporting both the values printed in the original work and the values produced by the code. The grader, however, checks the “code” coefficients only against the numbers printed in the paper, which is itself one of the task’s inputs. The files meant to show that the code was run are checked only for format: the execution log must not be empty, and the table’s source columns are never read. An agent can fill these files with placeholder content, copy the paper’s six numbers, and claim they came from rerunning the code.
Grader tampering. Task: image-model repair (dit_pipeline_cfg_alignment_fid_256_001). The agent has to fix a bug in a Diffusion Transformer image-generation process. The fix should make the pipeline behave like the authors’ intended implementation. When checking the solution, the grader loads the agent’s file in a separate process, runs five test cases against the reference, and accepts whatever score that process prints. Since the agent’s code runs as soon as the file is loaded, a three-line file that prints a perfect score and exits scores 100% without fixing the bug.
Errors
All defects in the 22 tasks listed below cause false positives. Some of these tasks also have other errors that we do not list.
| Task | Error type | Notes |
|---|---|---|
| tris_crackme | False Positive (public answer) | A crackme is a small program made as a reverse-engineering exercise: the agent must find a valid registration code for it. The key generator is in the public grader code, so writing two registry values (a name and its code) scores 100% without any of the intended work. |
| legal_ma_consistency_audit_01 | False Positive (public answer) | The public task description reveals all four filing inconsistencies the audit must find. The public grading code also gives away the page numbers and wording needed for full credit, so an agent can build a perfect report without reviewing the filings. |
| merfish_image_decoding_segmentation_1 | False Positive (public answer, lax grader) | The agent must decode gene signals and identify cell nuclei that it sees in microscope images. The hidden answer key is a public file from the starfish project, which is an open-source library that contains images of cells and matching it gives 65% of the score. The other 35% doesn’t check the nuclei against a reference. Taken together, agents can score 100% without analyzing any images. |
| abm_hangzhou_metro | False Positive (public answer) | The agent must write a simulator of passenger trips on the Hangzhou metro and submit its trip records and a report. The grader’s metro simulator is public, so calling it and copying the results scores 100%. |
| wsi_tumor_localization_1 | False Positive (public answer) | The agent is given a scanned tissue slide and must report one coordinate near the center of the tumor. The task card disallows network use but the harness never enforces the rule. The tumor outline is the public annotation of slide tumor_001 from CAMELYON16, a public set of lymph node tissue scans with expert tumor outlines. An agent can just download it and score 100%. |
| power_10kv_feeder_reliability_001 | False Positive (public answer) | A public script in the benchmark’s GitHub repository rebuilds the hidden answer key exactly from the public inputs. Simply running it scores 100%. |
| ketcher_smiles_reproduction | False Positive (public answer) | The task card prohibits internet use, but the harness never actually enforces this. Furthermore, the task’s only software is Ketcher, an online app for molecule editing. The hidden answer is the SMILES string (a text code for a chemical structure) of the target molecule, which the RCSB Protein Data Bank publishes on its website. Pasting the SMILES scores 100% without drawing the structure. |
| amber_three_stage_mmgbsa_workflow_instance_1 | False Positive (public answer) | The public tests include a complete set of four files that earns full credit without running the requested simulation. Copying it (or writing any value in the accepted band) scores 100% without doing the work. |
| legal_dr_fees_01 | False Positive (public answer) | See Representative Errors. |
| particle_filter_nonlinear_tracking | False Positive (lax grader) | The agent must track a moving object from noisy bearing and range measurements by writing a particle filter, a method that estimates a position by weighting many random guesses. The grader never runs the filter, so submitting the agent’s own simulated truth scores 100%. |
| yeast_colony_detection | False Positive (lax grader) | The agent must detect yeast colonies in an image and report their positions. Colony positions are checked by a distribution test that can be passed with one far outlier. Points stacked at one spot plus one outlier score 100%. |
| Analog_Active | False Positive (lax grader) | The agent must design a circuit and report its currents and voltages. The grader does not check whether the circuit’s parts are connected or run the circuit simulation itself. Made-up voltage and current results can score 100%. |
| atwood_2022_measles_vaccine_reproduction | False Positive (lax grader) | See Representative Errors. |
| internal_employee_agent_instance_1 | False Positive (lax grader) | The agent must answer employee questions about HR and IT policy. The task scores 100% when at least 92 of 102 hidden checks pass. These checks look only for topic words in the answer and for tool names that the agent claims it used, so answers with factually wrong HR and IT policy facts can score 100%. |
| prostate_imrt_matrad_reproduction | False Positive (lax grader) | The agent must reproduce a treatment plan for prostate radiation. The grader gives full credit if the agent scores 70 out of 98 points. A made-up dose with no treatment planning can earn 73 of 98 points, passing the task with full credit. This is because only one check, worth 12 points, compares the agent’s dose with the correct reference dose. |
| butterfly_flap_animation | False Positive (lax grader) | The agent is asked to animate an image of a butterfly with flapping wings, but the grader does a poor job of checking whether the animation is correct. Squeezing the whole, uncut image sideways four times along a wavy path scores 100%. |
| atlas_outpost_graybox_navigation | False Positive (lax grader) | The agent must build a geometric game level and export it as a 3D model file (glTF). The grader reads only object names in the exported file and text markers in the other files. A file of empty objects with the right names, plus the marker strings, scores 100% with no level geometry. |
| mp_checkpoint_consolidation_v2 | False Positive (grader tampering) | The grader runs a program from the agent’s input folder. Editing that program can make any submission score 100%. |
| cailian_road_highway_alignment_2 | False Positive (grader tampering) | The grader runs from a folder the agent can write to. Editing it to report success scores 100%. |
| dit_pipeline_cfg_alignment_fid_256_001 | False Positive (grader tampering) | See Representative Errors. |
| lenacapavir_sar_table2_extraction | False Positive (grader tampering) | The grader runs from a fixed Temp folder that the agent can write to. A specifically constructed Python file planted there could take over the grader and make any answer score 100%. |
| hst_acs_wfc_visit_reduction | False Positive (grader tampering) | The agent’s reduction script is run after the hidden answer key is placed on the same machine. A script that copies them scores 100% without reducing the images. |
Unflagged Tasks
Here, we report the other sampled tasks. We cannot conclude that these tasks are error free: metabase_bi_dashboard_01, taxform_4_1, ar_full_300, equity_research_summary, bpmn_category_governance_restructuring_l3, sec_10k_financial_parsing, cp_test_gen_1, k8s_migration_1, ranking_node_feature_parity_recovery_instance_1, cfr_game_theory_equilibrium, ecg_rhythm_conduction_ptbxl, healthcare_tcga_luad_survival_kras, nhanes_confounder_sensitivity_analysis, limited_angle_ct_dps_reconstruction, ltmle_targeted_bootstrap_simulation_study, 2d_drawings_to_3d_building_model, gcode, openroad_sky130_ibex_pnr_signoff, exact_diag_heisenberg_j1j2, mose2_bse_absorption_soc, silicon_bse_absorption, cell_tracking_instance_1, tp53_locus_variant_histone_browser_svg, music_transcription, project_migration, scene2_resample, yi_manuscript_translation_1, mota_exploration.
Rubric
1. Reviewability
| Level | Meaning | Status |
|---|---|---|
| Full | All tasks and scoring logic inspectable,5 and harness/API settings used for each model (reasoning effort, token/time limits, tool access, system prompts) are fully disclosed | [proceed to 2] |
| Partial | A representative sample of tasks and scoring logic inspectable, with full harness/API settings disclosed | [proceed to 2] |
| Inadequate | Limited, biased, or no inspectable tasks or scoring logic, harness/API settings undisclosed | [stop → NEI] |
2. Scoring
This is the minimum standard required to be Verified; failing any of these items results in a Flawed verdict.
| Scoring | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Either errors affect ≥20%6 of the total score in the inspected sample,7 or there is an issue that corrupts grading at scale |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Benchmark Consistency | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Leaderboard has incomparable results from different versions |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Elicitation | |
|---|---|
| Examples | Model elicitation is extremely constraining and is not the focus of the benchmark:
|
| Default threshold for Flawed | Substantially reduced performance compared to reasonable alternatives for the tasks |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
| Bias in Evaluation Setup | |
|---|---|
| Examples |
|
| Default threshold for Flawed | Material model-specific advantage found |
| Status | PassFlag [stop → Flawed]Not reviewed |
| Notes | — |
3. Evaluation Quality
This is the standard we would like all benchmarks to meet, but it is not necessarily disqualifying to omit or fail these items.
| Question | Status | Notes |
|---|---|---|
| Elicitation and resource adequacy: Are the resources given to models (reasoning token/turn budget, tool access, etc.) sufficient for them to perform near their ceiling? | SufficientConstrainingUnreasonably constrainingUnknownNot reviewed | — |
| Scaffold fairness: What scaffold does the leaderboard report? | Shared common scaffoldMix of model-specific and common scaffoldsModel-specific scaffoldsNot reviewed | — |
| Is there evidence/risk of contamination? | Not reviewed | — |
| Has human completability been assessed? | All tasksRepresentative set of tasksPoor implementation (Unrepresentative set of tasks, unreasonable set of participants)Not establishedNot reviewed | — |
| Score range (if possible to estimate) | — floor, — ceilingNot reviewed | — |
| Statistical adequacy: how many runs/model (≥ 5 recommended for error bars) | — runs/modelUnknownNot reviewed | — |
| Construct validity | Measures stated capabilitiesPartially measures stated capabilitiesDoes not measure stated capabilitiesNot reviewed | — |
Disclaimer
This is a partial review. We stopped inspecting once we discovered errors that exceeded our published threshold for a flawed benchmark.
We inspected 50 tasks, stratified by category weight. Tasks not inspected may contain additional defects.
Developer Response
Link to developer response: agents-last-exam.org/blogs/verifier-check
-
The dataset in v1.1 is made of 122 public representative tasks from the website’s reported 1500 total. The leaderboard only scores agents on public tasks. The developers say that they will move a new batch of tasks from the private pool to the public set every six months, but v1.1 retired 30 tasks and added none compared to v1.0. Most errors we found come from how the graders and environments are built, not from the content of a specific task, so if private tasks use the same framework, which we could not check, we would expect similar errors there.
-
The categories are: business_finance, computing_math, health_medicine, engineering, physical_sciences, life_sciences, visual_media, psychology_neuro, education_info, legal, other, transport_safety, social_sciences. V1.1 retired the two agriculture_env tasks present in the previous dataset.
-
Opus also flagged errors that affect only part of a task’s score, that are harder to confirm, or that wouldn’t award deserved score to correct work. We did not investigate these further, but we expect the true error rate to be higher than 44%.
-
Prompts: legal_dr_fees_01, atlas_outpost_graybox_navigation, and lenacapavir_sar_table2_extraction. Input file: atwood_2022_measles_vaccine_reproduction. Task cards: wsi_tumor_localization_1 and ketcher_smiles_reproduction.
-
Either publicly or privately to reviewers under an NDA.
-
A benchmark is rated Flawed if scoring defects could affect ≥20% of the benchmark’s full score. For benchmarks with equally weighted all-or-nothing tasks, our threshold corresponds to ≥20% of tasks having defects. For benchmarks with partial scoring or unequal task weights, our threshold corresponds to ≥20% of the total possible score affected by defects. For single task benchmarks, our threshold is the reported score having defects that could affect it by ≥20%.
-
For benchmarks with >50 available tasks, we will sample a random set of 50 (stratified by category, when present). If the observed error rate is 15–25%, we will expand the sample to 100. For benchmarks with ≤50 available tasks, all available tasks will be assessed.
About Benchmark Reviews
Epoch AI’s Benchmark Reviews are independent reviews of external AI benchmarks. Our documentation describes the rubric behind this verdict, how we choose which benchmarks to review, and answers frequently asked questions.