Interpretation

WeirdML tests rapid ML engineering on a novel problem formulation, executed under a 120-second, single-GPU compute ceiling, with feedback across five attempts, scored on the best attempt. The benchmark uses an unusual non-agentic setting, and separate analysis of two models showed significant score increases with a conventional agentic setup. A high scorer can be trusted to: understand a stated data/problem description, select and implement a reasonable PyTorch architecture, and achieve high accuracy under a tight and disclosed compute budget with multiple attempts for a problem type whose exact formulation it hasn’t seen before. The benchmark does not assess whether this capability extends to general adaptive ML engineering on unfamiliar problems and datasets.

Potential confounds

• Best-of-five scoring: Score reflects the best of five attempts, not a solution the model committed to as being the best. This inflates the score relative to any deployment where a model must commit to one answer.
• Ambiguous system prompt: The system prompt is ambiguous, stating “you have several iterations” instead of clarifying that the limit is five. This biases results against models that assume they will have more time to explore. The behavior is most common in the undisclosed task “number_patterns”.

Task Analysis

We were able to inspect all prompts and data for all 19² of the tasks (including those that are normally kept private). The prompt and a small subset of the training data are publicly available for the four disclosed tasks³. The remaining 13 tasks are identified only by name (e.g. Splash Hard). Our review will focus on the public tasks, with any findings in the private set kept abstract to minimize contamination risk.

‘Weird’ primarily describes the task formulations rather than the underlying ML methods. The four disclosed tasks are largely bespoke versions of familiar ML problem classes:

• Shapes (Easy/Hard) is permutation-invariant classification over point sets
• Digits is semi-supervised classification with class imbalance
• Chess predicts game outcomes from real game move sequences (beginners with Elo <1300)

Hard variants mainly remove shortcuts rather than adding conceptual novelty. The datasets used are a mixture of known and novel, not exclusively novel as described by the benchmark. Some disclosed tasks (e.g., Shapes Easy) are saturated among frontier models.

Performance Ceilings/Floors

• Task ceiling: The known per-task upper bound is 94.78% (the average, over the 17 tasks, of the best score any model has ever achieved on each task). We estimate the per task ceiling ranges from 80% to 100% across the 17 tasks, with an average of around 95%.
• Task floors:
  • Shapes (Easy/Hard/Variable) = 20%
  • Chess = 50%
  • Digits Unsupervised = 16.7% uniform, though class imbalance could make a majority-class guess stronger⁴
  • Kolmogorov Shuffle = 20%
  • Classify Sentences = 10%
  • Classify Shuffled = 20%
  • Insert Patches = 8%
  • Blunders (Easy/Hard) = 20%
  • Digits Generalize = 10%
  • XOR (Easy/Hard) = 10%
  • Splash (Easy/Hard) = 10%
  • Number Patterns = 20%
• Human baseline: None.

Elicitation and Scaffolding

• Coverage: 152 models present in the public results file at time of review; updated regularly with new models.
• Elicitation setup: 5 submissions per run; model receives terminal output + evaluation feedback after each of the first 4 submissions; each submission executes in a Docker container on a fixed TITAN V GPU (12GB) with a 120-second timeout (down from 600s in v1); ~5 runs/model in v2 (only 2 for several expensive models), vs. 15 runs/model in v1 (5 for o1-preview). Approximate run cost is tracked.
• Scaffold variants: The standard leaderboard uses one shared scaffold. A separate Agentic WeirdML experiment is the only scaffold variant discussed. In it, GPT-5.5 and Claude Opus 4.7 were evaluated using codex_cli and claude_code, respectively, inside the Inspect AI framework in a Docker sandbox with no internet access and a budget of 20M total tokens per task. The scaffold improved performance compared to the non-agentic baseline: Claude Opus 4.7 improved from 76.4% to 87.9% and GPT-5.5 went from 84.9% to 88.3%.

Limitations

The write-up describes 6 public tasks and 13 new tasks, but scoring covers 17 tasks. Shuffle (Easy/Hard), which are image-patch jigsaw/permutation tasks on Fashion-MNIST/Imagenette, were dropped due to prompt issues, as described on X⁵.

Shuffle Easy, which was dropped due to prompt issues, has a documented shortcut: individual patches often reveal their position without solving the intended cross-patch reasoning problem. Of the hidden tasks, two have known shortcuts, which GPT 6 Astra (max) is able to exploit⁶.

In the agentic experiment, frontier models discovered they could train outside the timed grading harness entirely and submit only pre-computed weights for inference within the 120-second window. This complicates cross-model comparison in that setting, but does not affect the standard (non-agentic) leaderboard.

Two of the four stated goals of the benchmark are not explicitly tested:

• “Understanding the data” (Goal 1) is not separately measured; several disclosed tasks (Shapes Hard, Shuffle Hard, Digits) explicitly tell the model the properties it is nominally supposed to discover, so the score partly tests acting on stated properties rather than independent discovery. A subset of tasks are prone to shortcuts or generic solutions that achieve reasonable results without a deep understanding of the data.
• “Improve from feedback” (Goal 3) is not separately measured; because the final score is the best of five submissions, additional attempts can improve scores even without using feedback. The removal of v1’s max@k decomposition means the benchmark cannot quantify how much improvement comes from feedback versus variation across independent attempts.

Additional Results

Our analysis of the public results found a correlation of r=0.88 between log median submission execution time and accuracy across the 17 tasks. The correlation is ambiguous since we can’t distinguish between more accurate models using more compute and more accurate models calibrating their budget to the available compute.⁷

Rubric

Disclaimer

The developer declined to respond.

About Benchmark Reviews

Epoch AI’s Benchmark Reviews are independent reviews of external AI benchmarks. Our documentation describes the rubric behind this verdict, how we choose which benchmarks to review, and answers frequently asked questions.

View benchmarkRead the documentation and FAQs