Benchmark scores are well correlated, even across domains

Benchmark data is obtained from our Benchmarking Hub. In general, benchmarking evaluations are sparse; the models evaluated on one benchmark will not be evaluated on every other benchmark (especially for benchmarks released far apart in time). To create a more easily interpretable visualization with reasonable confidence intervals, we filter our benchmarks to a subset where no pair of benchmarks has fewer than 5 common models evaluated on each.

We calculate simple Spearman rank correlations over model scores for each pair of benchmarks. We use rank correlation rather than Pearson correlations, since the non-linear nature of benchmark progress and saturation suggests that relationships between benchmarks will be non-linear.

After filtering to a subset of benchmarks where all pairwise comparisons share at least 5 common models, we are left with 15 benchmarks. Among these benchmarks, we find an average rank correlation of 0.73.

We manually classified benchmarks as belonging to four categories: Math, Coding, Reasoning, and Other. We then calculated the median pairwise correlation among benchmarks from the same category, and compared that to the median among benchmarks from different categories. These values were 0.68 and 0.79, respectively.

We additionally calculate correlations between each of the benchmarks and the Epoch Capabilities Index. Correlations are strong overall, as is expected, since ECI uses each of these benchmarks as input. We found a median correlation of 0.90, with a range of 0.46 to 0.96.

Two factors play a large role in the specific correlation values we observe. First, correlation estimates are based on relatively few observations, and these observations are subject to selection biases. Second, because general capabilities are strongly correlated with training compute and release date, the range of models evaluated on a benchmark matters substantially. Benchmarks spanning a wide range of release dates—especially with sparse coverage—will tend to show higher correlations with other benchmarks that share this feature.

For example, evaluations on METR Time Horizons span from GPT-2 XL (2019) to Opus 4.5 (2025). Models far apart in time will rank similarly on almost any benchmark, leading to high correlations. Conversely, Fiction.liveBench covers only models released since December 2024—a narrow enough window that benchmarks might disagree on rankings for some models.