MethodologyMethodology
ECI
The Epoch Capabilities Index (ECI) uses two statistical frameworks: Item Response Theory (IRT) and Bayesian modeling. IRT was originally developed for educational testing, enabling comparisons between students even when they took different exams with varying difficulty levels. Bayesian modeling is a statistical approach that naturally incorporates uncertainty in its outputs.
To calculate the ECI, we use a set of results for a variety of models and benchmarks which is periodically updated as new data becomes available. The goal is to use the results to learn three parameters for individual models and benchmarks:
- \(C_m\), the model capability (which we call the ECI),
- \(D_b\), the benchmark difficulty (which we internally call the ‘EDI’),
- \(\alpha_b\), the benchmark slope.
These parameters are combined in a standard IRT logistic function:
\[\textrm{performance}(m,b) = \sigma(\alpha_b[C_m - D_b]) = \frac{1}{1 + e^{-\alpha_b[C_m - D_b]}}.\]This states that a model’s performance on a benchmark depends on its capability relative to the benchmark’s difficulty, scaled by the benchmark’s slope. A model is expected to score 50% when its capability equals the benchmark’s difficulty. If the model is more capable than the benchmark is difficult, it is expected to score above 50%. If it is less capable, it is expected to score below 50%.
The benchmark slope determines how quickly performance changes around the point where capability equals difficulty. A steeper benchmark slope quickly separates models whose capabilities are close to its difficulty level: a model needs fewer ECI points to increase its expected score on that benchmark. Models well above or below that level all score near 100% or 0%, so the benchmark tells them apart poorly. A shallower slope distinguishes between models more slowly and across a wider capability range. Consider, for example, two benchmarks with the same difficulty, EDI = 145.4, but different slopes:
| Performance measure | WeirdML (slope 0.08) | ARC-AGI (slope 0.21) |
|---|---|---|
| Expected performance at ECI 140 | 39% | 25% |
| Expected performance at ECI 145 | 49% | 48% |
| Expected performance at ECI 150 | 59% | 72% |
| ECI points to go from 20% to 80% expected performance | ~35 ECI points | ~13 ECI points |
Table 1: ECI values needed to reach expected performance with different benchmark slopes.
Bayesian set-up
We fit model capability (\(C_m\)), benchmark difficulty (\(D_b\)), and benchmark slope (\(\alpha_b\)) simultaneously using Bayesian modeling. This gives us a range of plausible values, or a probability distribution, for each model’s capability (ECI) and each benchmark’s difficulty (EDI). We report each model’s ECI as the mean of its distribution, with a 90% credible interval running from the 5th to the 95th percentile. The only data we use is model performance on benchmarks; we make no assumptions about how capability or difficulty relate to training time, release date, organization, or compute.
Performance scores range from 0 to 1, so we use a beta distribution as the likelihood function, which is well suited since its support is \((0, 1)\). The beta distribution can’t produce values exactly at 0 or 1, so before fitting, we clip observed scores to the range \([0.001, 0.999]\). We include a model only if at least four of its benchmark scores fall between 0.1 and 0.9 after normalization; models that qualify keep all of their scores, including those near 0 or 1. The IRT performance equation gives an average performance estimate for a model on a given benchmark:
\[\textit{expected performance} := \mu_{m,b} = \sigma(\alpha_b[C_m - D_b]).\]We also need to model the standard deviation that we expect around this average performance estimate. We scale the beta distribution so that each benchmark has standard deviation \(\sigma_b\) at \(\mu_{m,b} = \frac{1}{2}\). This noise then reduces as the expected performance gets close to 0 or 1, with the form:
\[\textit{standard deviation} = 2\sigma_b\sqrt{\mu_{m,b}(1 - \mu_{m,b})}\]\(\sigma_b\) is allowed to vary per benchmark, to accommodate the fact that different benchmarks have different amounts of noise in their results.
To give us the two properties described above, our likelihood function is defined as:
\[\textit{score of model } m \textit{ on benchmark } b \sim \mathrm{Beta}\left(\mu_{m,b}\,\frac{1 - 4\sigma_b^2}{4\sigma_b^2},\ (1 - \mu_{m,b})\,\frac{1 - 4\sigma_b^2}{4\sigma_b^2}\right).\]Technical details
Priors
Capability and difficulty have wide, flat priors. Slope and standard deviation are hierarchical: we learn their shared scales from the data using weakly informative hyperpriors.
We fit the model and set all priors on a “raw” scale, using the priors below. After fitting, we map each posterior draw linearly onto the ECI scale so that Claude 3.5 Sonnet sits at 130 and GPT-5 at 150. These two anchor models therefore have no uncertainty, and every other model’s interval describes its position relative to them.
Model parameters, priors and hyperpriors
| Parameter | Prior | Notes |
|---|---|---|
| \(C_m\) (capability) | \(\mathrm{Uniform}(-50, 50)\) | |
| \(D_b\) (difficulty) | \(\mathrm{Uniform}(-50, 50)\) | \(\sum_b D_b = 0\): sum-to-zero constraint fixes location. |
| \(\alpha_b\) (slope) | \(\mathrm{LogNormal}(0, \tau_\alpha)\) | \(\prod_b \alpha_b = 1\): keeps the geometric mean of \(\alpha\) at 1 to fix the scale. |
| \(\sigma_b\) | \(\mathrm{LogNormal}(\ln \sigma_{\mathrm{median}}, \tau_\sigma)\) | Truncated between 0 and 0.5 to ensure beta parameters are strictly positive. |
| \(\tau_\alpha\) | \(\mathrm{LogNormal}(\ln 0.5, 0.5)\) | |
| \(\sigma_{\mathrm{median}}\) | \(\mathrm{LogNormal}(\ln 0.1, 0.5)\) | |
| \(\tau_\sigma\) | \(\mathrm{LogNormal}(\ln 0.5, 0.5)\) |
Sampling
We implement the model in PyMC and fit it using the NUTS sampler, with four chains, a target acceptance rate of 0.95, and a max tree depth of 10. PyMC’s default setup uses a diagonal mass matrix for sampling, which works but needs about 12,000 draws per chain in this model because capability, difficulty, and slope are strongly correlated.
To speed up the sampling, we implement a two-step process: a pilot followed by a main fit. We first run a short pilot using a diagonal mass matrix, with 1,000 tuning steps and 1,000 draws per chain, and use these draws to estimate the correlations between parameters. The main fit then uses these correlations to build a dense mass matrix, which is more effective for sampling. The main fit then uses 500 tuning steps and 1,000 draws per chain. We implemented this to speed up sampling, but it does not change the results.
We require no divergences, a bulk ESS of at least 400 for every parameter, and \(\hat{R}\) of at most 1.01 as convergence standards on every fit. If the main fit misses these convergence checks, it reruns with more draws, up to 16,000 per chain. As a last resort, we simply fit the model with the default diagonal mass matrix, with up to 32,000 draws per chain.