Too few scores per model for bootstrapping to be justified

Our previous methodology estimated uncertainty in the ECI results with bootstrapping, reporting 90% confidence intervals around the point estimate. We aimed to assess how each model’s ECI could have differed if we had observed a slightly different set of benchmark results.

We used 500 bootstrap samples, each holding the set of models fixed and resampling each model’s benchmark results with replacement, keeping its number of observations the same, and refitting the entire model for every sample. Each sample yields slightly different ECI scores, and the 5th and 95th percentiles form the confidence interval bounds.

Each model’s results are resampled independently, with replacement, from its own scores: ten for the median model and as few as four. A bootstrap works when there is enough data for each resample to look like a fresh set of results. On average, each resample leaves out about a third of a model’s scores and counts others more than once. With so few scores per model, this shifts a model’s ECI considerably and the spread across resamples is not a reliable measure of the real uncertainty in its ECI.

Sampling the posterior instead of finding one best fit

The posterior of a Bayesian model usually cannot be calculated directly, so instead we draw samples from it using Markov Chain Monte Carlo (MCMC)². Each sample is one plausible set of every model’s capability and every benchmark’s difficulty, slope, and noise. Taken together, the samples approximate the posterior. Instead of selecting a single best-fitting parameter combination, these samples approximate a probability distribution for each model’s ECI. We report the mean of this distribution as the model’s ECI (after per-sample rescaling to set Claude 3.5 Sonnet to 130 and GPT-5 to 150) and its 90% credible intervals³ are precisely the 90% probability mass intervals of this distribution running from the 5th to the 95th percentile.

As all parameters are jointly estimated, we can look at the uncertainty in functions of the ECI, such as the gap in ECI scores between two models, to assess the probability that one is more capable than another. We present an example of this type of gap analysis in the section ‘How likely one model is to be more capable than another’.

The beta likelihood

We have described how the IRT model gives a predicted score for each model on each benchmark. To fit it statistically, we need to measure how well the predicted scores match the observed scores. In Bayesian modeling, this comes from the likelihood: for any set of parameter values, it says how probable each observed score is, given the model’s predicted score.

The previous frequentist methodology, which minimized the squared prediction error, is comparable to a Bayesian model with a normal distribution that has the same noise for every predicted score. When using least-squares minimization, predicting 50% against an observed 55% is penalized just as much as predicting 90% against 95%. However, it seems reasonable to expect the noise to shrink as the predicted score gets close to 0 or 100%. Another issue is that while the predicted score function itself always stays between 0 and 1, a normal likelihood with constant noise assigns probability to scores below 0 and above 1, which is undesirable (Figure 2).

We can replace the normal distribution assumption with a beta distribution for the likelihood function. The beta is a natural choice as it already has support strictly between 0 and 1, and its noise shrinks as scores approach 0 or 1⁴ (Figure 3).

We set the predicted score function as the expected value (or mean) of our beta likelihood function, fit capability, difficulty, and slope parameters, and include benchmark-specific noise parameters, which we denote \(\sigma_b\) and discuss further in the next section. Then our likelihood is defined:

where \(\mu_{m,b} = \mathrm{predicted\_score}(m,b) = \sigma(\alpha_b[C_m - D_b])\).

The mean of this likelihood is \(\mu_{m,b}\), and the standard deviation is \(2\sqrt{\mu_{m,b}(1 - \mu_{m,b})}\,\sigma_b\). The standard deviation (how much noise/error we expect in the benchmark results compared to the predictions) is maximized at \(\mu_{m,b} = \frac{1}{2}\) where it just equals \(\sigma_b\), and it shrinks to 0 as \(\mu_{m,b}\) goes to 0 or 1.

Finally, the beta distribution assigns no probability mass at exactly 0 or 1, so before fitting, we clip observed scores to [0.001, 0.999]. The ECI results are not sensitive to the exact range used here.

Each benchmark gets its own noise level

Benchmark-specific noise is another change we introduced to better match the model to patterns in the data. By ‘noise’ we mean how far a model’s observed scores tend to deviate from their predicted scores.

For example, a model predicted to score 50% typically lands within 46–54% on METR Time Horizons (taken as a % score on the task suite), while on VPCT it typically lands within 32–68%. In the previous frequentist fit, having the same penalty for every predicted-observed pair of scores effectively assumes that every benchmark is equally noisy. As our example shows, some benchmarks are noisier than others; by introducing the \(\sigma_b\) parameter, we can account for this in the model (Figures 4 and 5).

Note that even a benchmark with very little noise might still carry little information about a model that scores 99% on it, if, for example, almost any strong model would score near 99% too. A noisier benchmark on which the model scores 50% can tell us more. A score’s informativeness depends on the benchmark’s slope and how far the model’s capability is from the benchmark’s difficulty as well as its level of noise.

Fixing the scale with the full set of benchmarks

We fit the model on an internal ‘raw’ scale and then convert every sample to the final ‘ECI scale’ with a linear transformation that sets Claude 3.5 Sonnet to 130 and GPT-5 to 150. We do not fit on the ECI scale directly because the priors are more intuitive to set on the raw scale. Due to the way the predicted scores are calculated, they depend only on the difference between model capability and benchmark difficulty, multiplied by the benchmark’s slope. This means we could add a constant value to all the difficulty and capability parameters without changing any of the predicted scores, or similarly rescale everything by a constant factor and adjust the slopes. We need to remove situations like this, where different parameter values yield identical predictions, to sensibly fit our model.

In the previous frequentist model, we fixed the raw scale by setting WinoGrande’s benchmark slope at 1, alongside the regularization discussed above, which weakly set the mean of model capabilities and benchmark difficulty to zero⁵. In the new Bayesian model, we fix the geometric mean of benchmark slopes to 1: \(\prod_b \alpha_b = 1\) and center benchmark difficulties at zero: \(\sum_b D_b = 0\). This establishes the raw scale using the full set of benchmarks rather than relying on a single benchmark.

After fitting the model, we apply the same linear transformation as before to report results on the ECI scale, fixing Claude 3.5 Sonnet’s ECI at 130 and GPT-5’s ECI at 150. These anchors let us compare results over time, for example, when we receive more benchmark scores or a new model is released. Because these two anchors fix the scale, each model’s uncertainty interval describes its capability relative to Claude 3.5 Sonnet and GPT-5. The anchors themselves have no uncertainty.

Priors

In Bayesian modeling, we use priors, which are probability distributions that represent any existing knowledge we have for the plausible values. Setting priors describes what we expect before seeing the data. For the ECI, we do not have strong evidence of plausible values, and we do not want preconceived notions to have a substantial influence on results; instead, we want them to be data-driven. We therefore use flat priors on capability and difficulty, and loose hierarchical priors on the slopes and noise, which are probability distributions whose parameters (such as mean and variance) are themselves learned from the data.

For capability \(C_m\) and difficulty \(D_b\), we want them to be roughly on the same scale. We set flat priors for these parameters, giving every value in a wide range the same probability, since we make no assumptions about how model capability or benchmark difficulty might be distributed. In particular, we don’t want the prior to constrain the rate of model progress that the model discovers. We then have

We bound the priors so the posterior stays well defined, but set the bounds far wider than the data can reach: when converted from the raw scale to the ECI scale with the current fit, they span ECI values of −390 to +670, while the models in our data sit between about 45 and 169.

For the slope parameter, we do have some prior knowledge: we expect benchmarks to cluster around a typical rate of change, and a benchmark whose scores change 100 times faster or slower than average would be surprising. A natural prior for a slope parameter such as \(\alpha_b\) is the log-normal: it is strictly positive, it treats “twice as fast” and “half as fast” symmetrically, and it can span several orders of magnitude. Rather than fix how spread out the slopes are, we learn that from the data with a hyperprior. We set \(\tau_\alpha\) to be the scale parameter, and set 0 as the location parameter, so that

Finally, for the per-benchmark noise parameter, we use a log-normal with hyperpriors for the location and spread:

with

Because these priors have their own learned parameters, the model is hierarchical. We chose fixed hyperparameters via grid search; the results are not sensitive to them.

Frontier ECIs are similar while weak models move down

The Bayesian model affects two groups of models very differently, with GPT-4 as the dividing line. GPT-4 is the earliest model in our frontier trend and the weakest frontier model in our dataset (ECI 126 under the previous frequentist method, 122 with the Bayesian fit). Models below this were behind the frontier when they were released.

Models at ECI 125 and above

For the 176 models at or above ECI 125, the two fits agree closely. Figure 7 plots each model’s new ECI against its previous ECI. The points sit on the diagonal (R² 0.995), and the two fits differ by 1.1 ECI points by root mean squared error (RMSE) and 0.9 by mean absolute difference. Every model’s new ECI lies inside its previous 90% confidence interval, typically a quarter of the way from its center to its edge, and the new intervals are 13% narrower on average. Models above ECI 150 tend to move up slightly: 87% of them do, by 0.7 points on average and at most 2.3.

Models below ECI 125

The 76 models below ECI 125 all move down, by 13.7 points on average, and by 27 points for the 21 models previously below 100 (Figure 8; Figure 10 shows the 20 weakest models). The weaker a model, the further it moves: the change stretches the bottom of the scale rather than shifting these models by a fixed amount.

Part of this change comes from dropping the previous fit’s regularization: the penalty on large parameter values and their bounds. That penalty pulled every capability toward zero on the raw scale, which mattered little for models with many scores but held up very weak models measured on a few, mostly saturated, benchmarks. Refitting the previous frequentist model without the penalty and bounds moves models below ECI 100 down by 10.5 points (from 27 under the Bayesian fit) and models between 100 and 125 down by 3.6 points (from 8.8). The rest of the change comes from the new model itself. We could have kept a penalty to prevent this change, but we did not think it was a justified feature of the original frequentist model: it pulls a model toward the average simply because it has few scores. It also doesn’t affect anything the ECI is typically used for, since these models sit well below the frontier.

The table below compares the two groups.

Figures 9 and 10 show each model’s ECI and 90% interval under both fits, for the 20 highest-ranked models and the 20 weakest. All 20 top models lie within their previous intervals; 18 move up slightly, by 0.7 points on average, and all 20 get narrower intervals. The 20 weakest models (by previous ECI) all move down, by 27 points on average, and all 20 get wider intervals.

Frontier trends get slightly steeper

We refit the frontier trend with the same criteria as our prior work analyzing trends in the ECI⁶, using the Bayesian model. We fit separate trends for reasoning and non-reasoning models, ending the non-reasoning trend at GPT-4.5.

Figure 11 shows each era’s trend under the frequentist and Bayesian fits, and the table below gives both eras’ trends.

Both trends get about one to two ECI points per year steeper, which is small compared to their uncertainty. This is likely due mainly to a change of scale: the Bayesian fit places the two anchor models, Claude 3.5 Sonnet and GPT-5, closer together in capability compared to the frequentist fit. Because both fits fix the gap between these two anchors at 20 ECI points, the Bayesian scale stretches relative to the frequentist one. For example, GPT-4 and Claude Opus 5.5 have a 46.9-point ECI gap in the Bayesian fit, compared with 43.0 in the frequentist fit.

Within each era, earlier frontier models move down slightly and later ones up: GPT-4 drops 4.0 points while GPT-4.5 drops 0.2, and the first half of the reasoning-era frontier moves by −0.1 on average against +0.5 for the second half. The two fits also pick slightly different frontier models because each draws from its own estimates. Each frontier model’s new ECI lies within its previous 90% interval (Figure 12).

Similar accuracy, better calibrated out of sample

The main reason for moving to a Bayesian model was to get better-justified uncertainty estimates. In this section, we assess both the accuracy of the predicted scores and the calibration of their uncertainty, comparing the previous frequentist approach with the new Bayesian one.

Accuracy measures how far predicted scores are from the observed scores, using RMSE and mean absolute error (MAE). Calibration measures whether the uncertainty is right. For each observed score, the fit gives a range that should contain it 90% of the time. If the fit is well calibrated, about 90% of observed scores fall inside their range; much more means the ranges are too wide, and much less means they are too narrow. The frequentist fit gives only a single predicted score, with no range, so to compare we treat each prediction as a normal distribution with the same spread (SD 0.076) for every score, as described in the section on the beta likelihood. We check calibration for all scores, and separately for scores near 0 and 1, where a single noise level for every score fits worst. In-sample results check each score against a fit that included it; leave-one-out results check it against a fit that didn’t, which tests how well each fit predicts scores it has not seen.

In-sample, the frequentist fit has a slightly lower error; however, out of sample, the Bayesian model is slightly better. The Bayesian intervals contain 89.6% of out-of-sample scores, against 84.2% for the frequentist fit. Near 0 and 1, the frequentist intervals contain almost every score because they extend below 0 and above 1 into impossible values, as discussed in the section on the beta likelihood. The Bayesian intervals stay within the possible range and contain 80.5% of scores near 0 out of sample.

How likely one model is to be more capable than another

The Bayesian fit gives a probability distribution for each model’s ECI. Figure 13 shows these distributions for the 20 highest-ranked models. Wider distributions mean more uncertainty. These distributions are still fairly wide and often overlap: GPT-6 Astra’s 90% interval runs from 166.2 to 172.7, and Claude Opus 5.5’s from 165.8 to 172.0 (Figure 14).

We are often much more certain about the gap between two nearby models than their separate intervals suggest. Much of each model’s uncertainty is shared across models, as it comes from uncertainty in the gap between the two anchor models, Claude 3.5 Sonnet and GPT-5. Because that gap is fixed to be exactly 20 ECI points, a draw in which the anchors sit closer together on the raw scale stretches the whole scale, and models near each other move together, so the gap between them is less uncertain than either model’s ECI. The 90% interval for the gap between GPT-6 Astra and Claude Opus 5.5 runs from −0.9 to +2.1, half the width of either model’s own interval.

Because we estimate each model’s capability together, we can now compare models directly. Figure 15 shows the five models closest behind GPT-6 Astra, which has a 72% probability of being more capable than Claude Opus 5.5, but a 99% probability of being more capable than Claude Fable 5.1. Figure 16 shows the full probability distribution of these two gaps. We hope to present more results like this alongside the ECI in the future.