Our data comes from Epoch AI’s Biology AI Models dataset, a subset of our AI Models dataset focused on models trained on biological data. For each model, the dataset has a corresponding publication date, training compute quantity, and intended purpose, among other fields.

There are 361 models in the dataset, of which 163 have compute estimates. Of those, 66 are labeled ‘Protein or nucleotide language models” and 97 are other types of models (which we group together into “specialized models”). After separating protein language models and specialized models, we filtered to the rolling top-4 models by training compute in each category. After this filtering, we were left with 18 foundation models and 17 specialized models.

We have made a focused effort to increase coverage of biology AI models, but coverage is still low for these models, particularly prior to 2020. We performed a systematic literature review for models published since 2020, while pre-2020 model inclusion was limited to those with established significance in the field. This lack of coverage might bias our estimates of compute scaling rates. However, since models trained on large amounts of compute tend to receive attention due to their good performance and cost, we think it is unlikely that we missed models significantly above the trendline. For this reason we believe the overall slowdown pattern is not caused by missing data.

Our analysis is concerned with the frontier of scaling in each category. While we typically define frontier models as those in the top 10 by training compute at their time of publication, here we use the top 4 to balance the goal of capturing only frontier models with a need for reliable statistical analysis. Due to the relatively small population of biological models, the rolling top 1 model would provide a very small dataset (11 models per category), and the rolling top 10 included models far away from the frontier. The slopes and particular dates of the breakpoint of our fitted trends are sensitive to choice of top N; results are presented in the table below.

To determine the best fit lines, we ran a log-linear regression on each category of models. We evaluated segmented regression models which allowed either zero or one breakpoints. We did not test models with a discontinuity at the breakpoint, since we have too few datapoints to accurately measure the presence and size of a potential discontinuity. For each model type, we tested all possible breakpoints at a monthly resolution, for line segments that span at least 5 data points. The minimum number of data points per segment is somewhat arbitrary, but does not have a significant impact on the median breakpoint date.

After testing breakpoints, we selected the breakpoint which minimized the Bayesian information criterion (BIC). To estimate confidence intervals in the results, we perform a bootstrap analysis, resampling from our original dataset with replacement and repeating the process described above for 1000 iterations. After evaluating each regression model type, we chose the model type which minimized BIC. The model with a single breakpoint obtained lower BICs than a simple exponential trend with no breaks. In particular, for protein language models the difference in BIC was -2.09 nats (90% confidence interval: -37.04 to 4.18), and for specialized models it was -6.67 nats (90% confidence interval: -39.0 to 4.05).

Code for our analysis is available at Github.