Our data come from Epoch AI’s Notable AI Models dataset. We focus our analysis on frontier LLMs, defined as those that were among the top-5 largest by compute at the time they were published. We then filter to models released after 2020, and those which have estimates of training duration, leaving a dataset of 38 models

Separately, we use previous estimates for trends in algorithmic progress, hardware price-performance, and training budgets to estimate the longest training time under a simple model of opportunity costs. The hardware progress estimate is based on data from Epoch AI’s ML Hardware dataset, while algorithmic progress and training budgets each use Epoch AI’s Notable AI models dataset.

To obtain an estimate for the trend in frontier LLM training time, we fit a simple model regressing the logarithm of training time on publication date. We bootstrap this regression to obtain a 90% confidence interval.

Our main analysis of the longest training runs focuses on the “hardware and software progress” scenario of our earlier report, “The Longest Training Run”. We update the numbers from that analysis using our latest estimates for each of the underlying trends. Additionally, we propagate our uncertainty over the underlying trends through the model, to get a confidence interval on the longest training duration.

To obtain a confidence interval on the range of dates at which training time might stop growing, we randomly sample from both the training time trend and the longest run bootstraps, calculate the implied date at which they meet, and report the 5th and 95th percentile of these dates.

Key results can be found in the following table:

A notebook containing our analysis can be found here.

Our main analysis focuses on a scenario where hardware and software progress continues at historic rates. Our previous analysis of the longest training run also examined the effect of increasing training budgets. However, we think this effect may be limited in today’s environment; with engineering effort, we believe it is possible to add newly acquired chips mid-training run. We still believe that hardware progress matters, since developers still face an opportunity cost to waiting for better hardware any time they buy chips.

Our analysis of the longest training run assumes that progress in algorithms and hardware is smooth and continuous. We do not expect our conclusions would substantially change if we relaxed these assumptions by accounting for discontinuous steps on release of new GPU designs or algorithm breakthroughs.