We use Epoch AI’s Notable Models database and pull relevant fields including publication date, number of parameters, estimated training compute, and training dataset size. To focus on models trained from scratch, we exclude non‑language systems as well as any fine‑tuned, continually trained, or distilled variants, since their token‑per‑parameter ratios reflect downstream adaptations rather than pre‑training dynamics.

For transformer-based models lacking reported dataset sizes but have available compute estimates (C) and active parameter counts (N), we infer their tokens to active param ratio by rearranging the relation C=6 N D to:

We then include these estimated values of tokens per parameter in our overall trend analysis.

We fit an exponential growth model by performing a linear regression on log(tokens per parameter) against model publication date. The resulting linear fit is statistically significant and shows a positive correlation between tokens per active parameter and model publication date.

To generate confidence intervals, we used bootstrap sampling with replacement (n=500). For each bootstrap sample, we resampled the 33 observations with replacement, refit the exponential growth model, and collected the resulting slope estimates. We then calculated the 5th and 95th percentiles of the bootstrap slope distribution to construct 90% confidence intervals.

The annual growth factor derived from the bootstrap median is 3.1x per year, with a 90% confidence interval of [2.1x, 4.9x].

The reliability of the findings are directly tied to the accuracy and completeness of the training dataset size, parameter count, and training compute estimates in our Notable AI models database.

An important limitation of our analysis is the lack of data on closed models. Many do not disclose key details such as training compute, parameter counts, or cumulative training tokens, which prevents us from estimating their tokens‑per‑parameter ratios.