Over 20 AI models have been trained at the scale of GPT-4

Overview

To search for AI models exceeding 10²⁵ FLOP we examined three sources:

• Recent releases from leading AI labs
• Top-scoring models on AI benchmarks and leaderboards
• The most-downloaded large models on the open model repository, Hugging Face

We performed an exhaustive review of recent model releases from AI developers known to have access to enough AI hardware for large-scale training: Google (including Google DeepMind), Meta, Microsoft, OpenAI, xAI, NVIDIA, and ByteDance. Other organizations that have developed models in the Epoch database of notable AI models were examined briefly for recent, large-scale model releases.

To identify other large models developed by other, lesser-known labs, we examined common language benchmarks such as MMLU, MATH and GPQA, as well as the Chatbot Arena and HELM leaderboards. We searched for any scores that matched or exceeded the performance of lower-performing models known to be trained with over 10²⁵ FLOP (i.e. the minimum score across the original GPT-4, Inflection-2, or Gemini Ultra). We also searched for large models developed by Chinese developers. To do this, we examined the China-focused CompassRank and CompassArena leaderboards. We looked at all models scoring above models known to be trained with just under 10²⁵ FLOP, such as Yi Lightning and Llama-3 70B.

Models that met these criteria were added to our wider database of AI models. We then estimated their training compute, and whether it might exceed 10²⁵ FLOP. For several AI models, developers provided insufficient information for us to directly estimate training compute. In this case, we estimated training compute from model performance. For language models, we did this using imputation based on benchmark scores. For image and video models, where fewer standardised benchmark scores were available, we relied on other measures of output quality, such as user preference testing.

There are no standardized benchmarks for frontier image and video models, meaning we could not use benchmark scores to search for these models. For example, recent video models such as Veo 2 were evaluated in their release materials by human preference of their outputs versus competitors. To address this, we individually examined the leading video generation and drawing services. We used user preferences and developers’ compute resources to assess whether such models might exceed the 10²⁵ FLOP threshold.

Data

We have identified 25 publicly disclosed models estimated to have been trained with at least 10²⁵ FLOP. Our database of Notable AI Models documentation describes how we estimate training compute. Where possible, we use details such as parameter count and dataset size, or training hardware quantity and duration. For models without reported details, we instead estimate training compute based on benchmark performance. We discuss this further under Assumptions.

We do not separately count different releases built on top of models using 10²⁵ FLOP. For example, we do not count different Llama-3.1 405B variants and finetunes as separate 10²⁵ FLOP models. Implementation of the EU AI Act is ongoing, but current discussions suggest regulation of 10²⁵ FLOP models will not separately cover finetunes, so long as the resulting model does not “significantly modify” the risk level of the base system (see Section 1.1 of Novelli et al., 2024).

Models included in this dataset:

Additionally, we identified 19 models which we believe to be under 10²⁵ FLOP, but cannot confidently rule out from being over the threshold. These include models where our estimates’ confidence intervals overlap the 10²⁵ FLOP threshold, and models where we had insufficient details to make a compute estimate, but suspect the model might exceed 10²⁵ FLOP based on its leaderboard ranking and developer’s resources.

Assumptions

For several AI models, developers provided insufficient information for us to directly estimate training compute. In this case, we estimated training compute from model performance. For language models, we did this using imputation based on benchmark scores. For image and video models, where fewer standardised benchmark scores were available, we relied on other measures of output quality, such as user preference testing.

Estimating compute from benchmark performance is more speculative than direct calculation from training details, but benchmark performance and training compute are correlated. This can provide valuable evidence when developers don’t provide enough information to apply other approaches. For the candidate language models we considered, we followed a straightforward methodology:

1. Collect reported performance on benchmarks such as MMLU, GPQA, the SEAL Leaderboards, etc. We collected this for several AI models with known compute, in addition to the ones we were trying to estimate.
2. Fit a per-benchmark sigmoid curve for the relationship between training compute and benchmark performance. Algorithmic efficiency improves over time, so we limited the “ground truth” datapoints to models that were at least as compute-efficient as the Llama-3 family, i.e. with higher benchmark performance versus compute than the fir for Llama-3 models only. The models for which we were estimating compute (e.g. Claude 3.5 Sonnet, Gemini Pro 1.5, GPT-4o) were likely to be more compute-efficient than the Llama-3 series.
3. Estimate training compute given benchmark performance for the models without known training compute. For each model, our estimate is the median compute estimate across all available benchmarks.
4. We did not use this approach to estimate compute for “reasoning” models such as o1, as they may show a different relationship between training compute and benchmark performance.

To assess this method’s accuracy, we performed leave-one-out cross-validation across several different AI models with known training compute. Benchmark-derived estimates of compute are, on average, within a factor of 2.2x of ground truth. Hence they should be treated as fairly speculative, but nevertheless provide useful information for estimating training compute.

Image and video generation models at this scale could not be identified using benchmark scores, as they often do not report benchmark scores or report on different benchmarks between models. Instead, we examined the qualitative performance of leading AI image and video products since 2023 to find models likely to be above 10²⁵ FLOP.

To guess the compute scale of image generation models, we compared against Stable Diffusion 3, with training compute reported less than 10²³ FLOP. Image generation models released within a year, with worse performance, are likely to have used less training compute, and are therefore not included in this list. To guess the compute scale of video generation models, we compared against Meta Movie Gen, with training compute reported slightly under 10²⁵ FLOP. Video generation models released within a year, with worse performance, are likely to have used less training compute and are therefore not included in this list.

OpenAI’s Sora and Google DeepMind’s Veo 2 have shown superior qualitative performance to Meta Movie Gen, including in user preference testing. Along with the compute resources available to OpenAI and Google DeepMind, their performance suggests the models are likely to have been trained with more than 10²⁵ FLOP. Based on available GPU resources and performance, we believe that other leading video generation models were not trained with more than than 10²⁵ FLOP, although Kling-1.5 performs similarly to Meta Movie Gen, and we include it in the table of models which we cannot confidently rule out.

We searched for models over 10²⁵ FLOP in other domains, including audio generation, biology, and game-playing AI. However, we didn’t identify any publicly disclosed models at this scale of training compute.