Biology AI Models Documentation

Inclusion

The dataset focuses on biological ML models: those that are trained on biological data, including biological sequences, molecular structures or data about molecular properties, among others. Here, we detail criteria for inclusion, and give an overview of how the data have been collected.

Criteria

To be included in the dataset, an ML model must satisfy all inclusion criteria:

• there must be reliable documentation of its existence and relevance to machine learning;
• the model must include a learning component and cannot be a non-learned algorithm;
• the model must have been trained, it cannot be a theoretical description without experimental results;
• the model must be directly and explicitly trained on biological data, including:
  • biological sequence data;
  • biomolecule structure data;
  • fitness, pathogenicity or other biological properties of proteins or other biomolecules;
  • cell-level data (cell type, expression levels, spatial or imaging data…).

Search process

This data has been collected mainly from a literature review, although some models have been collected from other sources like high-profile models from leading industry labs, bibliographies of notable papers, and ad hoc suggestions from contributors.

Coverage

As of February 21, 2025, the dataset contains 350 models, 154 of which have compute estimates. The dataset does not provide exhaustive coverage of biological models. We attempt to cover the most historically relevant models, as well as significant models released in 2023 and 2024, but we expect some important models to be missing from our dataset.

Estimation

Some fields within the database require estimation, because they are often not straightforwardly reported within papers or other sources. Here, we detail how estimation works for compute, model size, dataset size, and the metadata on estimate confidence.

Estimating compute

Training compute is one of the most important pieces of information in our dataset, as reflected in its usage across Epoch AI’s research and elsewhere. However, estimating compute can be challenging. Here we outline how compute estimation is performed in the notable models dataset.

Compute is measured in units of floating point operations (FLOP). For older models, sometimes the relevant operations were integer operations - in this case we report these instead. We do not apply any multiplier to adjust for operations potentially being more valuable under different tensor formats or precisions, for example FP16 versus FP32 or BF16. Some sources report compute in multiplications-and-addition operations, fused multiply-adds (FMAs), or similar. We treat one multadd/FMA as being equivalent to two FLOP to match typical reporting of chip performance.

For a given model in the database, training compute is provided as the total training compute, including pretraining, and including pretrained base models used as components. Finetuning compute is recorded in its associated column. Finetuning is distinguished by authors’ descriptions of the training as finetuning, or unambiguous use of a pretrained model in a distinct phase of training.

In the simplest case, training compute is directly reported in a paper, and we enter this figure into the database. When compute is not reported, we use two main methods to estimate it:

1. Hardware details and usage.
2. Counting the operations based on model architecture and data.

When there is enough information to count the operations, this is preferred in our dataset, because typically hardware-based estimates require assumptions about utilization, which may reduce the estimates’ accuracy.

Hardware details and usage

Hardware details and usage is a relatively straightforward way to estimate compute, when the necessary details are known:

1. The usage in chip-time, e.g. “trained on a cluster of 128 TPUv3 instances for two days” means 256 chip-days = 128 chips × 2 days. Sometimes this is reported as separate chips and time used, other times this may be reported directly in chip-time. When it is not reported, we may create estimates from publicly-known information, comparison to typical training runs, etc.
2. The type of hardware used, e.g. NVIDIA H100, TPUv3. Ideally, this is reported in the paper. Otherwise, for more speculative estimates, one may have to make assumptions based on institution and year, e.g. that Google would have used TPUs of the corresponding generation in that year.
3. The type of number representation used, e.g. FP32, FP16, BF16. Ideally, this is reported in the paper. When not reported, it can often be guessed. For example, the number representation was typically FP32 for models trained before 2019.

Once these details are known, the corresponding peak FLOP/s performance by hardware and number representation can be found from hardware documentation, or from the tool below. Finally, utilization rates account for real training runs falling significantly short of peak performance due to memory bottlenecks, network latency, etc. Typical utilization rates for large distributed training runs are around 30-50%. When these are not reported, they are estimated by reference to comparable models from a similar time period.

Counting the operations

Counting the number of operations is often useful for older research, where hardware and usage details might be unavailable. The above formula sets out a widely-applicable heuristic for training compute of dense models. This works by first estimating required FLOP for a forward pass, which is approximately twice the number of connections. This can be modified by sparsity such as Mixture-of-Experts: in this case, the heuristic should use the number of connections in the number of active experts.

The forward pass FLOP is then multiplied by three to account for backward passes, as the ratio between forward and backward passes is 1:2 for non-recurrent dense models. Finally, this is multiplied by the number of passes performed on the data - the number of training examples multiplied by the number of epochs the model was trained. For transformer-based language models, this formula is equivalent to the commonly-used heuristic: Compute = 6 × # of parameters × # of training examples × # of epochs.

Sometimes, the FLOP for a forward pass is reported directly in a paper. In this case, this value can be used directly instead of 2 × # of connections. Otherwise, the FLOP for a forward pass are evaluated by summing FLOP over the network’s layers. These are set out in Table 3.

Benchmark performance

When details about model architecture, training data, hardware, and development time are scarce, it may be informative to compare the model’s performance on benchmarks to that of other models. Scaling laws can predict benchmark performance improvements against compute when scaling a given model family (for example coding performance for GPT-4 scaling and ARC Challenge for Llama-3). When there are differences in model/data/training, benchmark performance is less predictable from compute, but nevertheless remains correlated.

This process of estimating training compute from benchmark performance can be improved by aggregating performance across many benchmarks, especially when several or many models with known training compute have been evaluated on those benchmarks.

The procedure is roughly as follows:

• Collect a dataset of many benchmarks (e.g. MMLU, GPQA, BigCodeBench) and models (e.g. Llama 3, Mistral Large, Nemotron 4) with known training compute and benchmark scores.
• For each benchmark, fit a curve that best matches the benchmark scores of each model as a function of their training compute.The curve-fitting procedure uses sigmoid functions, based on our study How Predictable is Language Model Benchmark Performance?
• Using the fitted curves, impute an x-value (training compute) from the y-values (benchmark scores) for models with unknown training compute. This represents the approximate training compute necessary to achieve the benchmark scores demonstrated by those models.
  • The resulting estimates are cross-validated to ensure that the fitted values are reasonable even if some benchmark evaluation datapoints are held out.
• If applicable, combine any other information about the compute resources available to the developers with the evidence from the benchmarks, to obtain overall estimates of the compute used to train the models.
• It may be helpful to constrain fitting to models with the most similar algorithmic efficiency. For example, when using this approach to collect information on leading LLMs from 2024, we constrained fitting to models with algorithmic efficiency similar to, or better than, the Llama 3 family.

This process is demonstrated in a public Colab notebook, Compute Estimation from Benchmark Scores. Because these compute estimates are already based on benchmark performance, they should be excluded from analyses of the relationship between benchmarks and compute. Such compute estimates can be filtered using the Training compute estimation method field.

Estimating model size

Parameter counts are often reported by the model developer, but if parameter count is not stated, it can sometimes be estimated based on provided architectural details. Similar to estimating compute, estimating parameter count requires finding a description of the architecture, i.e. type, number, and configuration of the layers, then calculating the parameters in each layer and summing them. Table 5 lists the parameter counts for different layers. Alternatively, if an architecture implementation is available in code, it can be simpler to load an architecture in code and count the number of parameters.

Estimating dataset size

Dataset size is measured in the number of datapoints used as training examples, as outlined in Table 7. The objective here is to provide an intuitive idea of how large a dataset is. We record the number of distinct datapoints, not multiplied by the number of epochs. The number of epochs is recorded separately. Table 8 provides worked examples across several of these ML problems.

Language dataset sizes are usually reported in terms of tokens or gigabytes. These are converted to words per Table 9. Language dataset sizes are usually reported in terms of tokens or gigabytes. These are converted to words per Table 9. These factors are based on the OpenAI GPT-3 tokenizer for Western languages, and manual inspection that tokenizers typically have one token per word in Mandarin, Japanese and Korean. The ratio is tokenizer-dependent, meaning that when estimating a dataset’s size in words, one should consider whether the tokenizer might have a substantially different ratio. Speech recognition data are often expressed in terms of duration or syllables, which are converted to words per Table 10. Speech recognition data are often expressed in terms of duration or syllables, which are converted to words per Table 10.

Estimating power draw

The field “Training power draw (W)” contains the power draw of the hardware used to train the model, measured in watts. This field is filled in when the training hardware type and quantity are known, and is calculated as follows:

The power draw is calculated for the processing hardware (GPUs or TPUs), by multiplying the TDP (thermal design power) of the processor by the number of processors.

The power draw of the computing hardware is then scaled up to account for overhead due to networking equipment and data center efficiency:

• We multiply by 2.03x to account for non-GPU server hardware (networking, switches, and CPUs), based on the specifications of NVIDIA DGX H100 servers.

• We also multiply by data center PUE (power usage effectiveness) to obtain the total power drawn from the grid while training the model. PUE uses an average value of 1.09x in 2024 and this value decreases by about 4% per year, based on a quarterly historical PUE record of Google data centers.

Estimating confidence

As discussed in Records, the confidence statuses specify the following bounds as 90% confidence intervals:

• Confident - ±3x (~0.5 orders of magnitude).
• Likely - ±10x (1 order of magnitude).
• Speculative - ±31x ( ~1.5 orders of magnitude).

Confidence applies to the recorded values for Training compute, Parameters, and Training dataset size. It describes confidence for the most uncertain of these values, for the ones that have a non-empty entry.

To estimate confidence statuses, we consider which parts of an estimate are uncertain, and how large the uncertainty is.

• If details (compute, model size, and dataset size) are all directly reported, then the value is Confident. There is little room for error.
• If a detail is estimated without any assumptions having to be made, then the value is Confident. For example, if hardware type, quantity, training time, number format, and utilization are all reported, then the ensuing compute estimate is unambiguous.
• When details are ambiguous, and an assumption has to be made, we consider the uncertainty in that assumption.
• For example, it is often necessary to estimate utilization when estimating training compute from hardware details. Given the typical language modeling range is 0.3-0.5, this estimate should fall within the Confident category.
• Further ambiguity may move estimates into the Likely category. For example, MedBERT was trained for one week using one V100 GPU, but the authors do not report the arithmetic precision or usage of tensor cores during training, which could affect the compute usage by a factor of 4x.
• Finally, some estimates are based almost entirely on credible ranges for (unreported) key parameters such as training time and hardware. These typically fall into the Speculative category. An example of this is GPT-4, where our compute estimate is based on secondhand reporting that lets us roughly estimate training duration and hardware.