Domain-specific ECIDomain-specific ECI
Overview
Domain-specific ECIs show how LLMs’ ECI scores are influenced by the selection of benchmarks used to calculate them.
The domain-specific ECI is based on the general ECI but only incorporates benchmarks from a specific domain, such as software engineering (SWE) or math. It combines the benchmark parameter estimates (benchmark difficulty and slope) from the general ECI, which are calculated through a Bayesian model, and then updates the model capability parameters, restricted to the LLMs’ scores on selected benchmarks. See the methodology section below for the full details. The Cyber ECI is an exception, as it also incorporates cybersecurity benchmarks that are not part of the general ECI; see the Cyber ECI section below.
The results are given on a scale comparable to the general ECI, so if an LLM has a higher Math ECI than general ECI, this means it performs better on math benchmarks than non-math benchmarks. However this scaling means that the results cannot be used to assess long term progress trends in different domains, as they are all scaled to increase at the same overall pace as the general ECI (but it is possible to detect one-off capability jumps, or for e.g. one family of LLMs to go from underperforming to overperforming within a domain).
Methodology
While the benchmark parameters are taken directly from the general ECI outputs, the re-fitting of the model capability is based on the previous, frequentist methodology for the ECI. This is done to ensure the resulting values are comparable to the general ECI values, but means that the results cannot be used to assess long term progress trends in different domains. For the Cyber ECI, this same procedure is applied starting from a “frequentist + cyber” joint refit rather than the published general ECI fit; see the Cyber ECI section below.
See the general ECI documentation for the Bayesian setup that we get the benchmark parameter values from. Below we outline the previous, frequentist approach that we use to update the LLM capability parameters, restricted to the domain-specific benchmark scores.
The core of the model is a simple logistic function:
\[\textit{predicted score}(m,b) = \sigma(\alpha_b[C_m - D_b]) = \frac{1}{1 + e^{-\alpha_b[C_m - D_b]}}.\]Where \(C_m\) is the model’s capability, \(D_b\) is the benchmark’s difficulty, and \(\alpha_b\) is a slope parameter related to the distribution of difficulty across questions within the benchmark. Each value of these parameters gives an LLM m’s predicted score on a benchmark b. We fit the model by finding the best values of the capability parameters \(C_m\) that minimize the squared error of the predicted score against the observed benchmark scores in our dataset, while keeping the benchmark parameters \(D_b\) and \(\alpha_b\) fixed from the general ECI posterior means. We also include a small regularization term to improve fitting efficiency. The full equation that we optimise is:
\[\begin{aligned} \min_{C_m}\ & \sum_{n=1}^{N} \left(\textit{predicted score}_n - \textit{observed score}_n\right)^2 \\ & + \frac{0.1}{M + 2B - 2}\left(\sum_{b=1}^{B} \left(\alpha_b^2 + D_b^2\right) + \sum_{m=1}^{M} C_m^2\right). \end{aligned}\]Here N is the total number of observations we have (one model score on an individual benchmark), B is the total number of benchmarks, and M is the total number of models. The scale of the resultant values is arbitrary; we rescale so that Claude 3.5 Sonnet is fixed at 130, and GPT-5 is fixed at 150, in order to allow for consistent scoring over recent models in a way that balances communicating our uncertainty with providing detailed information.
To report 90% uncertainty intervals, we first take a posterior draw from the general ECI, then generate 10 bootstrap samples by sampling with replacement for each LLM from its observed benchmark results within the domain-specific subset. We typically have 1,000 posterior draws, which yields 10,000 bootstrap samples in total. Keeping the benchmark difficulty and slope parameters from the posterior draws fixed, we refit the model capability using the minimisation above to obtain new model capability estimates. We then use the 5th and 95th percentiles of these bootstrap refits to construct the confidence intervals.
Cyber ECI
The Cyber ECI is constructed differently from other domain-specific ECIs, because the general ECI’s benchmark pool contains almost no cybersecurity benchmarks. We first refit a new “frequentist + cyber” version of the ECI, using the previous, frequentist approach and incorporating 34 additional cyber benchmarks, then follow the standard domain-specific methodology described above, starting from that joint fit. The general ECI values shown alongside the Cyber ECI are still the published general ECI values, and some LLMs covered by the cyber data (e.g. Claude Mythos Preview (Early)) have a Cyber ECI but no general ECI. Because the Cyber ECI starts from this separate joint fit, it may lag behind updates to the general ECI. The joint fit provides 200 bootstrap samples of its benchmark parameters, so cyber CIs are based on 2,000 total samples.
Data
Since the domain-specific ECI incorporates the benchmark parameters from the general ECI, it is influenced by all the data covered in its data section.
The processed benchmark results used to calculate the Cyber ECI are available in CSV format.
For which benchmarks are categorized as SWE or Math, see the benchmark hub.
FAQs
See general ECI FAQs for general information about the ECI
Why do some LLMs have a general ECI but not a Math ECI or SWE ECI?
We require at least 2 benchmarks within a domain to calculate its domain-specific ECI to avoid overly noisy results. Some LLMs might pass the 4-benchmark minimum for inclusion in the general ECI but don’t have enough to be given a domain-specific ECI.
Why do some LLMs have a Cyber ECI but no general ECI?
Unlike the other domain-specific ECIs, the Cyber ECI incorporates cyber benchmarks that are not part of the general ECI. An LLM with enough cyber benchmark results can therefore be given a Cyber ECI even if it does not meet the 4-benchmark minimum for a general ECI. These LLMs are marked with a “No General ECI” note in the explorer. See the Cyber ECI section above for more details.
How should I interpret the results? e.g. What does it mean for an LLM to have a Math ECI of 160 but a general ECI of 155?
This means that the LLM’s performance on math benchmarks is what we would expect from an LLM with general ECI 160, but it performs less well on other benchmarks, resulting in a general ECI of only 155.
Do the results mean that math and software engineering capabilities have been improving at the same rate as the general ECI?
No, the methodology we use causes the domain-specific ECI values to all be scaled to the same level as the general ECI. See the methodology section for details.
Did you add additional math or software engineering benchmarks to construct the SWE ECI or Math ECI?
No, the benchmarks used to calculate the math and SWE domain-specific ECIs are a subset of the benchmarks used to construct the general ECI. The Cyber ECI is the exception: it incorporates many cyber benchmarks that are not part of the general ECI (see the Cyber ECI section above).
Acknowledgements
We thank Alexander Barry for his work on the domain-specific ECI. For acknowledgements for the overall ECI see here.