LLM responses to benchmark questions are getting longer over time

Our data comes from Epoch’s Benchmarking Hub. We focus on model responses to GPQA Diamond, MATH level 5 benchmarks, and OTIS Mock AIME 2024-2025, since these have the most coverage and elicit similar response lengths. Across both benchmarks, we have data on 77 unique models, evaluated a total of 235 times. Of these models, 11 are reasoning models, 65 are non-reasoning models, and 1 (Claude 3.7 Sonnet) is evaluated both with and without extended reasoning. For reasoning models, we include thinking tokens in the total token count.

Some reasoning models are offered with varying levels of “effort”. For these models, we used only benchmark runs with the highest level of reasoning effort each model is capable of. Across OpenAI’s reasoning models, moving from “medium” to “high” effort resulted in a 1.6x increase in output tokens.

We perform log-linear regressions of token output length on time, with separate models for reasoning and non-reasoning models. To generate confidence intervals, we bootstrap sample with replacement (n=500), then identify the 5th and 95th percentile slopes and predictions across samples. Our results were as follows, with 90% confidence intervals in parentheses:

While results for our non-reasoning model subset are quite stable, the slope for non-reasoning models is quite sensitive. Omission of individual models can shift the overall trend by significant amounts.

• We assume that exponential growth is an appropriate model for the trend in output response length. To test this, we compared against a linear model, and found that for both reasoning and non-reasoning models, the exponential fit was weakly (but not statistically significantly) preferred to the linear fit.
• We assume that trends in output length for the three benchmarks we focused on are indicative of trends among other benchmarks. We intend to examine results for other internal benchmark evaluations as they become available.