We draw the underlying changes from OpenAI’s public Codex repository. The repository is squash-merged, so each commit on the default branch whose subject ends in “(#N)” corresponds to one merged pull request. We analyze 7,524 distinct merged pull requests from April 2025 through June 2026, contributed by roughly 500 distinct GitHub accounts. We take each PR’s net diff from a local git checkout, and fetch its branch commits’ authors and dates from the GitHub GraphQL API.

While the Codex repository features contributions from 497 distinct accounts, we focus our analysis on a subset of 41 contributors who appear as official repository collaborators, defined by a GitHub author_association value of “COLLABORATOR (write access)” on their pull requests. We exclude automated accounts like dependabot and github-actions.

We also systematically remove PRs whose estimate reflects code not authored in this repository: wholesale imports (code developed elsewhere and dropped in as a single commit, such as the initial import of the Rust Codex implementation) and relands (code re-applied from an earlier reverted PR). Because the LLM judges only see one PR at a time and cannot see that code in these PRs was authored elsewhere or already counted, their inclusion would overstate the amount of work done. This deterministic, title-based rule removes six of the 7,524 PRs and leaves the trend unchanged.

We score each PR with an ensemble of three frontier models (Claude Opus 4.8, GPT-5.5, and Gemini 3.1 Pro), run without extended reasoning through each provider’s batch API, and take the per-PR median. For each estimate, the model reads the PR title, description, per-file line counts, branch commit messages, and a sample of the diff distributed evenly across changed files, then emits a single time estimate. A PR’s hours are split evenly across the contributor-days its commits touch, and each contributor-day sums the allocations landing on it. To get a contributor’s effort level on any given day, we total each contributor’s estimated hours by Pacific-time calendar day, forming contributor-days (distinct author-and-day pairs).

The three models agree closely on ordering (pairwise log-scale correlations of 0.95-0.98) but less on scale: on the median PR, the highest of the three estimates is 2.5 times the lowest. See the “Assumptions and limitations” section below for details on ablations to our LLM judge methodology.

We bucket contributor-day hours into the following categories: <6 hours, 6 to 12 hours, 12 to 24 hours, 24 to 48 hours, and ≥48 hours. Estimates falling exactly on the border of two buckets are sorted into the higher bucket.

Among the 41 collaborators, the share of working days reaching the upper effort levels rises across the period:

* Partial quarters: the repository launched in mid-April 2025, and our data runs through mid-June 2026. Intervals are 90% Wilson score intervals, treating each contributor-day as an independent observation.

While our results are suggestive of AI uplift, they fall short of causal evidence. We lack a counterfactual comparison for the time it would have taken a non-AI assisted software engineer, and thus rely on noisy estimates from LLM judges. To reduce noise, we use the median result as a rough estimate, and group them into broad buckets.

Further, the implicit uplift shown here corresponds to an upper bound on time saved. Without AI, engineers wouldn’t replicate the observed output – they would build less, or build differently. Since some of the code written could have been prohibitively costly to replicate exactly without AI, the gap between actual and “no AI assistance” time costs overestimates the actual gain in productivity. See METR’s post “Task Substitution and Uplift” for a more detailed analysis of this effect.

To identify core collaborators on the Codex repo, we rely on GitHub author-association flags and identity signals. Contributors whose repository status changed over time may be misclassified. We also attribute a PR’s working hours evenly across its contributor-days, inferring those days from branch-commit timestamps. We lack a source of ground truth to validate this approximation against.

Compared to METR’s original method, we use pull request diffs and metadata rather than full agent transcripts. As such, METR’s validation of the approach – which found that LLM predictions were consistently within a factor of 2 to 3, when compared against human estimates – does not transfer directly to our numbers.

We tested two interventions to the judging infrastructure, which had minimal effect: extended thinking, and using more fully featured scaffolding. Across 20 randomly selected PRs, using high thinking effort changed the median estimate by only 10.1%. After bucketing results into categories of 0–6, 6–12, 12–24, 24–48, and 48+ hours, the assigned bucket changed only once out of those 20 cases. In that case, the estimate rose from 5.5 hours to 6.0 hours. We also produced estimates for the same 20 PRs with high thinking effort, file reading tools, and access to the full PR diff rather than a sample. Relative to the high thinking test, the additional tools and context resulted in a median change of 0% across PRs.