Included benchmarks

For the initial release of Benchmark Reviews, we attempted to create a representative sample of benchmarks across error types and domains. Benchmarks that require review by experts with domain knowledge take additional time as we hire external subject matter experts.

We plan to prioritize reviews with the following characteristics going forward:

• Impact: Evaluates a task we care about and/or safety-relevant capabilities
• Reach: Widely cited, in recent system cards, or underrated
• Diversity: Covers an under-covered capability in our current suite of benchmark reviews

Many benchmarks do not publicly release enough information for us to be able to sufficiently evaluate them against our rubric. These will be marked as Not enough information. For benchmarks where we have been given access to private information, we will disclose what we were able to evaluate. In order to avoid leakage, we will only include generic information in these reviews, such as scoring error types without task-level analysis.

We will not review Epoch-created benchmarks due to conflict of interest. This does not mean we think our benchmarks are without errors¹. We may commission external review of our benchmarks in the future. We generally do not expect to be compensated for reviewing benchmarks, and we will clearly note situations where this is not the case.