Claude 3.5 Sonnet (October 2024)
ECI#149
133Claude 3.5 Sonnet (October 2024) is a closed-weights model from Anthropic, released October 22, 2024. It has an ECI score of 133 and ranks 149th of 240 tracked models. At the time of its release, it ranked 3rd of 80 tracked models.
Claude 3.5 Sonnet (October 2024) vs select alternatives
| Domain | Benchmark | Claude 3.5 Sonnet (October 2024) | |||
|---|---|---|---|---|---|
ECI scoreBest score 173 | 133 | 130 | 135 | 130 | |
| Ranking | #149 | #157 | #140 | #156 | |
| Conceptual reasoning | SimpleBenchBest score 95% | ||||
| Mathematics | MATH Level 5Best score 98% | ||||
OTIS Mock AIME 2024-2025Best score 100% | |||||
| Science | Humanity's Last ExamBest score 55% | ||||
GPQA DiamondBest score 97% | |||||
| Software engineering | METR Time HorizonsBest score 17 hr 25 min | 20 min 31 sec | 11 min 24 sec | — | — |
Aider PolyglotBest score 88% | |||||
WeirdML (v2)Best score 94% | |||||
| Other | LiveBenchBest score 82% |
