Back to models

Gemini 1.5 Pro (Sept 2024)

ECI#156
130
Google DeepMind
Sep. 24, 2024
Closed weights

Gemini 1.5 Pro (Sept 2024) is a closed-weights model from Google DeepMind, released September 24, 2024. It has an ECI score of 130 and ranks 156th of 240 tracked models. At the time of its release, it ranked 3rd of 78 tracked models.

Gemini 1.5 Pro (Sept 2024) vs select alternatives

DomainBenchmark
Gemini 1.5 Pro (Sept 2024)
Google DeepMind
Sep. 24, 2024
Closed weights
Gemini 1.5 Pro (May 2024)
Google DeepMind
May 14, 2024
Closed weights
o1-mini
OpenAI
Sep. 12, 2024
Closed weights
Claude 3.5 Sonnet
Anthropic
Jun. 20, 2024
Closed weights
ECI scoreBest score 173
130125135130
Ranking#156#173#140#157
Conceptual reasoning
SimpleBenchBest score 95%
27%
n/a
18%
28%
ARC-AGI-2Best score 95%
1%
n/a
1%
n/a
DTBenchBest score 99%
59%
56%
n/a
68%
Mathematics
MATH Level 5Best score 98%
70%
41%
89%
52%
OTIS Mock AIME 2024-2025Best score 100%
23%
7%
47%
7%
Science
GPQA DiamondBest score 97%
57%
46%
62%
54%
Software engineering
WeirdML (v2)Best score 94%
22%
n/a
36%
31%
World knowledge
MMLUBest score 88%
87%
86%
n/a
87%

Model metadata

General
OrganizationGoogle DeepMind
Release dateSep. 24, 2024
Model accessibilityClosed weights
CountryUnited States of America
Knowledge cutoff2024-05-01
Provider notesGemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Pricing
Input price—
Output price—
Training
ParametersUnknown
Training computeUnknown
Training hardwareGoogle TPU v4
Training timeUnknown
Training dataset sizeUnknown