Back to models

Gemini 1.5 Pro (Sept 2024)

ECI#146
132
Google DeepMind
Sep. 24, 2024
Closed weights

Gemini 1.5 Pro (Sept 2024) is a closed-weights model from Google DeepMind, released September 24, 2024. It has an ECI score of 132 and ranks 146th of 249 tracked models. At the time of its release, it ranked 3rd of 89 tracked models.

Input price—
Output price—
ParametersUnknown
Training computeUnknown

Gemini 1.5 Pro (Sept 2024) vs select alternatives

Reasoning
DomainBenchmark
Gemini 1.5 Pro (Sept 2024)
Google DeepMind
Sep. 24, 2024
Closed weights
Gemini 1.5 Pro (May 2024)
Google DeepMind
May 14, 2024
Closed weights
o1-mini
OpenAI
Sep. 12, 2024
Closed weights
Claude 3.5 Sonnet
Anthropic
Jun. 20, 2024
Closed weights
ECI scoreBest score 166
132127136130
Ranking#146#169#130#151
Conceptual reasoning
SimpleBenchBest score 81.9%
27.1%
n/a
18.1%
27.5%
ARC-AGI-2Best score 95.0%
0.8%
n/a
0.8%
n/a
DTBenchBest score 98.4%
59.0%
56.5%
n/a
67.8%
Mathematics
MATH Level 5Best score 98.1%
70.4%
40.7%
89.2%
51.7%
OTIS Mock AIME 2024-2025Best score 100.0%
23.1%
6.8%
46.9%
6.5%
Science
GPQA DiamondBest score 95.8%
57.2%
45.9%
62.4%
54.0%
Software engineering
WeirdML (v2)Best score 92.9%
22.2%
n/a
36.3%
31.0%
World knowledge
MMLUBest score 88.1%
86.9%
85.9%
n/a
86.5%

Model metadata

General
OrganizationGoogle DeepMind
Release dateSep. 24, 2024
Model accessibilityClosed weights
CountryUnited States of America
Knowledge cutoff2024-05-01
Provider notesGemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Pricing
Input price—
Output price—
Training
ParametersUnknown
Training computeUnknown
Training hardwareGoogle TPU v4
Training timeUnknown
Training dataset sizeUnknown