Search epoch.ai
MATH Level 5
MATH Level 5 is a subset of the broader MATH dataset, which comprises problems from various mathematics competitions including the AMC 10, AMC 12, and AIME. As described in the paper: “These competitions span decades and assess the mathematical problem-solving ability of the best young mathematical talent in the United States. Unlike most prior work, most problems in MATH cannot be solved with a straightforward application of standard K-12 mathematics tools.”
The full MATH dataset contains 12,500 problems (7,500 training and 5,000 test). The problems in MATH are assigned difficulty levels from 1 to 5. MATH Level 5 specifically consists of the 1,324 level 5 questions from the MATH test set, representing the most challenging problems in the dataset.
The MATH dataset is widely used and reported on by model developers when evaluating their models’ mathematical reasoning capabilities. We selected the Level 5 subset for our evaluations because it’s not yet saturated by current models, allowing us to focus on the most informative and challenging questions without having to run all 5,000 test questions.
Methodology
We use the following prompt:
Solve the following math problem step by step. The last line of your response should be of the form “ANSWER: $ANSWER” (without quotes) where $ANSWER is the answer to the problem.
{question}
Remember to put your answer on its own line at the end in the form “ANSWER: $ANSWER” (without quotes) where $ANSWER is the answer to the problem, and you do not need to use a \boxed command.
If the model’s answer is not of the form “ANSWER: $ANSWER”, it receives no points for that question. We use three distinct scorers to assess model answers:
normalized_string_match: A normalized string match scorer compares the model’s answer to the target using simple rules. For example, it identifies “\frac1b” and “\frac{1}{b}” as the same answer.sympy_equiv: A symbolic equivalence (sympy) scorer extends beyond string matching by recognizing mathematically equivalent expressions. For example, it correctly identifies “5/2” and “2.5” as the same answer.model_graded_equiv: A model-based scorer employs gemini-1.5-flash-002 to evaluate whether the model’s answer matches the target answer.
Our plots use the model_graded_equiv scorer, except where explicitly noted otherwise.
