We are holding a series of workshops to develop problems for our benchmark FrontierMath: Open Problems. The goal is to address two pressing issues in AI math benchmarking.

• Grading AI proofs is extremely laborious. FrontierMath: Open Problems focuses on constructive problems where solutions can be verified programmatically.
• AI math achievements to date have been hard to contextualize. FrontierMath: Open Problems focuses on open problems of real interest to working mathematicians, with problem significance characterized on a uniform scale.

This is a chance to focus AI resources on problems you’d like to see solved — including very interesting, very hard problems.

See the pilot problem set for examples.

Logistics

Where: San Francisco, Los Angeles, Toronto, Princeton, NYC, Boston, London
When: late May and early June (see below)

Workshops will feature an opening talk, small group brainstorming, large group discussion, and time to draft problem proposals. Lunch and coffee will be served.

Collaboration is encouraged. Participants will work in groups and may propose multiple problems. Afterward, our selection committee will decide which problems to include in the benchmark. A prize of $1,000 will be awarded for each selected problem, split among the group that proposed it. Proposers will also have the opportunity to contract with Epoch to write up the problem and develop the solution verifier code.

Attendees will receive a three-month subscription to ChatGPT Pro. Regional travel (e.g., train) will be reimbursed.

Benchmark problems will be public. Verifier code will be made available for purchase to AI companies. These companies must disclose any solutions they find, and the mathematicians who contributed the problem will have joint publication rights. Learn more about verifier access.

Sign up

Attendance is by application. All working mathematicians (grad student, postdoc, professor) are encouraged to apply. Applications will be reviewed on a rolling basis while space remains.