BTF-3 (Bench to the Future 3), created by FutureSearch, evaluates probabilistic forecasting agents by “pastcasting”: agents forecast 1,907 real, already-resolved questions — 1,515 binary and 392 numeric — while researching against a frozen snapshot of the internet, so the answers cannot be looked up. Because the questions have known resolutions, forecasts can be scored immediately and precisely, without waiting for events to resolve.

We source results from the public FutureSearch evals page. Our chart reports the pooled score on the Brier scale, a weighted average of the binary Brier score and the numeric ranked probability score, where lower is better and 0 is a perfect forecast. The binary and numeric subscores and 95% confidence intervals are kept in the data export.

Agents research each question against FutureSearch’s frozen offline web corpus before forecasting. The same model can appear both under FutureSearch’s own forecasting agent and self-driving through the vendor’s agent SDK, and we keep those as separate entries distinguished by harness. FutureSearch’s “SOTA” row is an ensemble of their agent runs rather than a single model, so it is excluded.