ForecastGym: A Live, Controlled, and Replayable Benchmark for Forecasting Agents
Organizations: Carnegie Mellon University Pittsburgh, Pennsylvania, USA · Jilin University Changchun, China
Abstract
Agents built on large language models (LLMs) increasingly forecast real-world events by searching the Web and reasoning over what they find. How well such an agent forecasts depends on both the retrieved evidence and the model, making control over that evidence essential for evaluation. Live evaluations can mix retrieval with reasoning when each system searches on its own, while retrospective search can leak later information through date filters or today's ranking of historical pages. Comparing models, search sources, and memory strategies requires both fixed evidence and a record of how it evolved before resolution. We introduce ForecastGym, which records what four commercial web search APIs return for open prediction-market questions every two days and replays this frozen evidence offline once outcomes are known, comparing models, memory, and captured search sources under a fixed evidence-acquisition policy. Its first release covers 150 questions, 614 evidence states, one per question and collection time, and 131k search results. Rerunning the same queries after settlement returns results that all but state the outcome for up to 92% of the questions, despite the tested date filters. On the frozen evidence, all tested LLMs trail the market in forecasting accuracy. Our diagnostic analysis suggests that missed public information contributes more to the gap in high-gap states than misreading does. Switching the search API changes a model's Brier score by up to 0.043, more than the spread across models, while persistent notes score similarly to rereading all evidence at half the input tokens. The evidence a forecaster sees can matter as much as the model that reads it.
Figures & tables
| Benchmark | Live | Controlled | Replayable | Source-aware | |
|---|---|---|---|---|---|
| Open | Reusable | Multi-state | Memory | Providers | |
| ForecastQA ( Jin et al., 2021 ) | – | ✓ | – | – | – |
| ForecastBench ( Karger et al., 2025 ) | ✓ | – | – | – | – |
| FutureX ( Zeng et al., 2025 ) | ✓ | – | – | – | – |
| Prophet Arena ( Yang et al., 2025 ; Prophet Arena, 2026 ) | ✓ | ✓ | – | ✓ | |
| BTF ( Wildman et al., 2025 ) | – | ✓ | – | – | – |
| Argument | Variant | Evidence | Memory |
| Main | |||
| Model | Four LLMs | as Main | as Main |
| Source | Four providers | as Main | |
| Evidence | (i) Closed-book | none, last state | none |
| (ii) Final only | , last state | none | |
| Memory | (iii) Previous only |
| Leaking | Revealed | Brier | |||
|---|---|---|---|---|---|
| Results | raw | filtered | raw | filtered | filtered |
| Live snapshot, Tavily ※ | 3% | 10% | 0.224 | ||
| Tavily | 42% | 18% | 87% | 60% | 0.137 ∗ |
| Exa | 31% | 27% | 90% | 92% | 0.080 ∗ |
| Serper | 32% | 17% | 73% | 43% | 0.194 |
| Parallel | 37% | n/a | 90% | n/a | – |
| Forecaster | Last-state Brier | Accuracy | Closed-book | Gap |
|---|---|---|---|---|
| GPT-5.6 Terra | 0.217 | 0.642 | 0.260 | 0.022 |
| Claude Sonnet 5 | 0.244 | 0.597 | 0.279 | 0.046 |
| Gemini 3.8 Flash | 0.218 | 0.628 | 0.300 | 0.028 |
| DeepSeek-V4.1-Flash | 0.220 | 0.660 | 0.258 | 0.028 |
| GPT-6.1 Sol † | 0.208 | 0.653 | 0.266 | 0.018 |
| Market | 0.176 | 0.719 |
| Variant | TBS | Last | Update | Agree | Ratio |
|---|---|---|---|---|---|
| Main | 0.228 | 0.218 | 0.045 | 0.59 | 0.76 |
| Previous only | 0.232 | 0.221 | 0.049 | 0.59 | 0.83 |
| No memory | 0.237 | 0.230 | 0.084 | 0.50 | 1.24 |
| All so far | 0.232 | 0.227 | 0.059 | 0.49 | 0.91 |
| BM25 search | 0.241 | 0.249 | 0.075 | 0.46 | 1.10 |
| Days to event | |||||
|---|---|---|---|---|---|
| Provider | Last-state | Accuracy | TBS | 0–1 | 4–7 |
| Tavily | 0.218 | 0.628 | 0.228 | 0.220 | 0.232 |
| Exa | 0.213 | 0.646 | 0.232 | 0.211 | 0.243 |
| Serper | 0.255 ∗ | 0.594 | 0.252 ∗ | 0.246 | 0.250 |
| Parallel | 0.212 | 0.639 | 0.229 | 0.209 | 0.246 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Questions | |
|---|---|
| Candidates screened / eligible | 6,754 / 279 |
| Candidates, Polymarket / Kalshi | 5,099 / 1,655 |
| Sports share, Polymarket / Kalshi | 95.5% / 92.2% |
| Selected (Kalshi / Polymarket) | 150 (104 / 46) |
| Horizon 2–5 / 6–9 / 10–14 days | 23 / 85 / 42 |
| Activity tier low / medium / high | 47 / 51 / 52 |
| Model | Released | Cutoff |
|---|---|---|
| GPT-5.6 Terra | 9 Jul 2026 | Feb 2026 |
| Claude Sonnet 5 | 30 Jun 2026 | Jan 2026 |
| Gemini 3.8 Flash | 2 Sep 2026 | Mar 2026 |
| DeepSeek-V4.1-Flash | 10 Sep 2026 | not disclosed |
| GPT-6.1 Sol † | 29 Sep 2026 | Apr 2026 |
| Provider | Coverage | Relevant | Dated | Price |
|---|---|---|---|---|
| Tavily | 0.68 | 66% | 0% | $16 |
| Exa | 0.64 | 51% | 39% | $15 |
| Serper | 0.53 | 42% | 55% | $1 |
| Parallel | 0.64 | 56% | 56% | $5 |
| Forecaster | Last state | All states |
|---|---|---|
| Best single model | 0.217 | 0.222 |
| Mean of four | 0.219 | 0.224 |
| Median of four | 0.215 | 0.222 |
| Blend, | 0.189 | 0.205 |
| Blend, | 0.179 | 0.200 |
| Market | 0.176 | 0.199 |
| Trader | Trades | Sold at next state | Held to settlement |
|---|---|---|---|
| Blind rule | |||
| GPT-5.6 Terra | 231 | [ , ] | [ , ] |
| Claude Sonnet 5 | 278 | [ , ] | [ , ] |
| Gemini 3.8 Flash | 214 | [ , ] | [ , ] |
| DeepSeek-V4.1-Flash | 224 | [ , ] | [ , ] |
| Trading mode, own exits | |||
| Brier | Wrong side of 0.5 | ||||||
| Forecaster | Toward | Mixed | Against | Toward | Mixed | Against | |
| GPT-5.6 Terra | 0.133 | 0.266 | 0.387 | 216 | 17% | 32% | 50% |
| Claude Sonnet 5 | 0.140 | 0.301 | 0.449 | 223 | 13% | 36% | 50% |
| Gemini 3.8 Flash | 0.126 | 0.277 | 0.405 | 219 | 13% | 34% | 52% |
| DeepSeek-V4.1-Flash | 0.120 | 0.273 | 0.429 | 206 | 12% | 33% | 54% |
| Market | 0.137 | 0.222 | 0.313 | 185 | 22% | 29% | 49% |
| Condition | Last [95% CI] | TBS [95% CI] | |||
|---|---|---|---|---|---|
| Previous only | 0.003 [ 0.004, 0.012] | 0.004 [ 0.002, 0.009] | 0.003 | 0.025 | 13% |
| No memory | 0.013 [ 0.003, 0.032] | 0.010 [0.002, 0.018] | 0.008 | 0.042 | 21% |
| All so far | 0.009 [ 0.001, 0.022] | 0.004 [ 0.003, 0.012] | 0.000 | 0.029 | 15% |
| BM25 search | 0.031 [0.006, 0.058] | 0.013 [0.004, 0.023] | 0.012 | 0.059 | 28% |
Forecasting Benchmarks
Forecasting Benchmarks are datasets and evaluation protocols for comparing predictions of future time-series values, events, and outcomes.
See all 248 papers