Agents built on large language models (LLMs) increasingly forecast real-world events by searching the Web and reasoning over what they find. How well such an agent forecasts depends on both the retrieved evidence and the model, making control over that evidence essential for evaluation. Live evaluations can mix retrieval with reasoning when each system searches on its own, while retrospective search can leak later information through date filters or today's ranking of historical pages. Comparing models, search sources, and memory strategies requires both fixed evidence and a record of how it evolved before resolution. We introduce ForecastGym, which records what four commercial web search APIs return for open prediction-market questions every two days and replays this frozen evidence offline once outcomes are known, comparing models, memory, and captured search sources under a fixed evidence-acquisition policy. Its first release covers 150 questions, 614 evidence states, one per question and collection time, and 131k search results. Rerunning the same queries after settlement returns results that all but state the outcome for up to 92% of the questions, despite the tested date filters. On the frozen evidence, all tested LLMs trail the market in forecasting accuracy. Our diagnostic analysis suggests that missed public information contributes more to the gap in high-gap states than misreading does. Switching the search API changes a model's Brier score by up to 0.043, more than the spread across models, while persistent notes score similarly to rereading all evidence at half the input tokens. The evidence a forecaster sees can matter as much as the model that reads it.
Figures & tables
Figure 1. Design principles of ForecastGym . Four design properties: unresolved questions at collection, the same time-valid corpus for all methods, replayable evidence trajectories, and separate search providers under a shared initial query plan.
Benchmark
Live
Controlled
Replayable
Source-aware
Open
Reusable
Multi-state
Memory
Providers
ForecastQA ( Jin et al., 2021 )
–
✓
–
–
–
ForecastBench ( Karger et al., 2025 )
✓
–
–
–
–
FutureX ( Zeng et al., 2025 )
✓
–
–
–
–
Prophet Arena ( Yang et al., 2025 ; Prophet Arena, 2026 )
✓
∘
✓
–
✓
BTF ( Wildman et al., 2025 )
–
✓
–
–
–
Table 1. Forecasting benchmarks by the principles of Figure 1 . Live: open questions at collection. Controlled: reusable fixed evidence. Replayable: multiple states and controlled memory studies. Source-aware: controlled provider comparisons. ∘ : partial.
Figure 2. The ForecastGym lifecycle described in Section 3 . Phase 1 records what four search APIs return every two days for open questions, with each result stored with its source, query, and retrieval time. Phase 2 replays the frozen states to a forecasting agent behind three safeguards and changes one argument of Eq. ( 1 ) at a time. Orange cards follow one question of the first batch.
Figure 3. One question of the first batch, replayed by Gemini 3.8 Flash on Tavily’s evidence. The market price is never shown to the model. Left: the question, its domain, horizon, two evidence states, the outcome No, and per-provider result counts. Right: forecasts over time. With persistent notes Gemini falls from 0.36 to 0.22 and without memory from 0.38 to 0.35, while the Kalshi price stays near 0.64 until 17 September and then falls to 0.15 before the cutoff.
Argument
Variant
Evidence
Memory
Main
Et(k)
pt−1,Rt−1,Nt−1
Model f
Four LLMs
as Main
as Main
Source k
Four providers
Et(k′)
as Main
Evidence E
(i) Closed-book
none, last state
none
(ii) Final only
Et , last state
none
Memory M
(iii) Previous only
Et
pt−1,Rt−1
Table 2. Single-factor interventions on Eq. ( 1 ). Each row changes one argument of the main condition and keeps the others as in Main. Evidence: what the agent reads at state t . Memory: what it carries from the state before.
Figure 4. The four providers return largely different pages under the shared query policy, averaged over all states. Jaccard overlap: shared URLs over all URLs of the pair. Heat map of pairwise Jaccard overlaps between providers, from 0.09 to 0.18, and bars showing that 53 to 71 percent of each provider's URLs come from no other provider.
Leaking
Revealed
Brier
Results
raw
filtered
raw
filtered
filtered
Live snapshot, Tavily ※
3%
10%
0.224
Tavily
42%
18%
87%
60%
0.137 ∗
Exa
31%
27%
90%
92%
0.080 ∗
Serper
32%
17%
73%
43%
0.194
Parallel
37%
n/a
90%
n/a
–
Table 3. Post-hoc search on 60 settled questions, with raw and date-filtered results. ∗ Significantly below the live snapshot.
Forecaster
Last-state Brier ↓
Accuracy ↑
Closed-book ↓
Gap
GPT-5.6 Terra
0.217
0.642
0.260
+ 0.022
Claude Sonnet 5
0.244
0.597
0.279
+ 0.046
Gemini 3.8 Flash
0.218
0.628
0.300
+ 0.028
DeepSeek-V4.1-Flash
0.220
0.660
0.258
+ 0.028
GPT-6.1 Sol †
0.208
0.653
0.266
+ 0.018
Market
0.176
0.719
Table 4. Main condition with Tavily’s evidence, at each question’s last state. Gap: to the market over all states, significant for all. † Released after the batch resolved.
Figure 5. Where the models’ Brier gap to the market lies. Panel a shows shares of the gap by question type against each type’s share of states; panel b shows causes across the 111 states trailing the market by more than 0.1, drawn from 49 questions.
Figure 6. Brier score by days until the event in the main condition with Tavily’s evidence. Dashed: closed-book. † Released after the batch resolved.
Variant
TBS ↓
Last ↓
Update
Agree
Ratio
Main
0.228
0.218
0.045
0.59
0.76
Previous only
0.232
0.221
0.049
0.59
0.83
No memory
0.237
0.230
0.084
0.50
1.24
All so far
0.232
0.227
0.059
0.49
0.91
BM25 search
0.241
0.249
0.075
0.46
1.10
Table 5. Memory variants of Gemini 3.8 Flash. Last: last-state Brier. Update: mean change between states. Agree: share in the market’s direction. Ratio: size relative to the market’s.
Figure 7. Last-state Brier scores and model ranks without memory. Hollow markers: Tavily; filled markers: Exa. Kendall’s τ=0 across four models. ∗ Only significant change. † Not ranked.
Days to event
Provider
Last-state ↓
Accuracy ↑
TBS ↓
0–1
4–7
Tavily
0.218
0.628
0.228
0.220
0.232
Exa
0.213
0.646
0.232
0.211
0.243
Serper
0.255 ∗
0.594
0.252 ∗
0.246
0.250
Parallel
0.212
0.639
0.229
0.209
0.246
Table 6. Gemini 3.8 Flash with each provider’s evidence. The last two columns give the Brier score 0–1 and 4–7 days before the event. ∗ Worse than Tavily.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Questions
Candidates screened / eligible
6,754 / 279
Candidates, Polymarket / Kalshi
5,099 / 1,655
Sports share, Polymarket / Kalshi
95.5% / 92.2%
Selected (Kalshi / Polymarket)
150 (104 / 46)
Horizon 2–5 / 6–9 / 10–14 days
23 / 85 / 42
Activity tier low / medium / high
47 / 51 / 52
Appendix
Table 7. Statistics of the first batch. Document: a unique URL and content pair. Per-state and per-question counts are means.
Model
Released
Cutoff
GPT-5.6 Terra
9 Jul 2026
Feb 2026
Claude Sonnet 5
30 Jun 2026
Jan 2026
Gemini 3.8 Flash
2 Sep 2026
Mar 2026
DeepSeek-V4.1-Flash
10 Sep 2026
not disclosed
GPT-6.1 Sol †
29 Sep 2026
Apr 2026
Appendix
Table 8. Evaluated models. † Released after the batch resolved, with cutoff from third parties.
Provider
Coverage
Relevant
Dated
Price
Tavily
0.68
66%
0%
$16
Exa
0.64
51%
39%
$15
Serper
0.53
42%
55%
$1
Parallel
0.64
56%
56%
$5
Appendix
Table 9. Search providers at the last state. Coverage rating, relevant and dated items, and list price per 1,000 requests.
Forecaster
Last state ↓
All states ↓
Best single model
0.217
0.222
Mean of four
0.219
0.224
Median of four
0.215
0.222
Blend, w=0.5
0.189
0.205
Blend, w=0.2
0.179
0.200
Market
0.176
0.199
Appendix
Table 10. Brier score of model ensembles and model–market blends, where w is the models’ weight.
Trader
Trades
Sold at next state
Held to settlement
Blind rule
GPT-5.6 Terra
231
−13.8 [ −23.4 , −2.5 ]
−5.7 [ −28.7 , 19.1 ]
Claude Sonnet 5
278
−12.4 [ −20.7 , −3.6 ]
−3.7 [ −23.7 , 18.8 ]
Gemini 3.8 Flash
214
−12.7 [ −23.2 , −0.4 ]
−3.8 [ −28.4 , 25.2 ]
DeepSeek-V4.1-Flash
224
−10.9 [ −20.8 , 0.8 ]
−5.4 [ −28.1 , 20.4 ]
Trading mode, own exits
Appendix
Table 11. Profit in cents per dollar staked, with 95% intervals over questions.
Figure 8. Gemini hedges toward the market when shown its price. Forecast minus market price at the 575 scored states that have a market price, with the price hidden and shown. The dashed line marks ignoring the price, and the horizontal axis marks copying it.
Brier ↓
Wrong side of 0.5
Forecaster
Toward
Mixed
Against
n
Toward
Mixed
Against
GPT-5.6 Terra
0.133
0.266
0.387
216
17%
32%
50%
Claude Sonnet 5
0.140
0.301
0.449
223
13%
36%
50%
Gemini 3.8 Flash
0.126
0.277
0.405
219
13%
34%
52%
DeepSeek-V4.1-Flash
0.120
0.273
0.429
206
12%
33%
54%
Market
0.137
0.222
0.313
185
22%
29%
49%
Appendix
Table 12. Forecasts by the direction of a state’s evidence (main condition, Tavily). The Brier columns use the LLM judge’s labels. The wrong-side split counts weak support as mixed and requires at least moderate strength for toward. n : number of wrong-side forecasts.
Figure 9. Model disagreement and accuracy by days until the event. (a) Disagreement among the four models, as mean pairwise difference and standard deviation, and (b) Brier of their mean forecast and of the market.
Condition
Δ Last [95% CI]
Δ TBS [95% CI]
Gˉ
∣G∣
∣G∣>0.05
Previous only
+ 0.003 [ − 0.004, 0.012]
+ 0.004 [ − 0.002, 0.009]
+ 0.003
0.025
13%
No memory
+ 0.013 [ − 0.003, 0.032]
+ 0.010 [0.002, 0.018]
+ 0.008
0.042
21%
All so far
+ 0.009 [ − 0.001, 0.022]
+ 0.004 [ − 0.003, 0.012]
± 0.000
0.029
15%
BM25 search
+ 0.031 [0.006, 0.058]
+ 0.013 [0.004, 0.023]
+ 0.012
0.059
28%
Appendix
Table 13. Paired differences from the main condition (positive: worse). Δ Last: last-state Brier. Gt : forecast minus the main condition’s at the same state, with mean Gˉ and mean absolute value ∣G∣ .
Figure 10. Three case questions, all resolved Yes. Forecasts at each state (main condition, Tavily) and the market in red on (a) Norges Bank’s hike, (b) a Metascore above 90, and (c) the Eagles beating the Titans. Three small line plots of the four models' forecasts and the market price over the evidence states of three questions.