Large Language Models (LLMs) exhibit approximate recall: they retain coarse associations about past events while losing fidelity on specific details. Past research on LLMs has quantified different models' forecasting ability in real-world events using prediction markets, but has not studied how losing said fidelity impacts their forecasting abilities. For example: while an LLM correctly recalls that the answer to: ''Did Anna Sawai win the Best Drama Actress in the 76th Emmy Awards?'' is ''Yes,'' it incorrectly states the event date as September 22 rather than September 15. In this paper, we introduce a novel metric called the Recall Fidelity Score to study how imprecision in internalized knowledge propagates to forecasting capability on future events. On resolved historical markets, we find that higher local recall fidelity around semantically similar past events predicts lower forecasting error on resolved markets, and the same signal predicts smaller distance from open markets prices.
Figures & tables
Specification
Intercept
βRm,d
R2
M1: Rm,d
0.226∗∗∗
−0.204∗∗∗
0.398
M2: +Dentropy
0.286∗∗∗
−0.222∗∗∗
0.429
M3: +Model FE
0.285∗∗∗
−0.223∗∗∗
0.691
Table 1: Domain-level regression of recall Brier scores on Recall Fidelity Score Rm,d across N=36 (model, domain) cells. Entropy defined in A.8
Figure 1: Recall Fidelity Scores (null = 0) and Log Loss for all models grouped by domain. Elections and Economics domains seem to produce strongest recall across models. Exact figures reported in A.9
Figure 2: Confident-wrong recall produces high Brier scores despite reduced confidence, while correct and partial recall yield near-zero Brier at high confidence. Confident-wrong cells are separated from all other states by an order of magnitude in Brier.
Figure 3: Mean forecast log-loss (from unresolved markets) versus mean Recall Fidelity color coded by model across nine domains.
Figure 4: Local recall error-type decomposition. Coefficients and 95% confidence intervals come from a market-level OLS regression of forecast Brier loss on the shares of the ten nearest historical recall markets falling into each recall category, with model and forecast-market fixed effects and standard errors clustered by condition ID.
Figure 5: Regression coefficients for local historical recall across open-market Brier-proxy specifications. Coefficient on Rm,i,10local with 95% confidence intervals using standard errors clustered by condition ID shown. Higher local recall significantly predicts lower squared forecast-market disagreement and lower absolute forecast-market distance.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Open-market proxy regression coefficients with 95% confidence intervals by condition ID. Local historical recall is negatively associated with squared and absolute forecast-market disagreement in market fixed-effect specifications, while the domain-level specification shows that market entropy is strongly positively associated with forecast-market disagreement.
Panel
Outcome
Main regressor
Controls / FE
β^ (SE)
p
N / clusters
Historical local
Bm,i
Rm,i,10local
Model + market FE
−0.0652(0.0231)
0.0048
4,897/1,264
Historical weighted
Bm,i
Rm,i,10local,w
Model + market FE
−0.0710(0.0225)
0.0016
4,897/1,264
Domain M1
Bˉm,d
Rm,d
None
−0.2038(0.0430)
<0.001
36 / –
Domain M2
Bˉm,d
Rm,d
Dd
−0.2216(0.0445)
<0.001
36 / –
Domain M3
Bˉm,d
Rm,d
Model FE + Dd
−0.2227(0.0361)
<0.001
36 / –
Appendix
Table 2: Historical resolved-market and aggregate domain-level recall regressions. The reported coefficient is always the coefficient on the recall-fidelity regressor. In Domain M2, γ^D=−0.1185 ( SE=0.0891 , p=0.193 ); in Domain M3, γ^D=−0.1192 ( SE=0.0691 , p=0.0948 ).
Panel
Outcome
Term
Controls / FE
β^ (SE)
p
N / clusters
Open market FE
Bm,j
Rm,j,10local
Model + market FE
−0.0284(0.0108)
0.0084
5,724/1,897
Open market FE
Am,j
Rm,j,10local
Model + market FE
−0.0448(0.0152)
0.0032
5,724/1,897
Open market FE
Lm,j
Rm,j,10local
Model + market FE
−0.0606(0.0696)
0.384
5,724/1,897
Open domain FE
Bm,j
Rm,j,10local
Model + domain FE + H(πj)
0.0071(0.0154)
0.642
5,724/1,897
Open domain FE
Bm,j
H(πj)
Model + domain FE
0.0721(0.0137)
<0.001
5,724/1,897
Open domain FE
Lm,j
Rm,j,10local
Model + domain FE + H(πj)
0.0140(0.0544)
0.797
5,724/1,897
Appendix
Table 3: Open-market proxy regressions. Market fixed effects compare models on the same unresolved market. Domain fixed effects with entropy controls test whether local recall explains broader domain-level variation after accounting for market-price uncertainty.
Figure 7: Murphy Breakdown of recall Brier for each model. A lower reliability score is more favorable while a higher resolution score is more favorable.
Model
n
Brier
Uncertainty
Reliability
Resolution
Null Rate
Claude Opus 4.7
1150
0.1017
0.2110
0.0102
0.0031
0.3174
Claude Sonnet 4.6
1150
0.1406
0.2110
0.0239
0.0052
0.3252
Gemini 3.1 Pro
1150
0.0877
0.2110
0.0080
0.0042
0.2591
GPT-5.5
0 993
0.0816
0.2132
0.0075
0.0022
0.2115
Appendix
Table 4: Murphy decomposition components and null prediction rates by model. Lower values are better for Brier, Reliability, and Null Rate, while higher values are better for Resolution.
Forecasters are evaluated by backtesting, which replays resolved questions and grades the probability the system would have assigned before the outcome was known. For LLMs, two channels leak the answer into this test. A model that retrieves can surface reports written after the event, turning forecasting into a lookup, and each new model is trained on data closer to the event, so a question that lay in the future for last year's models sits inside this year's training data. Either way, the test grades recall while claiming to grade foresight. We introduce Hindcast, which closes both leaks by grading a model as if it stood at a chosen past date t0, before the outcome existed in either channel. Hindcast replays resolved Polymarket prediction markets against a frozen snapshot of public Reddit, lets the model read only posts written before t0, and scores each forecast against both what happened and the market's own price at t0, itself a human forecast made from the same past information. Because the cutoff is set per market and the snapshot never changes, the evaluation re-runs on new markets as models improve, without going stale. Once the leak is closed, retrieval still helps most models, but only where Reddit discussed the event beforehand. Where the archive carried only speculation, retrieval hurts.
Xiao Ye, Jacob Dineen, Evan Zhu +3
School of Computing and Augmented Intelligence, Arizona State University
We introduce TimeSeek, a benchmark for studying how the reliability of agentic LLM forecasters changes over a prediction market's lifecycle. We evaluate 10 frontier models on 150 CFTC-regulated Kalshi binary markets at five temporal checkpoints, with and without web search, for 15,000 forecasts total. Models are most competitive early in a market's life and on high-uncertainty markets, but much less competitive near resolution and on strong-consensus markets. Web search improves pooled Brier Skill Score (BSS) for every model overall, yet hurts in 12% of model-checkpoint pairs, indicating that retrieval is helpful on average but not uniformly so. Simple two-model ensembles reduce error without surpassing the market overall. These descriptive results motivate time-aware evaluation and selective-deference policies rather than a single market snapshot or a uniform tool-use setting.
Agents built on large language models (LLMs) increasingly forecast real-world events by searching the Web and reasoning over what they find. How well such an agent forecasts depends on both the retrieved evidence and the model, making control over that evidence essential for evaluation. Live evaluations can mix retrieval with reasoning when each system searches on its own, while retrospective search can leak later information through date filters or today's ranking of historical pages. Comparing models, search sources, and memory strategies requires both fixed evidence and a record of how it evolved before resolution. We introduce ForecastGym, which records what four commercial web search APIs return for open prediction-market questions every two days and replays this frozen evidence offline once outcomes are known, comparing models, memory, and captured search sources under a fixed evidence-acquisition policy. Its first release covers 150 questions, 614 evidence states, one per question and collection time, and 131k search results. Rerunning the same queries after settlement returns results that all but state the outcome for up to 92% of the questions, despite the tested date filters. On the frozen evidence, all tested LLMs trail the market in forecasting accuracy. Our diagnostic analysis suggests that missed public information contributes more to the gap in high-gap states than misreading does. Switching the search API changes a model's Brier score by up to 0.043, more than the spread across models, while persistent notes score similarly to rereading all evidence at half the input tokens. The evidence a forecaster sees can matter as much as the model that reads it.
Yicheng Tao, Yuante Li, Zipei Fan
Carnegie Mellon University Pittsburgh, Pennsylvania, USA · Jilin University Changchun, China