LLM agents are increasingly used for live forecasting, where they retrieve up-to-date information and produce estimates for unresolved future events. However, current agentic forecasting often relies on implicit narrative aggregation: agents collect evidence, discuss it in prose, and often assign a probability without an explicit update path from evidence to forecast. This limits both forecasting accuracy and auditability. We propose AuditForecast, an agentic scaffold for structured probabilistic forecasting. AuditForecast first anchors the forecast with a suitable quantitative baseline model, uses model-guided data retrieval to derive a base probability, and then applies situational factor updates outside the model's scope through mechanical aggregation in odds space. This turns forecasting from a prose-based judgment into a structured process with explicit intermediate objects. Across multiple live forecasting benchmarks, AuditForecast improves forecasting accuracy and calibration relative to strong agentic baselines, surpasses market-implied references in several settings, and outperforms substantially more expensive deep-research agents while remaining Pareto-dominant in the cost--accuracy tradeoff. Beyond performance gains, AuditForecast produces an auditable forecasting report that makes forecast construction explicit and supports systematic post hoc analysis.
Figures & tables
Figure 1: A narrative forecast from GPT-5.4 under Think & Search agent framework: the agent retrieves evidence, balances it narratively, and assigns a probability without an explicit derivation.
Figure 2: Left: the agent workflow under AuditForecast . Right: the corresponding auditable forecasting report. Highlighted text emphasizes key information and steps in the forecasting process.
Method
Polymarket
Kalshi
Stocks
Crypto
Sports
Overall
Finance
Entertain.
Sports
Overall
GPT-5.4
Think&Search
0.1063
0.1599
0.2538
0.1342
0.0588
0.1276
0.2258
0.1028
Market-Implied
0.1265
0.1582
0.2540
0.1494
0.0759
0.1073
0.2263
0.1060
AuditForecast
0.0950
0.1257
0.2517
0.1225
0.0524
0.0943
0.2119
0.0870
GPT-5.4-mini
Table 1: Brier score ( ↓ ) on Polymarket and Kalshi. The Market-Implied reference for Gemini-3-flash differs from that of the GPT backbones because it is evaluated on a different live time range.
Method
Level 1
Level 2
Overall
GPT-5.4
Think&Search
0.2879
0.1091
0.1274
AuditForecast
0.0932
0.0921
0.0922
GPT-5.4-mini
Think&Search
0.0970
0.1251
0.1223
AuditForecast
0.0965
0.1239
0.1211
Table 2: Brier score ( ↓ ) on FutureX.
Figure 3: Brier score ( ↓ ) of AuditForecast with and without factor updates. Factor updates consistently improve forecasting performance.
Figure 4: Effect of reasoning effort on forecasting accuracy and search usage. Results are shown for GPT-5.4 on Polymarket questions.
Figure 5: Comparison with deep research agents. Left: Brier score ( ↓ ) versus cost per question. Right: average number of web searches per question.
Method
Threshold Forecast
Chronos-2
0.1237
TimesFM
0.1241
AuditForecast p0
0.0980
AuditForecast
0.0938
Table 3: Brier score ( ↓ ) on value-threshold forecasts.
Method
Avg. Web Searches
p0 Brier ( ↓ )
Model-guided retrieval
11.7
0.0802
Retrieval-then-modeling
23.9
0.0882
Table 4: Ablation of model-guided retrieval.
Method
Brier ( ↓ )
AuditForecast
0.0764
w/o scope checking
0.0817
w/o independence checking
0.0824
Narrative aggregation
0.0815
Probability-space aggregation
0.0807
Table 5: Ablation of scope checking, independence checking, and aggregation.
Figure 6: GPT-5.4 under AuditForecast on stock range forecasts. Left: Data-source distribution within the Parametric CDF method for p0 . Right: Brier score of the dominant data-source subtypes.
Figure 7: Factor auditing results for AuditForecast on stock range questions with GPT-5.4 as the backbone.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Polymarket
Kalshi
FutureX
Total
GPT-5.4
562
220
49
831
GPT-5.4-mini
562
220
49
831
Gemini-3-flash
391
251
67
709
Appendix
Table 6: Dataset statistics.
With Check
w/o Check
Avg. # Factors
1.12
2.62
Avg. Factor Push ( ∣Δlog-odds∣ )
0.153
0.244
Brier
0.088
0.104
Appendix
Table 7: Effect of independence checking on factor retention, odds updates, and forecasting accuracy.
Judge
With Check
w/o Check
GPT-5.6
5.8%
12.7%
Claude-Opus-4.8
8.7%
12.7%
Appendix
Table 8: Double-counting rates under two LLM judges.
Method
Polymarket
Kalshi
GPT-5.4
Free-form scaffold
0.1241
0.1596
AuditForecast
0.1151
0.1462
GPT-5.4-mini
Free-form scaffold
0.1244
0.2089
AuditForecast
0.1157
0.1898
Appendix
Table 9: Brier score ( ↓ ) on Polymarket and Kalshi.
Metric
GPT-5.4
GPT-5.4-mini
Direction consistency
86%
88%
LR variability ( ∣log-LR∣ std.)
0.074
0.030
Appendix
Table 10: Stability of likelihood ratios across naturally occurring factor paraphrases.
Large language models (LLMs) now support forecasting systems that combine language-based reasoning with temporal data, evidence retrieval, external tools, and iterative prediction. We investigate LLM-based forecasting agents, meaning systems in which a language model contributes to a scored prediction about a future or currently unobserved target. We organize architectures into three groups. Standalone LLM workflows operate on encoded time series or event context. Tool- and retrieval-augmented agents incorporate external evidence. Hybrid systems pair LLMs with statistical or foundation models. We then review training methods and evaluation protocols. We examine negative as well as positive evidence, including sensitivity to small input perturbations, ablations in which the LLM component does not improve accuracy, and benchmark gains that may reflect contamination instead of temporal reasoning. We cover applications in finance, weather, health, energy, and operations, and we summarize the benchmarks and datasets used for evaluation. The evidence indicates that measurement is a central limitation. Future work requires calibration under distribution shift, contamination-resistant live evaluation, explicit reporting of cost and accuracy together, and methods for handling feedback between deployed forecasts and the outcomes being forecast.
Xiaogang Xu, Jiaqi Tang, Jianmin Chen +12
Zhejiang University, China · The Hong Kong University of Science and Technology, Hong Kong · Northwestern Polytechnical University, China +4
We introduce Forecast-Dojo, a replayable environment for benchmarking and training LLM forecasting agents. It combines resolved prediction-market questions with dated news, allowing agents to research an event and revisit their predictions at successive historical dates. The same tasks and tools support repeated evaluation, collection of training interactions, and feedback from recorded outcomes without waiting for new events to resolve. Forecast-Dojo contains 1,568 Polymarket events, split by time into training and evaluation periods, and 18.8M dated news articles. In an evaluation of 12 models, research tools lower Brier score for all 12. Forecasts also improve as events unfold, with the largest gains at steps where more newly dated evidence is recorded. Every model still trails historical market forecasts in both Brier score and accuracy. A belief notebook carried between dates lowers research cost but does not consistently improve forecast quality. Beyond evaluation, Forecast-Dojo provides interaction trajectories and outcome feedback for agent learning, with supervised fine-tuning as a proof of concept.
Liqin Ye, Haorui Wang, Fardin Ahmed +8
Georgia Institute of Technology · Amazon · University of Florida
We propose the Bayesian Linguistic Forecaster (BLF), which is a search-based ReAct style agent for (binary) forecasting that augments its context with a linguistic 'belief state', consisting of a probability estimate and a \emph{structured} natural-language evidence summary. (The belief update is 'Bayesian-inspired' rather than exact Bayesian inference, although we do include a comparison to a more explicit Bayesian approach using LLM-elicited likelihood surrogates.) On 400 historical ForecastBench question-date instances, BLF has the highest overall Brier Index among the compared external agents, and is the only compared method to significantly outperform a crowd+empirical-prior baseline in terms of overall performance. Our backtesting framework combines date-aware search tools, leakage auditing, and paired comparisons for variance control to provide a rigorous empirical framework for comparing agentic forecasting systems.
Kevin Murphy
Department of Computer Science University of British Columbia Vancouver, BC, Canada