From Narrative to Auditable Forecasts: A Structured Scaffold for Agentic Forecasting

Yuanpu Cao; Yongkang Du; Yurui Chang; Lu Lin; Jinghui Chen

cs.CLSep 5, 2026

From Narrative to Auditable Forecasts: A Structured Scaffold for Agentic Forecasting

Authors: Yuanpu Cao, Yongkang Du, Yurui Chang, Lu Lin, Jinghui Chen

Organizations: The Pennsylvania State University

Abstract

LLM agents are increasingly used for live forecasting, where they retrieve up-to-date information and produce estimates for unresolved future events. However, current agentic forecasting often relies on implicit narrative aggregation: agents collect evidence, discuss it in prose, and often assign a probability without an explicit update path from evidence to forecast. This limits both forecasting accuracy and auditability. We propose AuditForecast, an agentic scaffold for structured probabilistic forecasting. AuditForecast first anchors the forecast with a suitable quantitative baseline model, uses model-guided data retrieval to derive a base probability, and then applies situational factor updates outside the model's scope through mechanical aggregation in odds space. This turns forecasting from a prose-based judgment into a structured process with explicit intermediate objects. Across multiple live forecasting benchmarks, AuditForecast improves forecasting accuracy and calibration relative to strong agentic baselines, surpasses market-implied references in several settings, and outperforms substantially more expensive deep-research agents while remaining Pareto-dominant in the cost--accuracy tradeoff. Beyond performance gains, AuditForecast produces an auditable forecasting report that makes forecast construction explicit and supports systematic post hoc analysis.

Figures & tables

Figure 1: A narrative forecast from GPT-5.4 under Think & Search agent framework: the agent retrieves evidence, balances it narratively, and assigns a probability without an explicit derivation.

Figure 2: Left: the agent workflow under AuditForecast . Right: the corresponding auditable forecasting report. Highlighted text emphasizes key information and steps in the forecasting process.

Method	Polymarket				Kalshi
Method	Stocks	Crypto	Sports	Overall	Finance	Entertain.	Sports	Overall
GPT-5.4
Think&Search	0.1063	0.1599	0.2538	0.1342	0.0588	0.1276	0.2258	0.1028
Market-Implied	0.1265	0.1582	0.2540	0.1494	0.0759	0.1073	0.2263	0.1060
AuditForecast	0.0950	0.1257	0.2517	0.1225	0.0524	0.0943	0.2119	0.0870
GPT-5.4-mini

Table 1: Brier score (

\downarrow

) on Polymarket and Kalshi. The Market-Implied reference for Gemini-3-flash differs from that of the GPT backbones because it is evaluated on a different live time range.

Method	Level 1	Level 2	Overall
GPT-5.4
Think&Search	0.2879	0.1091	0.1274
AuditForecast	0.0932	0.0921	0.0922
GPT-5.4-mini
Think&Search	0.0970	0.1251	0.1223
AuditForecast	0.0965	0.1239	0.1211

Table 2: Brier score (

\downarrow

) on FutureX.

Figure 3: Brier score (

\downarrow

) of AuditForecast with and without factor updates. Factor updates consistently improve forecasting performance.

Figure 4: Effect of reasoning effort on forecasting accuracy and search usage. Results are shown for GPT-5.4 on Polymarket questions.

Figure 5: Comparison with deep research agents. Left: Brier score (

\downarrow

) versus cost per question. Right: average number of web searches per question.

Method	Threshold Forecast
Chronos-2	0.1237
TimesFM	0.1241
AuditForecast $p_{0}$	0.0980
AuditForecast	0.0938

Table 3: Brier score (

\downarrow

) on value-threshold forecasts.

Method	Avg. Web Searches	$p_{0}$ Brier ( $\downarrow$ )
Model-guided retrieval	11.7	0.0802
Retrieval-then-modeling	23.9	0.0882

Table 4: Ablation of model-guided retrieval.

Method	Brier ( $\downarrow$ )
AuditForecast	0.0764
w/o scope checking	0.0817
w/o independence checking	0.0824
Narrative aggregation	0.0815
Probability-space aggregation	0.0807

Table 5: Ablation of scope checking, independence checking, and aggregation.

Figure 6: GPT-5.4 under AuditForecast on stock range forecasts. Left: Data-source distribution within the Parametric CDF method for

p_{0}

. Right: Brier score of the dominant data-source subtypes.

Figure 7: Factor auditing results for AuditForecast on stock range questions with GPT-5.4 as the backbone.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Polymarket	Kalshi	FutureX	Total
GPT-5.4	562	220	49	831
GPT-5.4-mini	562	220	49	831
Gemini-3-flash	391	251	67	709

Appendix

Table 6: Dataset statistics.

	With Check	w/o Check
Avg. # Factors	1.12	2.62
Avg. Factor Push ( $\lvert\Delta\log\text{-odds}\rvert$ )	0.153	0.244
Brier	0.088	0.104

Appendix

Table 7: Effect of independence checking on factor retention, odds updates, and forecasting accuracy.

Judge	With Check	w/o Check
GPT-5.6	5.8%	12.7%
Claude-Opus-4.8	8.7%	12.7%

Appendix

Table 8: Double-counting rates under two LLM judges.

Method	Polymarket	Kalshi
GPT-5.4
Free-form scaffold	0.1241	0.1596
AuditForecast	0.1151	0.1462
GPT-5.4-mini
Free-form scaffold	0.1244	0.2089
AuditForecast	0.1157	0.1898

Appendix

Table 9: Brier score (

\downarrow

) on Polymarket and Kalshi.

Metric	GPT-5.4	GPT-5.4-mini
Direction consistency	86%	88%
LR variability ( $\lvert\log\text{-LR}\rvert$ std.)	0.074	0.030

Appendix

Table 10: Stability of likelihood ratios across naturally occurring factor paraphrases.

Explore similar work

LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications
Aug 24, 2026Xiaogang Xu, Jiaqi Tang, Jianmin Chen +12LLM Agents LLM-Based Time Series Forecasting
Zhejiang University, China · The Hong Kong University of Science and Technology, Hong Kong · Northwestern Polytechnical University, China +4
Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents
Sep 24, 2026Liqin Ye, Haorui Wang, Fardin Ahmed +8LLM Agent Evaluation Forecasting Benchmarks
Georgia Institute of Technology · Amazon · University of Florida
Agentic Forecasting with Structured Linguistic Beliefs
Apr 20, 2026Kevin MurphyForecasting Benchmarks Probabilistic Forecasting
Department of Computer Science University of British Columbia Vancouver, BC, Canada

Aug 24, 2026, cs.AIJ/K move · Enter open · S save

LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications

Xiaogang Xu, Jiaqi Tang, Jianmin Chen, Yingying Yan+11

Zhejiang University, China · The Hong Kong University of Science and Technology, Hong Kong · Northwestern Polytechnical University, China+4

LLM Agents LLM-Based Time Series Forecasting LLM Agent Evaluation LLM Agent Training Forecasting Benchmarks

Open paper AskPDF arXiv

Large language models (LLMs) now support forecasting systems that combine language-based reasoning with temporal data, evidence retrieval, external tools, and iterative prediction. We investigate LLM-based forecasting agents, meaning systems in which a language model contributes to a scored prediction about a future or currently unobserved target. We organize architectures into three groups. Standalone LLM workflows operate on encoded time series or event context. Tool- and retrieval-augmented agents incorporate external evidence. Hybrid systems pair LLMs with statistical or foundation models. We then review training methods and evaluation protocols. We examine negative as well as positive evidence, including sensitivity to small input perturbations, ablations in which the LLM component does not improve accuracy, and benchmark gains that may reflect contamination instead of temporal reasoning. We cover applications in finance, weather, health, energy, and operations, and we summarize the benchmarks and datasets used for evaluation. The evidence indicates that measurement is a central limitation. Future work requires calibration under distribution shift, contamination-resistant live evaluation, explicit reporting of cost and accuracy together, and methods for handling feedback between deployed forecasts and the outcomes being forecast.

Abstract

Explore similar work

LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications

Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents

Agentic Forecasting with Structured Linguistic Beliefs