The Corroboration Illusion: When More News Makes LLM Forecasts Less True

Yuan Lu; Yukuan Zhang

cs.CLSep 5, 2026

The Corroboration Illusion: When More News Makes LLM Forecasts Less True

Authors: Yuan Lu, Yukuan Zhang

Organizations: Peking university Beijing, China · University of Central Florida Orlando, FL, USA

Abstract

Large language models (LLMs) are increasingly used to forecast real-world events by retrieving and reasoning over news. We show that this dependence on an open, crawlable news corpus creates a new attack surface: an adversary who can merely publish articles--without access to the retriever, the model, or the user's queries--can systematically move the forecaster's output probabilities. We formalize news-corpus poisoning of probabilistic forecasters, a threat model distinct from prior RAG poisoning, which targets factual answers or opinion polarity rather than calibrated probabilities. We evaluate the attack on 500 resolved ForecastBench questions against a 17.4M-article Common Crawl News corpus with a strict crawl-date cutoff, using three retrieval-augmented forecasters built on open 7-8B models. A single LLM-written article per question flips 56% of forecasts across the 0.5 boundary; five articles flip 69-73% and shift probabilities by +0.13 to +0.22 net of a neutral-article placebo, degrading the Brier score from 0.18 to 0.37. The effect is monotone in the number, retrieval rank, query similarity, and context share of injected articles, transfers across model families, and is unaffected by the claimed publisher. We then evaluate three natural defenses--source allow-lists, isolate-then-aggregate forecasting, and perplexity filtering--and show that each has a cheap bypass: spoofed publishers, majority poisoning, and higher-temperature generation, respectively. Our results indicate that probabilistic LLM judgments inherit the full fragility of the information supply chain they consume.

Figures & tables

Fig. 1: Motivation for news-corpus poisoning. A retrieval-augmented forecaster observes the world only through an open news corpus. An attacker who can merely publish fabricated articles can alter the retrieved evidence and induce a high-confidence probability shift—without access to the model, retriever, or user queries.

Work	Target output	Ground truth	Injection	Cost curve
PoisonedRAG	answer string	yes	corpus	ASR vs. $k$
FlippedRAG	opinion polarity	no	corpus	–
Time-series adv. [ 14 ]	numeric forecast	yes	input perturbation	–
Trading-agent attacks [ 15 , 5 ]	trade / PnL	yes	prompt / headlines	–
Probing [ 11 ]	probability	yes	1 article in prompt	–
FutureX §5.2	event prediction	5 cases	prompt URL	–

TABLE I: Threat models of adjacent work versus ours.

Fig. 2: Overview of the news-corpus poisoning attack and its impact on an LLM forecasting pipeline. The attacker injects coordinated synthetic news into an open corpus; articles that survive retrieval and relevance filtering are summarized and incorporated into the final forecast. The resulting probability shift is illustrative.

Forecaster	Brier	ECE	mean $p$	AUC
Qwen-vanilla, no retrieval	0.159	0.183	0.33	0.707
Qwen-vanilla, RAG	0.250	0.343	0.49	0.722
Qwen-calibrated, RAG	0.184	0.242	0.39	0.744
Hermes, no retrieval	0.192	0.270	0.42	0.719
Hermes, RAG	0.226	0.325	0.48	0.750
Base rate (0.15)	0.127	–	–	–

TABLE II: Clean forecaster performance (

n{=}500

). Market prices at freeze: Brier 0.074.

Forecaster	Attack ( $k{=}5$ )	$\Delta p$ [95% CI]	net $\Delta p$ [CI]	flip [CI]	net flip	Brier
Qwen-vanilla	placebo	+0.036	–	30% / 17%	–	0.250 $\to$ 0.272
	YES	+0.215 [+0.20,+0.23]	+0.179 [+0.16,+0.19]	74% [68%,79%]	+44pp	0.250 $\to$ 0.451
	NO	+0.097 [+0.07,+0.12]	+0.133 [+0.11,+0.16]	62% [56%,68%]	+45pp	0.250 $\to$ 0.217
	flip	+0.286 [+0.27,+0.31]	+0.195 [+0.18,+0.21]	69% [65%,73%]	+46pp	0.250 $\to$ 0.346
Qwen-calibrated	placebo	+0.070	–	23% / 24%	–	0.184 $\to$ 0.227
	YES	+0.290 [+0.27,+0.31]	+0.221 [+0.20,+0.24]	77% [73%,82%]	+54pp	0.184 $\to$ 0.432

TABLE III: Attack effect at

k{=}5

(

n{=}500

; bootstrap 95% CIs). “net” subtracts the neutral-article placebo. Flip rates are over eligible questions (clean forecast on the wrong side of 0.5); for the placebo row we report up/down flip rates.

Fig. 3: Attack cost curves (Qwen-vanilla, YES direction). Left: mean $\Delta p$ and flip rate versus injected articles per question $k$ ; dotted line is the placebo. Middle/right: $\Delta p$ versus best rank and versus maximum query–article cosine of the injected articles ( $k{=}5$ ).

Defense	Attack	$\Delta p$	flip
none (0.250)	YES $k{=}5$	$+0.215$	74%
allow-list (0.248)	YES $k{=}5$ , obscure publ.	$-0.004$	14% †
allow-list	YES $k{=}5$ , spoofed publ.	$+0.213$	77%
isolate-aggregate (0.222)	YES $k{=}1$	$+0.028$	18% †
isolate-aggregate	YES $k{=}5$	$+0.132$	52%
isolate-aggregate	flip $k{=}5$	$+0.130$	40%

TABLE IV: Defenses against the attack (Qwen-vanilla). Clean Brier in parentheses; flip rates over eligible questions.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Source	$n$	base rate	clean Brier	net $\Delta p$ YES	flip YES	$\Delta p$ flip	flip rate
polymarket	334	0.12	0.242	+0.193	74%	+0.309	72%
manifold	89	0.20	0.259	+0.165	79%	+0.281	67%
metaculus	57	0.23	0.275	+0.125	58%	+0.174	53%
infer	20	0.15	0.285	+0.158	88%	+0.235	70%

Appendix

TABLE V: Per-source results (Qwen-vanilla,

k{=}5

Fig. 4: $\Delta p$ versus poison share of kept documents (Qwen-vanilla, YES, $k{=}5$ ).

Fig. 5: $\Delta p$ versus best rank, maximum similarity, and poison share for Qwen-calibrated (top) and Hermes-3-8B (bottom), YES direction, $k{=}5$ .

Fig. 6: Perplexity of injected versus organic articles. Default generation is trivially separable; sampling at $T{=}1.2$ overlaps the organic distribution.

Question	dir.	$y$	market	clean $p$	attacked $p$	poison kept
Tether insolvent in 2025?	YES	0	0.09	0.15	0.95	5
Eigenlayer market cap (FDV) between $20-25b one day after launch?	YES	0	0.06	0.15	0.85	5
Will Nice win the 2025–26 French Ligue 1?	YES	0	0.01	0.25	0.85	4
Will China unban Bitcoin by June 30?	YES	0	0.01	0.15	0.85	4
Will Denis Villeneuve’s next film be related to Dune?	NO	1	0.83	0.95	0.15	4
Will Igor Kurganov win the 2025 National Heads-Up Poker Championship?	YES	0	0.00	0.25	0.85	5

Appendix

TABLE VI: Additional case studies (Qwen-vanilla,

k{=}5

LLM Security

Large language model (LLM) security studies attacks, privacy risks, and defenses across model training, inference, and deployment.

See all 324 papers

Jul 15, 2026

Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters

Forecasters are evaluated by backtesting, which replays resolved questions and grades the probability the system would have assigned before the outcome was known. For LLMs, two channels leak the answer into this test. A model that retrieves can surface reports written after the event, turning forecasting into a lookup, and each new model is trained on data closer to the event, so a question that lay in the future for last year's models sits inside this year's training data. Either way, the test grades recall while claiming to grade foresight. We introduce Hindcast, which closes both leaks by grading a model as if it stood at a chosen past date

t_0

, before the outcome existed in either channel. Hindcast replays resolved Polymarket prediction markets against a frozen snapshot of public Reddit, lets the model read only posts written before

t_0

, and scores each forecast against both what happened and the market's own price at

t_0

, itself a human forecast made from the same past information. Because the cutoff is set per market and the snapshot never changes, the evaluation re-runs on new markets as models improve, without going stale. Once the leak is closed, retrieval still helps most models, but only where Reddit discussed the event beforehand. Where the archive carried only speculation, retrieval hurts.

Xiao Ye, Jacob Dineen, Evan Zhu +3

School of Computing and Augmented Intelligence, Arizona State University

LLM Evaluation Forecasting Benchmarks Temporal Data Leakage

Ask PDF

Sep 27, 2026

Two Heads Are Better Than One: Aggregating Weaker LLMs for Better Forecasts

Large language models (LLMs) are increasingly used to forecast real-world events, but access to the strongest individual forecaster may be costly or otherwise constrained. We study weak-to-strong forecast aggregation: can individually weaker LLM forecasters be aggregated to outperform a stronger forecaster? Using ForecastBench (Karger et al., 2025), we evaluate 70 LLM forecasters across 16 comparison groups, each with more than 1,000 shared subquestions, yielding 1,121 weaker-model pairs. Within each group, we identify the strongest individual by test Brier score and evaluate aggregates composed exclusively of weaker forecasters, with aggregation weights learned on separate training data. We find substantial evidence of weak-to-strong improvement. Learned linear pooling identifies a weaker pair that matches or outperforms the strongest individual in 11 of 16 groups and comes within 5% of its Brier score in all 16 groups. We also find that these improvements do not rely on having a near-best constituent and are generally accompanied by good calibration. Additional analyses show that adding more models does not consistently improve performance, and competitive weaker-model aggregates also remain available under practical constraints.

Cheng Peng, Ruixi Luo, Zhi Chen +1

Chinese University of Hong Kong

Language Model Ensembles Probabilistic Forecasting

Ask PDF

Oct 15, 2026

AlphaBet: Reinforcement Learning for LLM Event Forecasting Agents

Forecasting an unresolved event requires finding relevant evidence and deciding how strongly it supports each outcome. Yet outcome-based training of Large Language Model (LLM) forecasters often begins with preassembled evidence at a single prediction date, leaving evidence acquisition outside the learning loop. We introduce AlphaBet, an agentic reinforcement learning (RL) framework that trains a single LLM policy to search, reason, and report probabilities across multiple pre-resolution dates. Each dated forecast is scored by a proper scoring rule against the eventual outcome, allowing one resolved event to supervise decisions under different information states. We decompose forecasting regret into an evidence-acquisition gap and a probability-reporting gap, and show that Group Relative Policy Optimization (GRPO)'s standard-deviation normalization can distort the reporting target under multi-class proper scores. On GLM-4.5-Air, AlphaBet reduces the Brier score from 0.731 to 0.626 on Forecast-Dojo and from 0.612 to 0.528 on FutureX. Controlled ablations attribute these gains to multi-date training and show that they come from improvements in both information acquisition and probability reporting. Our empirical results validate that GRPO's std normalization degrades proper-scoring-rule performance while leaving top-outcome accuracy largely unchanged.

Liqin Ye, Rongzhi Zhang, Xin Liu +5

Georgia Institute of Technology · Amazon

Probabilistic Forecasting RL for LLM Agents Event Time Prediction Proper Scoring Rules

Ask PDF

Abstract

LLM Security

Explore similar work

Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters

Two Heads Are Better Than One: Aggregating Weaker LLMs for Better Forecasts

AlphaBet: Reinforcement Learning for LLM Event Forecasting Agents