cs.CLSep 5, 2026

The Corroboration Illusion: When More News Makes LLM Forecasts Less True

Authors: Yuan Lu, Yukuan Zhang

Organizations: Peking university Beijing, China · University of Central Florida Orlando, FL, USA

Abstract

Large language models (LLMs) are increasingly used to forecast real-world events by retrieving and reasoning over news. We show that this dependence on an open, crawlable news corpus creates a new attack surface: an adversary who can merely publish articles--without access to the retriever, the model, or the user's queries--can systematically move the forecaster's output probabilities. We formalize news-corpus poisoning of probabilistic forecasters, a threat model distinct from prior RAG poisoning, which targets factual answers or opinion polarity rather than calibrated probabilities. We evaluate the attack on 500 resolved ForecastBench questions against a 17.4M-article Common Crawl News corpus with a strict crawl-date cutoff, using three retrieval-augmented forecasters built on open 7-8B models. A single LLM-written article per question flips 56% of forecasts across the 0.5 boundary; five articles flip 69-73% and shift probabilities by +0.13 to +0.22 net of a neutral-article placebo, degrading the Brier score from 0.18 to 0.37. The effect is monotone in the number, retrieval rank, query similarity, and context share of injected articles, transfers across model families, and is unaffected by the claimed publisher. We then evaluate three natural defenses--source allow-lists, isolate-then-aggregate forecasting, and perplexity filtering--and show that each has a cheap bypass: spoofed publishers, majority poisoning, and higher-temperature generation, respectively. Our results indicate that probabilistic LLM judgments inherit the full fragility of the information supply chain they consume.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters

    Jul 15, 2026Xiao Ye, Jacob Dineen, Evan Zhu +3LLM EvaluationForecasting Benchmarks

  2. AlphaBet: Reinforcement Learning for LLM Event Forecasting Agents

    Oct 15, 2026Liqin Ye, Rongzhi Zhang, Xin Liu +5Probabilistic ForecastingRL for LLM Agents