Can LLMs Predict the Future? A Brier Score Analysis of Prediction Markets

Yuanbo Li; Zekun Li; Xiaoyan cong

cs.AISep 26, 2026

Can LLMs Predict the Future? A Brier Score Analysis of Prediction Markets

Authors: Yuanbo Li, Zekun Li, Xiaoyan cong

Organizations: Brown University Providence, RI 02912, USA

Abstract

We study whether model upgrades improve probability estimates for prediction-market questions. Our Resolved Market Forecasting (RMF) benchmark contains 3,000 resolved binary questions across nine domains, on which we evaluate six Claude and Qwen model variants using a question-only, zero-shot protocol. We assess Brier scores relative to an empirical base-rate predictor, examine their Murphy decomposition, and compare models through paired differences, with results stratified by event category and timing relative to training cutoffs. In the reported post-cutoff stratum, the four Claude models achieve Brier scores of 0.183-0.192, improving on the base-rate reference by 0.024-0.033. Qwen 32B does not significantly outperform that reference, although its paired Brier is 0.024 lower than that of the 7B checkpoint. The evaluated Claude version and tier upgrades yield no significant improvement. Within-model differences across event categories exceed the observed differences among Claude variants. These results show why forecasting scores should be interpreted alongside simple probability baselines and question composition: under this protocol, newer versions or higher model tiers do not consistently produce more accurate probabilities.

Figures & tables

Model	API / HF ID	Released	Vendor cutoff	n pre	n post
Sonnet 4.5	claude-sonnet-4-5-20250929	2025-09	2025-07	$1{,}498$	$1{,}502$
Sonnet 4.6	claude-sonnet-4-6	2025-12	2026-01	$1{,}142$	$1{,}152$
Opus 4.1	claude-opus-4-1-20250805	2025-08	2025-03	$1{,}498$	$1{,}502$
Opus 4.6	claude-opus-4-6	2025-11	2025-08	$1{,}498$	$1{,}501$
Qwen2.5-7B	Qwen/Qwen2.5-7B-Instruct	2024-09	$\sim$ 2023-10	$1{,}498$	$1{,}502$
Qwen2.5-32B-AWQ	Qwen/Qwen2.5-32B-Instruct-AWQ	2024-09	$\sim$ 2023-10	$1{,}498$	$1{,}502$

Table 1: Models evaluated on the

n{=}3{,}000

sample. “n pre” and “n post” use the paper’s shared boundary (pre

\leq

2024-12-31, post

\geq

2025-06-30). Cutoff dates for Sonnet 4.5 and Opus 4.1 are best-effort estimates from the corresponding release notes (Anthropic does not always publish an exact cutoff for intermediate model versions). For the Qwen2.5 checkpoints, both buckets fall after Qwen’s own October 2023 cutoff and are therefore both strictly post-cutoff for those models; we still report counts under the paper’s shared boundary so the rows are directly comparable.

Model	$n$	Brier	Base rate	$\Delta_{\text{BR}}$
Sonnet 4.5	$1{,}502$	$.186$	$.216$	$+.030^{\ast}$
Sonnet 4.6	$1{,}152$	$.189$	$.217$	$+.028^{\ast}$
Opus 4.1	$1{,}502$	$.183$	$.216$	$+.033^{\ast}$
Opus 4.6	$1{,}501$	$.192$	$.216$	$+.024^{\ast}$
Qwen 7B	$1{,}502$	$.234$	$.216$	$-.018^{\ast}$
Qwen 32B	$1{,}502$	$.210$	$.216$	$+.006$

Table 2: Post-cutoff Brier across all six models.

\Delta_{\text{BR}}

: base-rate Brier

-

model Brier (positive

=

model beats base rate). ∗

=

base rate outside the model’s

95\%

CI. All four Claude models clear the base rate by

0.02

–

0.03

Brier; Qwen 32B is indistinguishable from it; Qwen 7B is significantly worse.

Within-family version (older $\to$ newer)
Contrast	$\Delta$
Sonnet 4.5 $\to$ 4.6	$+.004$
Opus 4.1 $\to$ 4.6	$+.009^{\ast}$
Within-family size (smaller $\to$ larger)
Qwen 7B $\to$ 32B	$-.024^{\ast}$
Cross-family capacity (Sonnet $\to$ Opus)

Table 3: Paired Brier difference

\Delta

(later minus earlier; negative

=

later is better) on post-cutoff markets. ∗

=

95% bootstrap CI excludes zero. Only Qwen 7B

\to

32B improves significantly; within-Claude contrasts are null or significantly negative. Full CIs in App. D .

Model	Best cat	Brier	Worst cat	Brier
Sonnet 4.5	weather	$.126$	sports	$.195$
Sonnet 4.6	geopolitics	$.138$	sports	$.205$
Opus 4.1	geopolitics	$.102$	sports	$.205$
Opus 4.6	weather	$.129$	geopolitics	$.229$
Qwen 7B	geopolitics	$.149$	sci/tech	$.274$
Qwen 32B	geopolitics	$.143$	other	$.225$

Table 4: Within-model post-cutoff Brier spread across categories (

n{\geq}20

per cell). The spread is several times the largest paired capacity effect from § 4.1 ; best and worst categories differ across models. Full per-cell numbers in App. E .

Category	Sonnet 4.5	Sonnet 4.6	Opus 4.1	Opus 4.6	Qwen 7B	Qwen 32B
crypto	$.176$ (178)	$.170$ (136)	$.164$ (178)	$.165$ (177)	$.203$ (178)	$.192$ (178)
entertainment	$.150$ (14)	$.036$ (10)	$.150$ (14)	$.160$ (14)	$.199$ (14)	$.219$ (14)
finance	$.197$ (14)	$.187$ (11)	$.101$ (14)	$.218$ (14)	$.193$ (14)	$.202$ (14)
geopolitics	$.147$ (34)	$.138$ (28)	$.102$ (34)	$.229$ (34)	$.149$ (34)	$.143$ (34)
other	$.198$ (745)	$.200$ (566)	$.198$ (745)	$.202$ (745)	$.248$ (745)	$.225$ (745)
politics	$.183$ (64)	$.165$ (48)	$.179$ (64)	$.219$ (64)	$.220$ (64)	$.205$ (64)

Table 5: Post-cutoff Brier by event category and model. Per-category

n

in parentheses (varies slightly for Sonnet 4.6 because of parse failures). Italic cells (

n<20

) are directional only.

Model	n post	Pre	Post	$\Delta_{\text{BR}}$
Sonnet 4.5	$1{,}502$	$.196$ $[.184,.208]$	$.186$ $[.176,.196]$	$+.030^{\ast}$
Sonnet 4.6	$1{,}152$	$.220$ $[.205,.236]$	$.189$ $[.178,.201]$	$+.028^{\ast}$
Opus 4.1	$1{,}502$	$.211$ $[.198,.224]$	$.183$ $[.173,.193]$	$+.033^{\ast}$
Opus 4.6	$1{,}501$	$.211$ $[.198,.225]$	$.192$ $[.181,.203]$	$+.024^{\ast}$
Qwen 7B	$1{,}502$	$.264$ $[.251,.277]$	$.234$ $[.222,.246]$	$-.018^{\ast}$
Qwen 32B	$1{,}502$	$.231$ $[.220,.243]$	$.210$ $[.200,.220]$	$+.006$

Table 6: Brier marginals across all six models, both periods.

\Delta_{\text{BR}}

: base-rate minus model Brier on post-cutoff (positive = model beats base rate). ∗ = base rate outside the model’s post-cutoff

95\%

CI.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Pair (older vs. newer)	Both pre	Split	Both post	Total
Sonnet 4.5 (cutoff 2025-07) vs. Sonnet 4.6 (cutoff 2026-01)	$1{,}150$	$480$	$664$	$2{,}294$
Opus 4.1 (cutoff 2025-03) vs. Opus 4.6 (cutoff 2025-08)	$1{,}498$	$51$	$1{,}450$	$2{,}999$

Appendix

Table 7: Pairwise cutoff coverage on the

n{=}3{,}000

sample, restricted to markets jointly parseable by both models in the pair. “Split” is the memorization-gap window: only the newer member could plausibly have ingested these outcomes during training. The Opus pair has very few split markets (

n{=}51

) because the

n{=}3{,}000

sample was constructed under the shared 2024-12-31 / 2025-06-30 boundary and the Opus 4.1 to Opus 4.6 gap (2025-03 to 2025-08) falls almost entirely inside the excluded buffer; consequently, the Opus pair primarily contrasts both-pre and both-post performance, not the memorization-gap window.

Component	Pre	Post	$\Delta$
Reliability $\downarrow$	$.023$ $[.016,.034]$	$.012$ $[.007,.021]$	$-.011$
Resolution $\uparrow$	$.048$ $[.040,.061]$	$.034$ $[.027,.045]$	$-.014$
Uncertainty	$.233$	$.215$	$-.018$
Brier $\downarrow$	$.207$	$.193$	$-.014$

Appendix

Table 8: Murphy decomposition of Opus Brier score on the

n{=}3{,}000

sample. Resolution drops on novel events (less discrimination) but reliability improves (better calibration).

Contrast	$n$	$\Delta$	[95% CI]
Sonnet 4.5 $\to$ 4.6	$1{,}152$	$+.004$	$[-.004,+.011]$
Opus 4.1 $\to$ 4.6	$1{,}501$	$+.009^{\ast}$	$[+.001,+.017]$
Qwen 7B $\to$ 32B	$1{,}502$	$-.024^{\ast}$	$[-.035,-.014]$
Sonnet 4.6 vs. Opus 4.6	$1{,}152$	$+.004$	$[-.005,+.014]$
Sonnet 4.5 vs. Opus 4.1	$1{,}502$	$-.003$	$[-.011,+.004]$

Appendix

Table 9: Paired Brier difference with

95\%

bootstrap CI for the five contrasts in Table 3 .

n

is the intersection size.

	low	mid	high	gap
Sonnet 4.5
pre	$.217$	$.176$	$.190$	$-.026$
post	$.195$	$.177$	$.214$	$+.019$
$\Delta_{\text{gap}}$				$+.045$
Sonnet 4.6
pre	$.221$	$.226$	$.216$	$-.005$

Appendix

Table 10: Brier by log-volume salience tertile, cutoff, and model.

Group	$n$ pairs	Pre	Post
Within Claude (4 models)	$6$	$.53$	$.75$
Within Qwen (2 models)	$1$	$.51$	$.56$
Cross-family (Claude $\times$ Qwen)	$8$	$.32$	$.53$
Overall	$15$	$.42$	$.62$

Appendix

Table 11: Mean per-event squared-error correlation by pair group, on the all-six-model intersection. Within-Claude correlations rise sharply post-cutoff; within-Qwen does not.

Probabilistic Forecasting

Probabilistic forecasting predicts distributions or events with quantified uncertainty to support reliable decisions under uncertainty.

See all 231 papers

Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters
Jul 15, 2026Xiao Ye, Jacob Dineen, Evan Zhu +3LLM Evaluation Forecasting Benchmarks
School of Computing and Augmented Intelligence, Arizona State University
TimeSeek: Temporal Reliability of Agentic Forecasters
Apr 5, 2026Hamza Mostafa, Om Shastri, Dennis LeeLLM Agent Reliability Forecasting Benchmarks
The Wharton School, University of Pennsylvania
PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Data
Apr 3, 2026Pu Cheng, Juncheng Liu, Yunshen LongForecasting Benchmarks LLM Evaluation
College of Electrical Engineering, Sichuan University, Chengdu, Sichuan, China · School of Economics, Sichuan University, Chengdu, Sichuan, China

Jul 15, 2026J/K move · Enter open · S save

Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters

Xiao Ye, Jacob Dineen, Evan Zhu, Shijie Lu+2

School of Computing and Augmented Intelligence, Arizona State University

LLM Evaluation Forecasting Benchmarks Temporal Data Leakage

Open paper AskPDF arXiv

Forecasters are evaluated by backtesting, which replays resolved questions and grades the probability the system would have assigned before the outcome was known. For LLMs, two channels leak the answer into this test. A model that retrieves can surface reports written after the event, turning forecasting into a lookup, and each new model is trained on data closer to the event, so a question that lay in the future for last year's models sits inside this year's training data. Either way, the test grades recall while claiming to grade foresight. We introduce Hindcast, which closes both leaks by grading a model as if it stood at a chosen past date

t_0

, before the outcome existed in either channel. Hindcast replays resolved Polymarket prediction markets against a frozen snapshot of public Reddit, lets the model read only posts written before

t_0

, and scores each forecast against both what happened and the market's own price at

t_0

, itself a human forecast made from the same past information. Because the cutoff is set per market and the snapshot never changes, the evaluation re-runs on new markets as models improve, without going stale. Once the leak is closed, retrieval still helps most models, but only where Reddit discussed the event beforehand. Where the archive carried only speculation, retrieval hurts.

Abstract

Probabilistic Forecasting

Explore similar work

Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters

TimeSeek: Temporal Reliability of Agentic Forecasters

PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Data