TimeSeek: Temporal Reliability of Agentic Forecasters

Hamza Mostafa; Om Shastri; Dennis Lee

cs.AIApr 5, 2026

TimeSeek: Temporal Reliability of Agentic Forecasters

Authors: Hamza Mostafa, Om Shastri, Dennis Lee

Organizations: The Wharton School, University of Pennsylvania

Abstract

We introduce TimeSeek, a benchmark for studying how the reliability of agentic LLM forecasters changes over a prediction market's lifecycle. We evaluate 10 frontier models on 150 CFTC-regulated Kalshi binary markets at five temporal checkpoints, with and without web search, for 15,000 forecasts total. Models are most competitive early in a market's life and on high-uncertainty markets, but much less competitive near resolution and on strong-consensus markets. Web search improves pooled Brier Skill Score (BSS) for every model overall, yet hurts in 12% of model-checkpoint pairs, indicating that retrieval is helpful on average but not uniformly so. Simple two-model ensembles reduce error without surpassing the market overall. These descriptive results motivate time-aware evaluation and selective-deference policies rather than a single market snapshot or a uniform tool-use setting.

Figures & tables

Characteristic	Value
Markets / Checkpoints / Predictions	150 / 5 / 750 per condition
Categories	5 (30 markets each)
Timeline	Oct 2025 – Jan 2026
Trading volume	$5.6k –$ 41.3M (med. $35k)
Duration (days)	2 – 337 (med. 89)
Outcomes	73% NO, 27% YES

Table 1: TimeSeek dataset characteristics.

Model	Open+1	25%	50%	75%	Close-1	Pooled
Claude Opus 4.5	+0.167	+0.059	-0.136	-0.165	-0.829	-0.068
GPT-5.2	+0.085	-0.095	-0.344	-0.595	-0.960	-0.252
Gemini 3 Pro	-0.084	-0.250	-0.373	-0.510	-0.734	-0.312
Grok 4.1-fast	-0.003	-0.071	-0.314	-0.621	-0.909	-0.267
Kimi-k2.5	+0.118	-0.022	-0.311	-0.579	-0.942	-0.214
Kimi-k2	+0.011	-0.113	-0.601	-0.912	-0.823	-0.367

Table 2: BSS across temporal checkpoints with web search. Bold = beating market (BSS

>0

Model	Open+1	25%	50%	75%	Close-1	Pooled
Claude Opus 4.5	+0.189	+0.378	+0.540	+0.816	+1.383	+0.520
GPT-5.2	-0.040	+0.108	+0.130	+0.087	+0.893	+0.140
Gemini 3 Pro	-0.080	+0.218	+0.246	+0.525	+1.693	+0.325
Grok 4.1-fast	+0.214	+0.457	+0.721	+0.679	+1.710	+0.592
Kimi-k2	-0.025	+0.234	-0.135	-0.146	+1.388	+0.135
Kimi-k2.5	+0.110	+0.180	+0.172	+0.195	+0.777	+0.218

Table 3:

\Delta

BSS (agentic minus baseline). Bold = search helped. Shaded = search hurt—model was better without tools.

Searches	$n$	BSS
1–3	48	-0.811
4–7	152	+0.109
8–12	238	+0.007
13+	312	-0.115

Table 4: Search count vs. BSS (Claude). Performance is non-monotonic across query-count buckets.

Model	Easy	Medium	Hard	Toss-up	Overall
Claude Opus 4.5	-0.692	-0.188	+0.074	+0.301	-0.068
GPT-5.2	-1.069	-0.433	-0.001	+0.217	-0.252
Gemini 3 Pro	-0.829	-0.365	-0.346	+0.047	-0.312
Grok 4.1-fast	-0.905	-0.256	-0.201	+0.002	-0.267
Kimi-k2.5	-1.125	-0.316	+0.077	+0.179	-0.214
Kimi-k2	-0.906	-0.555	-0.296	+0.098	-0.366

Table 5: BSS stratified by market difficulty (

d=|p_{m}-0.5|

): Easy

d>0.4

, Medium

0.2<d\leq 0.4

, Hard

0.1<d\leq 0.2

, Toss-up

d\leq 0.1

. Bold = beating market.

Model	Pearson $r$
Intellect-3	0.193
Claude Opus 4.5	0.211
Trinity Large	0.218
Kimi-k2.5	0.222
DeepSeek v3.2	0.256
Qwen3-235B	0.261

Table 6: Error correlation (

|\hat{p}-y|

vs.

|p_{m}-y|

). Lower

r

indicates less overlap between model and market error patterns. All

p<10^{-6}

Model	Politics	Sports	Macro	Sci/Tech	Financial
Claude Opus 4.5	-0.379	+0.294	+0.163	-0.154	-0.609
GPT-5.2	-0.484	-0.088	+0.021	-0.378	-0.482
Gemini 3 Pro	-0.598	-0.227	+0.033	-0.575	-0.217
Grok 4.1-fast	-0.427	-0.197	-0.182	-0.190	-0.426
Kimi-k2.5	-0.596	+0.124	-0.066	-0.213	-0.625
Kimi-k2	-0.421	-0.342	-0.023	-0.369	-0.788

Table 7: BSS by category (agentic condition). Bold = beating markets.

Model	Politics	Sports	Macro	Sci/Tech	Financial
Claude Opus 4.5	-0.107	+0.657	+1.571	+0.217	+0.072
GPT-5.2	-0.479	+0.293	+0.870	-0.170	+0.074
Gemini 3 Pro	+0.135	+0.218	+1.084	-0.267	+0.604
Grok 4.1-fast	+0.209	+0.440	+1.392	+0.263	+0.792
Kimi-k2.5	-0.384	+0.660	+0.711	-0.113	-0.129
Kimi-k2	-0.030	+0.252	+0.868	-0.169	-0.430

Table 8:

\Delta

BSS by category (agentic minus baseline). Bold = search helped. Shaded = search hurt. Bottom row: models helped / total and category average.

Explore similar work

Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters
Jul 15, 2026Xiao Ye, Jacob Dineen, Evan Zhu +3LLM Evaluation Forecasting Benchmarks
School of Computing and Augmented Intelligence, Arizona State University
Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents
Sep 24, 2026Liqin Ye, Haorui Wang, Fardin Ahmed +8LLM Agent Evaluation Forecasting Benchmarks
Georgia Institute of Technology · Amazon · University of Florida
Foresight Arena: An On-Chain Benchmark for Evaluating AI Forecasting Agents
May 1, 2026Maksym Nechepurenko, Pavel ShuvalovAI Agent Evaluation Forecasting Benchmarks
Director of Research, Devnull, Dubai, UAE · Chief Technology Officer, Devnull, Dubai, UAE

Jul 15, 2026J/K move · Enter open · S save

Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters

Xiao Ye, Jacob Dineen, Evan Zhu, Shijie Lu+2

School of Computing and Augmented Intelligence, Arizona State University

LLM Evaluation Forecasting Benchmarks Temporal Data Leakage

Open paper AskPDF arXiv

Forecasters are evaluated by backtesting, which replays resolved questions and grades the probability the system would have assigned before the outcome was known. For LLMs, two channels leak the answer into this test. A model that retrieves can surface reports written after the event, turning forecasting into a lookup, and each new model is trained on data closer to the event, so a question that lay in the future for last year's models sits inside this year's training data. Either way, the test grades recall while claiming to grade foresight. We introduce Hindcast, which closes both leaks by grading a model as if it stood at a chosen past date

t_0

, before the outcome existed in either channel. Hindcast replays resolved Polymarket prediction markets against a frozen snapshot of public Reddit, lets the model read only posts written before

t_0

, and scores each forecast against both what happened and the market's own price at

t_0

, itself a human forecast made from the same past information. Because the cutoff is set per market and the snapshot never changes, the evaluation re-runs on new markets as models improve, without going stale. Once the leak is closed, retrieval still helps most models, but only where Reddit discussed the event beforehand. Where the archive carried only speculation, retrieval hurts.

Abstract

Explore similar work

Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters

Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents

Foresight Arena: An On-Chain Benchmark for Evaluating AI Forecasting Agents