PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Data

Pu Cheng; Juncheng Liu; Yunshen Long

q-fin.CPApr 3, 2026

PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Data

Authors: Pu Cheng, Juncheng Liu, Yunshen Long

Organizations: College of Electrical Engineering, Sichuan University, Chengdu, Sichuan, China · School of Economics, Sichuan University, Chengdu, Sichuan, China

Abstract

Predicting real-world events from live market signals demands systems that fuse qualitative news with quantitative order-book dynamics under strict temporal discipline -- a challenge existing benchmarks fail to capture. We present PolyBench, a multimodal benchmark derived from Polymarket that records point-in-time cross-sections of 38,666 binary prediction markets spanning 4,997 events, synchronously coupling each snapshot with a Central Limit Order Book (CLOB) state and a real-time news stream. Using PolyBench, we evaluate seven state-of-the-art Large Language Models -- spanning open- and closed-source families -- generating 36,165 predictions under identical, timestamp-locked market states collected between February 6 and 12, 2026. Our multidimensional framework assesses directional accuracy, our proposed Confidence-Weighted Return (CWR), Annualized Percentage Yield (APY), and Sharpe ratio via realistic order-book execution simulation. The results reveal a pronounced performance divergence: only two of seven models achieve positive financial returns -- MiMo-V2-Flash at 17.6% CWR and Gemini-3-Flash at 6.2% CWR -- while the remaining five incur losses despite uniformly high stated confidence. These findings highlight the gap between surface-level language fluency and genuine probabilistic reasoning under live market uncertainty, and establish PolyBench as a contamination-proof, financially-grounded evaluation standard for future LLM research. Our dataset and code available at https://github.com/PolyBench/PolyBench.

Figures & tables

Figure 1: Confidence-Weighted Return (CWR) timeline reflecting empirical portfolio evolution. Annotated markers denote outsized individual trade returns resulting from correct, high-confidence predictions on low-probability events. Only MiMo-V2-Flash and Gemini-3-Flash sustain positive trajectories, isolating predictive alpha from market noise.

Method	Markets	w/ Polymarket	w/ Order Book	w/ News	CWR Eval.
Time-LLM [ 13 ]	–	$\times$	$\times$	$\times$	$\times$
AgentBench [ 16 ]	–	$\times$	$\times$	$\times$	$\times$
FinBERT [ 1 ]	–	$\times$	$\times$	✓	$\times$
BloombergGPT [ 21 ]	–	$\times$	$\times$	✓	$\times$
Autocast [ 25 ]	6,707	$\times$	$\times$	✓	$\times$
Prophet Arena [ 22 ]	1,367	$\times$	$\times$	✓	$\times$

Table 1: Comparison of prediction-market and forecasting benchmarks. PolyBench is the only benchmark combining Polymarket data, live order books, aligned news, and return-based evaluation at scale.

Figure 2: The four-stage PolyBench construction pipeline: (1) market collection via the Polymarket Gamma API, (2) multi-modal fetch of news and order-book snapshots, (3) LLM batch analysis, and (4) ground-truth resolution matching.

Field	Description
decision	The specific target option identifier
outcome	The exact predicted position mapping
strategy	A classification of the trading rationale (e.g., value_bet , news_catalyst )
confidence	A scalar conviction score ( $c\in[0.0,1.0]$ )
est_resolution_date	The model’s estimated temporal timeline for market settlement
reasoning	A concise, specific rationale validating the position

Table 2: Structured JSON schema requirements for model predictions.

Component	Count	Notes
Events	4,997
Active	2,904	Open at collection time
Resolved	2,093	Closed / expired
Markets	38,666
Active	26,784	Open at collection time
Inactive	11,882	Settled or delisted

Table 3: PolyBench dataset statistics.

Model Name	Company	Service Provider	Quantization	Cost	Open Source
GPT-OSS-120B	OpenAI	DeepInfra	FP4	$ 1.34	Yes
DeepSeek-V3.2	DeepSeek	Atlas Cloud	FP8	$ 3.32
MiMo-V2-Flash	Xiaomi	Xiaomi	FP8	$ 1.72
MiniMax-M2.5	MiniMax	Inceptron	FP8	$ 8.43
Trinity-Large-Preview	Arcee AI	Prime	-	Free
Gemini-3-Flash-Preview	Google	Vertex AI	-	$ 11.50	No

Table 4: Evaluated models grouped by open-source availability, detailing respective companies, API providers, quantization formats, and their costs.

Model	Accuracy $\downarrow$	Avg. Conf.	Non-CWR	CWR	APY	Sharpe
Gemini-3-Flash	75.0%	0.855	4.1%	6.2%	1486.5%	0.01
MiMo-V2-Flash	62.1%	0.859	11.1%	17.6%	4067.3%	0.02
Grok-4.1-Fast	59.1%	0.862	-15.9%	-12.1%	-5820.4%	-0.17
GPT-OSS-120B	58.0%	0.836	-19.4%	-17.8%	-7075.8%	-0.22
DeepSeek-V3.2	53.5%	0.838	-15.3%	-11.4%	-5571.8%	-0.03
Trinity-Large	42.7%	0.871	-12.4%	-9.2%	-4527.8%	-0.02

Table 5: Overall LLMs performance on PolyBench. (Base Lot Size=$10)

Figure 3: Impact of base lot size ( $L$ ) on Confidence-Weighted Return (CWR) for MiMo-V2-Flash and Gemini-3-Flash . As the investment budget scales from $10 to $1,000, algorithmic execution slippage against the limited top levels of the historical order book rapidly decays theoretical alpha. — Figure 3: Impact of base lot size ( $L$ ) on Confidence-Weighted Return (CWR) for MiMo-V2-Flash and Gemini-3-Flash . As the investment budget scales from $10 to$ 1,000, algorithmic execution slippage against the limited top levels of the historical order book rapidly decays theoretical alpha.

Figure 4: Dual-metric radar chart comparing empirical Confidence-Weighted Return (CWR) and Average Declared Confidence across eight event domains. The disparity highlights LLM miscalibration; models uniformly maintain high confidence ( $c\geq 0.8$ ) across all domains, yielding severe negative returns in volatile sectors such as Crypto .

Model	Value Bet	News Cat.	Arbitrage	Stable Yld.	Invalid Strat.
Gemini-3-Flash	80.2% ( 7.2% , 0.01)	40.0% ( 20.7% , 0.03)	0.8% ( 8.1% , 0.07)	28.5% ( -0.1% , 0.00)	8
MiMo-V2-Flash	94.8% ( 19.2% , 0.02)	46.7% ( 50.0% , 0.04)	5.7% (-31.1%, -0.37)	0.7% ( 0.4% , 1.03)	-
Grok-4.1-Fast	99.3% (-11.7%, -0.17)	37.1% (6.7%, 0.04)	3.5% (-10.8%, -0.11)	0.2% (-49.9%, -0.70)	-
GPT-OSS-120B	99.9% (-17.8%, -0.22)	39.4% (-19.3%, -0.21)	1.1% ( -2.7% , -0.05)	2.5% (-0.2%, -0.02)	-
DeepSeek-V3.2	98.4% (-10.9%, -0.03)	34.0% (19.0%, 0.02)	15.3% (-35.4%, -0.37)	20.7% (-22.6%, -0.29)	-
Trinity-Large	98.2% (-9.7%, -0.03)	38.1% (16.6%, 0.01)	5.4% (-10.3%, -0.10)	0.6% (-38.5%, -0.71)	103

Table 6: Strategy utilization (%) paired with subsequent financial performance (CWR%, Sharpe Ratio), alongside execution instruction adherence (Invalid Strat.) in overall accuracy order.

Explore similar work

Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters
Jul 15, 2026Xiao Ye, Jacob Dineen, Evan Zhu +3LLM Evaluation Forecasting Benchmarks
School of Computing and Augmented Intelligence, Arizona State University
TimeSeek: Temporal Reliability of Agentic Forecasters
Apr 5, 2026Hamza Mostafa, Om Shastri, Dennis LeeLLM Agent Reliability Forecasting Benchmarks
The Wharton School, University of Pennsylvania
Prediction Arena: Benchmarking AI Models on Real-World Prediction Markets
Mar 28, 2026Jaden Zhang, Gardenia Liu, Oliver Johansson +3AI Agent Evaluation Forecasting Benchmarks
Arcada Labs · Harvard University

Jul 15, 2026J/K move · Enter open · S save

Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters

Xiao Ye, Jacob Dineen, Evan Zhu, Shijie Lu+2

School of Computing and Augmented Intelligence, Arizona State University

LLM Evaluation Forecasting Benchmarks Temporal Data Leakage

Open paper AskPDF arXiv

Forecasters are evaluated by backtesting, which replays resolved questions and grades the probability the system would have assigned before the outcome was known. For LLMs, two channels leak the answer into this test. A model that retrieves can surface reports written after the event, turning forecasting into a lookup, and each new model is trained on data closer to the event, so a question that lay in the future for last year's models sits inside this year's training data. Either way, the test grades recall while claiming to grade foresight. We introduce Hindcast, which closes both leaks by grading a model as if it stood at a chosen past date

t_0

, before the outcome existed in either channel. Hindcast replays resolved Polymarket prediction markets against a frozen snapshot of public Reddit, lets the model read only posts written before

t_0

, and scores each forecast against both what happened and the market's own price at

t_0

, itself a human forecast made from the same past information. Because the cutoff is set per market and the snapshot never changes, the evaluation re-runs on new markets as models improve, without going stale. Once the leak is closed, retrieval still helps most models, but only where Reddit discussed the event beforehand. Where the archive carried only speculation, retrieval hurts.

Abstract

Explore similar work

Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters

TimeSeek: Temporal Reliability of Agentic Forecasters

Prediction Arena: Benchmarking AI Models on Real-World Prediction Markets