Prediction Arena: Benchmarking AI Models on Real-World Prediction Markets

Jaden Zhang; Gardenia Liu; Oliver Johansson; Hileamlak Yitayew; Kamryn Ohly; Grace Li

cs.LGMar 28, 2026

Prediction Arena: Benchmarking AI Models on Real-World Prediction Markets

Authors: Jaden Zhang, Gardenia Liu, Oliver Johansson, Hileamlak Yitayew, Kamryn Ohly, Grace Li

Organizations: Arcada Labs · Harvard University

Abstract

We introduce Prediction Arena, a benchmark for evaluating AI models' predictive accuracy and decision-making by enabling them to trade autonomously on live prediction markets with real capital. Unlike synthetic benchmarks, Prediction Arena tests models in environments where trades execute on actual exchanges (Kalshi and Polymarket), providing objective ground truth that cannot be gamed or overfitted. Each model operates as an independent agent starting with $10,000, making autonomous decisions every 15-45 minutes. Over a 57-day longitudinal evaluation (January 12 to March 9, 2026), we track two cohorts: six frontier models in live trading (Cohort 1, full period) and four next-generation models in paper trading (Cohort 2, 3-day preliminary). For Cohort 1, final Kalshi returns range from -16.0% to -30.8%. Our analysis identifies a clear performance hierarchy: initial prediction accuracy and the ability to capitalize on correct predictions are the main drivers, while research volume shows no correlation with outcomes. A striking cross-platform contrast emerges from parallel Polymarket live trading: Cohort 1 models averaged only -1.1% on Polymarket vs. -22.6% on Kalshi, with grok-4-20-checkpoint achieving a 71.4% settlement win rate - the highest across any platform or cohort. gemini-3.1-pro-preview (Cohort 2), which executed zero trades on Kalshi, achieved +6.02% on Polymarket in 3 days - the best return of any model across either cohort - demonstrating that platform design has a profound effect on which models succeed. Beyond performance, we analyze computational efficiency (token usage, cycle time), settlement accuracy, exit patterns, and market preferences, providing a comprehensive view of how frontier models behave under real financial pressure.

Figures & tables

Figure 1: Trading cycle flow showing the sequence of operations from market data sync through metrics calculation. Each cycle follows a structured process ensuring agents have current information and can make informed decisions.

Figure 2: Final account values as of March 9, 2026 for all Cohort 1 models. The dotted line marks the $10,000 starting capital.

Model	Final Value	Total PnL	Ph. 1 Ret.	Total Ret.	Win Rate (Early Exit)	Max DD
glm-4.7	$8,398	$-$ $1,601	$-7.2\%$	$-16.0\%$	18.9%	16.3%
grok-4-20-checkpoint	$7,999	$-$ $2,001	$-4.4\%$	$-20.0\%$	31.5%	30.9%
gpt-5.2	$7,950	$-$ $2,050	$-11.9\%$	$-20.5\%$	20.9%	18.4%
claude-opus-4-5-20251101	$7,409	$-$ $2,591	$-7.2\%$	$-25.9\%$	24.4%	25.9%
gemini-3-pro-preview	$6,950	$-$ $3,050	$-25.3\%$	$-30.5\%$	24.3%	30.8%
grok-4-1-fast-reasoning	$6,925	$-$ $3,075	$-26.8\%$	$-30.8\%$	21.1%	30.8%

Table 1: Overall performance of Cohort 1 (Legacy) models on Prediction Arena across the full 57-day evaluation period (Jan 12–Mar 9, 2026). Models are ranked by final account value. Ph. 1 Return covers Jan 12–Feb 14 (34 days); Total Return covers the full period.

Figure 3: Settlement win rate comparison across all Cohort 1 models (full 57-day period). Win rate measures the fraction of resolved positions that were profitable.

Figure 4: Full 57-day account value evolution for Cohort 1 (Jan 12–Mar 9, 2026). Phase 2 is shaded grey. Note the peak of grok-4-20-checkpoint at $+15.5\%$ ( $\$11{,}554$ ) on February 6 before its subsequent drawdown. — Figure 4: Full 57-day account value evolution for Cohort 1 (Jan 12–Mar 9, 2026). Phase 2 is shaded grey. Note the peak of grok-4-20-checkpoint at $+15.5\%$ ( $\$ 11{,}554$ ) on February 6 before its subsequent drawdown.

Figure 5: Cross-generation comparison: Cohort 1 Phase 1 returns vs. Cohort 2 paper-trading returns. gemini-3.1-pro-preview not displayed as it performed zero Kalshi trades.

Model	Start Date	Days	Account Value	Return	Note
GPT-5.4	Mar 6, 2026	3	$10,121.96	$+1.22\%$	Paper trading, 5 trades
Claude Opus 4.6	Mar 6, 2026	3	$9,988.90	$-0.11\%$	Paper trading, 46 trades
GLM-5	Mar 6, 2026	3	$9,590.86	$-4.09\%$	Paper trading, 12 trades
Gemini 3.1 Pro Preview	Mar 6, 2026	3	$10,000.00	$0.00\%$	No Kalshi trades; see Table 3

Table 2: Cohort 2 (Next-Gen) preliminary Kalshi results (Mar 6–9, 2026, 3 days; paper trading, simulated capital). Account values and returns reflect Kalshi activity only; Polymarket results are in Table 3 . Not directly comparable to Cohort 1 live-trading figures. Data insufficient for statistical inference; presented as an early directional signal only.

Figure 6: Cohort 1 Polymarket live-trading returns (Feb 9–Mar 9, real capital). Return values annotated above each bar. The concurrent Kalshi average ( $-22.6\%$ ) is shown for reference.

Figure 7: Cohort 2 Polymarket paper-trading returns (Mar 6–9, simulated capital). gemini-3.1-pro-preview achieves the highest return across either cohort ( $+6.02\%$ ).

Model	Cohort	Period	Return	Win Rate	Trades	Note
grok-4-20-checkpoint	1	Feb 9–Mar 9	$-1.85\%$	$71.4\%$	18	Highest WR across all settings
gemini-3-pro-preview	1	Feb 9–Mar 9	$-1.81\%$	$14.3\%$	155	High volume, low accuracy
gpt-5.2	1	Feb 9–Mar 9	$-0.42\%$	$43.5\%$	127	Nearly flat
glm-4.7	1	Feb 9–Mar 9	$-0.09\%$	$10.5\%$	95	Nearly flat despite low WR
claude-opus-4-5-20251101	1	Feb 9–Mar 9	$-2.68\%$	$33.3\%$	86
grok-4-1-fast-reasoning	1	Feb 9–Mar 9	$0.00\%$	N/A	0	No trades executed

Table 3: Polymarket trading results for both cohorts. Cohort 1 runs use real capital on live markets; Cohort 2 runs use simulated capital (paper trading). Win rate is computed over settled positions only; N/A indicates fewer than 5 settled positions.

Figure 8: Initial prediction accuracy comparison across models, showing the percentage of first trades on markets that were ultimately correct.

Figure 9: Exit strategy comparison showing settlement rates versus early exit rates across models, revealing distinct trading philosophies.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Settlement win rate heatmap by model and market category. Darker cells indicate higher win rates. Weather markets dominate settlement volume; model differences are most pronounced in Financial and Weather categories.

Explore similar work

Foresight Arena: An On-Chain Benchmark for Evaluating AI Forecasting Agents
May 1, 2026Maksym Nechepurenko, Pavel ShuvalovAI Agent Evaluation Forecasting Benchmarks
Director of Research, Devnull, Dubai, UAE · Chief Technology Officer, Devnull, Dubai, UAE
PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Data
Apr 3, 2026Pu Cheng, Juncheng Liu, Yunshen LongForecasting Benchmarks LLM Evaluation
College of Electrical Engineering, Sichuan University, Chengdu, Sichuan, China · School of Economics, Sichuan University, Chengdu, Sichuan, China
When do prophets profit in prediction markets?
Jul 7, 2026Anri Gu, Nicole Kagan, Alec Sun +2Probabilistic Forecasting Proper Scoring Rules
University of Chicago · Kalshi Research · New York University, Shanghai

May 1, 2026J/K move · Enter open · S save

Foresight Arena: An On-Chain Benchmark for Evaluating AI Forecasting Agents

Maksym Nechepurenko, Pavel Shuvalov

Director of Research, Devnull, Dubai, UAE · Chief Technology Officer, Devnull, Dubai, UAE

AI Agent Evaluation Forecasting Benchmarks Probabilistic Forecasting Proper Scoring Rules Prediction Markets

Open paper AskPDF arXiv

Evaluating the true forecasting ability of AI agents requires environments that are resistant to environments resistant to overfitting, free from centralized trust, and grounded in incentive-compatible scoring. Existing benchmarks either rely on static datasets vulnerable to training-data contamination, or measure trading PnL -- a metric conflating predictive accuracy with timing, sizing, and risk appetite. We introduce Foresight Arena, the first permissionless, on-chain benchmark for evaluating AI forecasting agents on real-world prediction markets. Agents submit probabilistic forecasts on binary Polymarket markets via a commit-reveal protocol enforced by Solidity smart contracts on Polygon PoS; outcomes are resolved trustlessly through the Gnosis Conditional Token Framework. Performance is measured by the Brier Score and a novel Alpha Score -- proper scoring rules that incentivize honest probability reporting and isolate predictive edge over market consensus. We provide a formal analysis: closed-form variance for per-market Alpha, the connection to Murphy's classical Brier decomposition, and a power analysis characterizing the number of rounds required to reliably distinguish agents of different skill levels. We show that detecting a true edge of

α^* = 0.02

at 80% power requires approximately 350 resolved binary predictions (50 rounds of 7 markets), while

α^* = 0.01

requires four times more. We complement these analytical results with a deterministic, seed-controlled simulation study calibrated to literature-reported Brier-score ranges, illustrating how Murphy decomposition distinguishes well-calibrated agents from market-tracking agents that fail through reduced resolution. Live results from the deployed benchmark will be reported in a future revision. All smart contracts and evaluation infrastructure are open-source.

Abstract

Explore similar work

Foresight Arena: An On-Chain Benchmark for Evaluating AI Forecasting Agents

PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Data

When do prophets profit in prediction markets?