JockeyBench: Evaluating LLM Agents on Horse-Racing Decision-Making

Zhongwei Xie; Xingsu Li; Jingqiao Yu; Zihao Wang; Hong ChiTat; Zhaowei Wang; Weizhi Du; Yuxuan Liu

cs.AIOct 14, 2026

JockeyBench: Evaluating LLM Agents on Horse-Racing Decision-Making

Authors: Zhongwei Xie, Xingsu Li, Jingqiao Yu, Zihao Wang, Hong ChiTat, Zhaowei Wang, Weizhi Du, Yuxuan Liu,

Organizations: The Hong Kong University of Science and Technology · Tsy Capital Limited · Independent researcher · Hong Kong Baptist University

Abstract

Repeated event forecasting requires extracting predictive signals from noisy evidence and evaluating their value across successive decisions. We introduce JockeyBench, a benchmark and evaluation protocol that studies this problem through historical Japanese horse racing. The protocol combines time-gated evidence, priced tools, paired frozen-program controls, and two workflows: executing an LLM-written predictor or acquiring evidence for a fresh per-race LLM forecast. Evaluation separates forecast quality, benchmark-specific net utility, and retrospective betting returns and risk under fixed decision rules. Across 28 model--scaffold configurations on 324 races, paired tests identify seven revision gains and two declines after multiple-testing correction. Yet every configuration has negative mean profit under capped-Kelly settlement. In a four-model comparison, the reusable-program workflow has higher mean betting profit and lower research fees, although both workflows lose on average. Further diagnostics reveal that mean probability bias reverses on selected bets and that profitable individual runs can depend on concentrated returns. These findings motivate evaluating predictive competence, adaptation, resource use, and decision value separately when assessing agents for repeated event forecasting.

Figures & tables

Figure 1: Mean bankroll across repeats for selected configurations under retrospective capped-Kelly staking. Initial bankroll is JPY 100,000; the full-Kelly fraction is capped at 1% of day-start bankroll per race. Returns use historical race outcomes and official JRA WIN dividends, which incorporate the standard 20% WIN takeout. No live bets were placed. Labels identify selected curves and give their final ranks among all 28 configurations. Settlement details: Appendix A.5.4 .

Benchmark	Time-gated data	Predictive code	Feedback loop	Outcome revision	Dual workflows	Payoffs	Risk metrics	Priced tools	Frozen control
ForecastBench	$\checkmark$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$
Prophet Arena	$\checkmark$	$\times$	$\times$	$\times$	$\times$	$\checkmark$	$\checkmark$	$\times$	$\times$
PolyBench	$\checkmark$	$\times$	$\times$	$\times$	$\times$	$\checkmark$	$\checkmark$	$\times$	$\times$
Vending-Bench	$\times$	$\times$	$\checkmark$	$\times$	$\times$	$\checkmark$	$\times$	$\times$	$\times$
CostBench	$\times$	$\times$	$\checkmark$	$\times$	$\times$	$\times$	$\times$	$\checkmark$	$\times$
MLE-bench	$\times$	$\checkmark$	$\checkmark$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$

Table 1: Design features of related evaluations. ✓: included;

\times

: absent or unreported in the reviewed protocol. Outcome revision updates programs using released results; dual workflows compares program-generated and per-event LLM forecasts. Payoffs include returns, profit, or net worth; risk metrics include drawdown or Sharpe. Full criteria: Appendix A.6 .

Figure 2: Two forecasting workflows under the same replay clock. Program executes an LLM-written forecasting program, whereas Adaptive Acquisition executes an LLM-written acquisition program before a fresh per-race LLM forecast. Dashed arrows denote next-day revision after outcome release.

	Forecast quality				Delivery	Utility	Daily revision		Cost	Run setup
Model / reference	Log gain $\uparrow$ mean $\pm$ SD	S log gain $\uparrow$ mean	Top-1 % $\uparrow$	Top-3 % $\uparrow$	Predictor %	Net $\uparrow$ /race	A $-$ S [95% CI]	Validated versions	Fees /race	Access
GPT-6.1-Sol	$\mathbf{0.525}$ $\pm 0.002$	$\mathbf{0.525}$	$\underline{28.5}$	$\mathbf{48.3}$	100	$\underline{+59.5}$	$+0.000\;[+0.000,\,+0.000]$	1.0	14.1	Codex
GPT-6-Astra	$\underline{0.523}$ $\pm 0.008$	$\underline{0.523}$	$\mathbf{30.0}$	$\underline{48.0}$	100	$\mathbf{+61.3}$	$+0.000\;[+0.000,\,+0.000]$	1.0	13.6	Codex
Claude Opus 5	$0.484$ $\pm 0.023$	$0.485$	$28.2$	$46.3$	100	$+54.3$	$-0.001\;[-0.008,\,+0.006]$	4.0	14.9	Claude Code
GPT-5.6-Sol	$0.473$ $\pm 0.023$	$0.468$	$27.8$	$45.9$	100	$+50.6$	$+0.004\;[-0.005,\,+0.014]$	4.2	17.0	Codex
GPT-6-Sol	$0.471$ $\pm 0.019$	$0.462$	$27.3$	$45.4$	100	$+50.2$	$+0.009\;[-0.000,\,+0.018]$	4.2	16.7	Codex

Table 2: Main Program results: 324 races, ten race days, 138 matched pairs. A: daily revision; S: frozen initial program. Agent metrics describe A unless labeled otherwise. Log gain (nats relative to uniform): A mean

\pm

between-run SD; S paired-run mean. A

-

S: paired mean log-gain difference. Top-1/Top-3: top-pick win rate/precision at three. Predictor: non-uniform program delivery rate. Validated versions: mean accepted programs per run, including the initial program. Net/fees: score-yen per race, excluding provider charges.

\uparrow

: higher is better. Bold/underline: highest/second-highest agent means (ties retained); ranks are descriptive. Darker forecast/delivery/net cells indicate higher values within columns; teal/orange A

-

S cells mark significant gains/losses.

\dagger

: Holm-adjusted

p<.05

across 28 tests; 95% CIs are pointwise and day-clustered. Closing market is an external reference; dashes: not applicable. Run counts: Appendix Table 6 .

Reference	6 Jan–3 Feb 2024		Jan–Jun 2024	From 5 Oct 2024	Predictor
	Log gain $\uparrow$	Top-1 accuracy $\uparrow$	Log gain $\uparrow$	Log gain $\uparrow$
Agent-interface references: R1–R3 fitted on 87 support races
R0: uniform	0.000	0.071	0.000	0.000	Equal probabilities
R1: race-card features	0.155	0.157	0.160	0.088	Conditional logit
R2: card + recent history	0.343	0.238	0.363	0.239	Conditional logit
R3: expanded factual features	0.298	0.210	0.310	0.167	Conditional logit

Table 3: Forecasting references across three periods in 2024: main evaluation (324 races), Jan–Jun (1,800 races, including main), and autumn (323 races over ten race days). Log gain is in nats relative to uniform; Top-1 is the win rate of top-ranked non-scratched runners (0–1).

\uparrow

: higher is better. Darker cells indicate higher values; log-gain columns share a scale, while accuracy is scaled separately. R0 reports empirical accuracy under a seeded pseudorandom ranking. Agent-inspired features were selected post hoc. Methods: Appendix A.4 .

Figure 3: Mean cumulative retrospective betting P&L by workflow.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Synthetic example of preprocessing and information filtering. The runner, identifiers, dates, and record values are invented. Struck-through fields are excluded, and the dashed result is withheld until release. The card and form views illustrate identifier replacement, relative dates, typed values, and explicit missing-value reasons.

Group	Fields	Content
Race	23	distance; surface; direction; course layout and variant; venue; meeting day; age class; eligibility condition; weight rule; grade; declared field size; turf grass type and height; rolling and antifreeze treatment; rainfall; forecast weather; forecast turf and dirt going; inner, middle and outer turf condition
Runner	25	carried weight; draw; sex; age in days; blinkers; body weight and its change (when on the card); layoff duration; transport; earned and class prize money; starts and days since stabling; class-drop, rest-reason and hoof codes; late-start rate; sire and damsire turf and dirt top-two rates and average distance; sire and damsire IDs
Workout	13	workout distance; early, middle and late split times; effort and urging; rider role; training partner’s result, class, age and effort; course type; distance type
Past start	30	finishing position; abnormal result; distance; surface; going; venue; condition; grade; age class; field size; draw; carried weight; body weight and its change; first and last three-furlong times; margin to the winner; fourth-corner position and path; path; overall, early, middle and late trouble; race pace and own pace; running style; slow start; jockey and trainer IDs
Connections	6	jockey and trainer starts, wins and top-three finishes over the preceding 365 days, counted from released results
Withheld	–	odds and popularity (current and past races); dividends; expert indices, ratings and marks; predicted positions; aptitude, class and track-bias assessments; free text; real names; absolute dates

Appendix

Table 4: Fields available to the agent and its program. Each card also carries the entrant number and pseudonymous horse, jockey and trainer IDs; categorical codes come without a codebook, and unknown values are null with a reason.

Display name	Recorded model identifier	Route	Effort	Model responses per run
GPT-6-Astra	gpt-6-astra	Codex	high	26.0
Claude Fable 5.1	claude-fable-5-1	Claude Code	high	52.0
GPT-5.6-Sol	gpt-5.6-sol	Codex	high	74.0
GPT-6-Sol	gpt-6-sol	Codex	high	86.7
Claude Opus 5	claude-opus-5	Claude Code	high	47.7
Claude Opus 5.5	claude-opus-5-5	Claude Code	high	30.3

Appendix

Table 5: Recorded identifiers, requested reasoning settings, and mean model responses per A trajectory in the earlier 25-configuration export. The main cohort contains 28 configurations. Responses count model outputs; data-tool calls may be batched differently by route. Each configuration combines a model with its execution scaffold. Exact run IDs, initial-program hashes, tariff and schedule fingerprints are in the accompanying manifest; model IDs retain the recorded provider aliases.

Model	Runs A/S
GPT-6.1-Sol	5/5
GPT-6-Astra	5/5
Claude Opus 5	4/4
GPT-5.6-Sol	5/5
GPT-6-Sol	5/5
Claude Fable 5.1	4/4

Appendix

Table 6: Run counts for the main Program cohort in Table 2 . A: daily revision; S: frozen initial program. Runs A/S counts complete 324-race trajectories in each arm. The cohort contains 138 matched pairs across 28 configurations, listed in the same order as the main table.

Analysis	Coverage	Sample
Main Program comparison	28 configurations	138 A/S pairs
Delivery archive	25 configurations	Earlier export
Flat-WIN archive	22 models	54 Program trajectories
Workflow forecasts	4 models	31 trajectories
Workflow betting and costs	4 models	Same 31 trajectories
Matched acquisition controls	2 models	3 Frozen/Adaptive pairs

Appendix

Table 7: Samples used by the principal analyses.

Figure 5: Forecast quality and net score for every daily-revision run (28 configurations, 138 runs). Dots are individual runs and diamonds their configuration means; open circles in the left panel are the paired frozen-program means. The dotted line marks the closing market’s log gain (left); the market has no net score. Runs share the same outcomes, so their spread reflects session variability rather than independent replications.

	Sealed prediction outcomes					Session diagnostics
Model	Predictor	Silent unif.	Explicit unif.	No program	Fallback	SDK misuse	No save
GPT-6-Astra	972	0	0	0	0	1	0
Claude Fable 5.1	972	0	0	0	0	0	0
GPT-5.6-Sol	972	0	0	0	0	0	0
GPT-6-Sol	972	0	0	0	0	0	0
Claude Opus 5	972	0	0	0	0	0	0

Appendix

Table 8: Delivery and session diagnostics for an earlier export of 25 configurations (77 A trajectories; 24,948 forecasts). The first five columns partition forecasts into non-uniform predictors, exactly uniform program outputs, explicit uniform choices, missing programs, and execution fallbacks. SDK misuse and No save count events and sessions, respectively. This is not the full main cohort.

	F18	FAGENT		FV5
Train $N$	Logit	Logit	Logit $\oplus$ HGB	Logit	HGB	Logit $\oplus$ HGB
100	0.411	0.470	0.438	0.394	0.327	0.431
300	0.442	0.487	0.462	0.492	0.405	0.493
1,000	0.461	0.501	0.494	0.536	0.466	0.535
3,000	0.462	0.502	0.510	0.577	0.571	0.597
10,000	0.463	0.503	0.519	0.584	0.606	0.613

Appendix

Table 9: Smoothed log gain on the 324-race main window by training-set size and feature representation. F18: 18 hand-built features; FAGENT: agent-inspired features chosen post hoc; FV5: broad factual features (definitions in Appendix A.4 ). Logit: conditional logit; HGB: histogram gradient boosting;

\oplus

: logit averaging before softmax. Training subsets differ in size and historical composition; columns vary feature representation and fitting method.

Reference	January–June 2024	Autumn 2024
Uniform (R0)	0	0
Race-card features (R1)	0.160	0.088
Card and recent history (R2)	0.363	0.239
Expanded factual features (R3)	0.310	0.167
18 hand-built features + logit (F18)	0.457	0.369
Agent-inspired feature ensemble (FAGENT)	0.496	0.405

Appendix

Table 10: Smoothed log gain on two additional 2024 windows (nats relative to uniform; higher is better). January–June contains 1,800 races, including the 324-race main window. Autumn contains 323 races over ten days from 5 October. Offline references use 20,730 pre-2024 training races for January–June and a separate pool of 23,345 races through 29 September for Autumn. These are reference-model evaluations, not additional agent trajectories.

DeepSeek-V4-Flash
		Probabilistic score	Ranking accuracy
Workflow	$n$	Smoothed log gain (nats)	Top-1 (%)	Top-3 precision (%)
Program	6	$0.232\;(0.128)$	$20.63\;(3.18)$	$39.25\;(3.48)$
Adaptive Acquisition	6	$0.256\;(0.107)$	$21.35\;(2.39)$	$40.91\;(2.86)$
GLM-5.3-Flash
Program	5	$0.286\;(0.225)$	$23.40\;(2.43)$	$42.45\;(2.34)$

Appendix

Table 11: Forecast quality by workflow. Values are trajectory means, with sample standard deviations in parentheses; – denotes

n=1

Figure 6: Trajectory-level forecast quality by workflow.

Synthetic scores (score-yen)
Metric	DeepSeek R1		DeepSeek R2		GPT-6-Astra R1
	Frozen- $P_{0}$	Adaptive	Frozen- $P_{0}$	Adaptive	Frozen- $P_{0}$	Adaptive
Gross score	$14{,}502.13$	$13{,}853.16$	$14{,}641.41$	$13{,}766.93$	$23{,}255.99$	$22{,}071.05$
Weighted fees	$12{,}609.19$	$6{,}586.46$	$11{,}202.53$	$10{,}259.94$	$8{,}758.57$	$7{,}777.62$
Net score	$1{,}892.94$	$7{,}266.70$	$3{,}438.88$	$3{,}506.99$	$14{,}497.42$	$14{,}293.44$
Forecast quality

Appendix

Table 12: Matched scores, forecast quality, and betting outcomes.

Measure	Frozen $P_{0}$	Adaptive R1	Reduction
Logical calls	4,568	392	4,176
Native acquisitions	3,500	287	3,213
Newly charged cells	1,136,800	92,904	1,043,896
Weighted data fee	5,684.00	464.52	5,219.48
Weighted call fee	875.00	71.75	803.25
Weighted total	6,559.00	536.27	6,022.73

Appendix

Table 13: DeepSeek R1 condition-statistics accounting.

Model (rounds pooled)	Win ROI %	[95% bootstrap]	Place ROI %
GPT-6-Astra (3)	-5.0	[-21, +11]	-16.3
Claude Fable 5.1 (3)	-12.7	[-27, +2]	-15.6
GPT-5.6-Sol (3)	-7.6	[-24, +10]	-14.9
GPT-6-Sol (3)	-8.1	[-25, +10]	-15.9
Claude Opus 5 (3)	-10.0	[-26, +7]	-14.1
Claude Opus 5.5 (3)	-10.5	[-25, +5]	-15.9

Appendix

Table 14: Retrospective settlement at official dividends (takeout included): flat 100-yen WIN or PLACE tickets on the top-ranked runner, 324 races per round. The 23 September archive is intersected with the main manifest by exact run ID and Program protocol: 54 A trajectories from 22 models. Archived per-model bootstrap intervals are retained only after verifying their replicate lists. Resampling is by race, without day dependence. All-model pooled WIN ROI: -19.7%.

Figure 7: Mean cumulative profit across repeats from flat JPY 100 WIN stakes at official dividends (28 configurations, 138 runs). Shading shows drawdown. Footers report repeat counts, annualized profit-based Sharpe ratio (SR), maximum drawdown as a signed JPY loss, and the longest run below the prior peak. † marks configurations that never delivered a non-uniform predictor.

Flat JPY 100 WIN
			Return		Mean-path risk
Model	Workflow	$n$	Profit (JPY)	ROI (%)	Max. DD (JPY)	Annualized P&L SR
DeepSeek	Program	6	$-7{,}738$	$-23.88$	$7{,}738$	$-11.73$
DeepSeek	Adaptive Acquisition	6	$-8{,}175$	$-25.23$	$9{,}415$	$-11.23$
GLM	Program	5	$-5{,}732$	$-17.69$	$7{,}446$	$-7.76$
GLM	Adaptive Acquisition	1	$-8{,}700$	$-26.85$	$9{,}470$	$-8.99$

Appendix

Table 15: Retrospective betting returns and mean-path risk on 324 races.

n

counts trajectories with 324 scored forecasts. Profit is mean terminal P&L. ROI pools profits and stakes. Drawdown and annualized P&L SR use the equal-weight mean per-race P&L path, rather than averaged individual risks. P&L SR is not account-return Sharpe. Appendix A.5.5 defines the measures and completion accounting and discusses annualization qualifications ( Sharpe, 1994 ; Lo, 2002 ) .

Figure 8: Retrospective betting P&L across trajectories.

Figure 9: Return concentration in DeepSeek Adaptive Acquisition.

Model	Workflow	$n$	Gross score	Weighted fees	Net score
DeepSeek-V4-Flash	Program	6	$11{,}793.52$	$4{,}553.46$	$7{,}240.05$
DeepSeek-V4-Flash	Adaptive Acquisition	6	$12{,}793.60$	$9{,}104.15$	$3{,}689.45$
GLM-5.3-Flash	Program	5	$14{,}420.59$	$7{,}299.45$	$7{,}121.14$
GLM-5.3-Flash	Adaptive Acquisition	1	$13{,}861.49$	$10{,}974.81$	$2{,}886.68$
GPT-6-Astra	Program	5	$24{,}254.02$	$4{,}408.87$	$19{,}845.15$
GPT-6-Astra	Adaptive Acquisition	2	$22{,}661.31$	$8{,}428.89$	$14{,}232.42$

Appendix

Table 16: Mean workflow scores per trajectory (score-yen).

Figure 10: Workflow rewards, research fees, and net scores.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,217 papers

Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents
Sep 24, 2026Liqin Ye, Haorui Wang, Fardin Ahmed +8LLM Agent Evaluation Forecasting Benchmarks
Georgia Institute of Technology · Amazon · University of Florida
AlphaBet: Reinforcement Learning for LLM Event Forecasting Agents
Oct 15, 2026Liqin Ye, Rongzhi Zhang, Xin Liu +5Probabilistic Forecasting RL for LLM Agents
Georgia Institute of Technology · Amazon
TimeSeek: Temporal Reliability of Agentic Forecasters
Apr 5, 2026Hamza Mostafa, Om Shastri, Dennis LeeLLM Agent Reliability Forecasting Benchmarks
The Wharton School, University of Pennsylvania

Sep 24, 2026J/K move · Enter open · S save

Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents

Liqin Ye, Haorui Wang, Fardin Ahmed, Rongzhi Zhang+7

Georgia Institute of Technology · Amazon · University of Florida

LLM Agent Evaluation Forecasting Benchmarks LLM Agent Training

Open paper AskPDF arXiv

Forecasting requires large language model (LLM) agents to gather evolving evidence, revise their beliefs, and make calibrated predictions over time. Yet evaluating and improving these capabilities is challenging because real-world events unfold continuously and outcomes may take months to resolve. We introduce Forecast-Dojo, a replayable environment that reconstructs resolved prediction-market events as sequences of dated forecasting steps, enabling reproducible evaluation, longitudinal analysis, and training from agent interactions. It comprises 1,568 Polymarket events split temporally into training and evaluation sets, alongside 18.8 million dated news articles. Across 12 models, research tools consistently improve forecasting performance, and greater capture of new evidence is associated with larger improvements over time. Nevertheless, all models trail historical market forecasts, while carrying a belief notebook across dates reduces research costs without consistently improving forecast quality. We supervise fine-tuned Qwen3-30B-A3B-Thinking-2507 on collected trajectories from Qwen3-235B-A22B-Thinking-2507 to demonstrate the environment's training utility. The student's accuracy is improved from 34.8% to 42.8% on the evaluation set.

Abstract

LLM Agent Evaluation

Explore similar work

Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents

AlphaBet: Reinforcement Learning for LLM Event Forecasting Agents

TimeSeek: Temporal Reliability of Agentic Forecasters