TraderBench: How Robust Are AI Agents in Adversarial Capital Markets?

Xiaochuang Yuan; Hui Xu; Silvia Xu; Cui Zou; Jing Xiong

cs.AIFeb 27, 2026

TraderBench: How Robust Are AI Agents in Adversarial Capital Markets?

Authors: Xiaochuang Yuan, Hui Xu, Silvia Xu, Cui Zou, Jing Xiong

Organizations: UC Santa Cruz

Abstract

Evaluating AI agents in finance faces two key challenges: static benchmarks require costly expert annotation yet miss the dynamic decision-making central to real-world trading, while LLM-based judges introduce uncontrolled variance on domain-specific tasks. We introduce TraderBench, a benchmark that addresses both issues. It combines expert-verified static tasks (knowledge retrieval, analytical reasoning) with adversarial trading simulations scored purely on realized performance-Sharpe ratio, returns, and drawdown-eliminating judge variance entirely. The framework features two novel tracks: crypto trading with four progressive market-manipulation transforms, and options derivatives scoring across P&L accuracy, Greeks, and risk management. Trading scenarios can be refreshed with new market data to prevent benchmark contamination. Evaluating 13 models (8B open-source to frontier) on ~50 tasks, we find: (1) 8 of 13 models score ~33 on crypto with <1-point variation across adversarial conditions, exposing fixed non-adaptive strategies; (2) extended thinking helps retrieval (+26 points) but has zero impact on trading (+0.3 crypto, -0.1 options). These findings reveal that current agents lack genuine market adaptation, underscoring the need for performance-grounded evaluation in finance.

Figures & tables

Figure 1: Overview of the TraderBench two-agent architecture. The Evaluator Agent generates tasks from six datasets and sends them to the Candidate Agent via the A2A protocol. The Candidate Agent uses an LLM and six MCP servers to access financial data, execute code, and simulate trades. Responses are scored by dataset-specific evaluators.

Server	Function	Key Feature
SEC EDGAR	10-K/10-Q filings	Temporal locking (no lookahead)
Yahoo Finance	Market data	Lookahead detection
Sandbox	Python execution	Sandboxed computation
Options	Black-Scholes pricing	Greeks & IV calculation
Trading Sim	Paper trading	Slippage modeling
Risk	Portfolio risk metrics	VaR, Sharpe/Sortino

Table 1: The six MCP servers available to the Candidate Agent.

Section	Datasets	Tasks	Evaluator	Tool Use
Knowledge Retrieval	BizFinBench, PRBench	18	Exact match + LLM rubric	Required
Analytical Reasoning	Synthetic (CFA-level)	9	LLM rubric (3 components)	Not required
Options Trading	Options Alpha	9	4-dim scoring	Required
Crypto Trading	Crypto (4 transforms)	6	Performance-based	Minimal

Table 2: Overview of the four TraderBench evaluation sections (25% each). Tool Use indicates whether MCP server interaction is required to answer correctly.

Transform	Description
Baseline	Clean historical price data. Control condition for measuring base performance.
Noisy	Gaussian price noise ( $\sigma=2\%$ ) and sporadic volume spikes ( $3\times$ normal). Tests microstructure robustness.
Meta	Combined noise patterns with trend modifications, false breakouts, and support/resistance violations.
Adversarial	Coordinated false signals targeting common strategies: moving average crossovers, RSI divergences, and MACD signal injections.

Table 3: Adversarial transform conditions applied to historical crypto price data. Each transform progressively increases the difficulty of signal extraction.

Model	Type	Size	Provider
Proprietary
Gemini-3-Pro	Proprietary	–	Google
Kimi-K2.5	Proprietary	–	Moonshot AI
GPT-5.2	Proprietary	–	OpenAI
GPT-4o	Proprietary	–	OpenAI
Grok 4.1 Fast	Proprietary	–	xAI

Table 4: Models evaluated. All access the same MCP server infrastructure through the Candidate Agent. Web search and thinking ablations are discussed separately.

Model	Overall	KR	AR	Opt	Cry
Gemini-3-Pro	64.3	52.4	94.8	63.2	46.6
Grok 4.1 Fast	63.7	61.6	87.9	72.2	33.2
GPT-5.2	61.9	50.6	87.3	62.1	47.4
Kimi-K2.5	54.4	37.1	71.0	62.6	46.8
GLM-4.7-Flash	53.9	38.9	82.1	61.0	33.5
GPT-4o	53.3	32.3	80.4	55.6	44.9

Table 5: Main results. Models sorted by overall score (mean of four sections at 25% each). Bold : best in column. Underline : second best. KR = Knowledge Retrieval, AR = Analytical Reasoning, Opt = Options Trading, Cry = Crypto Trading.

Figure 2: Overall TraderBench scores for all 12 models, sorted by performance. The dashed line marks the 50/100 midpoint. Frontier proprietary models (left) clearly separate from smaller open-source models (right).

Figure 3: Analytical Reasoning (self-contained) vs. Knowledge Retrieval (tool-dependent) scores. Models sorted by AR; hatched bars denote ablation variants (GPT-5.2+WS = web search, Qwen3-32B-T = thinking mode). Several base models score 75+ on reasoning but below 15 on retrieval. Both ablation variants show large KR gains, confirming that tool access and tool-use planning drive retrieval performance.

Figure 4: Options trading sub-scores across all 12 models. P&L accuracy (80–93) consistently dominates Greeks precision (18–53), revealing a universal conceptual-vs-computational gap with a mean 54-point difference. Note the "competence mirage" where models correctly identify strategies but fail to quantify their risks.

Figure 5: Crypto trading scores by adversarial transform condition. Seven models ( bottom cluster ,

\sim

32–34) show virtually no variation, suggesting a fixed non-adaptive strategy. Five models ( top cluster , 45–52) actively trade; among these, only GPT-4o and Gemma3-27B show large transform-dependent variation.

Judge Model	Overall	KR	AR	Opt	Cry
Gemini-3-Flash	66.5	78.2	81.7	58.8	47.1
GPT-5.2 (baseline)	61.9	50.6	87.3	62.1	47.4
Claude Sonnet 4.5	55.2	49.4	68.3	55.7	47.4
Range	11.3	28.8	19.0	6.4	0.3

Table 6: Judge comparison: identical GPT-5.2 Candidate Agent responses evaluated by three different judge models. Even with only three judges, Knowledge Retrieval scores vary by nearly 29 points.

Figure 6: Per-section score distributions across three judge models evaluating identical responses. Crypto Trading (performance-based scoring) shows near-zero variance, while Knowledge Retrieval (LLM rubric scoring) shows substantial judge disagreement.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model	P&L	Strategy	Greeks	Risk
Gemini-3-Pro	88.9	74.4	36.7	52.8
Grok 4.1 Fast	88.9	75.0	53.3	71.7
GPT-5.2	82.2	71.1	36.7	58.3
Kimi-K2.5	86.7	72.8	33.3	57.8
GLM-4.7-Flash	84.4	66.1	41.7	51.7
GPT-4o	87.8	67.2	17.8	49.4

Appendix

Table 7: Full options trading sub-scores for all models.

Model	Baseline	Noisy	Meta	Adversarial
Gemini-3-Pro	46.9	45.8	46.9	47.2
Grok 4.1 Fast	32.9	33.8	33.5	32.7
GPT-5.2	47.2	46.7	47.0	49.2
Kimi-K2.5	46.8	46.0	47.3	47.9
GLM-4.7-Flash	33.3	33.9	33.1	33.2
GPT-4o	43.1	42.7	50.1	49.1

Appendix

Table 8: Full crypto trading scores by transform condition for all models.

Model	Prof	Trading	$\Delta$
Gemini-3-Pro	29.2	54.9	$-$ 25.7
Grok 4.1 Fast	84.8	52.7	$+$ 32.1
GPT-5.2	48.9	54.8	$-$ 5.9
Kimi-K2.5	30.7	54.7	$-$ 24.0
GLM-4.7-Flash	16.2	47.2	$-$ 31.0
GPT-4o	63.1	50.2	$+$ 12.9

Appendix

Table 9: Professional Tasks (GDPVal) scores and trading composite. Pearson correlations: Prof vs. Trading

r{=}0.36

, Prof vs. Crypto

r{=}0.16

, Prof vs. KR

r{=}0.62

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 510 papers

From Knowing to Doing: A Memory-Controlled Benchmark for LLM Trading Agents on Stock Markets
May 27, 2026Taojie Zhu, Wentao Zhao, Rui Sun +7Benchmark Contamination LLM Agent Evaluation
1Tsinghua University, 2Stepfun · 2Stepfun · 4Shanghai Jiao Tong University +3
FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use
Mar 9, 2026Jiaxuan Lu, Kong Wang, Yemin Wang +9LLM Agent Evaluation Tool-Using Language Agents
Shanghai AI Laboratory · Hunan University · Xiamen University +3
Learning to Trade Like an Expert: Cognitive Fine-Tuning for Stable Financial Reasoning in Language Models
Apr 18, 2026Yuchen Pan, Soung Chang LiewLLM Evaluation Financial Reasoning in Language Models
Department of Information Engineering, The Chinese University of Hong Kong

May 27, 2026J/K move · Enter open · S save

From Knowing to Doing: A Memory-Controlled Benchmark for LLM Trading Agents on Stock Markets

Taojie Zhu, Wentao Zhao, Rui Sun, Beidi Luan+6

1Tsinghua University, 2Stepfun · 2Stepfun · 4Shanghai Jiao Tong University+3

Benchmark Contamination LLM Agent Evaluation AI Agent Benchmarks Finance Algorithmic Trading

Open paper AskPDF arXiv

Evaluating whether large language model (LLM) agents can profit in capital markets is increasingly framed as end-to-end trading: place an agent in a historical market, let it trade, and measure portfolio returns. This setup is vulnerable to two evaluation failures. First, long backtests often overlap with the knowledge cutoffs of frontier LLMs, allowing memorized tickers, dates, prices, and market narratives to substitute for investment reasoning. Second, raw returns are a noisy proxy for stock-selection ability, since positive performance may come from market beta, style exposure, or favorable regimes rather than genuine alpha. We introduce KTD-Fin (Knowing-To-Doing Financial Benchmark), an end-to-end stock-market trading benchmark that addresses both issues. KTD-Fin uses a data-side masking protocol to anonymize key identifiers and calendar information consistently across prompts and tools, separating historical market memory from investment decision-making. It also incorporates a Barra-style performance attribution framework that decomposes portfolio returns into market, style, and stock-selection alpha components. Across ten frontier LLM agents evaluated on the Chinese CSI300 over a 2024--2026 window, masking substantially changes agent rationales, pushing them towards anonymized factor-based reasoning. Attribution analysis further shows that LLM agents' cumulative returns under leakage-controlled evaluation are largely explained by passive market and style exposure, with limited evidence of persistent stock-selection alpha. These findings suggest that financial LLM benchmarks should evaluate not only whether an agent makes money, but also whether the source of returns reflects transferable investment skill. We release KTD-Fin as a reproducible template for leakage-controlled and attribution-aware evaluation of LLM trading agents.

Abstract

AI Agent Evaluation

Explore similar work

From Knowing to Doing: A Memory-Controlled Benchmark for LLM Trading Agents on Stock Markets

FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use

Learning to Trade Like an Expert: Cognitive Fine-Tuning for Stable Financial Reasoning in Language Models