Quantitative research is an iterative loop: a researcher proposes a market hypothesis, implements it as an executable signal, tests it on historical data, and decides what to keep. Existing benchmarks score formulas, code, or individual trading decisions, leaving open whether large language models (LLMs) can complete this loop and produce strategies that work beyond the research period. We introduce Quant-Arena, a benchmark and public leaderboard for end-to-end quantitative research on real A-share market data. A model constructs features and stock-scoring code, receives feedback only on 2022--2023, and freezes one strategy for assessment on the sealed year 2024. A cash-account evaluator applies trading constraints and costs, including market impact estimated from a simulated order book. Scores are dominated by realized excess return and complemented by signal quality and the diversity of features used by the strategy. We evaluate nine frontier models, among which GPT-6 Astra scores highest; yet no model beats the equal-weight market in 2024. Quant-Arena is open to submissions from any model or agent harness.
Figures & tables
Figure 2: A model constructs features and stock-scoring code from raw market observations, revises candidates using research-period feedback, and freezes one strategy for assessment on the sealed period under common portfolio and execution rules.
Property
Specification
Stock universe
Selected universe of 3,042 stocks
Inputs
Daily market data and raw Level-2 quotes, orders, and trades
Research period
2022–2023
Out-of-sample period
Sealed year 2024
Table 1: Market data and temporal split used by the benchmark.
Table 2: Iterative results for every model and supported reasoning effort, Models are ordered by their best total score, and efforts increase within each model. Shaded rows mark each model’s highest-scoring effort, the configuration used by the diagnostics in Sections 3.3 – 3.6 . Scores are on a 0–100 scale. Financial and signal metrics refer to the sealed 2024 window; the gross equal-weight market ARR is 17.1%. MDD is a negative return. Bold marks the best value in each column.
Figure 3: Mean next-day return by daily score decile for the nine models’ best iterative configurations in 2024, in basis points (bp). H–L is the highest-minus-lowest decile spread, and the last entry in each header is the number of days with a signal.
Figure 4: Cash-account NAV of the iterative Astra run at xhigh effort in 2024, against the gross equal-weight market. Shaded bands mark days on which Astra holds stocks.
Model
Efforts
S
Window Δ
ARR (%)
Iter.
One-shot
Δ
ΔSin
ΔSout
Iter.
One-shot
Δ
GPT-6 Astra
xhigh / high
51.29
35.22
16.07
15.67
16.24
10.7
−4.8
15.5
GPT-6 Sol
xhigh / low
51.14
32.15
18.99
17.21
19.74
8.6
−7.6
16.2
GPT-5.6 Terra
max / none
47.58
42.14
5.44
3.81
6.14
8.4
9.5
−1.1
Claude Opus 5.5
medium / medium
44.28
42.61
1.67
9.59
−1.73
4.4
5.6
−1.2
Gemini 3.8 Flash
medium / medium
43.04
35.91
7.13
17.96
2.48
−2.9
1.6
−4.5
Table 3: Paired iterative and one-shot results, each at the supported effort with the highest total score within its setting (efforts listed as iterative / one-shot). Δ is iterative minus one-shot. Sin and Sout are the research-period and sealed-period scores, and ARR differences are in percentage points. The gross equal-weight market ARR is 17.1%.
Figure 5: Research-period progress of the nine iterative runs. (a) Best-so-far excess ARR gain over the first valid candidate; means use the runs available at each index. (b) First, best, and committed candidates on 2022–2023.
Figure 6: Holding-value-weighted style exposure of the nine models’ best iterative configurations in 2024, in cross-sectional standard deviations relative to the market. Astra’s estimates use its 61 invested days.
Figure 7: Cash-account NAV in 2024 for the nine models’ best iterative configurations and Kepler-1, against the gross equal-weight market.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Field
Specification
Market and universe
Selected A-shares; static universe of 3,042 stocks; missing and delisted stocks retained
Input profile
Raw daily and Level-2 market data through a date-scoped data interface; Python strategies
Windows
In-sample research 2022–2023 with authorized warmup history; sealed out-of-sample 2024; participants may define training and validation splits within the research period
Budget
At most 5 research evaluations
Feedback (iterative track)
Aggregate return, signal, and coverage statistics on the research window, plus execution errors
Portfolio
Top-75 equal-weight long portfolio, boundary ties retained; Top-200/400 diagnostics
Appendix
Table 4: Public specification of the selected A-share suite.
Table 5: One-shot results for every model and supported reasoning effort, Layout and shading follow Table 2 ; shaded rows mark each model’s highest-scoring one-shot effort. The one-shot setting receives no performance feedback.
Figure 8: Pairwise mean daily Spearman signal correlations (a) and Top-75 Jaccard overlaps (b) among the nine models’ best iterative configurations in 2024 (36 pairs). Portfolio overlap measures the selected score sets, rather than identical realized trades under execution constraints.
Large Language Models (LLMs) have shown strong capabilities in finance-oriented question answering, yet their evaluation on professional quantitative workflows remains fragmented and often stops at factual or document-grounded QA. We introduce QuantEval, a benchmark for evaluating whether LLMs can complete the compute--implement--validate loop required in quantitative finance. QuantEval contains 1,575 expert-validated instances across three complementary dimensions: (i) knowledge-based QA, (ii) quantitative mathematical reasoning, and (iii) quantitative strategy coding. The coding task is evaluated with a deterministic CTA-style backtesting framework that executes model-generated strategies and compares risk--return metrics against expert references, moving beyond syntax or template matching. We evaluate 13 open-source and proprietary LLMs, report prompt and reliability analyses, and observe substantial gaps relative to human experts, especially in multi-step reasoning and framework-constrained coding. We further conduct supervised fine-tuning and reinforcement learning experiments on domain-aligned data, showing that training can improve quantitative reasoning while also exposing trade-offs in factual recall. QuantEval releases the dataset, evaluation scripts, and full backtesting configuration to support reproducible research on LLMs for quantitative finance.
Zhaolu Kang, Junhao Gong, Wenqing Hu +13
Peking University · Tsinghua University · Fudan University +6
Large Language Models (LLMs) are evolving into autonomous trading agents, yet existing benchmarks often overlook the interplay between architectural reasoning and strategy consistency. We propose Strat-LLM, a framework grounded in Stratified Strategy Alignment. Operating in a live-forward setting throughout 2025, it integrates heterogeneous data including sequential prices, real-time news, and annual reports to eliminate look-ahead bias. Extensive stress tests on A-share and U.S. markets reveal: (1) reasoning-heavy models achieve peak utility in Free Mode via internal logic, whereas standard models require Strict Mode as a vital risk anchor; (2) alignment utility is regime-dependent, with Free and Guided modes capturing momentum in uptrending markets, while Strict Mode mitigates drawdowns in downtrends; (3) mid-scale models (35B) show optimal fidelity under strict constraints, whereas ultra-large models (122B) suffer an alignment tax under rigid rules but gain a performance premium in Guided Mode; (4) standard LLMs often fall into a high win-rate trap, optimizing for small gains at the expense of total returns, which can only be mitigated through deep reasoning or strict external guardrails. Project details are available at https://Strat-LLM.github.io.
Wenliang Huang, Zengyi Yu
School of Economics Zhejiang University of Technology Hangzhou, China · Faculty of Education East China Normal University Shanghai, China
Evaluating whether large language model (LLM) agents can profit in capital markets is increasingly framed as end-to-end trading: place an agent in a historical market, let it trade, and measure portfolio returns. This setup is vulnerable to two evaluation failures. First, long backtests often overlap with the knowledge cutoffs of frontier LLMs, allowing memorized tickers, dates, prices, and market narratives to substitute for investment reasoning. Second, raw returns are a noisy proxy for stock-selection ability, since positive performance may come from market beta, style exposure, or favorable regimes rather than genuine alpha. We introduce KTD-Fin (Knowing-To-Doing Financial Benchmark), an end-to-end stock-market trading benchmark that addresses both issues. KTD-Fin uses a data-side masking protocol to anonymize key identifiers and calendar information consistently across prompts and tools, separating historical market memory from investment decision-making. It also incorporates a Barra-style performance attribution framework that decomposes portfolio returns into market, style, and stock-selection alpha components. Across ten frontier LLM agents evaluated on the Chinese CSI300 over a 2024--2026 window, masking substantially changes agent rationales, pushing them towards anonymized factor-based reasoning. Attribution analysis further shows that LLM agents' cumulative returns under leakage-controlled evaluation are largely explained by passive market and style exposure, with limited evidence of persistent stock-selection alpha. These findings suggest that financial LLM benchmarks should evaluate not only whether an agent makes money, but also whether the source of returns reflects transferable investment skill. We release KTD-Fin as a reproducible template for leakage-controlled and attribution-aware evaluation of LLM trading agents.