QuantEval: A Benchmark for Financial Quantitative Tasks in Large Language Models

Zhaolu Kang; Junhao Gong; Wenqing Hu; Yingjie He; Kehan Jiang; Leqi Zheng; Shuo Yin; Zhicheng Fang

cs.CLJan 13, 2026

QuantEval: A Benchmark for Financial Quantitative Tasks in Large Language Models

Authors: Zhaolu Kang, Junhao Gong, Wenqing Hu, Yingjie He, Kehan Jiang, Leqi Zheng, Shuo Yin, Zhicheng Fang,

Organizations: Peking University · Tsinghua University · Fudan University · University of California, Los Angeles ·

Abstract

Large Language Models (LLMs) have shown strong capabilities in finance-oriented question answering, yet their evaluation on professional quantitative workflows remains fragmented and often stops at factual or document-grounded QA. We introduce QuantEval, a benchmark for evaluating whether LLMs can complete the compute--implement--validate loop required in quantitative finance. QuantEval contains 1,575 expert-validated instances across three complementary dimensions: (i) knowledge-based QA, (ii) quantitative mathematical reasoning, and (iii) quantitative strategy coding. The coding task is evaluated with a deterministic CTA-style backtesting framework that executes model-generated strategies and compares risk--return metrics against expert references, moving beyond syntax or template matching. We evaluate 13 open-source and proprietary LLMs, report prompt and reliability analyses, and observe substantial gaps relative to human experts, especially in multi-step reasoning and framework-constrained coding. We further conduct supervised fine-tuning and reinforcement learning experiments on domain-aligned data, showing that training can improve quantitative reasoning while also exposing trade-offs in factual recall. QuantEval releases the dataset, evaluation scripts, and full backtesting configuration to support reproducible research on LLMs for quantitative finance.

Figures & tables

Figure 1: Examples from the three core tasks in QuantEval: knowledge-based question answering, quantitative mathematical reasoning, and quantitative strategy coding.

Figure 2: The QuantEval data construction pipeline consists of three main stages: Data Preparation, Data Construction, and Validation.

Metric in Table 2	Goal	Definition / Interpretation
Executable Rate (%)	$\uparrow$	Fraction of generated strategies that execute end-to-end in the CTA framework with valid interfaces and no runtime failure.
Return MAE	$\downarrow$	Mean absolute error between generated and GT annualized return, computed over executable instances.
Drawdown MAE	$\downarrow$	Mean absolute error between generated and GT maximum drawdown.
Sharpe MAE	$\downarrow$	Mean absolute error between generated and GT Sharpe ratio.
Ret/Draw MAE	$\downarrow$	Mean absolute error between generated and GT return-to-drawdown ratio, capturing profitability versus downside risk.

Table 1: Strategy Coding metrics used in Table 2 . MAE is computed only over executable outputs; non-executable models are reported as “–”. Return/Drawdown MAE are in percentage points (pp); Sharpe and Ret/Draw MAE are unitless.

	Knowledge QA (Acc.%)			Reasoning (Acc.%)			Strategy Coding (CoT-only)
Model	w/	w/o	$\Delta$	w/	w/o	$\Delta$	Exec. Rate $\uparrow$	Return MAE $\downarrow$	Drawdown MAE $\downarrow$	Sharpe MAE $\downarrow$	Ret/Draw MAE $\downarrow$
Human Expert	91.75	91.75	0.0	89.05	89.05	0.0	100	1.52	1.10	0.07	0.09
Open-Source Models
Qwen3-4B	64.4	69.0	-4.6	27.7	33.9	-6.2	0.0	–	–	–	–
Qwen3-8B	65.5	65.5	0.0	33.9	29.2	+4.7	0.0	–	–	–	–
Qwen3-14B	75.9	78.2	-2.3	35.4	33.9	+1.5	0.0	–	–	–	–

Table 2: Unified evaluation on QuantEval . For Knowledge QA and Reasoning, we report accuracy (%) with Chain-of-Thought prompting (w/), without CoT (w/o), and the performance gap

\Delta=\text{w/}-\text{w/o}

. For Strategy Coding, we evaluate under CoT prompting and report Executable Rate (%) and MAE of four backtesting metrics. Return MAE and Drawdown MAE are reported in percentage points (pp) ; models with 0% executable rate are marked as “–” for MAE.

Training Stage	Knowledge QA	Reasoning	Strategy Coding (Sharpe MAE) $\downarrow$
Initial checkpoint	50.3	42.8	–
After SFT	53.2	47.8	0.85
After SFT + GRPO RL	52.2	48.4	0.72

Table 3: Performance progression of DianJin-R1-7B under extended training. Accuracy (%) is reported for QA and reasoning. For strategy coding, we report Sharpe MAE (lower is better).

Figure 3: Diagnostic RL-only performance curves of DianJin-R1-7B on external financial benchmark test sets. These curves illustrate optimization stability.

Model	Finance IQ	Fin Eval	Finova	QuantEval QA	QuantEval Reasoning
Qwen2.5-7B-Instruct	57.4	52.6	30.6	38.7	30.9
DianJin-R1-7B	62.3	58.7	41.9	50.3	42.8
DianJin-R1-7B + RL	69.2	64.2	43.7	51.8	44.1
DianJin-R1-7B + SFT	72.2	70.5	44.8	53.2	47.8
DianJin-R1-7B + SFT + RL	77.9	72.7	45.5	52.2	48.4

Table 4: Cross-benchmark evaluation of selected models on existing financial evaluation datasets and QuantEval. Accuracy (%) is reported for QA and reasoning tasks. QuantEval coding task is excluded here for comparability. QuantEval columns are highlighted to emphasize the larger difficulty gap relative to existing benchmarks.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Primary Sources	Data Types	Time Span / Coverage	License / Availability
Knowledge QA	Finance textbooks, academic papers, regulatory filings, curated glossaries	Text	N/A	Publicly available / licensed academic materials
Quantitative Reasoning	Yahoo Finance, Stooq, FRED, Quandl (public datasets)	Prices, rates, volatility indices, macro indicators	Varies by asset class (e.g., 2010–2024)	Publicly accessible; redistribution follows source licenses
Strategy Coding	Open-source strategy repositories, academic implementations, expert-written CTA templates	Python code + aligned historical market data	Source strategies vary; evaluation backtest fixed to 2010–2025	Open-source code under permissive licenses; expert templates released with benchmark

Appendix

Table 6: Data sources used for QuantEval. All benchmark instances rely on publicly accessible data or appropriately licensed materials. Commercial data vendors (e.g., Bloomberg) are not required for reproducing the benchmark.

Expert	Knowledge QA (%)	Reasoning (%)
Expert 1	91.2	88.5
Expert 2	93.0	89.7
Expert 3	90.5	87.9
Expert 4	92.3	90.1

Appendix

Table 8: Human expert performance on QuantEval. Accuracy is reported for Knowledge QA and Reasoning.

Metric	Mean	Std. Dev.
Executable Rate (%)	100	0
Return MAE (pp)	1.52	0.27
Drawdown MAE (pp)	1.10	0.24
Sharpe MAE	0.07	0.02
Ret/Draw MAE	0.09	0.02

Appendix

Table 9: Human expert variance on Strategy Coding tasks. Metrics are computed against the canonical GT reference using the same backtesting harness.

Name	Professional Background
Expert 1	Computer Science & Quantitative Research
Expert 2	Computer Science & Quantitative Research
Expert 3	Quantitative Trading Practitioner
Expert 4	Quantitative Trading Practitioner
Expert 5	Computer Science & Quantitative Research
Expert 6	Quantitative Trading Practitioner

Appendix

Table 11: Professional backgrounds of the experts involved in dataset construction and evaluation.

Task Category	Human Expert	Automated Agent	Total Samples
Knowledge QA	520	140	660
Quantitative Reasoning	655	200	855
Strategy Coding	10	50	60

Appendix

Table 12: Distribution of samples constructed by human experts and automated agents across the three core tasks in QuantEval.

Task	Expert Data	Agent Data	Agent $-$ Expert
Knowledge QA (Acc.%)	62.8	61.3	-1.5
Quantitative Reasoning (Acc.%)	55.6	53.9	-1.7
Strategy Coding (Sharpe MAE $\downarrow$ )	0.17	0.20	+0.03

Appendix

Table 13: Quality comparison on human-expert-generated and automated-agent-generated subsets. For QA and reasoning we report accuracy (%); for Strategy Coding we report Sharpe MAE (lower is better). The difference is computed as Agent Data minus Expert Data. Differences are small, indicating comparable data quality after expert validation.

Model	Knowledge QA Acc. 95% CI	Reasoning Acc. 95% CI	Coding Exec. 95% CI
Human Expert	91.8 $\pm$ 2.1	89.0 $\pm$ 2.1	[94.0, 100.0]
Qwen3-30B-A3B	48.5 $\pm$ 3.8	40.5 $\pm$ 3.3	[3.6, 18.1]
Claude Sonnet 4.5	86.0 $\pm$ 2.6	43.1 $\pm$ 3.3	[52.4, 75.8]
Gemini 2.5 Pro	66.7 $\pm$ 3.6	38.0 $\pm$ 3.3	[52.4, 75.8]
GPT-5	83.9 $\pm$ 2.8	55.0 $\pm$ 3.3	[40.9, 65.4]

Appendix

Table 14: Approximate 95% confidence intervals for representative systems. QA and reasoning intervals use binomial normal approximations with

n=660

and

n=855

, respectively. Strategy Coding executable-rate intervals use Wilson score intervals with

n=60

Type	Evaluation Prompt
Multiple Choice	You are an evaluation assistant.
	Please determine whether the model’s answer below is correct.
	Question: [question]
	Options: [options]
	Correct answer: [correct answer]
	Model output: [model output]

Appendix

Table 17: Prompt templates used for automated evaluation of model outputs on multiple-choice and open-ended questions.

Training Setting	Knowledge QA	Reasoning	Sharpe MAE
Base	50.3	42.8	–
25% SFT data	51.2	44.5	1.25
50% SFT data	52.4	46.2	1.02
100% SFT data	53.2	47.8	0.85
100% SFT + GRPO	52.2	48.4	0.72

Appendix

Table 23: Training data scale ablation for DianJin-R1-7B under increasing fractions of the SFT corpus. Accuracy (%) is reported for QA and reasoning; lower Sharpe MAE is better for Strategy Coding.

	Exec. Rate (%)			Sharpe MAE
Model	$T{=}0.0$	$T{=}0.3$	$T{=}0.7$	$T{=}0.0$	$T{=}0.3$	$T{=}0.7$
Claude Sonnet 4.5	65.0	64.2	63.3	0.16	0.17	0.17
Gemini 2.5 Pro	65.0	64.2	63.3	0.15	0.16	0.16
GPT-5	53.3	52.5	51.7	0.17	0.17	0.18

Appendix

Table 26: Strategy Coding performance across decoding temperatures. Lower temperatures yield slightly higher executable rates and marginally lower Sharpe MAE, but the model ordering Gemini

\approx

Claude

>

GPT-5 in executable rate and the open-source vs proprietary structural gap remain stable across all settings.

Figure 4: Representative Knowledge QA failure. The model reasons about counterparty credit exposure but inserts unrelated hallucinated content, illustrating that fluent CoT may contain unsupported intermediate statements.

Figure 5: Representative quantitative reasoning case. Models recall the modified-duration formula, but the example illustrates how answer mapping, rounding, and step consistency remain important in financial calculation tasks.

Figure 6: Representative Strategy Coding failure. The generated code contains finance-relevant indicator logic but suffers from formatting, indentation, interface, or framework-call issues that can prevent reliable execution.

Financial Language Model Evaluation

See all 23 papers

FINESSE-Bench: A Hierarchical Benchmark Suite for Financial Domain Knowledge and Technical Analysis in Large Language Models
May 14, 2026Dmitry Stanishevskii, Nini Kamkia, Alexey Khoroshilov +4Financial Language Model Evaluation LLM Evaluation
Lime FinTech
QuantCode-Bench: A Benchmark for Evaluating the Ability of Large Language Models to Generate Executable Algorithmic Trading Strategies
Apr 16, 2026Alexey Khoroshilov, Alexey Chernysh, Orkhan Ekhtibarov +2Agentic Code Generation Code Generation Evaluation
Lime
V-FiLLM: Verified Financial LLM Reasoning Benchmark
Aug 11, 2026Alicia Larsen, Victoire Laurent, Aulia Kharis Rakhmasari +2Numerical Reasoning in Language Models Synthetic Data Generation
ETH Zürich, Zürich · Aisot Technologies Ltd

May 14, 2026J/K move · Enter open · S save

FINESSE-Bench: A Hierarchical Benchmark Suite for Financial Domain Knowledge and Technical Analysis in Large Language Models

Dmitry Stanishevskii, Nini Kamkia, Alexey Khoroshilov, Dmitry Zmitrovich+3

Lime FinTech

Financial Language Model Evaluation LLM Evaluation Quantitative Finance Benchmark Design Finance Financial Reasoning Benchmarks

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly being applied to financial analysis, reporting, investment decision support, risk management, compliance, and professional training. However, robust evaluation of their domain competence in finance remains incomplete. Widely used open benchmarks such as FinQA, ConvFinQA, and TAT-QA have played an important role in advancing financial question answering and numerical reasoning, but they focus primarily on question answering over financial reports and do not provide an explicit hierarchy of professional difficulty. Broader resources, including FinanceBench, PIXIU, FinBen, and FLaME, expand the coverage of financial tasks, yet the problem of evaluating the transition from foundational knowledge to expert-level financial reasoning remains open. In this work, we present FINESSE-Bench, a suite of eight specialized benchmarks comprising 3,993 questions for hierarchical evaluation of financial competencies in LLMs. FINESSE-Bench combines exam-oriented datasets inspired by professional certifications (CFA-like Levels 1-3, CMT-like Level 2, and CFTe-like Level 1), applied trading task collections, and a Russian-language olympiad benchmark. This design enables evaluation of domain breadth, performance degradation as difficulty increases, the ability to solve computational tasks, and model behavior in specialized financial domains. We also describe a unified evaluation protocol covering multiple-choice questions, numerical answers, and short open-ended responses, together with an automated scoring scheme for freeform answers based on the LLM-as-judge paradigm. FINESSE-Bench is intended both as a complement to existing open financial benchmarks and as a tool for more substantive evaluation of professionally relevant financial competencies in large language models.

Abstract

Financial Language Model Evaluation

Explore similar work

FINESSE-Bench: A Hierarchical Benchmark Suite for Financial Domain Knowledge and Technical Analysis in Large Language Models

QuantCode-Bench: A Benchmark for Evaluating the Ability of Large Language Models to Generate Executable Algorithmic Trading Strategies

V-FiLLM: Verified Financial LLM Reasoning Benchmark