FinTradeBench: A Financial Reasoning Benchmark for LLMs

Yogesh Agrawal; Aniruddha Dutta; Md Mahadi Hasan; Santu Karmaker; Aritra Dutta

cs.CEMar 19, 2026

FinTradeBench: A Financial Reasoning Benchmark for LLMs

Authors: Yogesh Agrawal, Aniruddha Dutta, Md Mahadi Hasan, Santu Karmaker, Aritra Dutta

Organizations: University of Central Florida

Abstract

Financial analysis requires reasoning over heterogeneous signals, including company fundamentals from regulatory filings and trading signals derived from market prices. However, existing financial benchmarks focus almost exclusively on regulatory filings, neglecting market dynamics and cross-signal interactions. We introduce FinTradeBench, a benchmark of 1,400 questions covering NASDAQ-100 stocks over a ten-year window. To holistically evaluate financial reasoning, FinTradeBench includes questions in three reasoning categories: fundamentals-focused, trading-signal-focused, and hybrid questions that require cross-signal reasoning. Our benchmark starts with 150 expert-authored seed questions, which are then scaled to 1400. To ensure reliable evaluation, we create a Core400 subset in which every item is expert-reviewed. In zero-shot (No-RAG) response generation, we observe that all models perform substantially worse on trading-focused questions compared to fundamental-focused questions. To identify failure causes, we evaluate LLMs under controlled settings including retrieval-based baselines, tool-assisted computation, oracle conditions, and time-series-specialized pipelines. LLMs' capabilities to do computation and evidence selection contribute most to reasoning failures. Additionally, retrieval significantly improves trading-signal reasoning but does not improve fundamental reasoning, where identifying relevant signals remains the dominant failure mode. Finally, rather than relying on a small, fine-tuned model for time-series reasoning, it is better to summarize the time-series data and leverage LLMs' reasoning capabilities.

Figures & tables

Dataset	Primary evidence	Reasoning target	RAG/Open-book	Market TS	F+T	Evaluation style
FinQA ( Chen et al., 2021 )	Reports, tables	Numerical financial QA	$\times$	$\times$	$\times$	Program / exact match
TAT-QA ( Zhu et al., 2021 )	Tables + text	Tabular-text numerical QA	$\times$	$\times$	$\times$	Exact match / derivation
ConvFinQA ( Chen et al., 2022 )	Reports, tables	Conversational numerical QA	$\times$	$\times$	$\times$	Program / exact match
FinanceBench ( Islam et al., 2023 )	SEC filings, earnings calls	Open-book financial QA	✓	$\times$	$\times$	Human / reference answer
FinDER ( Choi et al., 2025 )	Financial documents	Retrieval-based financial QA	✓	$\times$	$\times$	Retrieval + answer scoring
FinTextQA ( Chen et al., 2024b )	Long-form financial text	Long-form financial QA	✓	$\times$	$\times$	Human + GPT-4 scoring

Table 1: Comparison of financial QA and LLM benchmarks. emphasizing the core aspects of the work such as retrieval/open-book evaluation, explicit reasoning over historical market time series (TS), and joint reasoning over company fundamentals and trading evidence (F+T). ✓ denotes explicit support,

\times

denotes no explicit support, and

\circ

denotes partial or indirect support.

Figure 2: FinTradeBench construction pipeline. The benchmark is built through data and signal design, a three-way financial reasoning taxonomy, expert-guided seed construction and calibration, and finally scalable question generation and quality control.

Figure 3: FinTradeBench evaluation framework. Models answer the final Core400 under general-QA, controlled-evidence, and time-series-specialist conditions. Generated responses and their evidence traces are evaluated using a common four-dimensional, evidence-aware judging framework. A financial expert then validates the automated judging.

Figure 6: Number of models whose score changes significantly under each intervention, by lane (two-sided Wilcoxon signed-rank test on per-question paired differences, Holm-corrected;

p_{\mathrm{Holm}}<0.05

). Numbers inside bars are model counts. For details, see Tables 22 and 23

Figure 7: Lane-level paired effects: (a) RAG

-

No-RAG on the common-evidence Core400; (b) Tool

-

Indicator and (c) Oracle

-

Indicator for the seven controlled models. Points are mean paired changes in 1–5 judge Likert score with 95% bootstrap CIs; the vertical line marks no change. † Hollow marker: only 12 paired Trading items under Tool.

Condition	Overall	F	T	FT
No-RAG	2.81	3.20	2.06	3.08
RAG	2.98	3.06	2.41	3.41
Indicator	4.69	4.77	4.44	4.83
Tool	4.57	4.42	4.61	4.69
Oracle	4.88	4.90	4.77	4.95
GPT-5.4-mini + TimesFM2.5	4.12 †	–	3.70	4.51

Table 4: GPT-5.4-mini across all experimental conditions. † TimesFM2.5 is evaluated only on the time-series-eligible questions, so its Overall excludes Fundamental.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Company	Period	Fundamental Signal	Dominant Signal
Amazon	1999–2001	Persistent losses, negative EPS	Market (narrative)
Nvidia	2016–2017	Moderate earnings, GPU revenue lag	Market (forward expectation)
Tesla	2020	Modest profitability, high P/E	Market (sentiment)
Tesla	Apr. 2025	EPS miss ( $0.27 vs.$ 0.42), revenue miss	Market (narrative rally)

Appendix

Table 5: Summary of fundamentals–market divergence episodes. “Dominant Signal” refers to which signal better characterised the market outcome at the time.

Check	Purpose	Stage
Golden indicators	Provide an expert-defined minimal evidence set for each question; used for construction auditing and controlled Oracle evidence, not as a complete answer template.	Question design / controlled evaluation
Numerical audit	Flag unsupported, contradicted, or unavailable numerical claims in construction-time candidate responses.	Benchmark construction
Human–LLM alignment	Verify that automated judge scores approximate expert scores	Judge calibration
Aligned judge validation	Validate the human-aligned LLM judge for benchmark scaling	Benchmark Scaling
Scaled-set fatal-flaw screen	Identify catastrophic construction failures in the broader 1,400-question collection; 42 responses required regeneration during benchmark scaling.	Broader benchmark construction
Canonical-data support screen	Remove Core candidates whose required company/time-period evidence is absent from the frozen canonical data universe.	450 $\rightarrow$ 403

Appendix

Table 6: Construction and validation checks in the FinTradeBench pipeline. The checks operate at different stages and should not be interpreted as interchangeable forms of validation.

Statistic	Value
Broader FinTradeBench collection	1,400
Expert-authored seed questions	150
Programmatically scaled questions	1,250
Core candidate pool	450
After canonical-data support screen	403
Final human-validated Core	400

Appendix

Table 7: FinTradeBench dataset and final evaluation-set statistics. The 1,400-question collection provides benchmark breadth, whereas the human-validated Core400 is the frozen set used for the final controlled evaluation.

Signal	Formula	Definition / Description
Trading Signals (Notation: $P_{t}$ = Price at time $t$ , $V_{t}$ = Volume, $R_{t}$ = Return, $N,k$ = Lookback periods, $\alpha$ = Smoothing factor)
MA (Moving Average)	$\frac{1}{N}\sum_{i=1}^{N}P_{t-i}$	Average price of stock over fixed lookback window. Smooths out short-term fluctuations to reveal longer-term trends.
EMA (Exp. Moving Average)	$\alpha P_{t}+(1-\alpha)\text{EMA}_{t-1}$	Weighted moving average emphasizing recent prices; reacts faster to new information ( $\alpha$ is the smoothing factor).
MACD	$\text{EMA}_{\text{short}}-\text{EMA}_{\text{long}}$	Momentum indicator based on difference between short- & long-term EMAs; captures trend strength & reversal signals.
RSI (Relative Strength Index)	$100-\frac{100}{1+\text{RS}}$ ( $\text{RS}=\frac{\text{Avg Gain}}{\text{Avg Loss}}$ )	Scaled measure of recent gains vs. losses; high RSI suggests overbought conditions, low RSI indicates oversold conditions.
OBV (On-Balance Volume)	$\text{OBV}_{t-1}+V_{t}\cdot\text{sgn}(P_{t}-P_{t-1})$	Cumulative volume measure linking price movement and trading volume; rising OBV indicates accumulation, falling OBV suggests distribution.

Appendix

Table 8: Summary of trading signals and company fundamentals used in the question design.

ID	Attributes	Description (Example Behaviour)
L1	Single-turn, low detail, instruction-style, role-specified	Instructs the model in simple one-sentence instructions focusing on the high-level goal.
L2	Single-turn, moderate detail, instruction-style, role-specified	Paragraph-style instructions expressing the high-level goal and sub-tasks that need to be performed to achieve the goal
L3	Step-by-step reasoning, moderate detail, decomposed	adds a structured reasoning template (bulleted-list-style): clarify goal, decompose into sub-questions, answer sub-parts using both contexts, and synthesize a final answer.
L4	Step-by-step reasoning, moderate detail, decomposed	“Level 3 Prompt” + “It provides a guideline on how LLMs will be evaluated.”
L5	Step-by-step reasoning, high detail, decomposed	“Level 4 Prompt” + “Provide additional relevant gathered via RAG”
L6	Step-by-step reasoning, Maximalist, high detail, evidence-citing, justified	“Level 5 Prompt” + “Provide explicit statement asking LLM to explain its own output.”

Appendix

Table 9: TELeR-inspired prompt variants used for multi-prompt generation during benchmark creation § 3 .

Lvl	Type	Strategy	RAG
L1	Baseline	Single-sentence high-level directive.	No
L2	Focus	Role specification + Paragraph style breakdown.	No
L3	CoT	Bulleted Step-by-Step Chain-of-Thought.	No
L4	Auditor	Adds explicit evaluation criteria (e.g., coherence).	No
L5	Context	Injects retrieved evidence and citations.	Yes
L6	Explain	Adds self-justification for data usage.	Yes

Appendix

Table 10: The TELeR Prompt Taxonomy used for Benchmarking and evaluation for the RAG and No-RAG setting in § 4 . The framework systematically increases prompt complexity, with Levels 5 and 6 designed to leverage RAG.

Panel of Table 11 — Table 11: Sample seed questions & corresponding golden answers; see details about key financial terms in Table 8 .

Stage	Total	F	T	FT
Candidate pool	450	163	124	163
Canonical-data screening	403	143	124	136
Human audit	400	143	122	135

Appendix

Table 12: FinTradeBench Core curation . We begin with 450 candidate items, then apply canonical-data support screening, followed by item-level human audit, producing the final human-validated Core400.

Human-audit criterion	Count	Rate
Question clarity: Pass	400/403	99.3%
Question clarity: Pass or Minor	403/403	100.0%
Reference answer: Correct	401/403	99.5%
Canonical evidence: Full support	399/403	99.0%
Golden indicators: Adequate	402/403	99.8%
Entity/time alignment: Pass	403/403	100.0%

Appendix

Table 13: Full human quality audit of Core403. High pass rates reflect the preceding canonical-data support screen. The audit verifies the filtered Core and should not be interpreted as a quality estimate for the broader 1,400-question collection.

Panel of Table 14 — Table 14: Sample bad responses caught during screening.

Condition	Fundamental (F)	Trading (T)	Hybrid (FT)
No-RAG	Question only	Question only	Question only
Raw-RAG	Point-in-time SEC evidence	Point-in-time raw OHLCV	Point-in-time SEC + raw OHLCV
Indicator-RAG	SEC + precomputed fundamental indicators	Precomputed trading indicators	SEC + fundamental + trading indicators
Tool-Use	SEC evidence	Raw OHLCV + deterministic tool routing	SEC + raw OHLCV + deterministic tool routing
Oracle	Gold-selected indicator evidence	Gold-selected indicator evidence	Gold-selected indicator evidence

Appendix

Table 15: Evidence conditions. Indicator-RAG does not use golden-indicator annotations for selection. Tool routing depends only on the question and declared tool inventory. Oracle supplies gold-selected evidence, not the benchmark reference response.

Figure 9: Controlled-condition evaluation. The question set, cutoff rules, model, answer budget, and judge are held fixed; only the evidence pathway changes. Indicator-RAG retrieves precomputed indicator and fundamental tables, Tool-Use lets the model query indicator values on demand, and Oracle supplies gold evidence as a probable upper bound, separating failures of evidence access from failures of reasoning.

System	Study	Exact model identifier	Provider / endpoint	Inference configuration
DeepSeek-V4-Flash ( DeepSeek-AI et al., 2026 )	General QA	DeepSeek-V4-Flash	Azure AI Foundry, OpenAI-compatible v1; deployment ftb-deepseek-v4-flash	Max output 1,200 tokens. The endpoint is invoked without custom temperature or top- $p$ . Visible <think> blocks are removed before scoring. API timeout 900 s with up to six retries.
GPT-5.4-mini ( OpenAI, 2026 )	General QA	gpt-5.4-mini	Azure AI Foundry, OpenAI-compatible v1; deployment ftb-gpt54mini	Max completion 1,200 tokens; reasoning_effort=medium . No custom temperature or top- $p$ is supplied to the endpoint. API timeout 900 s with up to six retries.
Gemini-3.8-Flash ( Google DeepMind, 2026 )	General QA	gemini-3.8-flash	Google Vertex AI v1 ( google-genai ); location global	Max output 1,200 tokens; thinking_level=MEDIUM . Deprecated temperature/top- $p$ controls are not supplied. API timeout 900 s with up to six retries.
Llama-4-Scout ( Meta AI, 2025 )	General QA + Controlled	Llama-4-Scout-17B-16E-Instruct	Azure AI Foundry, OpenAI-compatible v1; deployment ftb-llama4-scout	General QA: max output 10,000 tokens; candidate generation uses temperature $0.1$ , top- $p=0.9$ ; self-selection is deterministic (temperature $0$ , top- $p=1$ ). Controlled conditions use a 10,000-token answer ceiling; Tool-Use router ceiling 1,024 tokens.
Qwen3.8-27B ( Qwen Team, 2026 )	General QA + Controlled	Qwen3.8-27B	Local vLLM ( Kwon et al., 2023 ) , OpenAI-compatible loopback endpoint	General QA: max output 1,200 tokens; candidate generation temperature $0.1$ , top- $p=0.9$ ; deterministic self-selection; seed 42; thinking explicitly disabled. Controlled: max output 1,200 tokens; deterministic generation; router ceiling 120 tokens. Serving: BF16, max model length 32,768, GPU-memory utilization 0.90.
Ministral-3-8B ( Liu et al., 2026 )	General QA	Ministral-3-8B-Instruct-2512	Local vLLM ( Kwon et al., 2023 ) , OpenAI-compatible loopback endpoint	Max output 1,200 tokens; candidate generation temperature $0.1$ , top- $p=0.9$ ; deterministic self-selection; seed 42. Serving: BF16, max model length 32,768, GPU-memory utilization 0.90, Mistral tokenizer/config/load format.

Appendix

Table 16: Model identifiers, serving backends, and inference configuration. General-QA models generate four No-RAG candidates and two RAG candidates before intra-model self-selection. Controlled Indicator-RAG, Tool-Use, and Oracle conditions use a single final generation; Tool-Use additionally invokes the deterministic router. Llama-4-Scout is an open-weight Meta model evaluated through Azure AI Foundry. TimesFM 2.5 is used only to generate forecast features and does not directly answer FinTradeBench questions. At the time of research, Gemini 3.8 Flash leads vals Finance Agent v2 with 61.44

\%

accuracy AI (2026)

System / run	Execution mode	Inference hardware	Client allocation	Recorded completing-job wall time
Qwen3.8-27B ( Qwen Team, 2026 )	Local vLLM ( Kwon et al., 2023 )	1 $\times$ NVIDIA H100 80GB HBM3	16 CPU cores, 256 GB RAM	21:58:41 †
Ministral-3-8B ( Liu et al., 2026 )	Local vLLM ( Kwon et al., 2023 )	1 $\times$ NVIDIA H100 80GB HBM3	16 CPU cores, 256 GB RAM	02:27:38
Llama-4-Scout ( Meta AI, 2025 )	Azure AI Foundry API	Provider-managed; accelerator not exposed	16 CPU cores, 64 GB RAM	07:11:05 †
GPT-5.4-mini ( OpenAI, 2026 )	Azure AI Foundry API, General QA	Provider-managed; accelerator not exposed	16 CPU cores, 64 GB RAM	06:03:29 †
DeepSeek-V4-Flash ( DeepSeek-AI et al., 2026 )	Azure AI Foundry API	Provider-managed; accelerator not exposed	16 CPU cores, 64 GB RAM	02:19:38 †
Gemini-3.8-Flash ( Google DeepMind, 2026 )	Vertex AI API	Provider-managed; accelerator not exposed	16 CPU cores, 64 GB RAM	08:45:51 †

Appendix

Table 17: Compute configuration and recorded runtime. Times are Slurm wall-clock durations for the completing jobs associated with the frozen outputs; smoke tests and failed attempts are excluded. † These runners support resumption and may reuse rows generated by earlier jobs, so the reported time is the wall time of the completing job rather than cumulative compute expenditure. ‡ The four locally controlled baseline models were dispatched as 12 independent model

\times

condition jobs with generic Slurm job names ( ftb403_resume ). Their exact per-model wall times cannot be reconstructed reliably from the accounting export, so we do not assign potentially incorrect runtimes. § The two values correspond to the controlled-model judge batches for the two model groups. ¶ Common-evidence RAG/No-RAG responses were judged in parallel model–condition jobs; the longest observed completing job was 02:23:57. For API-hosted models, the listed CPU/RAM values describe the Slurm client, not the provider-side inference hardware.

Figure 10: Oracle–Indicator effects across seven controlled models. Positive values indicate improved performance when question-specific gold-indicator evidence replaces the broader Indicator-RAG evidence-selection process.

Stage	Unit reviewed	Human task	Role in pipeline
Seed construction and development calibration	Seed questions and generated responses	Experts author seed questions and golden indicators; generated responses are subsequently scored to calibrate the development-time automated evaluator.	Benchmark construction and evaluator development
Scaled-set fatal-flaw screening	Scaled question–answer pairs	Screen for catastrophic generation failures such as wrong entity, incomplete answers, or abrupt truncation; ambiguous cases are escalated.	Construction quality control
Core403 item-level audit	Benchmark items	Inspect the question, reference answer, golden indicators, canonical point-in-time evidence, and entity/time alignment.	Final benchmark curation: Core403 $\rightarrow$ Core400
Held-out judge validation	Anonymous model responses	Score responses using the frozen four-dimension rubric and record blinded A/B/Tie preferences.	Post-freeze evaluator validation only

Appendix

Table 18: Distinct human-review stages in FinTradeBench. The stages operate on different objects and serve different purposes. The held-out judge-validation labels are not used to modify the frozen evaluation procedure.

Dimension	MAE	$\rho$	Exact	$\pm 1$	QWK
Accuracy	0.17	0.95	83.3%	100.0%	0.945
Completeness	0.29	0.88	75.0%	95.8%	0.882
Relevance	0.40	0.86	68.8%	91.7%	0.809
Clarity	0.21	0.86	79.2%	100.0%	0.857
All rubric ratings	0.27	0.89	76.6%	96.9%	0.884

Appendix

Table 19: Held-out human–judge agreement. Statistics are computed over 48 responses and 192 individual rubric ratings. Exact and

\pm 1

denote the proportion of automated ratings identical to, or within one Likert point of, the human rating. QWK denotes quadratic-weighted Cohen’s

\kappa

Comparison	Pairs	Decisive	Strict	Direct.
Overall	24	15	70.8%	100.0%
Oracle vs. Indicator	8	4	62.5%	100.0%
RAG vs. No-RAG	8	6	87.5%	100.0%
TimesFM+GPT vs. ChatTS	4	3	75.0%	100.0%
Tool vs. Indicator	4	2	50.0%	100.0%

Appendix

Table 20: Pairwise preference agreement. Decisive denotes pairs for which the human reviewer selected A or B rather than Tie. Strict agreement requires identical A/B/Tie labels. Directional (Direct.) agreement is computed only over the 15 human-decisive pairs.

Quantity	Core403	Core400	Change
ChatTS overall	3.0386	3.0321	$-$ 0.0065
GPT+TimesFM overall	4.1158	4.1235	+0.0077
Paired time-series effect	1.0772	1.0914	+0.0142

Appendix

Table 21: Sensitivity to the final human-audit exclusions. The final three exclusions produce only small numerical changes in the time-series study and do not change its substantive conclusion.

Lane	$N$	$\Delta$	95% CI	W/T/L	$p_{\mathrm{Holm}}$
DeepSeek-V4-Flash
All	399	+0.23	[.15, .30]	53/17/31	$<$ .001
Fund.	143	$-$ 0.06	[ $-$ .19, .07]	39/15/46	.683
Trad.	121	+0.46	[.35, .57]	69/16/16	$<$ .001
Hybrid	135	+0.32	[.21, .44]	53/19/27	$<$ .001
Llama-4-Scout

Appendix

Table 22: Effect of retrieval (RAG

-

No-RAG) on the common-evidence Core400, per model and lane (Fund. = Fundamental, Trad. = Trading).

\Delta

: mean paired difference in 1–5 judge Likert score (multiply by 20 for the 0–100 scale); 95% CI: paired bootstrap; W/T/L: % of items where RAG wins, ties, or loses;

p_{\mathrm{Holm}}

: Wilcoxon signed-rank

p

, Holm-corrected across all 24 tests. Bold:

p_{\mathrm{Holm}}<0.05

		Tool $-$ Indicator			Oracle $-$ Indicator			Oracle $-$ Tool
Lane	$N$	$\Delta$	95% CI	$p_{\mathrm{Holm}}$	$\Delta$	95% CI	$p_{\mathrm{Holm}}$	$\Delta$	95% CI	$p_{\mathrm{Holm}}$
Qwen2.5-32B
All	399	$-$ 0.17	[ $-$ 0.27, $-$ 0.06]	.004	+0.26	[0.17, 0.36]	$<$ .001	+0.43	[0.33, 0.53]	$<$ .001
Fund.	143	$-$ 0.74	[ $-$ 0.90, $-$ 0.58]	$<$ .001	+0.01	[ $-$ 0.11, 0.13]	1.00	+0.75	[0.61, 0.89]	$<$ .001
Trad.	121	+0.64	[0.44, 0.85]	$<$ .001	+0.82	[0.62, 1.04]	$<$ .001	+0.18	[ $-$ 0.05, 0.40]	1.00
Hybrid	135	$-$ 0.28	[ $-$ 0.41, $-$ 0.15]	$<$ .001	+0.03	[ $-$ 0.10, 0.15]	1.00	+0.31	[0.15, 0.46]	$<$ .001

Appendix

Table 23: Full paired effects for the controlled conditions on the final Core400, per model and lane (Fund. = Fundamental, Trad. = Trading).

\Delta

: mean paired difference in 1–5 judge Likert score (multiply by 20 for the 0–100 scale); 95% CI: paired bootstrap;

p_{\mathrm{Holm}}

: two-sided Wilcoxon signed-rank

p

on per-question paired differences, Holm-corrected within each experimental batch over all model

\times

lane

\times

comparison tests (48 tests for Qwen2.5-32B, R1-Distill-Qwen-32B, Llama-3.1-8B and Phi-4; 36 for Llama-4-Scout, Qwen3.8-27B and GPT-5.4-mini). Bold:

p_{\mathrm{Holm}}<0.05

N

is the number of paired questions. † Qwen3.8-27B’s Tool condition has incomplete coverage, so comparisons involving Tool use fewer pairs: 290 overall and only 12 on Trading.

Lane	$N$	$\Delta_{(T+G)-C}$	95% CI
Overall	257	+1.091	[0.944, 1.245]
Trading	122	+0.525	[0.277, 0.775]
Hybrid	135	+1.604	[1.472, 1.739]

Appendix

Table 24: Paired time-series specialist effects on the Core400 eligible subset. Here,

\Delta_{(T+G)-C}

represents

(TimesFM+GPT)-ChatTS

values. All three comparisons remain significant after Holm correction.

Panel of Table 25 — Table 25: GPT-5.4-mini responses to a Fundamental seed question under every evaluated condition (TimesFM2.5 is evaluated only on time-series-eligible Trading and Hybrid questions). Bracketed numbers are judge scores (1–5 Likert mean). Scores are comparable only within a judging protocol: the per-condition-evidence judge grades each answer against the evidence that condition supplied, the common-evidence judge grades both RAG and No-RAG against identical canonical evidence. Excerpts are verbatim apart from removed Markdown formatting; […] marks omitted text. Gray notes summarize the judge’s rationale.

Panel of Table 26 — Table 26: GPT-5.4-mini responses to a Trading seed question under every evaluated condition. Notation as in Table 25 .

Panel of Table 27 — Table 27: GPT-5.4-mini responses to a Hybrid seed question under every evaluated condition. Notation as in Table 25 .

Criterion	Scale	Reviewer instruction / anchors
Audit Validation Agreement	0–1	Does the independent human review agree with the automated numerical audit’s is_numerically_accurate flag? 1 : Agree—the audit conclusion is correct. 0 : Disagree—the audit missed an error or incorrectly flagged a correct claim.
Factual & Numerical Accuracy	1–5	Based on the reviewer’s own assessment, together with the numerical audit, assign the final accuracy score. 5 : 100% correct. 1 : Contains significant, misleading numerical errors.
Completeness & Context	1–5	Does the answer fully address the question and correctly use and contextualize the golden indicators? 5 : Excellent; uses required metrics in a deep, integrated analysis. 1 : Superficial; misses most required metrics or necessary context.
Relevance & Utility	1–5	Is the included information relevant, and does the response avoid fluff or potentially misleading tangents? 5 : High precision; no fluff or harmful information. 1 : Cluttered with irrelevant or misleading content.
Clarity & Rationale	1–5	Is the answer clear and well structured, and does it explain its reasoning? 5 : Exceptionally clear and well reasoned. 1 : Confusing, poorly written, or reasoning is opaque.

Appendix

Table 28: Development-time human annotation rubric. This rubric was used during calibration of the construction-time automated evaluator. Reviewers additionally had optional fields for a human-written golden answer and free-text notes. Intermediate values between the endpoint anchors were available on the 1–5 scale but were not assigned additional verbal anchors in the original annotation instructions.

Screening category	Operational interpretation	Action
Wrong entity / company	Response analyzes a different company, ticker, or proxy rather than the entity requested by the question.	Regenerate
Wrong period / ticker–period mismatch	Response is inconsistent with the requested historical entity or period.	Regenerate
Unsupported material claims	Response contains material claims that are not supported by the benchmark evidence available for the item.	Regenerate / escalate if ambiguous
Conclusion inconsistent with supplied evidence	Final conclusion conflicts materially with the information or indicators associated with the item.	Regenerate / escalate if ambiguous
Incomplete answer	Response fails to answer a required part of the question; for example, a cross-company question is answered using only one company.	Regenerate
Abrupt truncation / sudden break	Generation terminates before completing the reasoning or answering the main question.	Regenerate

Appendix

Table 29: Construction-time fatal-flaw screening guide. This stage was a release-quality screen rather than an independent financial-reasoning score. Responses exhibiting catastrophic failures were regenerated and retained only after passing the screen; ambiguous cases were escalated for financial-expert review.

Dimension	5	4	3	2	1
Factual & Numerical Accuracy	Fully supported by supplied evidence; calculations and entity/time references are sound.	Mostly correct with a minor non-material issue.	Mixed: useful core analysis but meaningful errors or unsupported details.	Substantial factual or numerical problems.	Materially wrong, contradicted by evidence, or wrong entity/period.
Completeness & Context	Fully answers the question and covers evidence needed for the conclusion.	Substantively complete with a minor omission.	Partially complete; misses a meaningful aspect.	Major omissions.	Largely non-responsive.
Relevance & Utility	Directly addresses the question with useful synthesis.	Relevant with small digressions.	Useful but noticeably generic, padded, or indirect.	Weakly relevant.	Mostly irrelevant or non-responsive.
Clarity & Rationale	Concise, clear, and well justified.	Clear with minor presentation issues.	Understandable but wordy, repetitive, or mechanical.	Difficult to follow or poorly justified.	Confusing or disjointed.
Pairwise fields
Pairwise preference (A / B / Tie)		Entered only after all four ratings were completed independently for both responses.

Appendix

Table 30: Held-out human response-scoring rubric and pairwise fields. Reference metrics were presented only as helpful context and were explicitly not treated as a mandatory checklist. For subjective investment-style questions, a conclusion differing from another response could still receive a high score when it was supported by the supplied evidence and appropriately qualified.

Dimension	Labels	Rubric definition
Question Clarity	Pass; Minor issue; Major issue	Pass : clear and answerable at the historical point in time. Minor issue : small wording or precision issue that does not change the task. Major issue : ambiguous, inconsistent, based on an unanswerable or false premise, or not answerable from benchmark evidence.
Reference Answer Correctness	Correct; Minor correction; Major incorrect; Unverifiable	Correct : material claims and conclusion are supported. Minor correction : main conclusion remains valid but a small correction is needed. Major incorrect : material claims or conclusion conflict with evidence. Unverifiable : supplied evidence is insufficient. For subjective investment-style questions, the reviewer was instructed not to require a unique recommendation; the conclusion instead had to be defensible and appropriately qualified.
Evidence Support	Full; Partial; Unsupported; Evidence missing	Full : evidence is sufficient for the material claims and conclusion. Partial : supports the main conclusion but not all secondary claims. Unsupported : material claims are unsupported or contradicted. Evidence missing : no usable canonical evidence.
Golden-Indicator Adequacy	Adequate; Minor issue; Missing critical; Incorrect; N/A	Adequate : captures the key evidence. Minor issue : basically correct with a non-critical omission or extraneous item. Missing critical : an essential indicator is absent. Incorrect : materially mismatched. N/A : an indicator checklist is not meaningful for the item. Golden indicators were explicitly treated as a minimal evidence set, not a mandatory wording checklist.
Entity / Time Alignment	Pass; Unclear; Fail	Pass : entity and cutoff are aligned and no post-cutoff evidence is required. Unclear : alignment cannot be verified confidently. Fail : wrong entity or historical period, or future information is required or used.
Record fields

Appendix

Table 31: Core403 item-level human quality-audit rubric and record fields. The unit of annotation is a benchmark item, not a model response.

Financial Reasoning in Language Models

See all 13 papers

FINESSE-Bench: A Hierarchical Benchmark Suite for Financial Domain Knowledge and Technical Analysis in Large Language Models
May 14, 2026Dmitry Stanishevskii, Nini Kamkia, Alexey Khoroshilov +4Financial Language Model Evaluation LLM Evaluation
Lime FinTech
V-FiLLM: Verified Financial LLM Reasoning Benchmark
Aug 11, 2026Alicia Larsen, Victoire Laurent, Aulia Kharis Rakhmasari +2Numerical Reasoning in Language Models Synthetic Data Generation
ETH Zürich, Zürich · Aisot Technologies Ltd
Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements
Jul 22, 2026Xinke Tong, Xuanming Zhang, Tianyi Tang +10Numerical Reasoning in Language Models LLM Evaluation
Qwen Team, Alibaba Group · Tsinghua University · Alibaba Group

May 14, 2026J/K move · Enter open · S save

FINESSE-Bench: A Hierarchical Benchmark Suite for Financial Domain Knowledge and Technical Analysis in Large Language Models

Dmitry Stanishevskii, Nini Kamkia, Alexey Khoroshilov, Dmitry Zmitrovich+3

Lime FinTech

Financial Language Model Evaluation LLM Evaluation Quantitative Finance Benchmark Design Finance Financial Reasoning Benchmarks

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly being applied to financial analysis, reporting, investment decision support, risk management, compliance, and professional training. However, robust evaluation of their domain competence in finance remains incomplete. Widely used open benchmarks such as FinQA, ConvFinQA, and TAT-QA have played an important role in advancing financial question answering and numerical reasoning, but they focus primarily on question answering over financial reports and do not provide an explicit hierarchy of professional difficulty. Broader resources, including FinanceBench, PIXIU, FinBen, and FLaME, expand the coverage of financial tasks, yet the problem of evaluating the transition from foundational knowledge to expert-level financial reasoning remains open. In this work, we present FINESSE-Bench, a suite of eight specialized benchmarks comprising 3,993 questions for hierarchical evaluation of financial competencies in LLMs. FINESSE-Bench combines exam-oriented datasets inspired by professional certifications (CFA-like Levels 1-3, CMT-like Level 2, and CFTe-like Level 1), applied trading task collections, and a Russian-language olympiad benchmark. This design enables evaluation of domain breadth, performance degradation as difficulty increases, the ability to solve computational tasks, and model behavior in specialized financial domains. We also describe a unified evaluation protocol covering multiple-choice questions, numerical answers, and short open-ended responses, together with an automated scoring scheme for freeform answers based on the LLM-as-judge paradigm. FINESSE-Bench is intended both as a complement to existing open financial benchmarks and as a tool for more substantive evaluation of professionally relevant financial competencies in large language models.

Abstract

Financial Reasoning in Language Models

Explore similar work

FINESSE-Bench: A Hierarchical Benchmark Suite for Financial Domain Knowledge and Technical Analysis in Large Language Models

V-FiLLM: Verified Financial LLM Reasoning Benchmark

Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements