FinReasoning: A Hierarchical Benchmark for Reliable Financial Research Reporting

Yiyun Zhu; Yidong Jiang; Ziwen Xu; Yinsheng Yao; Dawei Cheng; Jinru Ding; Jie Xu

cs.CLFeb 25, 2026

FinReasoning: A Hierarchical Benchmark for Reliable Financial Research Reporting

Authors: Yiyun Zhu, Yidong Jiang, Ziwen Xu, Yinsheng Yao, Dawei Cheng, Jinru Ding, Jie Xu

Organizations: School of Computer Science and Technology, Tongji University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China

Abstract

Large language models (LLMs) are increasingly deployed in financial research workflows, where their role is evolving from single-model assistance for human analysts toward autonomous collaboration among multiple agents. Yet real-world deployments still expose factual errors, numerical inconsistencies, and shallow analysis, which can distort assessments of corporate fundamentals and trigger severe economic losses. While existing benchmarks have begun to evaluate such failures, they score all aspects of the generated analysis in one pass, failing to distinguish whether a model fails at foundational stages like auditing and correction, or underperforms at generating research-grade insights. Consequently, it obscures capability bottlenecks and the specialized strengths essential for multi-agent role assignment. To address these gaps, we introduce FinReasoning, a hierarchical benchmark that decomposes the core capabilities of financial research into semantic consistency, data alignment, and deep insight. We further propose a fine-grained evaluation framework that strengthens hallucination-correction assessment and incorporates a 12-indicator rubric for core analytical skills. FinReasoning reveals clear capability stratification across model types. Closed-source models (like Doubao-Seed-1.8) perform strongly overall and are better suited for core reasoning agents in multi-agent financial systems; open-source general models (like Qwen3-235B) show clear capability divergence and consistently underperform on Semantic Consistency, making them less suited for quality-sensitive generation tasks; financial-domain models (like Fin-R1) generate moderate insights but lack foundational auditing skills. Our work has already been deployed in pilot tests across several real-world scenarios. The resource is available at https://github.com/TongjiFinLab/FinReasoning.

Figures & tables

Figure 1: An overview of the taxonomy of FinReasoning benchmark, which is organized into three complementary tracks. Semantic Consistency track evaluates text-grounded error detection and correction. Data Alignment track assesses structured data grounding and executable rule-based reasoning. Deep Insight track measures research-grade analyses. Each track is constructed with multi-level tasks to diagnose LLMs’ capabilities from factual reliability to analytical reasoning.

Benchmarks	Task Type		Core Competence			Evaluation Protocol			Baselines
Benchmarks	Obj.	Subj.	Calculation	Consistency	Long Text	Staged Eval.	LLM Judge	Expert Rubric	Baselines
TAT-QA [ 45 ]	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	5
FinQA [ 7 ]	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	11
SEC-QA [ 21 ]	$\boldsymbol{\times}$	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$	4
FinAuditing [ 36 ]	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\checkmark}$	$\boldsymbol{\checkmark}$	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	13
FAITH [ 43 ]	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	16

Table 1: Comparison of FinReasoning with existing financial benchmarks. The criteria columns of the table include: (1) task type (objective vs. subjective/open-ended), (2) core competence (calculation, consistency, long text), (3) evaluation protocol (staged evaluation, LLM judge, expert rubric), (4) total evaluation samples, and (5) baselines tested.

Figure 2: An overview of the FinReasoning benchmark, presenting its data sources, the construction procedures for each task, and the data statistics including the number of sub-tasks within each track.

Track	number	Length (Q)			Length (A)
Track	number	avg.	max	min	avg.	max	min
Semantic Consistency	1800	8783	12183	5475	6767	12317	4326
Data Alignment	1800	57	220	23	1	1	1
Deep Insight	1200	138	347	33	1483	2823	653

Table 2: An overview of FinReasoning statistics, including the number of samples and the average, maximum, and minimum lengths of questions and answers.

Track	Evaluation Metrics	Method
Semantic Consistency	Error Location	Sentence-F1
	Error Explanation	BERTScore, SimCSE, LLM-as-a-Judge
	Correct Content	BERTScore, SimCSE, LLM-as-a-Judge
Data Alignment	Success Rate	Rate
	Answer Accuracy	Accuracy
	Retrieval Accuracy	F1-Score (Data ID, Field)

Table 3: Evaluation Metrics Overview. This table details the metrics and methodologies. For LLM-judge, we use a fused protocol combining DeepSeek-V3 and Qwen3-235B.

Model	Semantic Consistency					Data Alignment					Deep Insight						Overall
Model	Term.	Fact	Logic	Avg.	Norm.	Ver.	Cal.	Reason.	Avg.	Norm.	ARS	F&C	C&A	JCD	Avg.	Norm.	Norm.
Doubao-Seed-1.8	63.9	74.4	63.3	67.2	100.0	93.5	66.6	55.6	71.9	92.1	93.5	87.2	86.2	95.7	90.6	98.3	96.8
GPT-5	64.3	71.4	57.6	64.4	94.9	88.3	72.8	68.5	76.5	100.0	92.4	84.6	86.7	81.8	86.4	91.1	95.4
Kimi-K2	49.5	66.0	59.1	58.2	83.7	90.2	69.3	63.2	74.2	96.1	93.4	87.4	89.5	96.3	91.6	100.0	93.3
Claude-sonnet-4.5	54.9	69.0	51.8	58.6	84.4	85.3	64.6	62.7	70.9	90.4	92.2	87.1	84.5	96.2	90.0	97.3	90.7
GLM-4.6	59.0	73.4	62.5	65.0	96.0	89.4	67.5	58.4	59.4	70.7	92.6	88.1	87.3	96.1	91.0	99.0	88.6

Table 4: Performance comparison of LLMs across different tracks (Best in bold, second best underlined; the full leaderboard is ranked by the Overall Normalized Score). Ver.=Verification; Cal.=Calculation; Reason.=Reasoning; ARS=Architectural Richness and Strictness; FC=Factuality and Contextualization; CA=Completeness and Comparative Analysis; JCD=Justification and Causal Depth; Avg.=Average Score; Norm.=Normalized Score.

Task	Error Location	Error Explanation			Correct Content			Overall Score
Task	Sentence-F1	BERTScore	SimCSE	LLM	BERTScore	SimCSE	LLM	Overall Score
Terminology	20.6	66.0	67.5	49.9	36.7	35.7	50.4	40.9
Fact	40.0	64.8	69.3	57.7	52.8	51.5	58.5	52.7
Logic	21.5	61.9	62.7	62.8	26.4	25.8	62.2	40.7

Table 5: Detailed results across tasks in Semantic Consistency.

Figure 4: Visualized DI scores across tasks and model series.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Dimension	Explanation	Number
Semantic Consistency (SC)
Terminology	Improper Usage	Inconsistent terminology, reducing rigor	200
	Terminology Inconsistency	Use of colloquial or non-domain standard vocabulary	200
	Terminology Confusion	Words with similar semantics but different definitions	200
Fact	Relation Error	Errors in predicates, actions, or relationship descriptions	200
	Entity Error	Wrong key entities or subject-object reversal	200

Appendix

Table 7: Hierarchical Task Distribution of FinReasoning. This table provides a detailed breakdown of the tracks, categories, and dimensions, including the specific logic and sample size for each task.

Track	DeepSeek-V3 Weight	Qwen3-235B Weight
Semantic Consistency (SC)	0.58	0.42
Deep Insight (DI)	0.60	0.40

Appendix

Table 8: Optimal fusion weights and expert agreement by track.

Category / Dimension	Input Tokens (Avg)	Output Tokens (Avg)	Total Tokens (Avg)	Latency (ms, Avg)
Semantic Consistency
Terminology Confusion	5078.2	3385.0	8463.3	105947.4
Terminology Inconsistency	5322.2	4286.5	9608.8	96807.5
Improper Usage	5049.1	3540.5	8589.6	88156.3
Entity Error	5024.5	3462.3	8486.7	86872.0
Relation Error	5045.9	3573.3	8619.2	87682.7

Appendix

Table 9: Computational Cost and Scalability Statistics Across Tracks (average per sample). We report input/output/total tokens and latency for each category.

Models	Error Explanation		Correct Content
Models	Origin	Rephrase	Origin	Rephrase
Close-Source
GPT-5	85.09	83.31	88.12	87.43
GPT-4o	48.24	49.78	50.34	49.94
Gemini-3-pro	79.00	81.45	82.17	81.39
Claude-sonnet-4.5	80.65	79.69	83.82	80.92

Appendix

Table 10: Prompt sensitivity of LLM-as-a-Judge evaluation under original and rephrased prompts for the Terminology category in the Semantic Consistency track.

Models	Error Explanation		Correct Content
Models	Origin	Rephrase	Origin	Rephrase
Close-Source
GPT-5	80.14	79.76	82.38	85.43
GPT-4o	60.08	62.63	62.74	63.79
Gemini-3-pro	81.43	78.12	83.35	87.07
Claude-sonnet-4.5	82.12	83.45	85.22	87.80

Appendix

Table 11: Prompt sensitivity of LLM-as-a-Judge evaluation under original and rephrased prompts for the Fact category in the Semantic Consistency track.

Models	Error Explanation		Correct Content
Models	Origin	Rephrase	Origin	Rephrase
Close-Source
GPT-5	87.89	84.74	90.47	87.67
GPT-4o	67.15	68.27	68.71	72.19
Gemini-3-pro	87.15	89.34	88.32	91.65
Claude-sonnet-4.5	89.76	88.80	91.82	89.14

Appendix

Table 12: Prompt sensitivity of LLM-as-a-Judge evaluation under original and rephrased prompts for the Logic category in the Semantic Consistency track.

Models	ARS		F&C		C&A		JCD
Models	Origin	Rephrase	Origin	Rephrase	Origin	Rephrase	Origin	Rephrase
Close-Source
GPT-5	92.39	90.29	84.58	87.63	86.71	83.48	81.80	82.97
GPT-4o	79.96	84.19	77.64	75.83	74.78	75.33	76.93	72.87
Gemini-3-pro	82.14	79.78	79.01	80.95	76.65	75.88	80.67	79.56
Claude-sonnet-4.5	92.19	96.52	87.07	86.48	84.51	80.57	96.17	96.83

Appendix

Table 13: Prompt sensitivity of LLM-as-a-Judge evaluation under original and rephrased prompts for the Deep Insight track.

Figure 5: Comparative Evaluation of DeepSeek-V3 and Qwen3-235B in Semantic Consistency track.

Figure 6: Comparative Evaluation of DeepSeek-V3 and Qwen3-235B in Deep Insight track.

Figure 7: Bootstrap 95% confidence intervals across Semantic Consistency.

Figure 8: Bootstrap 95% confidence intervals across Data Alignment.

Figure 9: Bootstrap 95% confidence intervals across Deep Insight.

Dimension	Both Fail	Complete Success	Fix but Not Explain	Explain but Not Fix	Partial Success
Fact	33.2	32.9	9.4	3.6	20.9
Logic	31.6	39.0	10.1	6.4	13.0
Terminology	48.0	16.1	13.5	3.6	18.8

Appendix

Table 14: Average distribution of failure modes across the three dimensions of the Semantic Consistency track (%).

Figure 10: Failure Modes Distribution by Model.

Figure 11: Distribution of failure mode families in the data alignment track.

Figure 12: Model-wise counts of failure mode families, excluding fully correct cases.

Figure 13: Failure mode distribution across verification, calculation, and reasoning tasks.

Figure 14: Distribution of weak dimensions.

Figure 15: Diagnosis of high-frequency error patterns.

Figure 16: Performance differences across model tiers.

Models	Error Location Accuracy	Error Explanation			Correct Content			Overall Score
Models	Error Location Accuracy	BERTScore	SimCSE	LLM-as-a-Judge	BERTScore	SimCSE	LLM-as-a-Judge	Overall Score
Close-Source
GPT-5	41.82	67.83	73.96	85.09	69.81	68.68	88.12	64.33
GPT-4o	11.68	64.65	63.82	48.24	30.39	28.94	50.34	35.71
Gemini-3-pro	43.39	69.17	76.09	79.00	66.42	65.10	82.17	63.13
Claude-sonnet-4.5	26.98	69.33	73.51	80.65	53.78	52.10	83.82	54.90

Appendix

Table 15: Semantic Consistency evaluation results of the Terminology category. This table reports model performance on each evaluation rubric and the overall score (Best in bold, second best underlined).

Models	Error Location Accuracy	Error Explanation			Correct Content			Overall Score
Models	Error Location Accuracy	BERTScore	SimCSE	LLM-as-a-Judge	BERTScore	SimCSE	LLM-as-a-Judge	Overall Score
Close-Source
GPT-5	63.47	68.33	76.77	80.14	72.43	72.41	82.38	71.43
GPT-4o	36.41	62.69	63.68	60.08	63.88	62.01	62.74	53.81
Gemini-3-pro	62.59	69.41	77.84	81.43	71.19	69.08	83.35	71.12
Claude-sonnet-4.5	53.88	70.27	78.16	82.12	72.17	71.19	85.22	68.97

Appendix

Table 16: Semantic Consistency evaluation results of Fact category. This table reports model performance on each evaluation rubric and the overall score(Best in bold, second best underlined).

Models	Error Location Accuracy	Error Explanation			Correct Content			Overall Score
Models	Error Location Accuracy	BERTScore	SimCSE	LLM-as-a-Judge	BERTScore	SimCSE	LLM-as-a-Judge	Overall Score
Close-Source
GPT-5	37.21	64.45	72.32	87.89	45.38	45.80	90.47	57.55
GPT-4o	14.44	60.72	55.54	67.15	21.48	21.30	68.71	37.58
Gemini-3-pro	28.35	64.87	71.59	87.15	28.27	26.51	88.32	50.19
Claude-sonnet-4.5	24.37	66.47	72.52	89.76	36.55	36.37	91.82	51.84

Appendix

Table 17: Semantic Consistency evaluation results of the Logic category. This table reports model performance on each evaluation rubric and the overall score (Best in bold, second best underlined).

Models	Success Rate	Answer Accuracy	Data IDs F1	Fields F1	Overall Score
Close-Source
GPT-5	100.00	81.50	99.86	83.50	88.29
GPT-4o	100.00	81.83	100.00	89.35	90.39
Gemini-3-pro	100.00	87.17	100.00	95.82	94.33
Claude-sonnet-4.5	92.83	86.54	99.84	89.27	85.30
GLM-4.6	100.00	79.17	100.00	88.99	89.39

Appendix

Table 18: Data Alignment evaluation results of the Verification category. This table reports model performance on each evaluation rubric and the overall score (Best in bold, second best underlined).

Models	Success Rate	Answer Accuracy	Data IDs F1	Fields F1	Overall Score
Close-Source
GPT-5	100.00	72.25	81.32	64.74	72.77
GPT-4o	100.00	48.63	76.23	58.17	61.01
Gemini-3-pro	100.00	73.13	76.14	60.42	69.90
Claude-sonnet-4.5	95.63	64.24	78.25	60.28	64.63
GLM-4.6	98.75	71.15	74.92	58.95	67.49

Appendix

Table 19: Data Alignment evaluation results of the Calculation category. This table reports model performance on each evaluation rubric and the overall score (Best in bold, second best underlined).

Models	Success Rate	Answer Accuracy	Data IDs F1	Fields F1	Overall Score
Close-Source
GPT-5	100.00	63.00	64.67	77.80	68.49
GPT-4o	100.00	57.00	62.31	69.85	63.05
Gemini-3-pro	100.00	60.75	38.62	71.09	56.82
Claude-sonnet-4.5	99.50	60.80	52.18	75.99	62.68
GLM-4.6	100.00	55.75	46.75	72.69	58.40

Appendix

Table 20: Data Alignment evaluation results of the Reasoning category. This table reports model performance on each evaluation rubric and the overall score (Best in bold, second best underlined).

Models	Justification & Causal Depth				Factuality & Context				Completeness & Comparative Analysis				Architecture Richness & Strictness
Models	Score	RC	CD	Prof.	Score	FD	CR	CA	Score	CT	CP	AB	Score	FC	LH	EG
Close-Source
GPT-5	81.80	32.29	24.18	24.78	84.58	32.80	25.36	26.42	86.71	35.39	24.24	27.08	92.39	36.99	27.74	27.66
GPT-4o	76.93	31.95	21.52	23.46	77.64	31.84	22.08	23.72	74.78	31.52	19.62	23.64	79.96	32.00	23.98	23.98
Gemini-3-pro	80.67	33.73	21.62	25.32	79.01	32.21	22.16	24.64	76.65	32.43	20.38	23.84	82.14	32.88	24.66	24.60
Claude-sonnet-4.5	96.17	38.13	28.54	29.50	87.07	34.11	25.94	27.02	84.51	34.85	23.48	26.18	92.19	36.93	27.72	27.54

Appendix

Table 21: Deep Insight evaluation results across four categories: Justification & Causal Depth , Factuality & Context , Completeness & Comparative Analysis , and Architecture Richness & Strictness . The abbreviated rubrics are defined as follows: RC = Reasoning Chain, CD = Causal Depth, Prof. = Professionalism, FD = Factual Density, CR = Computational Rigor, CA = Contextual Analysis, CT = Critical Thinking, CP = Comparative Perspective, AB = Argument Balance, FC = Framework Completeness, LH = Logical Hierarchy, and EG = Expression Granularity. This table reports model performance on each rubric and the overall category score (Best in bold, second best underlined).

Dimension	Design Principles
Justification and causal depth	Thinking Model: Multi-stage transmission path analysis. Task Objective: Identify the initial driving variable in the report and require the model to infer its downstream impact on the firm’s financial performance or industry position. Prompting Technique: The instruction should force the model to build a long reasoning chain of Variable A $\rightarrow$ Intermediate Mechanism B $\rightarrow$ Financial Outcome C. Question Design Requirement: The background facts should include driving factors, such as policy changes or technological breakthroughs. The instruction should ask how such a factor penetrates the business layer and is ultimately reflected in the financial statements. Answer Scoring Rubric (100-point scale): [40] Logical chain: whether a clear and closed A $\rightarrow$ B $\rightarrow$ C chain is established without logical jumps. [30] Causal depth: whether the answer explains why the transmission is necessary or highly probable. [30] Professionalism: whether the terminology is precise and consistent with CFA-style analytical reporting.
Factuality and contextualization	Thinking Model: Data anchoring and quantitative verification. Task Objective: Evaluate the model’s ability to extract, understand, and apply hard data from the report. Prompting Technique: The instruction should require the model to use specific numerical values in the background facts, such as gross margin, CAGR, order amount, or market share, for secondary calculation or logical argumentation. Question Design Requirement: The background facts must contain at least 3–5 groups of core financial or operational data. The instruction should include requirements such as “quantitatively assess the extent to which … is supported by the above data” or “compare and analyze the marginal changes in …”. Answer Scoring Rubric (100-point scale): [40] Fact density: whether all key numerical values in the question are accurately cited and interpreted. [30] Computational rigor: whether the accounting and quantitative relations across data points are correct. [30] Contextual analysis: whether the answer explains the meaning of the data under the relevant industry context.
Completeness and comparative analysis	Thinking Model: Dialectical conflict and margin-of-safety analysis. Task Objective: Identify the tension between optimistic expectations and potential constraints in the report, and evaluate the model’s critical thinking ability. Prompting Technique: The instruction should force the model to identify risk disclosures, prerequisite conditions, or threats from competitors in the report. Question Design Requirement: The background facts must include both positive and negative information, such as new capacity expansion versus slowing industry demand growth. The instruction should ask the model to evaluate under what conditions the positive logic would be falsified. Answer Scoring Rubric (100-point scale): [40] Critical thinking: whether the answer identifies boundary conditions or downside risks under which the conclusion may fail. [30] Comparative perspective: whether horizontal peer comparison or vertical historical comparison is conducted. [30] Argument balance: whether the answer remains objective and avoids blind optimism.
Architectural richness and strictness	Thinking Model: Systematic industry analysis framework. Task Objective: Reorganize fragmented facts into a highly professional and structurally rigorous industry analysis. Prompting Technique: The instruction should require the model to use professional analytical frameworks, such as SWOT, Porter’s Five Forces, PEST, or the BCG Matrix, for integrated discussion. Question Design Requirement: The background facts should cover multi-dimensional information. The instruction should explicitly ask the model to provide a systematic and in-depth analysis using a professional analytical framework. Answer Scoring Rubric (100-point scale): [40] Framework completeness: whether an appropriate analytical framework is adopted and whether the coverage of dimensions is comprehensive. [30] Logical hierarchy: whether the response follows a clear overall–particular–overall structure with explicit nesting. [30] Granularity of expression: whether concepts are defined with sufficient depth and rigor, reaching the standard of top-tier institutions.

Appendix

Table 22: Dimension-specific gold-answer design principles for the Deep Insight track.

Sub-dimension	Weight	Scoring Criteria (0–5)
Logical chain	40%	0: No logical structure at all; statements are disconnected; reasoning does not hold.
		1: Only an A $\rightarrow$ B pattern appears, but without actual derivation; severe logical jumps; incomplete reasoning.
		2: A rudimentary A $\rightarrow$ B $\rightarrow$ C chain appears, but it is loose; key logic is missing; explanation is insufficient.
		3: A basic A $\rightarrow$ B $\rightarrow$ C closed loop is formed; the logic is clear but limited in depth.
		4: The A $\rightarrow$ B $\rightarrow$ C structure is complete and rigorous; causal progression is natural; no obvious flaws.
		5: Multi-layer reasoning chains (e.g., A $\rightarrow$ B $\rightarrow$ C $\rightarrow$ D); internally coherent; identifies premises and implicit assumptions; demonstrates a professional reasoning system.

Appendix

Table 23: Category-specific scoring rubric for Justification and Causal Depth.

Sub-dimension	Weight	Scoring Criteria (0–5)
Fact density	40%	0: Data are wrong, fabricated, or completely irrelevant to the question.
		1: Only a very small number of facts are cited; key data are missing; relevance is weak.
		2: Some key facts are cited, but coverage is incomplete; there are minor inaccuracies or a lack of context.
		3: The cited information is broadly accurate and covers the main points, but more comprehensive data support is still lacking.
		4: Factual references are comprehensive and accurate; key financial indicators or core evidence are included; information density is high.
		5: Data references are precise and rich; they cover core indicators, supporting indicators, trend facts, and so on; the answer reaches the level of a professional analytical report.

Appendix

Table 24: Category-specific scoring rubric for Factuality and Contextualization.

Sub-dimension	Weight	Scoring Criteria (0–5)
Critical thinking	40%	0: No critical discussion; no risks or assumption limits are identified.
		1: Merely repeats information mechanically; lacks awareness of risks or boundary conditions.
		2: Identifies some risks or limitations, but only at a surface level; lacks structured logic.
		3: Points out key assumptions, boundaries, or sources of risk; shows a basic critical perspective.
		4: Systematically analyzes risks and limitations; proposes insightful counterexamples or hedging logic.
		5: Demonstrates professional research- or investment-grade critical thinking; actively examines assumptions, boundaries, and reverse scenarios, and provides rigorous rebuttals and limitation analysis.

Appendix

Table 25: Category-specific scoring rubric for Completeness and comparative Analysis.

Sub-dimension	Weight	Scoring Criteria (0–5)
Framework completeness	40%	0: No structure; content is chaotic; no analytical framework is presented.
		1: There are scattered points, but they are unrelated to one another; the answer lacks even a basic structure.
		2: A preliminary framework is formed, but coverage is incomplete; key analytical dimensions are missing.
		3: The framework is relatively clear and covers the main dimensions; the overall structure is understandable.
		4: The framework is complete and systematic; dimensional coverage is comprehensive; the answer presents a professional analytical structure.
		5: Reaches the framework quality of consulting or investment-research reports; multiple dimensions are systematically linked; the structure is highly rigorous with clearly differentiated functional sections.

Appendix

Table 26: Category-specific scoring rubric for Architectural richness and strictness (ARS).

Explore similar work

Mar 9, 2026

Evaluating Financial Intelligence in Large Language Models: Benchmarking SuperInvesting AI with LLM Engines

Large language models are increasingly used for financial analysis and investment research, yet systematic evaluation of their financial reasoning capabilities remains limited. In this work, we introduce the AI Financial Intelligence Benchmark (AFIB), a multi-dimensional evaluation framework designed to assess financial analysis capabilities across five dimensions: factual accuracy, analytical completeness, data recency, model consistency, and failure patterns. We evaluate five AI systems: GPT, Gemini, Perplexity, Claude, and SuperInvesting, using a dataset of 95+ structured financial analysis questions derived from real-world equity research tasks. The results reveal substantial differences in performance across models. Within this benchmark setting, SuperInvesting achieves the highest aggregate performance, with an average factual accuracy score of 8.96/10 and the highest completeness score of 56.65/70, while also demonstrating the lowest hallucination rate among evaluated systems. Retrieval-oriented systems such as Perplexity perform strongly on data recency tasks due to live information access but exhibit weaker analytical synthesis and consistency. Overall, the results highlight that financial intelligence in large language models is inherently multi-dimensional, and systems that combine structured financial data access with analytical reasoning capabilities provide the most reliable performance for complex investment research workflows.

Akshay Gulati, Kanha Singhania, Tushar Banga +8

The Future University

LLM Evaluation LLM Reliability Finance

Ask PDF

Aug 11, 2026

V-FiLLM: Verified Financial LLM Reasoning Benchmark

While existing benchmarks have made substantial progress in evaluating LLMs across STEM domains, financial reasoning over structured data remains comparatively less explored. We introduce V-FiLLM, a framework that generates financial reasoning benchmarks from executable computation trees grounded in real tables, yielding items whose answers are correct by construction. Trees are evaluated symbolically to obtain ground truth and rendered into natural-language questions, removing any model from the labeling loop, so items can be generated at arbitrary scale without annotation cost and without inheriting a generator's error rate. V-FiLLM exposes four independently controllable axes of difficulty including computation depth, expression breadth, financial concept complexity, and context size. By evaluating on open-source models, we find that accuracy falls up to 51% as reasoning depth increases, and up to 47% points under adversarial numerical perturbations, highlighting remaining challenges in robust financial reasoning over tables. We further show that lightweight LoRA fine-tuning on verified chain-of-thought traces improves accuracy from 81.1% to 85.6% on held-out problems and outperforms the base model by 5% points on FinQA (Chen et al., 2022a), s), suggesting that targeted, low-cost adaptation is a promising direction for compositional reasoning in financial QA.

Alicia Larsen, Victoire Laurent, Aulia Kharis Rakhmasari +2

ETH Zürich, Zürich · Aisot Technologies Ltd

Numerical Reasoning in Language Models Synthetic Data Generation Financial QA

Ask PDF

Jun 2, 2026

BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents

Financial-research answers are decision-relevant only when another analyst can audit how they were produced: which source was chosen, which period and accounting definition were used, which assumptions were made, and how the calculation was performed. Existing finance benchmarks largely evaluate isolated subskills or final answers, leaving the auditable derivation itself under-measured. We introduce BigFinanceBench, a 928-item expert-authored benchmark of open-ended financial-research tasks in which each item pairs a ground-truth reference answer with a point-weighted rubric that decomposes the derivation into independently checkable steps. BigFinanceBench is workflow-grounded in that it evaluates the full derivation rather than only the final output. Across 36,241 rubric points, the benchmark supports partial-credit evaluation and localization of failures across the analyst workflow. Evaluating ten current frontier and open-weight agents, we find substantial headroom: the best system reaches only 58.8% rubric score, final-answer accuracy is a useful but lossy proxy for derivation quality, and model capability varies non-uniformly across financial workflows.

Alex Wang, Georg Meinhardt, Jacob Katz +4

1Rogo · 2OpenAI · †Work done while at Rogo.

Benchmark Design Financial QA Rubric-Based Evaluation Finance Agent Benchmarks

Ask PDF