FinRule-Bench: A Benchmark for Joint Reasoning over Financial Tables and Principles

Arun Vignesh Malarkkan; Manan Roy Choudhury; Guangwei Zhang; Vivek Gupta; Qingyun Wang; Yanjie Fu; Denghui Zhang

cs.AIMar 11, 2026

FinRule-Bench: A Benchmark for Joint Reasoning over Financial Tables and Principles

Authors: Arun Vignesh Malarkkan, Manan Roy Choudhury, Guangwei Zhang, Vivek Gupta, Qingyun Wang, Yanjie Fu, Denghui Zhang

Organizations: Arizona State University Tempe, Arizona, USA · Pine AI Singapore, Singapore · The College of William & Mary Williamsburg, Virginia, USA · Stevens Institute of Technology New Jersey, USA

Abstract

Large language models (LLMs) are increasingly applied to financial analysis, yet their ability to audit structured financial statements under explicit accounting principles remains poorly explored. Existing benchmarks primarily evaluate question answering, numerical reasoning, or anomaly detection on synthetically corrupted data, making it unclear whether models can reliably verify or localize rule compliance on correct financial statements. We introduce FinRule-Bench, a benchmark for evaluating diagnostic completeness in rule-based financial reasoning over real-world financial tables. FinRule-Bench pairs ground-truth financial statements with explicit, human-curated accounting principles and spans four canonical statement types: Balance Sheets, Cash Flow Statements, Income Statements, and Statements of Equity. The benchmark defines three auditing tasks that require progressively stronger reasoning capabilities: (i) rule verification, which tests compliance with a single principle; (ii) rule identification, which requires selecting the violated principle from a provided rule set; and (iii) joint rule diagnosis, which requires detecting and localizing multiple simultaneous violations at the record level. We evaluate LLMs under zero-shot and few-shot prompting, and introduce a causal-counterfactual reasoning protocol that enforces consistency between decisions, explanations, and counterfactual judgments. Across tasks and statement types, we find that while models perform well on isolated rule verification, performance degrades sharply for rule discrimination and multi-violation diagnosis. FinRule-Bench provides a principled and reproducible testbed for studying rule-governed reasoning, diagnostic coverage, and failure modes of LLMs in high-stakes financial analysis.

Figures & tables

Figure 1. FinRule-Bench Benchmark Overview. FinRule-Bench evaluates LLMs on financial auditing tasks that require both intra-table reasoning and joint reasoning across multiple financial statements under explicit accounting rules.

Statement Type	# Rules	Ground-Truth Records	Single-Violation Records	Multi-Violation Records
Balance Sheet (BS)	5	212	1060	100
Cash Flow (CF)	3	318	954	100
Statement of Equity (SE)	3	287	871	100
Income Statement (SI)	5	300	1500	100
Total	16	1117	4385	400

Table 1. Dataset statistics for FinRule-Bench. Each record corresponds to a company’s financial statement. Single-violation records contain exactly one violated rule, while multi-violation records contain multiple simultaneous violations.

Figure 2. Structural complexity distribution across financial statement types.

Model	Baseline Prompt	Rule Verification	Rule Identification	Diagnosis Step-1	Diagnosis Step-2
		Acc	Acc	Acc	Exact Match
GPT-4o	Zero-shot	0.640	0.592	0.713	0.234
	Few-shot	0.736	0.689	0.786	0.297
	Few-shot + CR	0.738	0.661	0.801	0.342
Gemini 2.5 Pro (Thinking)	Zero-shot	0.614	0.547	0.695	0.216
	Few-shot	0.629	0.612	0.728	0.259

Table 2. Overall Task performance on FinRule-Bench.

			Acc
Category	#Rules	#Instances	Zero-shot	Few-shot	Few-shot + CR
Arithmetic	2	499	0.916	0.657	0.759
Structural	5	424	0.735	0.794	0.791
Conditional	6	1357	0.597	0.978	0.991
Multi-record	3	510	0.916	0.822	0.790

Table 3. Rule Complexity Analysis on Rule Verification.

			Accuracy
Rule Category	#Rules	#Instances	Zero-shot	Few-shot	Few-shot + CR
Arithmetic	2	499	0.922	0.910	0.896
Structural	5	424	0.503	0.559	0.540
Conditional	6	1357	0.590	0.703	0.590
Multi-record	3	610	0.159	0.184	0.257

Table 4. Rule Complexity Analysis for Rule Identification.

			Metrics
Rule Category	#Rules	#Instances	Step-1 (Accuracy)	Step-2 (Micro-F1)
Arithmetic	2	79	0.797	0.545
Structural	5	136	0.824	0.586
Conditional	6	285	0.961	0.910
Multi-record	3	144	0.576	0.591

Table 5. Rule Complexity Analysis for Joint Rule Diagnosis.

Figure 3. Error distribution across prompting methods.

Figure 4. Performance-cost analysis for rule verification.

Figure 5. Performance-cost analysis for the joint rule diagnosis task - Step-1 anomaly detection and Step-2 localization.

Figure 6. Performance-cost analysis for rule identification.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7. Raw Financial Statement screenshot images

Section	Subsection	Item	Value
Assets	Current Assets	Cash and cash equivalents	8,588
		…	…
		Total current assets	18,961
	Non-Current Assets	Goodwill	7,660
		…	…
Total Assets			42,448

Appendix

Table 6. Example extracted Balance Sheet (MasterCard, December 31, 2023). The table illustrates hierarchical structure, subtotals, cross-section totals, and sign conventions used in FinRule-Bench. Individual line items are omitted for brevity.

Statement Type	# Records	Avg Rows	Min Rows	Max Rows	Avg Sections	Frac. w/ Subsections	Frac. w/ Subtotals	Avg Cols (Min-Max)
Balance Sheet (BS)	212	32.2	7	49	4.4	1.00	1.00	4.0 (4-4)
Cash Flow (CF)	318	34.4	7	60	3.4	1.00	1.00	4.0 (4-4)
Statement of Equity (SE)	287	10.1	1	29	10.0	1.00	0.98	8.5 (5-16)
Income Statement (SI)	300	22.4	2	40	7.9	1.00	0.99	3.0 (3-3)

Appendix

Table 7. Structural complexity statistics.

Sample	Cohen’s Kappa (%)	Jaccard Coefficient (%)
$\alpha_{1}$	93.3	97.5
$\alpha_{2}$	95.3	95.8

Appendix

Table 8. Inter-annotator agreement results for human verification

Statement	Method	Acc	Precision	Recall	F1-Score	Tokens (K)
BS	Zero-shot	0.749	0.824	0.890	0.855	967.7
	Few-shot	0.812	0.882	0.894	0.888	998.0
	Few-shot + CR	0.884	0.962	0.897	0.928	1003.1
CF	Zero-shot	0.402	0.696	0.361	0.476	982.5
	Few-shot	0.620	0.819	0.634	0.714	1009.2
	Few-shot + CR	0.643	0.849	0.638	0.728	1030.8

Appendix

Table 9. Performance on Rule Compliance Verification (GPT-4o)

Statement	Method	Acc	Precision	Recall	F1-Score	Tokens (K)
BS	Zero-shot	0.739	0.835	0.857	0.845	1272.0
	Few-shot	0.676	0.810	0.798	0.804	2617.9
	Few-shot + CR	0.473	0.768	0.526	0.624	3316.7
CF	Zero-shot	0.619	0.803	0.652	0.720	2067.7
	Few-shot	0.606	0.800	0.707	0.751	1595.3
	Few-shot + CR	0.628	0.886	0.578	0.700	2415.5

Appendix

Table 10. Performance on Rule Compliance Verification (Gemini 2.5 Pro)

Statement	Method	Acc	Precision	Recall	F1-Score	Tokens (K)
BS	Zero-shot	0.759	0.870	0.836	0.853	1109.4
	Few-shot	0.519	0.900	0.862	0.881	1142.3
	Few-shot + CR	0.297	0.840	0.739	0.786	2257.7
CF	Zero-shot	0.805	0.803	0.652	0.417	1045.6
	Few-shot	0.858	0.919	0.540	0.680	1070.7
	Few-shot + CR	0.907	0.953	0.632	0.760	2522.6

Appendix

Table 11. Performance on Rule Compliance Verification (Gemini 2.0)

Statement	Method	Acc	Precision	Recall	F1-Score	Tokens (K)
BS	Zero-shot	0.606	0.801	0.709	0.748	1006.1
	Few-shot	0.749	0.873	0.818	0.845	1036.6
	Few-shot + CR	0.759	0.959	0.743	0.837	2016.7
CF	Zero-shot	0.458	0.808	0.364	0.502	1027.8
	Few-shot	0.567	0.765	0.609	0.678	1053.9
	Few-shot + CR	0.579	0.762	0.638	0.695	2435.7

Appendix

Table 12. Performance on Rule Compliance Verification (LLaMA)

Statement	Method	Acc	Bal. Acc	M-Prec	M-Rec	M-F1	W-F1
BS	Zero-shot	0.663	0.663	0.751	0.663	0.641	0.641
	Few-shot	0.724	0.724	0.750	0.724	0.702	0.702
	Few-shot + CR	0.724	0.724	0.750	0.724	0.702	0.702
CF	Zero-shot	0.841	0.842	0.510	0.505	0.504	0.843
	Few-shot	0.868	0.861	0.516	0.516	0.516	0.868
	Few-shot + CR	0.782	0.782	0.645	0.563	0.587	0.816

Appendix

Table 13. Performance on Single Violated Rule Identification (GPT-4o)

Statement	Method	Acc	Bal. Acc	M-Prec	M-Rec	M-F1	W-F1
BS	Zero-shot	0.472	0.611	0.730	0.611	0.616	0.691
	Few-shot	0.428	0.674	0.708	0.674	0.652	0.671
	Few-shot + CR	0.491	0.729	0.759	0.729	0.666	0.668
CF	Zero-shot	0.937	0.938	1.000	0.938	0.968	0.968
	Few-shot	0.943	0.943	0.926	0.943	0.944	0.944
	Few-shot + CR	0.664	0.743	0.520	0.743	0.726	0.818

Appendix

Table 14. Performance on Single Violated Rule Identification ( Gemini 2.5 Pro)

Statement	Method	Acc	Bal. Acc	M-Prec	M-Rec	M-F1	W-F1
BS	Zero-shot	0.658	0.659	0.819	0.659	0.620	0.621
	Few-shot	0.678	0.678	0.694	0.678	0.618	0.618
	Few-shot + CR	0.704	0.708	0.782	0.708	0.665	0.664
CF	Zero-shot	0.862	0.862	0.883	0.862	0.864	0.864
	Few-shot	0.861	0.861	0.890	0.861	0.860	0.860
	Few-shot + CR	0.873	0.873	0.912	0.873	0.887	0.887

Appendix

Table 15. Performance on Single Violated Rule Identification (Gemini 2.0)

Statement	Method	Acc	Bal. Acc	M-Prec	M-Rec	M-F1	W-F1
BS	Zero-shot	0.540	0.702	0.781	0.702	0.707	0.748
	Few-shot	0.617	0.763	0.804	0.763	0.768	0.789
	Few-shot + CR	0.475	0.644	0.733	0.644	0.658	0.671
CF	Zero-shot	0.809	0.894	0.923	0.894	0.902	0.904
	Few-shot	0.687	0.814	0.842	0.814	0.819	0.821
	Few-shot + CR	0.699	0.823	0.861	0.823	0.830	0.833

Appendix

Table 16. Performance on Single Violated Rule Identification (LLaMA-3)

Task	Method	Step 1 Violation				Violation Classification
		Detection (Yes/No)				(Multi-label)
		Acc	Prec	Rec	F1	EM	Prec μ	Rec μ	F1 μ
BS	Zero-shot	77.0	79.4	96.0	86.91	15.0	49.3	84.7	62.3
	Few-shot	76.0	80.1	93.5	86.3	18.0	52.1	78.3	62.6
	Few-shot + CR	78.0	82.3	92.7	87.2	24.0	59.3	66.1	62.5
CF	Zero-shot	88.0	90.2	96.0	93.0	43.0	89.4	72.1	79.8

Appendix

Table 17. Performance Comparison of GPT-4o on Financial Statement Joint Multi-Step Diagnosis

Task	Method	Step 1 Violation				Violation Classification
		Detection (Yes/No)				(Multi-label)
		Acc	Prec	Rec	F1	EM	Prec μ	Rec μ	F1 μ
BS	Zero-shot	81.0	80.8	100.0	89.4	26.0	63.6	85.0	72.7
	Few-shot	83.0	82.5	100.0	90.4	35.0	66.5	84.4	74.4
	Few-shot + CR	81.0	80.8	100.0	89.4	28.0	59.3	83.8	69.4
CF	Zero-shot	84.0	83.8	100.0	91.2	57.0	81.5	90.4	85.7

Appendix

Table 18. Performance Comparison of Gemini 2.5 Pro on Financial Statement Joint Multi-Step Diagnosis

Task	Method	Step 1 Violation				Violation Classification
		Detection (Yes/No)				(Multi-label)
		Acc	Prec	Rec	F1	EM	Prec μ	Rec μ	F1 μ
BS	Zero-shot	78.0	82.2	92.5	87.1	21.0	59.8	69.8	63.9
	Few-shot	81.0	82.1	97.5	89.1	29.0	65.1	81.9	72.5
	Few-shot + CR	77.0	80.6	93.7	86.7	20.0	55.9	70.6	62.4
CF	Zero-shot	90.0	93.9	93.9	93.9	40.0	93.8	63.8	75.9

Appendix

Table 19. Performance Comparison of Gemini 2.0 flash on Financial Statement Joint Multi-Step Diagnosis

Task	Method	Step 1 Violation				Violation Classification
		Detection (Yes/No)				(Multi-label)
		Acc	Prec	Rec	F1	EM	Prec μ	Rec μ	F1 μ
BS	Zero-shot	80.0	80.0	100.0	88.9	8.0	45.6	84.4	59.2
	Few-shot	84.0	84.8	97.5	90.7	16.0	45.7	91.9	61.0
	Few-shot + CR	73.0	82.7	83.8	83.2	14.0	47.9	48.8	48.3
CF	Zero-shot	83.0	83.0	100.0	90.7	53.0	79.0	92.8	85.3

Appendix

Table 20. Performance Comparison of LLaMA-3 on Financial Statement Joint Multi-Step Diagnosis

Statement	Correct	Error Breakdown
Statement	Correct	Step-1 Error	Partial Det.	Partial Hall.	Inc. Loc.
BS	18%	22%	10%	41%	9%
CF	41%	9%	35%	4%	11%
SE	29%	19%	10%	34%	8%
SI	20%	18%	22%	31%	9%

Appendix

Table 21. Error Type Dist. for Joint Rule Diagnosis under Few-shot + Counterfactual Prompting (GPT-4o).

Category	Rule ID	Rule Description	FASB Reference	FASB Guideline Text
I. Balance Sheet Rules	BS-01	Total Assets = Total Liabilities + Total Ownership Interests (including parent company common and preferred equity, non-controlling interests, and any equity-classified mezzanine items).	FASB Concepts Statement No. 6 (CON 6)	Equity or net assets is the residual interest in the assets of an entity that remains after deducting its liabilities. In a business enterprise, the equity is the ownership interest.
	BS-02	Assets and liabilities must be classified into current and non-current categories.	210-10-45-5/210-10-05-4	A total of current liabilities shall be presented in classified balance sheets./ The balance sheets of most entities show separate classifications of current assets and current liabilities (commonly referred to as classified balance sheets) permitting ready determination of working capital.
	BS-03	The statement of cash flows must use descriptive terms like "cash" or "cash and cash equivalents".	230-10-45-4	The statement shall use descriptive terms such as cash or cash and cash equivalents rather than ambiguous terms such as funds.
	BS-04	Any appropriation of retained earnings must be presented separately within the shareholders’ equity section.	505-10-45-3	Appropriation of retained earnings is permitted, provided that it is shown within the shareholders’ equity section of the balance sheet and is clearly identified as an appropriation of retained earnings.
	BS-05	Treasury stock should be reported as a deduction from total shareholders’ equity.	505-10-05-1	The Equity Topic includes the following Subtopics: a. Overall b. Stock Dividends and Stock Splits c. Treasury Stock d. Subparagraph superseded by Accounting Standards Update No. 2018-07. e. Spinoffs and Reverse Spinoffs.
II. Income Statement Rules	SI-01	The results of operating activities must be reported separately from non-operating activities.	255-10-50-3	An entity shall disclose all of the following information for each of the five most recent years: a. Net sales and other operating revenues b. Income from continuing operations on a current cost basis c. ……

Appendix

Table 22. Financial Statement Rules Example

Explore similar work

FinVerBench: Benchmark Validity and Calibration in Large Language Model Financial Statement Verification
May 28, 2026Silu PandaLLM Evaluation Language Model Calibration
Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements
Jul 22, 2026Xinke Tong, Xuanming Zhang, Tianyi Tang +10Numerical Reasoning in Language Models LLM Evaluation
Qwen Team, Alibaba Group · Tsinghua University · Alibaba Group
V-FiLLM: Verified Financial LLM Reasoning Benchmark
Aug 11, 2026Alicia Larsen, Victoire Laurent, Aulia Kharis Rakhmasari +2Numerical Reasoning in Language Models Synthetic Data Generation
ETH Zürich, Zürich · Aisot Technologies Ltd

May 28, 2026J/K move · Enter open · S save

FinVerBench: Benchmark Validity and Calibration in Large Language Model Financial Statement Verification

Silu Panda

LLM Evaluation Language Model Calibration Benchmark Validity Finance

Open paper AskPDF arXiv

We introduce FinVerBench, a benchmark and validity study for financial statement verification: determining whether a set of corporate financial statements is numerically consistent from the information shown to the model. FinVerBench is built from SEC 10-K XBRL filings for 43 S&P 500 companies and defines a four-category error taxonomy covering arithmetic, cross-statement linkage, year-over-year, and magnitude perturbations. We attempt fifteen contemporary LLM evaluations and report fourteen complete runs; a Gemini 2.5 Pro run is excluded from the main comparison because 40/108 gateway calls failed. All binary metrics exclude underdetermined positive instances whose perturbed line item is not rendered, leaving a 105-instance observable diagnostic subset (43 clean, 62 error-injected). Under the original guided-checklist prompt on the unrounded diagnostic subset, nine of fourteen complete LLM runs produce 95-100% false positives on clean statements, while one run achieves 0% observed false positives. Benchmark rendering choices materially affect measured recall: on a realistic rounded variant of the same observable subset, the calibrated model's recall is 79.0% with 0% observed FPR, compared with 100.0% recall on the unrounded diagnostic variant. These results support a construct-validity conclusion rather than a final leaderboard: financial statement verification is not merely arithmetic detection, but calibrated judgment under incomplete observability, prompt-induced assumptions, and realistic numerical rendering. FinVerBench and all code are publicly available.

Abstract

Explore similar work

FinVerBench: Benchmark Validity and Calibration in Large Language Model Financial Statement Verification

Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements

V-FiLLM: Verified Financial LLM Reasoning Benchmark