Organizations: Arizona State University Tempe, Arizona, USA · Pine AI Singapore, Singapore · The College of William & Mary Williamsburg, Virginia, USA · Stevens Institute of Technology New Jersey, USA
Large language models (LLMs) are increasingly applied to financial analysis, yet their ability to audit structured financial statements under explicit accounting principles remains poorly explored. Existing benchmarks primarily evaluate question answering, numerical reasoning, or anomaly detection on synthetically corrupted data, making it unclear whether models can reliably verify or localize rule compliance on correct financial statements. We introduce FinRule-Bench, a benchmark for evaluating diagnostic completeness in rule-based financial reasoning over real-world financial tables. FinRule-Bench pairs ground-truth financial statements with explicit, human-curated accounting principles and spans four canonical statement types: Balance Sheets, Cash Flow Statements, Income Statements, and Statements of Equity. The benchmark defines three auditing tasks that require progressively stronger reasoning capabilities: (i) rule verification, which tests compliance with a single principle; (ii) rule identification, which requires selecting the violated principle from a provided rule set; and (iii) joint rule diagnosis, which requires detecting and localizing multiple simultaneous violations at the record level. We evaluate LLMs under zero-shot and few-shot prompting, and introduce a causal-counterfactual reasoning protocol that enforces consistency between decisions, explanations, and counterfactual judgments. Across tasks and statement types, we find that while models perform well on isolated rule verification, performance degrades sharply for rule discrimination and multi-violation diagnosis. FinRule-Bench provides a principled and reproducible testbed for studying rule-governed reasoning, diagnostic coverage, and failure modes of LLMs in high-stakes financial analysis.
Figures & tables
Figure 1. FinRule-Bench Benchmark Overview. FinRule-Bench evaluates LLMs on financial auditing tasks that require both intra-table reasoning and joint reasoning across multiple financial statements under explicit accounting rules.
Statement Type
# Rules
Ground-Truth Records
Single-Violation Records
Multi-Violation Records
Balance Sheet (BS)
5
212
1060
100
Cash Flow (CF)
3
318
954
100
Statement of Equity (SE)
3
287
871
100
Income Statement (SI)
5
300
1500
100
Total
16
1117
4385
400
Table 1. Dataset statistics for FinRule-Bench. Each record corresponds to a company’s financial statement. Single-violation records contain exactly one violated rule, while multi-violation records contain multiple simultaneous violations.
Figure 2. Structural complexity distribution across financial statement types.
Model
Baseline Prompt
Rule Verification
Rule Identification
Diagnosis Step-1
Diagnosis Step-2
Acc
Acc
Acc
Exact Match
GPT-4o
Zero-shot
0.640
0.592
0.713
0.234
Few-shot
0.736
0.689
0.786
0.297
Few-shot + CR
0.738
0.661
0.801
0.342
Gemini 2.5 Pro (Thinking)
Zero-shot
0.614
0.547
0.695
0.216
Few-shot
0.629
0.612
0.728
0.259
Table 2. Overall Task performance on FinRule-Bench.
Acc
Category
#Rules
#Instances
Zero-shot
Few-shot
Few-shot + CR
Arithmetic
2
499
0.916
0.657
0.759
Structural
5
424
0.735
0.794
0.791
Conditional
6
1357
0.597
0.978
0.991
Multi-record
3
510
0.916
0.822
0.790
Table 3. Rule Complexity Analysis on Rule Verification.
Accuracy
Rule Category
#Rules
#Instances
Zero-shot
Few-shot
Few-shot + CR
Arithmetic
2
499
0.922
0.910
0.896
Structural
5
424
0.503
0.559
0.540
Conditional
6
1357
0.590
0.703
0.590
Multi-record
3
610
0.159
0.184
0.257
Table 4. Rule Complexity Analysis for Rule Identification.
Metrics
Rule Category
#Rules
#Instances
Step-1 (Accuracy)
Step-2 (Micro-F1)
Arithmetic
2
79
0.797
0.545
Structural
5
136
0.824
0.586
Conditional
6
285
0.961
0.910
Multi-record
3
144
0.576
0.591
Table 5. Rule Complexity Analysis for Joint Rule Diagnosis.
Figure 3. Error distribution across prompting methods.
Figure 4. Performance-cost analysis for rule verification.
Figure 5. Performance-cost analysis for the joint rule diagnosis task - Step-1 anomaly detection and Step-2 localization.
Figure 6. Performance-cost analysis for rule identification.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7. Raw Financial Statement screenshot images
Section
Subsection
Item
Value
Assets
Current Assets
Cash and cash equivalents
8,588
…
…
Total current assets
18,961
Non-Current Assets
Goodwill
7,660
…
…
Total Assets
42,448
Appendix
Table 6. Example extracted Balance Sheet (MasterCard, December 31, 2023). The table illustrates hierarchical structure, subtotals, cross-section totals, and sign conventions used in FinRule-Bench. Individual line items are omitted for brevity.
Statement Type
# Records
Avg Rows
Min Rows
Max Rows
Avg Sections
Frac. w/ Subsections
Frac. w/ Subtotals
Avg Cols (Min-Max)
Balance Sheet (BS)
212
32.2
7
49
4.4
1.00
1.00
4.0 (4-4)
Cash Flow (CF)
318
34.4
7
60
3.4
1.00
1.00
4.0 (4-4)
Statement of Equity (SE)
287
10.1
1
29
10.0
1.00
0.98
8.5 (5-16)
Income Statement (SI)
300
22.4
2
40
7.9
1.00
0.99
3.0 (3-3)
Appendix
Table 7. Structural complexity statistics.
Sample
Cohen’s Kappa (%)
Jaccard Coefficient (%)
α1
93.3
97.5
α2
95.3
95.8
Appendix
Table 8. Inter-annotator agreement results for human verification
Statement
Method
Acc
Precision
Recall
F1-Score
Tokens (K)
BS
Zero-shot
0.749
0.824
0.890
0.855
967.7
Few-shot
0.812
0.882
0.894
0.888
998.0
Few-shot + CR
0.884
0.962
0.897
0.928
1003.1
CF
Zero-shot
0.402
0.696
0.361
0.476
982.5
Few-shot
0.620
0.819
0.634
0.714
1009.2
Few-shot + CR
0.643
0.849
0.638
0.728
1030.8
Appendix
Table 9. Performance on Rule Compliance Verification (GPT-4o)
Statement
Method
Acc
Precision
Recall
F1-Score
Tokens (K)
BS
Zero-shot
0.739
0.835
0.857
0.845
1272.0
Few-shot
0.676
0.810
0.798
0.804
2617.9
Few-shot + CR
0.473
0.768
0.526
0.624
3316.7
CF
Zero-shot
0.619
0.803
0.652
0.720
2067.7
Few-shot
0.606
0.800
0.707
0.751
1595.3
Few-shot + CR
0.628
0.886
0.578
0.700
2415.5
Appendix
Table 10. Performance on Rule Compliance Verification (Gemini 2.5 Pro)
Statement
Method
Acc
Precision
Recall
F1-Score
Tokens (K)
BS
Zero-shot
0.759
0.870
0.836
0.853
1109.4
Few-shot
0.519
0.900
0.862
0.881
1142.3
Few-shot + CR
0.297
0.840
0.739
0.786
2257.7
CF
Zero-shot
0.805
0.803
0.652
0.417
1045.6
Few-shot
0.858
0.919
0.540
0.680
1070.7
Few-shot + CR
0.907
0.953
0.632
0.760
2522.6
Appendix
Table 11. Performance on Rule Compliance Verification (Gemini 2.0)
Statement
Method
Acc
Precision
Recall
F1-Score
Tokens (K)
BS
Zero-shot
0.606
0.801
0.709
0.748
1006.1
Few-shot
0.749
0.873
0.818
0.845
1036.6
Few-shot + CR
0.759
0.959
0.743
0.837
2016.7
CF
Zero-shot
0.458
0.808
0.364
0.502
1027.8
Few-shot
0.567
0.765
0.609
0.678
1053.9
Few-shot + CR
0.579
0.762
0.638
0.695
2435.7
Appendix
Table 12. Performance on Rule Compliance Verification (LLaMA)
Statement
Method
Acc
Bal. Acc
M-Prec
M-Rec
M-F1
W-F1
BS
Zero-shot
0.663
0.663
0.751
0.663
0.641
0.641
Few-shot
0.724
0.724
0.750
0.724
0.702
0.702
Few-shot + CR
0.724
0.724
0.750
0.724
0.702
0.702
CF
Zero-shot
0.841
0.842
0.510
0.505
0.504
0.843
Few-shot
0.868
0.861
0.516
0.516
0.516
0.868
Few-shot + CR
0.782
0.782
0.645
0.563
0.587
0.816
Appendix
Table 13. Performance on Single Violated Rule Identification (GPT-4o)
Statement
Method
Acc
Bal. Acc
M-Prec
M-Rec
M-F1
W-F1
BS
Zero-shot
0.472
0.611
0.730
0.611
0.616
0.691
Few-shot
0.428
0.674
0.708
0.674
0.652
0.671
Few-shot + CR
0.491
0.729
0.759
0.729
0.666
0.668
CF
Zero-shot
0.937
0.938
1.000
0.938
0.968
0.968
Few-shot
0.943
0.943
0.926
0.943
0.944
0.944
Few-shot + CR
0.664
0.743
0.520
0.743
0.726
0.818
Appendix
Table 14. Performance on Single Violated Rule Identification ( Gemini 2.5 Pro)
Statement
Method
Acc
Bal. Acc
M-Prec
M-Rec
M-F1
W-F1
BS
Zero-shot
0.658
0.659
0.819
0.659
0.620
0.621
Few-shot
0.678
0.678
0.694
0.678
0.618
0.618
Few-shot + CR
0.704
0.708
0.782
0.708
0.665
0.664
CF
Zero-shot
0.862
0.862
0.883
0.862
0.864
0.864
Few-shot
0.861
0.861
0.890
0.861
0.860
0.860
Few-shot + CR
0.873
0.873
0.912
0.873
0.887
0.887
Appendix
Table 15. Performance on Single Violated Rule Identification (Gemini 2.0)
Statement
Method
Acc
Bal. Acc
M-Prec
M-Rec
M-F1
W-F1
BS
Zero-shot
0.540
0.702
0.781
0.702
0.707
0.748
Few-shot
0.617
0.763
0.804
0.763
0.768
0.789
Few-shot + CR
0.475
0.644
0.733
0.644
0.658
0.671
CF
Zero-shot
0.809
0.894
0.923
0.894
0.902
0.904
Few-shot
0.687
0.814
0.842
0.814
0.819
0.821
Few-shot + CR
0.699
0.823
0.861
0.823
0.830
0.833
Appendix
Table 16. Performance on Single Violated Rule Identification (LLaMA-3)
Task
Method
Step 1 Violation
Violation Classification
Detection (Yes/No)
(Multi-label)
Acc
Prec
Rec
F1
EM
Prec μ
Rec μ
F1 μ
BS
Zero-shot
77.0
79.4
96.0
86.91
15.0
49.3
84.7
62.3
Few-shot
76.0
80.1
93.5
86.3
18.0
52.1
78.3
62.6
Few-shot + CR
78.0
82.3
92.7
87.2
24.0
59.3
66.1
62.5
CF
Zero-shot
88.0
90.2
96.0
93.0
43.0
89.4
72.1
79.8
Appendix
Table 17. Performance Comparison of GPT-4o on Financial Statement Joint Multi-Step Diagnosis
Task
Method
Step 1 Violation
Violation Classification
Detection (Yes/No)
(Multi-label)
Acc
Prec
Rec
F1
EM
Prec μ
Rec μ
F1 μ
BS
Zero-shot
81.0
80.8
100.0
89.4
26.0
63.6
85.0
72.7
Few-shot
83.0
82.5
100.0
90.4
35.0
66.5
84.4
74.4
Few-shot + CR
81.0
80.8
100.0
89.4
28.0
59.3
83.8
69.4
CF
Zero-shot
84.0
83.8
100.0
91.2
57.0
81.5
90.4
85.7
Appendix
Table 18. Performance Comparison of Gemini 2.5 Pro on Financial Statement Joint Multi-Step Diagnosis
Task
Method
Step 1 Violation
Violation Classification
Detection (Yes/No)
(Multi-label)
Acc
Prec
Rec
F1
EM
Prec μ
Rec μ
F1 μ
BS
Zero-shot
78.0
82.2
92.5
87.1
21.0
59.8
69.8
63.9
Few-shot
81.0
82.1
97.5
89.1
29.0
65.1
81.9
72.5
Few-shot + CR
77.0
80.6
93.7
86.7
20.0
55.9
70.6
62.4
CF
Zero-shot
90.0
93.9
93.9
93.9
40.0
93.8
63.8
75.9
Appendix
Table 19. Performance Comparison of Gemini 2.0 flash on Financial Statement Joint Multi-Step Diagnosis
Task
Method
Step 1 Violation
Violation Classification
Detection (Yes/No)
(Multi-label)
Acc
Prec
Rec
F1
EM
Prec μ
Rec μ
F1 μ
BS
Zero-shot
80.0
80.0
100.0
88.9
8.0
45.6
84.4
59.2
Few-shot
84.0
84.8
97.5
90.7
16.0
45.7
91.9
61.0
Few-shot + CR
73.0
82.7
83.8
83.2
14.0
47.9
48.8
48.3
CF
Zero-shot
83.0
83.0
100.0
90.7
53.0
79.0
92.8
85.3
Appendix
Table 20. Performance Comparison of LLaMA-3 on Financial Statement Joint Multi-Step Diagnosis
Statement
Correct
Error Breakdown
Step-1 Error
Partial Det.
Partial Hall.
Inc. Loc.
BS
18%
22%
10%
41%
9%
CF
41%
9%
35%
4%
11%
SE
29%
19%
10%
34%
8%
SI
20%
18%
22%
31%
9%
Appendix
Table 21. Error Type Dist. for Joint Rule Diagnosis under Few-shot + Counterfactual Prompting (GPT-4o).
Category
Rule ID
Rule Description
FASB Reference
FASB Guideline Text
I. Balance Sheet Rules
BS-01
Total Assets = Total Liabilities + Total Ownership Interests (including parent company common and preferred equity, non-controlling interests, and any equity-classified mezzanine items).
FASB Concepts Statement No. 6 (CON 6)
Equity or net assets is the residual interest in the assets of an entity that remains after deducting its liabilities. In a business enterprise, the equity is the ownership interest.
BS-02
Assets and liabilities must be classified into current and non-current categories.
210-10-45-5/210-10-05-4
A total of current liabilities shall be presented in classified balance sheets./ The balance sheets of most entities show separate classifications of current assets and current liabilities (commonly referred to as classified balance sheets) permitting ready determination of working capital.
BS-03
The statement of cash flows must use descriptive terms like "cash" or "cash and cash equivalents".
230-10-45-4
The statement shall use descriptive terms such as cash or cash and cash equivalents rather than ambiguous terms such as funds.
BS-04
Any appropriation of retained earnings must be presented separately within the shareholders’ equity section.
505-10-45-3
Appropriation of retained earnings is permitted, provided that it is shown within the shareholders’ equity section of the balance sheet and is clearly identified as an appropriation of retained earnings.
BS-05
Treasury stock should be reported as a deduction from total shareholders’ equity.
505-10-05-1
The Equity Topic includes the following Subtopics: a. Overall b. Stock Dividends and Stock Splits c. Treasury Stock d. Subparagraph superseded by Accounting Standards Update No. 2018-07. e. Spinoffs and Reverse Spinoffs.
II. Income Statement Rules
SI-01
The results of operating activities must be reported separately from non-operating activities.
255-10-50-3
An entity shall disclose all of the following information for each of the five most recent years: a. Net sales and other operating revenues b. Income from continuing operations on a current cost basis c. ……
We introduce FinVerBench, a benchmark and validity study for financial statement verification: determining whether a set of corporate financial statements is numerically consistent from the information shown to the model. FinVerBench is built from SEC 10-K XBRL filings for 43 S&P 500 companies and defines a four-category error taxonomy covering arithmetic, cross-statement linkage, year-over-year, and magnitude perturbations. We attempt fifteen contemporary LLM evaluations and report fourteen complete runs; a Gemini 2.5 Pro run is excluded from the main comparison because 40/108 gateway calls failed. All binary metrics exclude underdetermined positive instances whose perturbed line item is not rendered, leaving a 105-instance observable diagnostic subset (43 clean, 62 error-injected). Under the original guided-checklist prompt on the unrounded diagnostic subset, nine of fourteen complete LLM runs produce 95-100% false positives on clean statements, while one run achieves 0% observed false positives. Benchmark rendering choices materially affect measured recall: on a realistic rounded variant of the same observable subset, the calibrated model's recall is 79.0% with 0% observed FPR, compared with 100.0% recall on the unrounded diagnostic variant. These results support a construct-validity conclusion rather than a final leaderboard: financial statement verification is not merely arithmetic detection, but calibrated judgment under incomplete observability, prompt-induced assumptions, and realistic numerical rendering. FinVerBench and all code are publicly available.
Do Large Language Models (LLMs) possess genuine structural reasoning, or merely rely on surface-level pattern matching? The financial domain, demanding numerical precision and multi-step logic over long contexts, is an ideal testbed. Existing benchmarks fail to capture real-world industrial complexity, predominantly relying on multiple-choice questions or single-hop QA over cropped tables while ignoring intricate cross-statement dynamics and temporal de-cumulation. To bridge this gap, we introduce FinIndices, a large-scale benchmark evaluating data-processing fidelity over uncropped financial statements (up to 32K tokens). Utilizing an automated synthesis pipeline with adversarial traps, FinIndices encompasses Single-Index computation and Table-Index tabulation to test complex domain, temporal, and caliber reasoning. Our evaluation reveals two severe LLM vulnerabilities. First, a "Knowledge Bottleneck": despite memorizing formulas during pre-training, models demonstrate fragile pattern matching. Removing explicit formula hints causes performance to collapse (e.g., Gemini-3.1-Pro drops from 70.70% to 38.22% on table tasks), exposing fatal flaws in temporal de-cumulation and stock-flow caliber mismatch. Second, a "Structural Bottleneck": the intense cognitive load of generating multi-metric, multi-period tables actively drains reasoning capacity. Under structural pressure, LLMs that flawlessly execute isolated derivations regress to shallow heuristics, such as fetching incorrect adjacent columns or substituting deep accounting adjustments with lazy literal arithmetic. Finally, Supervised Fine-Tuning (SFT) yields substantial zero-hint gains (+8.54% Single, +3.82% Table), validating that structured logic can be partially restored via data-centric alignment.
Xinke Tong, Xuanming Zhang, Tianyi Tang +10
Qwen Team, Alibaba Group · Tsinghua University · Alibaba Group
While existing benchmarks have made substantial progress in evaluating LLMs across STEM domains, financial reasoning over structured data remains comparatively less explored. We introduce V-FiLLM, a framework that generates financial reasoning benchmarks from executable computation trees grounded in real tables, yielding items whose answers are correct by construction. Trees are evaluated symbolically to obtain ground truth and rendered into natural-language questions, removing any model from the labeling loop, so items can be generated at arbitrary scale without annotation cost and without inheriting a generator's error rate. V-FiLLM exposes four independently controllable axes of difficulty including computation depth, expression breadth, financial concept complexity, and context size. By evaluating on open-source models, we find that accuracy falls up to 51% as reasoning depth increases, and up to 47% points under adversarial numerical perturbations, highlighting remaining challenges in robust financial reasoning over tables. We further show that lightweight LoRA fine-tuning on verified chain-of-thought traces improves accuracy from 81.1% to 85.6% on held-out problems and outperforms the base model by 5% points on FinQA (Chen et al., 2022a), s), suggesting that targeted, low-cost adaptation is a promising direction for compositional reasoning in financial QA.