FinGREAT: Evaluating LLMs on Financial Graph-guided Retrieval And Typing

Ashman Mehra; Bryan Routledge; Jane JaeYeon Pyo; Pierre Jinghong Liang

cs.LGOct 13, 2026

FinGREAT: Evaluating LLMs on Financial Graph-guided Retrieval And Typing

Authors: Ashman Mehra, Bryan Routledge, Jane JaeYeon Pyo, Pierre Jinghong Liang

Organizations: School of Computer Science, AI Accounting Lab, Carnegie Mellon University · Tepper School of Business, AI Accounting Lab, Carnegie Mellon University

Abstract

Financial disclosures express the underlying relationships of double-entry accounting through unstructured text. While financial LLM benchmarks predominantly emphasize question answering and classification, schema-guided evidence discovery remains relatively under-explored. We investigate whether open-weight LLMs can extract and classify evidence corresponding to predefined double-entry accounting relationships underlying the text. Using a financial accounting graph comprising 11 nodes (accounts) and 11 edges (transaction types), we evaluate mid-sized models from 5 open-weight families across 1,300 SEC 10-K Item 7 filings spanning 26 years, comparing their extraction performance against three proprietary models. Results reveal substantial differences in extraction volume, edge distributions, and cross-model agreement. Building on this study, we introduce a human-annotated relation classification benchmark comprising 483 passages across three financial accounting categories, evaluating 34 open-weight and 10 proprietary models. The strongest open-weight and proprietary models achieve 84.1% and 92.8% accuracy, respectively, with considerable variation across accounting categories. Our findings highlight the importance of evaluating evidence discovery and relation understanding separately, while investigating locally deployable models for scalable, privacy-sensitive financial analysis.

Figures & tables

Panel of Figure 1 — Figure 1: FinGREAT study design. Task I retrieves and assigns evidence to graph edges. Task II evaluates edge classification against human-reviewed answer sets.

Study	SG	ED	RM	SE	HR
FIRE ( 2024 )	✓	✗	✓	✗	✓
FinGEAR ( 2025 )	$\ast$	✓	✗	$\ast$	✗
FinMRAGBench ( 2026 )	✗	✓	✗	$\ast$	✗
FinKario ( 2026 )	✓	✓	✓	$\ast$	✗
FinTagging ( 2025b )	✓	$\ast$	✗	$\ast$	✗
FinReflectKG–EvalBench ( 2025 )	✓	✓	✓	$\ast$	$\ast$

Table 1: Financial benchmark comparison. SG: schema/structure guidance; ED: document-level evidence discovery; RM: relation mapping; SE: separate extraction/relation evaluation; HR: human-labeled relations. ✓ : explicit;

\ast

: partial; ✗ : not evaluated.

Panel of Figure 2 — Figure 2: Underlying accounting graph used in the task formulation. Cash Flow edges connect the blue nodes to net non-cash assets (yellow), Net Income edges connect the green nodes, and Equity edges involve the purple nodes. Directed edges represent bookkeeping flows. The flow identity is $\Delta\mathrm{Cash}+\Delta\mathrm{NCA}-\Delta\mathrm{CE}-\mathrm{NI}+\mathrm{Div}=0$ , where NCA denotes net non-cash assets, CE contributed equity, NI net income, and Div dividends.

Symbol	Definition
$G=(V,E)$	Accounting graph with 11 nodes, 11 edges
$e\in E$	An accounting relationship
$m\in\mathcal{M}$	An evaluated language model
$\mathcal{M}_{O},\mathcal{M}_{P}$	Open-weight and proprietary model sets
$\mathcal{Y}$	Set of years (2000–2025)
$\mathcal{C}_{y}$	Corpus of SEC 10-K Item 7 texts from year $y$

Table 2: Mathematical notation used in the task formulation and evaluation.

Reference model	Open-weight model	$P$	$O$	$K$	$A$	Retrieval Recall [ $K/P$ ]	Jaccard Similarity [ $K/(O+P-K)$ ]	Conditional Classification Precision [ $A/K$ ]
Gemini 3.1 Flash Lite	Gemma 4 31B	4,061	7,534	2,782	2,217	68.5	31.6	79.7
	Nemotron 3 Super	4,061	8,742	2,110	1,248	52.0	19.7	59.1
	Qwen 3.6 35B A3B	4,061	8,023	2,668	1,915	65.7	28.3	71.8
	Ministral 3 14B	4,061	19,983	3,170	1,674	78.1	15.2	52.8
	Phi 4	4,061	12,547	1,797	768	44.3	12.1	42.7
Claude Sonnet 5	Gemma 4 31B	14,802	7,534	5,602	4,577	37.8	33.5	81.7

Table 3: Cross-model agreement on 300 common filings (%).

P,O

: reference/open extraction counts.

K

: evidence matches.

A

: matches agreeing on the edge. Bold marks the best score per reference.

Model	Purity (%, $\uparrow$ )	$H(E\mid T)$ (bits, $\downarrow$ )	Homogeneity (%, $\uparrow$ )	Distinct tags	Used once (%)
Gemma 4 31B	81.5	0.790	74.2	165	55.2
Nemotron 3 Super	79.5	0.962	71.7	588	60.7
Qwen 3.6 35B A3B	85.7	0.775	74.2	92	41.3
Ministral 3 14B	63.5	1.665	50.3	2,213	46.6
Phi 4	65.8	1.376	59.8	3,897	54.3
[1pt/2pt] Gemini 3.1 Flash Lite	74.6	0.990	68.5	28	10.7

Table 4: Tag–edge associations. Arrows indicate preferred directions. Bold marks the best open-weight scores at displayed precision. Above/below the dotted line: 1,300/300 filings.

Panel of Table 5 — Table 5: Human-reviewed edge accuracy (%) on 483 passages. $n$ counts accepted-set membership. Category Overall uses the union of its edge cohorts. OCI: other comprehensive income. Medal colors rank category/overall scores separately within open-weight and proprietary groups. Black borders identify open-weight highlights.

Panel of Figure 3 — Figure 3: Annual percentage share of each model’s extractions across the three edge categories, using 50 filings per year. Panel y-axis ranges differ.

Panel of Figure 4 — Figure 4: Overall accuracy versus first-token latency (top) and input price (bottom), for models with available Artificial Analysis data as of 10 October 2026. Colors identify providers. Horizontal positions use $\log_{10}t$ for latency and $\log_{10}(1+c/0.01)$ for cost, with $t$ in seconds and $c$ in USD per million input tokens.

Panel of Figure 5 — Figure 5: Best overall classification accuracy within each model family (values from Table 5 ). Dashed lines show the strongest proprietary and open-weight models.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Edge	Relation	Financial Interpretation
E1	Financing–Net Non-cash Assets	Financing flows associated with changes in non-cash assets and liabilities, such as borrowing and debt repayment.
E2	Investment–Net Non-cash Assets	Investing flows associated with acquiring or disposing of non-cash assets.
E3	Operations–Net Non-cash Assets	Operating cash flows and their connection to working capital and non-cash adjustments.
E4	Revenue–Net Non-cash Assets	Revenue recognition and related changes in non-cash assets or liabilities.
E5	Expenses–Net Non-cash Assets	Expense recognition and related changes in non-cash assets or liabilities.
E6	Interest Expenses–Net Non-cash Assets	Interest expense and associated accruals or other non-cash balance changes.

Appendix

Table 6: Definitions of the 11 accounting edges. OCI denotes other comprehensive income.

Panel of Figure 6 — Figure 6: Apple’s Form 10-K contents and opening Item 7 page.

Industry division	Registrants	Filings
Manufacturing	463	486
Finance, insurance and real estate	253	269
Services	221	233
Transportation, communications and utilities	116	123
Retail trade	67	68
Mining	58	60

Appendix

Table 7: Full-corpus industry composition using current SEC SIC metadata. Registrants are distinct CIKs; filings are distinct CIK–accession pairs.

Total assets (USD)	Filings	Share (%)
Below $100 million	189	29.2
$100 million–below$ 1 billion	187	28.9
$1 billion–below$ 10 billion	192	29.7
At least $10 billion	79	12.2
Matched total	647	100.0

Appendix

Table 8: Asset-size distribution for 2012–2025. Percentages use the 647 matched filings; the 53 unknown values remain in coverage counts. These are not market-capitalization classes.

Year	Selected	Matched	Unknown
2012	50	42	8
2013	50	45	5
2014	50	47	3
2015	50	49	1
2016	50	47	3
2017	50	44	6

Appendix

Table 9: SEC API asset coverage by dataset year. The asset date is the balance-sheet date in the selected filing, not necessarily calendar year-end.

Statistic	Input characters
Minimum	2,543
25th percentile	36,616.5
Median	57,395.5
75th percentile	78,265
95th percentile	136,180.1
Maximum	501,163

Appendix

Table 10: Processed Item 7 input lengths across all 1,300 filings. Percentiles are computed over filings, not over model outputs.

Figure 7: Task I extraction viewer: source evidence, assigned relation, tags, and attributes.

Model	Temperature	Top-p	Output cap
Gemma 4 31B	0.7	0.9	16384
Nemotron 3 Super	1.0	0.95	16384
Qwen 3.6 35B A3B	0.7	0.9	9216
Ministral 3 14B	0.15	0.95	8192
Phi 4	0.15	0.95	5120
Gemini 3.1 Flash Lite	0.7	0.9	16384

Appendix

Table 11: Representative archived Task I configurations. Caps are generation tokens. “–” denotes a sampling field omitted by the provider request. Configuration values are not asserted to be uniform across every run.

Panel of Figure 8 — Figure 8: Human review interface: entry screen and passage-level relationship selection.

Reviewer pair	Exact sets (%)	Jaccard (%)
R1–R2	70.0	75.9
R1–R3	72.2	78.8
R2–R3	83.6	87.1

Appendix

Table 12: Inter-reviewer agreement on all 500 candidate passages.

Model	Accuracy (%)	95% CI
Claude-fable-5-1	92.8	90.2–94.9
Claude-haiku-4-5	84.5	81.1–87.8
Claude-opus-5-5	92.1	89.6–94.5
Claude-sonnet-5-5	91.3	88.7–93.8
Gemini-3.1-pro-preview	89.6	86.7–92.2
Gemini-3.5-flash-lite	82.4	78.9–85.7

Appendix

Table 13: Overall classification accuracy and source-filing cluster-bootstrap intervals on the common human-golden benchmark.

Edge	Passages	Majority (%)	Random (%)
E1	82	39.0	15.6
E2	70	2.9	12.7
E3	66	9.1	13.6
E4	11	0.0	14.9
E5	72	1.4	10.6
E6	37	0.0	10.6

Appendix

Table 14: Edge cohorts and exact baseline accuracies. Cohorts may overlap.

Gemini 3.1 Flash Lite
Model	CF Recall	CF CCP	NI Recall	NI CCP	EQ Recall	EQ CCP
Gemma 4 31B	66.5	75.2	66.2	85.7	77.7	69.4
Nemotron 3 Super	58.4	72.3	42.9	58.0	70.4	47.7
Qwen 3.6 35B A3B	62.5	74.1	62.8	76.5	78.2	58.4
Ministral 3 14B	80.5	67.1	74.3	51.2	85.9	40.3
Phi 4	44.7	62.2	38.2	33.4	61.4	42.1

Appendix

Table 15: Reference-relative retrieval recall and conditional classification precision (CCP), in percent. CF, NI, and EQ denote Cash Flow, Net Income, and Equity edges.

Model	Gemini	Claude	GPT
Gemma 4 31B	85.7	90.8	88.4
Nemotron 3 Super	72.5	78.6	75.7
Qwen 3.6 35B A3B	77.5	81.6	79.9
Ministral 3 14B	65.7	72.4	73.4
Phi 4	55.2	64.5	63.6
Majority ensemble	79.9	87.3	84.8

Appendix

Table 16: Edge agreement (%) on the same majority-decidable evidence: 364, 543, and 448 passages for the three references, respectively.

Model	Once	Twice	3–5	6–10	11–50	$>50$
Gemma 4 31B	91	15	11	11	12	25
Nemotron 3 Super	357	71	72	26	31	31
Qwen 3.6 35B A3B	38	12	7	2	9	24
Ministral 3 14B	1031	367	356	152	209	98
Phi 4	2116	707	572	219	206	77
Gemini 3.1 Flash Lite	3	0	2	3	6	14

Appendix

Table 17: Distinct model-supplied tags by their number of extraction occurrences. Open-weight models cover 1,300 filings, proprietary references 300.

Model	$N$	Changed	Change (%)	Task I	Task II
Gemma 4 31B	483	105	21.7	77.8	83.0
Qwen 3.6 35B A3B	483	173	35.8	69.6	77.0
Ministral-3-14B-Instruct	483	286	59.2	53.4	80.1
Phi 4	483	286	59.2	44.9	70.2

Appendix

Table 18: Edge-label changes and accepted-set accuracy (%) on the same 483 passages.

N

is the number compared. Task I and Task II accuracy columns are percentages.

Panel of Figure 9 — Figure 9: Annual allocation across all 11 edges. Each share uses the corresponding model and year’s total extractions.

Panel of Figure 10 — Figure 10: Vocabulary Jaccard (%) on 300 common filings. Darker teal indicates greater overlap between mechanically normalized tag sets. Vocabulary overlap provides a complementary cross-model view: Qwen–Gemini and Gemma–Gemini are the strongest open/proprietary pairs. The latter also shares Google’s model lineage, but shared training data cannot be inferred from lexical overlap alone.

Panel of Figure 11 — Figure 11: Failure-set Jaccard (%) for family-leading classifiers on the common human-golden cohort. Darker teal indicates more shared errors. Legend entries give each model’s error count.

Panel of Figure 12 — Figure 12: Best observed model in each family for Cash Flow, Net Income, and Equity edges. Bars show accepted-set accuracy. Dashed lines identify the best proprietary and open-weight models ( $<36$ B parameters).

Panel of Figure 13 — Figure 13: Accuracy versus first-token latency for every model with available Artificial Analysis data on 12 October 2026. Colors identify providers, markers identify models. The horizontal axis uses $\log_{10}t$ , with $t$ in seconds.

Panel of Figure 14 — Figure 14: Accuracy versus input-token price for every model with available Artificial Analysis data on 12 October 2026. Colors identify providers, markers identify models. Horizontal positions use $\log_{10}(1+c/0.01)$ , with $c$ in USD per million input tokens.

Explore similar work

Can Open-Weight Models Compete on Financial Text Comprehension?
Aug 9, 2026Jan SpörerLLM Evaluation Financial QA
University of St. Gallen, Switzerland
FinRCA-Bench: Benchmarking Evidence Retrieval and Reasoning for Financial AI Systems
Aug 19, 2026Pratik GhawateEvidence Retrieval Root Cause Analysis
Researcher
FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceability
Sep 21, 2026Wenqing Wang, Haitao Xiang, Xinyi Zhao +8Financial QA AI Agent Benchmarks
Ling Team, Inclusion AI · Peking University · China International Capital Corporation Limited

Aug 9, 2026J/K move · Enter open · S save

Can Open-Weight Models Compete on Financial Text Comprehension?

Jan Spörer

University of St. Gallen, Switzerland

LLM Evaluation Financial QA Open-Weight Language Models LLM Reliability

Open paper AskPDF arXiv

Open-weight language models from Chinese AI labs caught up on benchmarks relative to proprietary frontier models in recent months. Yet their reliability on real-world financial tasks remains largely untested. We updated the Financial Touchstone benchmark, which now has 2,967 question context-answer triplets across 495 international annual reports. We also apply a new set of models on the benchmark, expanding coverage from eleven to twenty models across ten providers, including recent open-weight models such as GLM 4.7, GLM 5, Kimi K2.6, and DeepSeek V3.2, as well as Alibaba's proprietary flagship Qwen3-Max. Anthropic's Claude Opus 4.6 achieves the highest accuracy (88.4%), while Google's Gemini 2.5 Pro maintains the lowest hallucination rate (0.08%). Notably, the open-weight Kimi K2.6 ranks third in accuracy, and the non-reasoning models GLM 5 and Mistral 3 rank fourth and fifth, challenging the assumption that reasoning architectures or proprietary weights are a prerequisite for strong financial comprehension. Information retrieval remains the primary bottleneck, accounting for 48.9% of all failures. We also document a new finding: geopolitical content filters in Chinese models refuse legitimate financial questions (0.08% of attempts), sometimes without clear reason, and the refusal behavior depends on the access route as much as on the model. The complete dataset and evaluation framework are publicly available.

Abstract

Explore similar work

Can Open-Weight Models Compete on Financial Text Comprehension?

FinRCA-Bench: Benchmarking Evidence Retrieval and Reasoning for Financial AI Systems

FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceability