Where A Small Language Model Helps in Invoice Categorisation, Understood Through Embedding Geometry

Emma Ceccherini; Daniel Lawson; Anjulika Salhan

stat.MLAug 18, 2026

Where A Small Language Model Helps in Invoice Categorisation, Understood Through Embedding Geometry

Authors: Emma Ceccherini, Daniel Lawson, Anjulika Salhan

Organizations: University of Bristol, U.K. · System Holdings Limited

Abstract

Categorising invoices into the correct General Ledger (GL) code underpins financial reporting and tax compliance. This is a skilled accounting judgement rather than a routine task: the correct category depends subtly on the nature of the purchasing business, the vendor and the invoice text. Whilst AI is increasingly being adopted across industries to automate tasks, including invoice categorisation, implementations built on in-house small language models (SLMs) can simultaneously reduce cost and improve data security, confidentiality, and interpretability. We investigate this approach by first analysing the pre-trained embedding geometry of a small sentence transformer (SBERT) and classic SLM (DeBERTa). The sentence-embedding space of this financial corpus is globally anisotropic but composed of locally isotropic clusters, extending prior token-level findings to sentence embeddings in a financial setting, and these clusters are strongly correlated with the vendor identity. SBERT fine-tuned on a single GPU reaches 0.96 accuracy on invoice classification, above both a zero-shot LLM and a vendor identity baseline, increasing performance for smaller, challenging categories and new clients. For this important generalisation problem, SBERT reaches 0.9 F1 with roughly 100 client-specific invoices, showing that an in-house SLM implementation is promising. Combining these results with geometric analysis shows that pre-trained embedding geometry is associated with classification performance and reveals a counterintuitive finding that a structured input that would help a human reader does not improve the SLM performance.

Figures & tables

Panel of Figure 1 — Figure 1: Fictitious example of the three input variants.

Figure 2: Frozen transformer visualisation of the first three principal components of the representation space of pre-trained SBERT with input B. Note the overlap between Purchases (green) and Directors Loan Account (light purple).

Model	A	B	C	$d_{model}$
SBERT	42	61	57	384
DeBERTa mean	13	7	6	768
DeBERTa CLS	10	1	1	768

Table 1: Effective dimensionality (80% variance explained) by model and input type.

Model	A	B	C
SBERT	0.365	0.282	0.496
DeBERTa mean	0.973	0.933	0.949
DeBERTa cls	0.999	0.992	0.993

Table 2: Frozen transformers average cosine similarity between any embedding pairs by model and input type.

Model	A	B	C
SBERT	54	59	60
DeBERTa mean	52	2	2
DeBERTa CLS	52	2	2

Table 3: Frozen transformers number of

K

-means clusters

|G|

selected by highest silhouette score, by model and input type.

		True labels				K-means clustering
		Uncentered		Centred		Uncentered		Centred
Type	Model	Within	Between	Within	Between	Within	Between	Within	Between
B	SBERT	0.478	0.249	0.003	0.000	0.708	0.265	-0.015	0.000
B	DeBERTa mean	0.938	0.930	0.013	0.009	0.942	0.841	0.002	0.001
C	SBERT	0.641	0.485	0.003	0.000	0.823	0.476	-0.031	0.000
C	DeBERTa mean	0.956	0.948	0.019	0.011	0.953	0.864	0.000	0.001

Table 4: Frozen transformer average within-class/cluster and between-class/cluster cosine similarity, using the true labels and the K-means clustering partition.

	DeBERTa CLS			DeBERTa mean			SBERT			Vendor	LLM
	A	B	C	A	B	C	A	B	C	baseline	baseline
Acc.	$0.39_{\pm 0.02}$	$0.62_{\pm 0.02}$	$0.45_{\pm 0.04}$	$0.51_{\pm 0.02}$	$0.63_{\pm 0.02}$	$0.62_{\pm 0.02}$	$0.70_{\pm 0.04}$	$\underline{0.87}_{\pm 0.01}$	$0.83_{\pm 0.02}$	$\textbf{0.89}_{\pm 0.01}$	$0.52_{\pm 0.02}$
M F1	$0.18_{\pm 0.08}$	$0.55_{\pm 0.02}$	$0.33_{\pm 0.06}$	$0.39_{\pm 0.03}$	$0.53_{\pm 0.04}$	$0.49_{\pm 0.02}$	$0.67_{\pm 0.06}$	$\underline{0.80}_{\pm 0.01}$	$0.77_{\pm 0.02}$	$\textbf{0.87}_{\pm 0.01}$	$0.39_{\pm 0.02}$
W F1	$0.31_{\pm 0.05}$	$0.62_{\pm 0.02}$	$0.44_{\pm 0.04}$	$0.49_{\pm 0.01}$	$0.63_{\pm 0.01}$	$0.62_{\pm 0.02}$	$0.69_{\pm 0.04}$	$\underline{0.85}_{\pm 0.01}$	$0.82_{\pm 0.02}$	$\textbf{0.89}_{\pm 0.01}$	$0.52_{\pm 0.02}$

Table 5: Frozen Transformers accuracy, macro F1 and weighted F1 for classification with frozen transformers for each model and input type. The LLM baseline is zero-shot, and the transformers use a trained linear head. We report the mean

\pm

s.d. over 5 splits. The best performance is in bold , the second best is underlined .

Model	Acc.	M F1	W F1
SBERT 2l	$0.945_{\pm 0.00}$	$0.915_{\pm 0.01}$	$0.944_{\pm 0.00}$
SBERT 4l	$0.950_{\pm 0.01}$	$0.924_{\pm 0.02}$	$0.950_{\pm 0.01}$
SBERT 6l	$0.950_{\pm 0.01}$	${0.926}_{\pm 0.01}$	$0.949_{\pm 0.01}$
SBERT 2l b.n.	$\mathbf{0.961}_{\pm 0.00}$	$\mathbf{0.939}_{\pm 0.01}$	$0.960_{\pm 0.00}$
SBERT 4l b.n.	$\mathbf{0.961}_{\pm 0.00}$	${0.938}_{\pm 0.02}$	$\mathbf{0.961}_{\pm 0.01}$
SBERT 6l b.n.	$\mathbf{0.961}_{\pm 0.01}$	$0.936_{\pm 0.02}$	$0.960_{\pm 0.01}$

Table 6: Post fine-tuning accuracy, macro F1 and weighted F1 across fine-tuning configurations and number of unfrozen final layers (l) for input type B with/without business-nature prefix (+b.n.) reporting mean

\pm

s.d. over 5 splits. The best performance is in bold .

Figure 3: Post fine-tuning visualisation of the first three principal components of the representation space of SBERT with input B. Compared to Figure 2 , note that classes appear more distributed over the sphere, with Purchases (green) in the same direction but separated from Directors Loan Accounts (light purple).

Figure 4: Left-out client experiment , Macro F1 for fine-tuned SBERT and DeBERTa as a function of the proportion of ‘new client’ data injected into the training set (solid line). The baselines (dashed lines) are from Section 4.2 .

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1: Distribution of invoices across the 42 accounting categories in the full dataset; the categories before the dashed line are included in the analysis.

Figure A2: Visualisation of the first three principal components of the representation space of pre-trained SBERT with input B. Points coloured by Vendor name for the 20 most popular vendors, grey otherwise. Legend removed for data privacy.

Panel of Figure 13 — Figure A3: Within and between cosine similarity using SBERT Frozen transformer on input B, showing all-class average, Purchases, Accountancy Fee and Light, Heat & Power. With the exception of Accountancy Fee, they all overlap showing that the direction of the angular variation in embedding space is important for successful classification.

Figure A4: Confusion matrix for SBERT fine-tuned with 2 layers and business name/nature prefix.

Account Category	Definition
Purchases	Cost of goods, raw materials or components bought for resale or for consumption in production during the period. A profit-and-loss item forming part of cost of sales; it records the buying flow, not stock held. Excludes capital assets and overhead services.
Directors’ Loan Account	Related-party control account for amounts a director personally lends to, or withdraws from, the company other than salary, dividend or reimbursed expenses. May be in credit (company owes the director — a liability) or overdrawn (director owes the company — an asset); overdrawn balances attract a s455 CTA 2010 charge and beneficial-loan implications.
Petrol and Oil	Fuel and lubricant costs incurred in operating business motor vehicles.
Sub	Abbreviation for Subcontractor. Payments for subcontractor or subcontract labour directly attributable to revenue-generating work.
Motor Repairs and Servicing	Maintenance, servicing and repair of company vehicles.
Subscriptions	Recurring professional-body memberships, journals and software or service subscriptions of an administrative nature.

Appendix

Table A1: Definitions for accounting categories used in the dataset. There is no standardised UK nominal ledger and nominal ledger code numbers are a convention of the different accounting software. The UK firm that provided our data uses a custom accounting categorisation and proprietary software. When an invoice contains line items spanning multiple categories, the invoice is assigned to the category with the largest total value, as is standard practice in this UK firm.

Type	Model	Within	Between
A	SBERT	0.503	0.348
A	DeBERTa CLS	0.999	0.999
A	DeBERTa mean	0.977	0.972
B	SBERT	0.478	0.249
B	DeBERTa CLS	0.995	0.993
B	DeBERTa mean	0.938	0.930

Appendix

Table A2: Average within-class and between-class cosine similarity, using the true labels.

Type	Model	Within	Between
A	SBERT	0.820	0.341
A	DeBERTa CLS	0.999	0.999
A	DeBERTa mean	0.991	0.970
B	SBERT	0.708	0.265
B	DeBERTa CLS	0.997	0.981
B	DeBERTa mean	0.942	0.841

Appendix

Table A3: Average within-cluster and between-cluster cosine similarity.

	Input A			Input B			Input C
Class	P	R	F1	P	R	F1	P	R	F1
Accountancy Fee	$0.91_{\pm 0.02}$	$0.99_{\pm 0.02}$	$0.95_{\pm 0.01}$	$0.79_{\pm 0.06}$	$0.99_{\pm 0.02}$	$0.88_{\pm 0.04}$	$0.64_{\pm 0.06}$	$0.99_{\pm 0.02}$	$0.78_{\pm 0.05}$
Cleaning of Premises	$0.74_{\pm 0.09}$	$0.55_{\pm 0.14}$	$0.63_{\pm 0.13}$	$0.92_{\pm 0.07}$	$0.65_{\pm 0.08}$	$0.76_{\pm 0.08}$	$0.93_{\pm 0.10}$	$0.58_{\pm 0.15}$	$0.71_{\pm 0.14}$
Directors’ Loan Account	$0.48_{\pm 0.38}$	$0.11_{\pm 0.08}$	$0.17_{\pm 0.13}$	$0.53_{\pm 0.36}$	$0.07_{\pm 0.05}$	$0.12_{\pm 0.08}$	$0.50_{\pm 0.50}$	$0.00_{\pm 0.06}$	$0.10_{\pm 0.10}$
Light, Heat & Power	$0.86_{\pm 0.13}$	$0.71_{\pm 0.24}$	$0.76_{\pm 0.20}$	$0.94_{\pm 0.07}$	$0.96_{\pm 0.05}$	$0.95_{\pm 0.02}$	$0.95_{\pm 0.07}$	$0.95_{\pm 0.08}$	$0.95_{\pm 0.04}$
Motor Repairs and Servicing	$0.77_{\pm 0.04}$	$0.87_{\pm 0.04}$	$0.82_{\pm 0.04}$	$0.90_{\pm 0.06}$	$0.91_{\pm 0.03}$	$0.90_{\pm 0.03}$	$0.92_{\pm 0.02}$	$0.87_{\pm 0.02}$	$0.90_{\pm 0.02}$

Appendix

Table A4: Classification Report (precision, recall, F1) - SBERT. We report the mean

\pm

s.d. over 5 splits.

	Input A			Input B			Input C
Class	P	R	F1	P	R	F1	P	R	F1
Accountancy Fee	$0.15_{\pm 0.15}$	$0.59_{\pm 0.54}$	$0.23_{\pm 0.23}$	$0.81_{\pm 0.08}$	$0.97_{\pm 0.02}$	$0.88_{\pm 0.05}$	$0.61_{\pm 0.24}$	$0.83_{\pm 0.04}$	$0.68_{\pm 0.17}$
Cleaning of Premises	$0.00_{\pm 0.00}$	$0.00_{\pm 0.00}$	$0.00_{\pm 0.00}$	$0.56_{\pm 0.27}$	$0.80_{\pm 0.04}$	$0.62_{\pm 0.17}$	$0.31_{\pm 0.19}$	$0.71_{\pm 0.22}$	$0.41_{\pm 0.19}$
Directors’ Loan Account	$0.01_{\pm 0.02}$	$0.03_{\pm 0.06}$	$0.01_{\pm 0.03}$	$0.15_{\pm 0.29}$	$0.04_{\pm 0.06}$	$0.06_{\pm 0.10}$	$0.20_{\pm 0.45}$	$0.01_{\pm 0.03}$	$0.03_{\pm 0.06}$
Light, Heat & Power	$0.22_{\pm 0.44}$	$0.16_{\pm 0.28}$	$0.10_{\pm 0.14}$	$0.57_{\pm 0.21}$	$0.65_{\pm 0.10}$	$0.60_{\pm 0.15}$	$0.10_{\pm 0.19}$	$0.20_{\pm 0.28}$	$0.13_{\pm 0.21}$
Motor Repairs and Servicing	$0.52_{\pm 0.09}$	$0.65_{\pm 0.14}$	$0.56_{\pm 0.05}$	$0.78_{\pm 0.05}$	$0.61_{\pm 0.04}$	$0.68_{\pm 0.03}$	$0.51_{\pm 0.04}$	$0.55_{\pm 0.06}$	$0.53_{\pm 0.03}$

Appendix

Table A5: Classification Report (precision, recall, F1) - DeBERTa CLS. We report the mean

\pm

s.d. over 5 splits.

	Input A			Input B			Input C
Class	P	R	F1	P	R	F1	P	R	F1
Accountancy Fee	$0.46_{\pm 0.07}$	$1.00_{\pm 0.00}$	$0.63_{\pm 0.07}$	$0.34_{\pm 0.05}$	$0.99_{\pm 0.02}$	$0.51_{\pm 0.05}$	$0.37_{\pm 0.06}$	$0.92_{\pm 0.10}$	$0.52_{\pm 0.07}$
Cleaning of Premises	$0.00_{\pm 0.00}$	$0.00_{\pm 0.00}$	$0.00_{\pm 0.00}$	$0.74_{\pm 0.06}$	$0.36_{\pm 0.09}$	$0.48_{\pm 0.09}$	$0.85_{\pm 0.22}$	$0.29_{\pm 0.08}$	$0.43_{\pm 0.11}$
Directors’ Loan Account	$0.22_{\pm 0.22}$	$0.04_{\pm 0.04}$	$0.07_{\pm 0.06}$	$0.23_{\pm 0.20}$	$0.08_{\pm 0.06}$	$0.11_{\pm 0.08}$	$0.07_{\pm 0.09}$	$0.03_{\pm 0.04}$	$0.04_{\pm 0.05}$
Light, Heat & Power	$0.11_{\pm 0.06}$	$0.18_{\pm 0.11}$	$0.12_{\pm 0.05}$	$0.55_{\pm 0.17}$	$0.47_{\pm 0.24}$	$0.50_{\pm 0.21}$	$0.43_{\pm 0.08}$	$0.58_{\pm 0.22}$	$0.49_{\pm 0.13}$
Motor Repairs and Servicing	$0.62_{\pm 0.07}$	$0.72_{\pm 0.08}$	$0.66_{\pm 0.03}$	$0.73_{\pm 0.02}$	$0.69_{\pm 0.07}$	$0.71_{\pm 0.04}$	$0.69_{\pm 0.05}$	$0.68_{\pm 0.07}$	$0.68_{\pm 0.03}$

Appendix

Table A6: Classification Report (precision, recall, F1) - DeBERTa mean. We report the mean

\pm

s.d. over 5 splits.

	Precision	Recall	F1-score
Accountancy Fee	$0.22_{\pm 0.05}$	$0.23_{\pm 0.05}$	$0.23_{\pm 0.05}$
Cleaning of Premises	$0.68_{\pm 0.22}$	$0.33_{\pm 0.08}$	$0.43_{\pm 0.10}$
Directors’ Loan Account	$0.00_{\pm 0.00}$	$0.00_{\pm 0.00}$	$0.00_{\pm 0.00}$
Light, Heat & Power	$0.29_{\pm 0.02}$	$1.00_{\pm 0.00}$	$0.45_{\pm 0.03}$
Motor Repairs and Servicing	$0.58_{\pm 0.08}$	$0.36_{\pm 0.05}$	$0.44_{\pm 0.06}$
Petrol and Oil	$0.93_{\pm 0.04}$	$0.62_{\pm 0.05}$	$0.75_{\pm 0.04}$

Appendix

Table A7: Classification report for the LLM baseline, mean

\pm

s.d. over 5 splits.

	Precision	Recall	F1-score
Accountancy Fee	$1.00_{\pm 0.00}$	$0.99_{\pm 0.02}$	$1.00_{\pm 0.01}$
Cleaning of Premises	$0.88_{\pm 0.16}$	$0.75_{\pm 0.08}$	$0.80_{\pm 0.09}$
Directors’ Loan Account	$0.96_{\pm 0.09}$	$0.29_{\pm 0.12}$	$0.44_{\pm 0.15}$
Light, Heat & Power	$0.95_{\pm 0.05}$	$0.98_{\pm 0.04}$	$0.96_{\pm 0.02}$
Motor Repairs and Servicing	$0.99_{\pm 0.01}$	$0.92_{\pm 0.02}$	$0.95_{\pm 0.01}$
Petrol and Oil	$0.77_{\pm 0.04}$	$0.88_{\pm 0.03}$	$0.82_{\pm 0.03}$

Appendix

Table A8: Classification report for the vendor baseline. We report the mean

\pm

s.d. over 5 splits.

Model	Acc.	M F1	W F1
SBERT - 2 l	0.880	0.845	0.882
SBERT - 4 l	0.876	0.845	0.874
SBERT - 6 l	0.917	0.875	0.914
SBERT - 2 l + b.n.	0.903	0.839	0.904
SBERT - 4 l + b.n.	0.931	0.874	0.929
SBERT - 6 l + b.n.	0.929	0.873	0.927

Appendix

Table A9: Accuracy, macro F1 and weighted F1 across fine-tuning configurations and number of unfrozen final layers (l) for input type A with and without business- nature prefix (+b.n.), for a single split. The best performance is in bold .

Model	Acc.	M F1	W F1
SBERT - 2 l	0.942	0.892	0.939
SBERT - 4 l	0.945	0.894	0.938
SBERT - 6 l	0.938	0.898	0.934
SBERT - 2 l + b.n.	0.940	0.895	0.937
SBERT - 4 l + b.n.	0.959	0.923	0.956
SBERT - 6 l + b.n.	0.959	0.931	0.956

Appendix

Table A10: Accuracy, macro F1 and weighted F1 across fine-tuning configurations and number of unfrozen final layers (l) for input type C with and without business- nature prefix (+b.n.), for a single split. The best performance is in bold .

	Precision	Recall	F1
Accountancy Fee	$1.00_{\pm 0.00}$	$0.99_{\pm 0.02}$	$1.00_{\pm 0.01}$
Cleaning of Premises	$0.83_{\pm 0.09}$	$0.91_{\pm 0.11}$	$0.86_{\pm 0.08}$
Directors’ Loan Account	$0.86_{\pm 0.05}$	$0.61_{\pm 0.12}$	$0.71_{\pm 0.07}$
Light, Heat & Power	$0.92_{\pm 0.09}$	$0.98_{\pm 0.04}$	$0.95_{\pm 0.04}$
Motor Repairs and Servicing	$0.97_{\pm 0.02}$	$0.96_{\pm 0.01}$	$0.97_{\pm 0.01}$
Petrol and Oil	$0.96_{\pm 0.02}$	$0.97_{\pm 0.02}$	$0.96_{\pm 0.01}$

Appendix

Table A11: Classification Report (precision, recall, F1) - SBERT fine-tuned with 2 unfrozen layers + business name prefix. We report the mean

\pm

s.d. over 5 splits.

	Precision	Recall	F1
Accountancy Fee	$1.00_{\pm 0.00}$	$0.99_{\pm 0.02}$	$1.00_{\pm 0.01}$
Cleaning of Premises	$0.83_{\pm 0.13}$	$0.91_{\pm 0.11}$	$0.86_{\pm 0.08}$
Directors’ Loan Account	$0.71_{\pm 0.11}$	$0.44_{\pm 0.14}$	$0.53_{\pm 0.08}$
Light, Heat & Power	$0.94_{\pm 0.07}$	$1.00_{\pm 0.00}$	$0.97_{\pm 0.04}$
Motor Repairs and Servicing	$0.99_{\pm 0.01}$	$0.95_{\pm 0.00}$	$0.97_{\pm 0.01}$
Petrol and Oil	$0.96_{\pm 0.01}$	$0.97_{\pm 0.02}$	$0.97_{\pm 0.01}$

Appendix

Table A12: Classification Report (precision, recall, F1) - SBERT fine-tuned with 6 unfrozen layers. We report the mean

\pm

s.d. over 5 splits.

	Precision	Recall	F1
Accountancy Fee	$0.98_{\pm 0.04}$	$0.99_{\pm 0.02}$	$0.99_{\pm 0.02}$
Cleaning of Premises	$0.98_{\pm 0.04}$	$0.99_{\pm 0.02}$	$0.99_{\pm 0.02}$
Directors’ Loan Account	$0.77_{\pm 0.06}$	$0.64_{\pm 0.2}$	$0.68_{\pm 0.09}$
Light, Heat & Power	$1.00_{\pm 0.00}$	$0.98_{\pm 0.04}$	$0.99_{\pm 0.02}$
Motor Repairs and Servicing	$0.98_{\pm 0.01}$	$0.95_{\pm 0.02}$	$0.97_{\pm 0.01}$
Petrol and Oil	$0.98_{\pm 0.03}$	$0.97_{\pm 0.02}$	$0.98_{\pm 0.02}$

Appendix

Table A13: Classification Report (precision, recall, F1) - DeBERTa fine-tuned with 2 unfrozen layers + business name prefix. We report the mean

\pm

s.d. over 5 splits.

Small Language Models

Small language models are compact neural language systems designed to deliver useful text and reasoning capabilities with limited compute, memory, and energy.

See all 277 papers

Apr 2, 2026

Analysis of LLM Performance on AWS Bedrock: Receipt-item Categorisation Case Study

This paper presents a systematic, cost-aware evaluation of large language models (LLMs) for receipt-item categorisation within a production-oriented classification framework. We compare four instruction-tuned models available through AWS Bedrock: Claude 3.7 Sonnet, Claude 4 Sonnet, Mixtral 8x7B Instruct, and Mistral 7B Instruct. The aim of the study was (1) to assess performance across accuracy, response stability, and token-level cost, and (2) to investigate what prompting methods, zero-shot or few-shot, are especially appropriate both in terms of accuracy and in terms of incurred costs. Results of our experiments demonstrated that Claude 3.7 Sonnet achieves the most favourable balance between classification accuracy and cost efficiency.

Gabby Sanchez, Sneha Oommen, Cassandra T. Britto +3

RMIT University, Melbourne, Australia

LLM Evaluation Classification LLM Prompting LLM Inference Efficiency Few-Shot Prompting

Ask PDF

Oct 7, 2026

Using Small Language Models to Reverse-Engineer Machine Learning Pipelines Structures

Context: Once defined a taxonomy of stages structuring Machine Learning (ML) pipelines (e.g. Data Preprocessing, Modeling...), extracting these stages from source code is key for better understanding ML practices. However, the diversity caused by the constant evolution of ML (e.g., algorithms, datasets) makes this task challenging. Existing approaches either rely on non-scalable manual labeling or on classifiers that do not properly support domain's diversity. These limitations call for more reliable solutions. Objective: We evaluate whether Small Language Models (SLMs) can leverage their code understanding and classification abilities to address these limitations, and enhance our understanding of practices in ML. Method: We conduct a confirmatory study based on two relevant reference works representing current limitations in the state-of-the-art. We first compare several SLMs using Cochran's Q test, then evaluate the best-performing model against reference studies via two McNemar's tests. An additional Cochran's Q test examines how taxonomy definition variations affect the SLM performance. Finally, goodness-of-fit tests compare ML practice insights from SLM classification with those from prior studies. Results: First, we found that the taxonomy wording significantly impacts classification performance. Second, the best performing SLM yielded good results, yet, without outperforming other classifiers. Third, the three classification methods led to significantly different insights, with varying effect sizes, when exploring practices of data scientists. Conclusions: Limitations of existing classification methods bias our understanding of ML practices. While current SLMs show promising results without prior fine-tuning, they still exhibit common limitations, in addition to inference high costs challenging their applicability in large-scale studies.

Nicolas Lacroix, Frederic Precioso, Mireille Blay-Fornarino +1

Université Côte d’Azur, Inria, CNRS, I3S, Sophia Antipolis, France · McSCert, McMaster University, Hamilton, ON, Canada

Small Language Models Software Engineering

Ask PDF

Oct 13, 2026

FinGREAT: Evaluating LLMs on Financial Graph-guided Retrieval And Typing

Financial disclosures express the underlying relationships of double-entry accounting through unstructured text. While financial LLM benchmarks predominantly emphasize question answering and classification, schema-guided evidence discovery remains relatively under-explored. We investigate whether open-weight LLMs can extract and classify evidence corresponding to predefined double-entry accounting relationships underlying the text. Using a financial accounting graph comprising 11 nodes (accounts) and 11 edges (transaction types), we evaluate mid-sized models from 5 open-weight families across 1,300 SEC 10-K Item 7 filings spanning 26 years, comparing their extraction performance against three proprietary models. Results reveal substantial differences in extraction volume, edge distributions, and cross-model agreement. Building on this study, we introduce a human-annotated relation classification benchmark comprising 483 passages across three financial accounting categories, evaluating 34 open-weight and 10 proprietary models. The strongest open-weight and proprietary models achieve 84.1% and 92.8% accuracy, respectively, with considerable variation across accounting categories. Our findings highlight the importance of evaluating evidence discovery and relation understanding separately, while investigating locally deployable models for scalable, privacy-sensitive financial analysis.

Ashman Mehra, Bryan Routledge, Jane JaeYeon Pyo +1

School of Computer Science, AI Accounting Lab, Carnegie Mellon University · Tepper School of Business, AI Accounting Lab, Carnegie Mellon University

Relation Extraction LLM Evaluation Evidence Retrieval Document Information Extraction Financial Services

Ask PDF

Abstract

Small Language Models

Explore similar work

Analysis of LLM Performance on AWS Bedrock: Receipt-item Categorisation Case Study

Using Small Language Models to Reverse-Engineer Machine Learning Pipelines Structures

FinGREAT: Evaluating LLMs on Financial Graph-guided Retrieval And Typing