Verifying corporate financial disclosures against the International Financial Reporting Standards (IFRS) is a core auditing task in which experts cross-reference disclosures with hundreds of fine-grained regulatory paragraphs: a laborious, costly, and error-prone process that motivates growing interest in large language models (LLMs) as compliance assistants. Yet single-model accuracy remains insufficient for high-stakes use, and naive ensembling is vulnerable to systematic per-model biases that can drag aggregate accuracy below that of the strongest individual annotator. This gap is addressed with a multi-stage framework, supported by a new benchmark of 19,998 IFRS regulation-disclosure pairs comprising 11,161 unanimous multi-LLM labels and 8,837 human annotations on items where the LLMs disagreed. The framework couples a three-stage domain-adapted sentence embedding pipeline (BERT masked language modeling on SEC EDGAR filings, unsupervised TSDAE and SimCSE adaptation, and supervised contrastive alignment) with a learned ensemble classifier that fuses frozen embeddings and LLM probability features from GPT-4.1, Claude Sonnet 4, and Gemini 2.5 Pro via Tree-structured Parzen Estimator (TPE) optimized gradient-boosted trees. On the disagreement set, the ensemble achieves 81.7% weighted F1, exceeding the strongest single LLM by 10.4 points and majority voting by 14.6 points, and outperforms eleven unsupervised aggregation baselines; the retrieval pipeline attains Recall@15 of 45.3% on unseen regulations. The dataset, model checkpoints, annotation platform, and classifier implementation will be publicly released.
Figures & tables
Figure 1: Proposed framework. Top-5 regulations per disclosure are retrieved by an EDGAR-adapted BERT encoder and labeled by a three-LLM panel; 8,837 disagreement items are reconciled with humans. An Optuna-tuned XGBoost meta-learner over LLM probabilities and a frozen SBERT pair embedding predicts the final label, benchmarked against baseline aggregators.
Model
R@5
R@15
MRR
Seen Regulations ( n=530 )
MLM only
8.9
20.9
0.060
+ TSDAE
7.7
19.8
0.053
+ SimCSE
7.4
15.8
0.046
+ TSDAE + Sup
11.9
27.5
0.080
+ SimCSE + Sup
11.3
26.4
0.073
Table 1: Retrieval performance on (3,489 regulations, 39,404 disclosures).
Figure 2: Method leaderboard: all 15 evaluated methods (three LLMs, eleven aggregators, the proposed classifier) ranked by weighted F1 against 8,837 independent human annotations. The learned ensemble classifier (starred) reaches 0.817, exceeding the best single LLM (Gemini, 0.713) and the best unsupervised aggregator (MACE, 0.722).
Method
Acc
W-F1
κ
ECE
Individual LLMs
GPT-4.1 OpenAI (2025)
0.183
0.095
0.019
0.811
Claude Sonnet 4 Anthropic (2025)
0.672
0.687
0.256
0.211
Gemini 2.5 Pro Google DeepMind (2025)
0.737
0.713
0.278
0.215
Aggregation
Hard Vote
0.641
0.671
0.254
–
Table 2: Classification benchmark on 8,837 disagreement items.
Figure 3: Per-class F1 across all 15 evaluated methods on 8,837 human-annotated items.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Standard
Regs
Pairs
C
NC
NA
Human %
IFRS 1
97
893
32
185
676
61
IFRS 2
138
2,556
297
798
1,461
45
IFRS 3
140
1,192
67
207
918
46
IFRS 5
61
607
26
101
480
45
IFRS 6
25
672
113
281
278
35
IFRS 7
227
2,395
26
985
1,384
42
Appendix
Table 3: Pairs per IFRS standard. Labels are human where available, otherwise the unanimous LLM label (C = Complied , NC = Not Complied , NA = Not Applicable ). Human % is the share of the standard’s pairs in the human-annotated disagreement subset.
Unanimous
Human
Pairs
11,161
8,837
Complied
547
914
Not Complied
4,383
1,394
Not Applicable
6,231
6,529
Appendix
Table 4: Class distribution by label source. The two subsets share no pair; the 2 remaining pairs are three-way analyst splits left unresolved and excluded from the human labels. Regulation texts average 44.3 words (55.1 BERT wordpieces) and disclosures 27.6 words (34.0 wordpieces).
Model
USD per 1,000 pairs
GPT-4.1
0.89
Claude Sonnet 4
1.41
Gemini 2.5 Pro
0.68
Panel total
2.97
Appendix
Table 5: Estimated annotation cost at list prices: 344 input tokens per call on average (254 of them the system prompt) and 25 output tokens, counted with the o200k tokenizer on the exact prompt over all 20,000 pairs. Excludes structured-output schema overhead and Gemini 2.5 Pro reasoning tokens, so it is a lower bound.
Parameter
Value
parse_mode
parse_page_with_agent
model
openai-gpt-4-1-mini
high_res_ocr
True
adaptive_long_table
True
outlined_table_extraction
True
output_tables_as_HTML
True
Appendix
Table 6: LlamaParse configuration for PDF ingestion.
Pred: No
Pred: N/A
Pred: Yes
True: No
0.990
0.002
0.008
True: N/A
0.472
0.527
0.001
True: Yes
0.147
0.016
0.837
Appendix
Table 7: GPT-4.1 confusion matrix vs. hard-vote consensus.
Pred: No
Pred: N/A
Pred: Yes
True: No
0.897
0.092
0.011
True: N/A
0.005
0.994
0.001
True: Yes
0.114
0.002
0.884
Appendix
Table 8: Claude Sonnet 4 confusion matrix vs. hard-vote consensus.
Pred: No
Pred: N/A
Pred: Yes
True: No
0.703
0.250
0.047
True: N/A
0.003
0.996
0.002
True: Yes
0.060
0.016
0.925
Appendix
Table 9: Gemini 2.5 Pro confusion matrix vs. hard-vote consensus.
Field
Type
Description
label
enum
Complied , Not Complied , or Not Applicable
rationale
text
Free-text justification (1–3 sentences)
time_spent
float
Auto-tracked annotation duration (seconds)
flagged_difficult
bool
Annotator-flagged ambiguous items
Appendix
Table 10: Human annotation schema.
Partition
Count
Labels
Used For
Unanimous (3/3)
11,161
0
Supervised training
Partial (2/3)
8,795
8,794
Evaluation
Three-way (1/1/1)
44
43
Evaluation
Appendix
Table 11: Data partitions. Items where LLMs agreed unanimously were never sent to humans (0 human labels), keeping the embedding training pool disjoint from the human-labeled evaluation set. The one-label shortfall in each disagreement row (8,795 vs. 8,794; 44 vs. 43) reflects two routed pairs on which the three analysts split three ways and which remained unresolved when the labeling round closed; they are excluded from the released labels (§ 3.3 ).
Parameter
Stage 0 (MLM)
Stage 1A (TSDAE)
Stage 1B (SimCSE)
Stage 2 (Supervised)
Base model
bert-base-cased
Stage 0 output
Stage 0 output
Stage 1 output
Epochs
3
1
1
4
Batch size
4 (accum. 4)
8
128
16
Learning rate
5e-5
3e-5
3e-5
2e-5
fp16
Yes
No
Yes
Yes
Warmup
—
10%
10%
10%
Appendix
Table 12: Hyperparameters for each stage of the embedding pipeline. MNRL = MultipleNegativesRankingLoss. EDGAR 10-K sentences are filtered to 150–400 characters from Section 7 (MD&A) of annual filings (1993–2020).
Table 15: Label source results on shared test set (1,326 items). All CIs overlap between LLM and Human rows.
Seen
Unseen
Model
Labels (n)
R@5 / R@15
R@5 / R@15
TSDAE+S
Human (1.6k)
6.2 / 14.6
8.0 / 36.0
TSDAE+S
LLM (4.9k)
4.7 / 12.0
16.0 / 40.0
SimCSE+S
Human (1.6k)
7.8 / 15.6
20.0 / 32.0
SimCSE+S
LLM (4.9k)
5.7 / 13.5
12.0 / 48.0
Appendix
Table 16: Label-source ablation on a fixed test set (1,326 items, 192 seen regs, 25 unseen regs). Each cell reports R@5 / R@15 (%). 95% bootstrap CIs omitted for space.
Predictor
Class
Prec.
Rec.
F1
OpenAI
yes
—
—
—
OpenAI
no
0.16
0.98
0.28
OpenAI
not applicable
0.80
0.02
0.04
Anthropic
yes
0.14
0.35
0.20
Anthropic
no
0.29
0.27
0.28
Anthropic
not applicable
0.83
0.82
0.83
Appendix
Table 17: Per-class precision, recall, and F1 for all predictors on 8,837 human-annotated items. OpenAI “yes” metrics are near-zero (only 233 of 8,837 predictions are “yes”).
Strategy
Acc
W-F1
M-F1
κ
ECE
Brier
Oracle (ceiling)
0.952
0.949
0.900
0.882
0.031
0.102
Learned Ensemble (proposed)
0.824
0.817
–
0.556
0.068
–
MACE Hovy et al. (2013)
0.748
0.722
0.561
0.312
–
–
CROWDLAB Goh et al. (2022)
0.791
0.720
–
0.340
–
–
Gemini single Google DeepMind (2025)
0.737
0.713
0.532
0.278
0.215
0.478
M-MSR Ma and Olshevsky (2020)
0.672
0.687
–
0.256
–
–
Appendix
Table 18: Classification benchmark on 8,837 human-annotated items, ranked by W-F1. Bold row: the learned ensemble classifier (Section 3.5 ). EM Calibrated under the textbook Tmax=5 bound is included to make the 17-point bound penalty visible (Appendix R ).
Tmax
Toai
Tant
Tgem
W-F1
ECE
5.0
5.00 ⋆
0.95
5.00 ⋆
0.503
0.461
10.0
10.00 ⋆
0.66
5.84
0.642
0.301
20.0
14.36
0.55
5.73
0.672
0.292
Appendix
Table 19: Temperature bound sensitivity. ⋆ marks fitted temperatures pinned at the upper bound. The [0.1,5.0] bound introduces a 17-point W-F1 penalty by constraining two of three models to the bound.
Financial reporting systems increasingly leverage Large Language Models (LLMs) to extract and summarize corporate disclosures. However, most existing approaches assume a single-market setting and overlook structural differences across jurisdictions. Variations in accounting taxonomies, tagging infrastructures (e.g., XBRL vs.\ PDF), and aggregation conventions introduce substantial challenges for semantic alignment and reliable verification. Here, we aim to bridge this gap. We present FinReporting, an agentic workflow for localized cross-jurisdiction financial reporting. The system constructs a unified canonical ontology spanning the income statement, balance sheet, and cash flow statement, and decomposes reporting into auditable stages, including filing acquisition, extraction, canonical mapping, and anomaly logging. Rather than treating LLMs as free-form generators, FinReporting employs them as constrained verifiers operating under explicit decision rules with evidence grounding. Evaluated on annual filings from the USA, Japan, and China, FinReporting improves consistency and reliability under heterogeneous reporting regimes. We further release an interactive demo that enables cross-market inspection and supports structured export of localized financial statements. Our demo is available at url{https://huggingface.co/spaces/BoomQ/FinReporting-Demo. A video describing our system is available at https://www.youtube.com/watch?v=f65jdEL31Kk.
We introduce FinVerBench, a benchmark and validity study for financial statement verification: determining whether a set of corporate financial statements is numerically consistent from the information shown to the model. FinVerBench is built from SEC 10-K XBRL filings for 43 S&P 500 companies and defines a four-category error taxonomy covering arithmetic, cross-statement linkage, year-over-year, and magnitude perturbations. We attempt fifteen contemporary LLM evaluations and report fourteen complete runs; a Gemini 2.5 Pro run is excluded from the main comparison because 40/108 gateway calls failed. All binary metrics exclude underdetermined positive instances whose perturbed line item is not rendered, leaving a 105-instance observable diagnostic subset (43 clean, 62 error-injected). Under the original guided-checklist prompt on the unrounded diagnostic subset, nine of fourteen complete LLM runs produce 95-100% false positives on clean statements, while one run achieves 0% observed false positives. Benchmark rendering choices materially affect measured recall: on a realistic rounded variant of the same observable subset, the calibrated model's recall is 79.0% with 0% observed FPR, compared with 100.0% recall on the unrounded diagnostic variant. These results support a construct-validity conclusion rather than a final leaderboard: financial statement verification is not merely arithmetic detection, but calibrated judgment under incomplete observability, prompt-induced assumptions, and realistic numerical rendering. FinVerBench and all code are publicly available.
As large language models (LLMs) are increasingly deployed in financial services, a single non-compliant interaction can expose institutions to regulatory penalties and direct consumer harm. Existing guard models are built around general harm taxonomies and overlook violations grounded in specific financial regulations. We address this gap with a regulation-driven pipeline that operates directly on regulatory documents, inducing a financial compliance risk taxonomy and synthesizing grounded training data without any predefined violation categories. Instantiating the pipeline on Chinese financial regulations, we release FinGuard-Bench, to our knowledge the first benchmark for financial regulatory compliance detection, with expert-annotated labels at both the query and response levels. We further train FinGuard, a financial compliance detection model built on Qwen3-8B and trained on the regulation-grounded data via supervised fine-tuning and self-play reinforcement learning. On FinGuard-Bench, FinGuard substantially outperforms all baselines, including dedicated guard models and much larger general-purpose LLMs such as Qwen3.5-397B-A17B and GPT-5.1. Furthermore, FinGuard also preserves general safety capabilities and adapts to unseen institution-specific policies using policy documents alone. We will publicly release the code, prompts, and resources used in this work on GitHub.
Huaixia Dou, Jie Zhu, Minghao Wu +5
Qwen DianJin Team, Alibaba Cloud Computing · Tongyi Lab, Alibaba Group · School of Computer Science and Technology, Soochow University