IFRS Compliance Verification: An Open Multi-LLM Benchmark and Content-Conditioned Ensemble Classifier
Organizations: Faculty of Business and IT, Ontario Tech University Oshawa, Ontario, Canada
Abstract
Verifying corporate financial disclosures against the International Financial Reporting Standards (IFRS) is a core auditing task in which experts cross-reference disclosures with hundreds of fine-grained regulatory paragraphs: a laborious, costly, and error-prone process that motivates growing interest in large language models (LLMs) as compliance assistants. Yet single-model accuracy remains insufficient for high-stakes use, and naive ensembling is vulnerable to systematic per-model biases that can drag aggregate accuracy below that of the strongest individual annotator. This gap is addressed with a multi-stage framework, supported by a new benchmark of 19,998 IFRS regulation-disclosure pairs comprising 11,161 unanimous multi-LLM labels and 8,837 human annotations on items where the LLMs disagreed. The framework couples a three-stage domain-adapted sentence embedding pipeline (BERT masked language modeling on SEC EDGAR filings, unsupervised TSDAE and SimCSE adaptation, and supervised contrastive alignment) with a learned ensemble classifier that fuses frozen embeddings and LLM probability features from GPT-4.1, Claude Sonnet 4, and Gemini 2.5 Pro via Tree-structured Parzen Estimator (TPE) optimized gradient-boosted trees. On the disagreement set, the ensemble achieves 81.7% weighted F1, exceeding the strongest single LLM by 10.4 points and majority voting by 14.6 points, and outperforms eleven unsupervised aggregation baselines; the retrieval pipeline attains Recall@15 of 45.3% on unseen regulations. The dataset, model checkpoints, annotation platform, and classifier implementation will be publicly released.
Figures & tables
| Model | R@5 | R@15 | MRR |
|---|---|---|---|
| Seen Regulations ( ) | |||
| MLM only | 8.9 | 20.9 | 0.060 |
| + TSDAE | 7.7 | 19.8 | 0.053 |
| + SimCSE | 7.4 | 15.8 | 0.046 |
| + TSDAE + Sup | 11.9 | 27.5 | 0.080 |
| + SimCSE + Sup | 11.3 | 26.4 | 0.073 |
| Method | Acc | W-F1 | ECE | |
|---|---|---|---|---|
| Individual LLMs | ||||
| GPT-4.1 OpenAI (2025) | 0.183 | 0.095 | 0.019 | 0.811 |
| Claude Sonnet 4 Anthropic (2025) | 0.672 | 0.687 | 0.256 | 0.211 |
| Gemini 2.5 Pro Google DeepMind (2025) | 0.737 | 0.713 | 0.278 | 0.215 |
| Aggregation | ||||
| Hard Vote | 0.641 | 0.671 | 0.254 | – |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Standard | Regs | Pairs | C | NC | NA | Human % |
|---|---|---|---|---|---|---|
| IFRS 1 | 97 | 893 | 32 | 185 | 676 | 61 |
| IFRS 2 | 138 | 2,556 | 297 | 798 | 1,461 | 45 |
| IFRS 3 | 140 | 1,192 | 67 | 207 | 918 | 46 |
| IFRS 5 | 61 | 607 | 26 | 101 | 480 | 45 |
| IFRS 6 | 25 | 672 | 113 | 281 | 278 | 35 |
| IFRS 7 | 227 | 2,395 | 26 | 985 | 1,384 | 42 |
| Unanimous | Human | |
| Pairs | 11,161 | 8,837 |
| Complied | 547 | 914 |
| Not Complied | 4,383 | 1,394 |
| Not Applicable | 6,231 | 6,529 |
| Model | USD per 1,000 pairs |
|---|---|
| GPT-4.1 | 0.89 |
| Claude Sonnet 4 | 1.41 |
| Gemini 2.5 Pro | 0.68 |
| Panel total | 2.97 |
| Parameter | Value |
|---|---|
| parse_mode | parse_page_with_agent |
| model | openai-gpt-4-1-mini |
| high_res_ocr | True |
| adaptive_long_table | True |
| outlined_table_extraction | True |
| output_tables_as_HTML | True |
| Pred: No | Pred: N/A | Pred: Yes | |
|---|---|---|---|
| True: No | 0.990 | 0.002 | 0.008 |
| True: N/A | 0.472 | 0.527 | 0.001 |
| True: Yes | 0.147 | 0.016 | 0.837 |
| Pred: No | Pred: N/A | Pred: Yes | |
|---|---|---|---|
| True: No | 0.897 | 0.092 | 0.011 |
| True: N/A | 0.005 | 0.994 | 0.001 |
| True: Yes | 0.114 | 0.002 | 0.884 |
| Pred: No | Pred: N/A | Pred: Yes | |
|---|---|---|---|
| True: No | 0.703 | 0.250 | 0.047 |
| True: N/A | 0.003 | 0.996 | 0.002 |
| True: Yes | 0.060 | 0.016 | 0.925 |
| Field | Type | Description |
|---|---|---|
| label | enum | Complied , Not Complied , or Not Applicable |
| rationale | text | Free-text justification (1–3 sentences) |
| time_spent | float | Auto-tracked annotation duration (seconds) |
| flagged_difficult | bool | Annotator-flagged ambiguous items |
| Partition | Count | Labels | Used For |
|---|---|---|---|
| Unanimous (3/3) | 11,161 | 0 | Supervised training |
| Partial (2/3) | 8,795 | 8,794 | Evaluation |
| Three-way (1/1/1) | 44 | 43 | Evaluation |
| Parameter | Stage 0 (MLM) | Stage 1A (TSDAE) | Stage 1B (SimCSE) | Stage 2 (Supervised) |
|---|---|---|---|---|
| Base model | bert-base-cased | Stage 0 output | Stage 0 output | Stage 1 output |
| Epochs | 3 | 1 | 1 | 4 |
| Batch size | 4 (accum. 4) | 8 | 128 | 16 |
| Learning rate | 5e-5 | 3e-5 | 3e-5 | 2e-5 |
| fp16 | Yes | No | Yes | Yes |
| Warmup | — | 10% | 10% | 10% |
| Model | R@1 | R@5 | R@10 | R@15 | MRR | MAP@15 | NDCG@15 |
|---|---|---|---|---|---|---|---|
| MLM only | 2.5 | 8.9 | 17.0 | 20.9 | 0.060 | 0.034 | 0.059 |
| + TSDAE | 2.1 | 7.7 | 14.5 | 19.8 | 0.053 | 0.034 | 0.058 |
| + SimCSE | 1.5 | 7.4 | 11.3 | 15.8 | 0.046 | 0.026 | 0.046 |
| + TSDAE + Sup | 2.5 | 11.9 | 22.1 | 27.5 | 0.080 | 0.050 | 0.084 |
| + SimCSE + Sup | 2.5 | 11.3 | 23.2 | 26.4 | 0.073 | 0.042 | 0.077 |
| Model | R@1 | R@5 | R@10 | R@15 | MRR | MAP@15 | NDCG@15 |
|---|---|---|---|---|---|---|---|
| MLM only | 3.8 | 11.3 | 21.7 | 28.3 | 0.086 | 0.076 | 0.118 |
| + TSDAE | 0.9 | 12.3 | 22.6 | 26.4 | 0.069 | 0.064 | 0.106 |
| + SimCSE | 0.9 | 12.3 | 15.1 | 27.4 | 0.078 | 0.070 | 0.110 |
| + TSDAE + Sup | 4.7 | 18.9 | 32.1 | 41.5 | 0.122 | 0.110 | 0.171 |
| + SimCSE + Sup | 3.8 | 20.8 | 33.0 | 45.3 | 0.137 | 0.125 | 0.188 |
| Model | Labels | Seen@5 | Unseen@15 |
|---|---|---|---|
| TSDAE+Sup | Human (1.6k) | 6.2 | 36.0 |
| TSDAE+Sup | LLM (4.9k) | 4.7 | 40.0 |
| SimCSE+Sup | Human (1.6k) | 7.8 | 32.0 |
| SimCSE+Sup | LLM (4.9k) | 5.7 | 48.0 |
| Seen | Unseen | ||
|---|---|---|---|
| Model | Labels (n) | R@5 / R@15 | R@5 / R@15 |
| TSDAE+S | Human (1.6k) | 6.2 / 14.6 | 8.0 / 36.0 |
| TSDAE+S | LLM (4.9k) | 4.7 / 12.0 | 16.0 / 40.0 |
| SimCSE+S | Human (1.6k) | 7.8 / 15.6 | 20.0 / 32.0 |
| SimCSE+S | LLM (4.9k) | 5.7 / 13.5 | 12.0 / 48.0 |
| Predictor | Class | Prec. | Rec. | F1 |
|---|---|---|---|---|
| OpenAI | yes | — | — | — |
| OpenAI | no | 0.16 | 0.98 | 0.28 |
| OpenAI | not applicable | 0.80 | 0.02 | 0.04 |
| Anthropic | yes | 0.14 | 0.35 | 0.20 |
| Anthropic | no | 0.29 | 0.27 | 0.28 |
| Anthropic | not applicable | 0.83 | 0.82 | 0.83 |
| Strategy | Acc | W-F1 | M-F1 | ECE | Brier | |
|---|---|---|---|---|---|---|
| Oracle (ceiling) | 0.952 | 0.949 | 0.900 | 0.882 | 0.031 | 0.102 |
| Learned Ensemble (proposed) | 0.824 | 0.817 | – | 0.556 | 0.068 | – |
| MACE Hovy et al. (2013) | 0.748 | 0.722 | 0.561 | 0.312 | – | – |
| CROWDLAB Goh et al. (2022) | 0.791 | 0.720 | – | 0.340 | – | – |
| Gemini single Google DeepMind (2025) | 0.737 | 0.713 | 0.532 | 0.278 | 0.215 | 0.478 |
| M-MSR Ma and Olshevsky (2020) | 0.672 | 0.687 | – | 0.256 | – | – |
| W-F1 | ECE | ||||
|---|---|---|---|---|---|
| 5.0 | 5.00 ⋆ | 0.95 | 5.00 ⋆ | 0.503 | 0.461 |
| 10.0 | 10.00 ⋆ | 0.66 | 5.84 | 0.642 | 0.301 |
| 20.0 | 14.36 | 0.55 | 5.73 | 0.672 | 0.292 |