Human-Centred, Risk-Constrained Evaluation of Financial Document Recognition: Evidence, Critical Errors, and Selective Automation
Organizations: Independent Researcher
Abstract
Financial document recognition is often assessed using character and word error rates, although these metrics do not express whether a system preserves the financial facts on which people and organisations act. This paper develops a human-centred, risk-constrained framework for evaluating financial document recognition as evidence recovery and decision support. It defines typed financial facts linked to visible source regions and separates verbatim transcription, deterministic normalization, and business interpretation. A layered quality model covers text, localization, layout, tables, fields, fact correctness, evidence grounding, uncertainty, robustness, and operational performance. Non-compensatory acceptance gates use one-sided uncertainty bounds for critical-field errors, preventing strong aggregate scores from masking failures in amounts, identifiers, signs, periods, or entity associations. Selective automation is formulated as a constrained routing problem that accounts for residual risk, human-review capacity, and review cost. The framework further specifies annotation and adjudication procedures, leakage-resistant benchmark partitions, statistical estimands, subgroup analyses, and a reproducibility card. Formal results establish the limitations of compensatory scoring, the non-zero upper risk bound after zero observed errors, and a loss sensitive decision rule for routing uncertain cases to human review. The contribution is theoretical and methodological; no new model, dataset, or empirical performance result is claimed.
Figures & tables
| Category | Representative content | High-priority evaluation dimensions |
|---|---|---|
| Bank statements | Account identifiers, opening/closing balances, transaction dates, descriptions, debit/credit, running balances | Sign and decimal fidelity; row-to-column association; transaction completeness; balance continuity; date and amount normalization |
| Invoices and credit notes | Supplier/buyer, invoice number, line items, tax, discounts, totals, payment terms | Key-field exact match; line-item table structure; tax arithmetic; currency and unit fidelity; credit-note sign semantics |
| Receipts and expense claims | Merchant, date/time, items, subtotal, tax, tip, total, payment method | Low-resolution robustness; total extraction; item/quantity association; locale-specific separators; cropped or curved images |
| Purchase orders and remittance advices | PO references, quantities, unit prices, remitted amounts, invoice references | Cross-document identifier matching; row alignment; partial payment semantics; quantity and unit precision |
| Tax and payroll documents | Taxpayer/employer IDs, periods, gross/net pay, deductions, rates, statutory fields | Identifier exactness; period and jurisdiction; negative/withheld values; sensitive-field completeness and access controls |
| Loan, credit, and insurance documents | Principal, interest rate, repayment schedule, fees, collateral, coverage, dates | Decimal and percentage fidelity; schedule/table reconstruction; term association; unit and effective-date semantics |
| Resource | Domain / modality | Useful annotations or tasks | Important limitation for the proposed evaluation |
|---|---|---|---|
| SROIE [ 2 ] | Scanned receipts | Text localization, OCR, key information extraction | Receipt-focused; does not represent the full range of statements, disclosures, or multi-page finance documents |
| CORD [ 3 ] | Indonesian receipts | Word boxes, text, hierarchical semantic receipt labels | Strong receipt parsing resource, but narrow document and language coverage |
| FUNSD [ 1 ] | Noisy scanned forms | Text, entities, semantic relations, layout | General forms rather than a finance-specific corpus |
| DocVQA [ 11 ] | General document images | Document visual question answering with image-grounded questions | QA accuracy is not a direct substitute for exhaustive OCR transcription |
| FinTabNet / GTE [ 12 ] | Financial-report tables | Table detection and cell-structure annotations | Table-centric; does not alone test full-page reading order, field extraction, or operational routing |
| PubTables-1M [ 16 ] | Scientific-document tables | Table structure, cells, headers, canonicalized structure | Valuable for structural methods, but its domain differs from financial statements |
| Item | Required disclosure |
|---|---|
| Population | Target workflow, document families, jurisdictions/languages, sampling frame, collection period, inclusion/exclusion criteria |
| Sampling | Sampling unit, strata, inclusion probabilities, oversampling policy, target deployment weights |
| Ground truth | Annotation manual/version, annotator training, independent double-annotation fraction, adjudication, ambiguity policy, agreement by field type |
| Schema | Field ontology, typed-value representation, units/currencies, polarity, periods, table relations, missingness states, evidence-link rules |
| Partitions | Document/customer/template/time grouping, duplicate detection, holdout rationale, contamination checks |
| Systems | Model and software versions, prompts/configuration, preprocessing, normalization, decoding settings, failure handling |
| Dimension | Primary metrics | Required diagnostics |
|---|---|---|
| Text fidelity | CER, WER | Micro/macro; punctuation and symbol policy; critical-token accuracy |
| Detection/localization | Precision, recall, F1 at IoU thresholds | Text-line/word/table region slices; missed-region rate |
| Layout/order | Pairwise reading-order F1; layout-class F1 | Multi-column and cross-page continuation; header/footer confusion |
| Tables | Cell F1; row/column/span accuracy; header association | Cell text exactness; numeric column assignment; merged cells |
| Fields/facts | Typed exact match; precision/recall/F1 | Amount, date, currency, sign, entity, period, identifier slices |
| Risk | Critical error rate; severity-weighted loss | Error counts by C0–C4; upper confidence bounds; gate violations |
Document Information Extraction
Document Information Extraction turns text, tables, forms, and images into structured, searchable data.
See all 305 papers