FinVerify: A Verification Framework for Financial Agents
Organizations: Independent Researcher · New York University · Columbia University
Abstract
Financial agents produce numerical claims that can be wrong even when their calculations execute correctly, due to mismatched reporting periods, unsupported sources, incompatible units, or incorrect financial definitions. We introduce FinVerify, an evidence-grounded framework for verifying financial agents' numerical claims. Based on an analysis of 190 incorrect trajectories from the Agent Finance Reasoning dataset, we define five error categories--temporal integrity, source fidelity, comparability, accounting integrity, and financial-model fidelity--and operationalize them as verification protocols. An LLM structures claims and supporting evidence, while deterministic validators check financial requirements; when authoritative formulas are available, Z3 checks consistency with grounded inputs. FinVerify accepts only if all applicable checks pass, rejects proven violations, and abstains when evidence is insufficient. We evaluate the framework against a Direct baseline and an LLM judge across five fixed-candidate benchmarks. On Agent Finance Reasoning, accepted-claim precision increases from 56.4% for the LLM judge to 76.7% at 81.2% coverage. On XBRLFiling, FinVerify achieves 99.8% accepted-claim precision and 97.0% decision accuracy; on FinanceBench-67, it reaches 87.9% accepted-claim precision versus 69.7% for the judge. Improvements are smaller on FinQA and TAT-QA, where authoritative evidence is limited. These results suggest that explicit, evidence-backed verification can improve financial-agent reliability while exposing trade-offs between correctness, false rejection, and abstention.
Figures & tables
| Category | Agent error | Count | Share |
|---|---|---|---|
| A: Temporal integrity | wrong, stale, future, or unavailable period | 3 | 1.6% |
| B: Source fidelity | missing, wrong, or unsupported evidence | 145 | 76.3% |
| C: Comparability | incompatible entity, unit, metric, or basis | 8 | 4.2% |
| D: Accounting integrity | arithmetic, sign, or accounting-identity error | 2 | 1.1% |
| E: Financial-model fidelity | wrong definition, method, formula, or interpretation | 32 | 16.8% |
| All agent errors | Trajectories labeled wrong by the dataset | 190 | 100.0% |
| Protocol, policy | Functions | Authority |
|---|---|---|
| A no_lookahead.v1 | resolve_as_of, resolve_publication, compare_cutoff | question; SEC FSDS/EDGAR; date order |
| B traceable_claim.v1 | resolve_citation, ground_operands, match_filed_value | frozen evidence; filed XBRL |
| C like_for_like.v1 | compare_period, compare_unit, compare_scope | typed XBRL contexts |
| C requested_unit.v1 | compare_requested_unit | question; typed operand scale |
| C percent_scale.v1 | compare_percent_scale | re-executed program; question |
| D claim_consistency.v1 | execute_program, check_identities | deterministic DSL; filed facts |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | Candidate generator | Evidence (+ FinVerify-only) | wrong | null | |
|---|---|---|---|---|---|
| Agent Finance Reasoning [ 35 ] | five backbones (dataset) | trace; + 10-K tables | 357 | 190 | – |
| FinQA (dev) | Haiku 4.5 | FinQA text and table | 600 | 117 | 24 |
| TAT-QA (dev, arith.) | Haiku 4.5 | TAT-QA table and paragraphs | 600 | 59 | 5 |
| FinanceBench-67 | Haiku 4.5 | filing passages; + SEC filing | 67 | 22 | 0 |
| XBRLFiling | Haiku 4.5 | redacted excerpt; + SEC filing | 600 | 99 | 3 |
| A Temporal | B Source | C Compar. | D Accounting | E Model | |
|---|---|---|---|---|---|
| Atomic-claim verifiers [ 7 , 8 , 9 ] | – | (✓) | – | – | – |
| LLM-as-judge [ 29 ] | – | (✓) | – | – | (✓) |
| Look-ahead benchmarks [ 31 , 32 , 28 , 33 ] | ✓ | – | – | – | – |
| FinGround [ 11 ] | (✓) | ✓ | (✓) | – | (✓) |
| VeriFin [ 10 ] | – | ✓ | (✓) | ✓ | (✓) |
| FinVerify (this work) | ✓ | ✓ | ✓ | ✓ | ✓ |
| Benchmark | Verifier | TA | FA | TR | FR | Abs | Precision % | Accuracy % | Cov. % |
| Agent Finance Reasoning ( =357, wrong=190) | |||||||||
| FinVerify | 69 | 21 | 135 | 65 | 67 | 76.7 | 70.3 | 81.2 | |
| LLM judge | 155 | 120 | 69 | 12 | 1 | 56.4 | 62.9 | 99.7 | |
| Direct | 167 | 190 | 0 | 0 | 0 | 46.8 | 46.8 | 100.0 | |
| FinQA ( =600, wrong=117, 24 null) | |||||||||
| FinVerify | 397 | 49 | 25 | 46 | 83 | 89.0 | 81.6 | 86.2 | |
Claim Verification
Claim verification assesses whether statements are supported, contradicted, or unresolved by relevant evidence.
See all 63 papers