FinVeriQA: Financial Research Agents for Question Answering: Routing, Retrieval, MCP Tool Use, and Provenance-Aware Evidence Verification
Organizations: San José State University, San José, CA, USA
Abstract
Large language models (LLMs) write fluent financial analyses but miscompute ratios, recall stale prices, and route questions to the wrong tool with high confidence. We ask which component should do each job in a financial question-answering assistant, and study it with FinVeriQA, an open-source multi-agent research prototype over SEC filings and live market data: a compact typed-decision model (Laya) routes each question and screens out advice requests, Python computes every number, retrieval and tools fetch evidence, Mistral-7B writes a cited answer, a code validator checks that cited IDs exist and that numbers match computed notes, and a Model Context Protocol (MCP) server exposes the pipeline. In live experiments, Laya routed better than Mistral-7B (0.85 vs. 0.52; overall 0.86 vs. 0.68, McNemar p = 0.0003); a 12B Mistral decided no better, and DeepSeek-R1-8B decided better (0.79 vs. 0.68, p = 0.0192) but about 9.9 times more slowly. Mistral-7B computed 29% of 100 financial calculations correctly against 100% for Python, a live quote cut price error from 210% to 0.00%, and hybrid retrieval reached 76% Hit@5 on 50 filing questions. Yet in a live case study, both agent answers passed validation while describing AMD sample data under NVIDIA's and Alphabet's names. A controlled upper-bound study on the FinRank benchmark explains why: when one figure in the shared evidence is corrupted, self-consistency stays confident (agreement 0.55 to 0.59 while accuracy falls from 0.50 to 0.00), and a shared-evidence check (the kind our validator performs) catches 35% of the errors, whereas a check against the original passage catches 77% (); for DeepSeek-R1-8B, 68% vs. 90%. Verification must be independent of the evidence the writer saw.
Figures & tables
| Agent | Reads | Writes | Engine |
|---|---|---|---|
| data_agent | selected company | revenue series; filing passages | code |
| analysis_agent | revenue series | notes N1–N3: QoQ, YoY, and trend, with exact numbers and sources | Python |
| classify_agent | filing passages | one label per passage (positive, neutral, negative); sentiment breakdown | Mistral |
| market_agent | selected company | live quote, tagged real_time (60 s expiry) | code + Finnhub |
| viz_agent | revenue series; sentiment breakdown | chart specifications, rendered with Altair | code |
| report_agent | notes, passages, quote, conversation state | the cited answer (3–6 sentences) | Mistral + validator |
| Setup | Overall (95% CI) | Routing | Sentiment |
|---|---|---|---|
| A rules | 0.62 [0.53, 0.71] | 0.65 | 0.57 |
| B Mistral | 0.68 [0.60, 0.77] | 0.52 | 0.93 |
| C Laya | 0.86 [0.79, 0.92] | 0.85 | 0.88 |
| D cascade | 0.74 [0.66, 0.83] | 0.62 | 0.93 |
| E agree | 0.65 [0.57, 0.75] | 0.52 | 0.85 |
| F hybrid | 0.88 [0.82, 0.94] | 0.85 | 0.93 |
| Mistral-7B | NeMo-12B | R1-8B | |
| Accuracy | 0.6800 | 0.68 | 0.7700 |
| 95% CI | [0.60, 0.77] | [0.60, 0.77] | [0.69, 0.85] |
| Intent accuracy | 0.5167 | — | 0.6667 |
| Intent macro-F1 | 0.3683 | 0.35 | 0.4784 |
| Sentiment accuracy | 0.9250 | 0.90 | 0.9250 |
| Sentiment macro-F1 | 0.9212 | — | 0.9244 |
| Engine | Accuracy | Time / calc | Hardware |
|---|---|---|---|
| Python | 100% | 0.0007 ms | CPU |
| gpt-5 (minimal effort) | 98% (49/50) | 1,323 ms | OpenAI API |
| Ministral 3 8B | 72% (74% on 50) | 0.3 s | Colab T4 |
| Mistral-7B-Instruct | 29% | 386 ms | V100 (PSC) |
| Condition | Mean error | Within 5% |
|---|---|---|
| Without data | 210% | 0 / 10 |
| With live quote | 0.00% | 10 / 10 |
| Question | Route and steps | What worked | What went wrong | |
|---|---|---|---|---|
| Q1 | “What are the main risk factors?” (GOOGL) | no router; cites F2, F3, F4 | points 1–6 are genuine risk factors (advertising dependence, privacy changes, ad blocking, competition, the economy, new-business bets), each cited | points 7–8, cited to F4, are a pending acquisition (Wiz, Intersect) and a $2.1B compensation charge: facts, not risks |
| Q2 | “Summarize management’s discussion of results.” (META) | no router; cites F1–F4 | the one MD&A paragraph (F3) gives 2025 total revenue, $200.97B | 3 of 4 paragraphs summarize other Items (business, risk summary, market risk); the figure is copied from text and not checked against XBRL |
| Q3 | “What is the tone of this filing?” (AMD 10-K) | no router; cites F1–F4, all at the end | a fair overall read (“cautious and informative”); all four Items extracted in full (Item 1A: 131,846 characters) | citations cannot be matched to claims; no per-passage labels; a filler last sentence |
| Q4 | “How did NVIDIA’s revenue and margins change last quarter?” | Laya numbers , 0.87; data, analysis, viz, report ran; classify, market skipped | every number matches a Python note (+++$ 42.0% over six quarters); verified on attempt 1 | AMD sample data presented as NVIDIA’s; margins never answered; P2 cited for a claim P2 does not contain |
| Q5 | “What’s the tone and outlook in Alphabet’s latest 10-K?” | Laya sentiment , 0.87; data reused from memory; classify, viz, report ran; analysis, market skipped | 8 passages labeled (5 positive, 3 negative), 7 of them clearly right; verified on attempt 1 | AMD sample data presented as Alphabet’s; answer repeats revenue notes N1–N3 instead of the 5/3 split; same P2 mismatch; sentiment chart drew no bar |
| Failure | Seen in | Proposed fix |
|---|---|---|
| Wrong company | Q4, Q5 | check each note’s ticker against the question; say “sample data” in the answer text; connect XBRL (K2) |
| Citation ID points to the wrong passage | Q4, Q5 (P2) | one passage numbering for prompt and Sources; a claim–passage support check |
| Off-topic content | Q1 (2 of 8 points), Q2 (3 of 4 paragraphs) | send only the Item asked about (1A for risks, 7 for MD&A) |
| Part of the question unanswered | Q4 (margins), Q5 (5/3 split) | margin notes; pass the sentiment breakdown to the report agent |
| Stale items reused | Q5 ( amd_* , N1–N3) | key reuse by company, period, filing, and route |
| Unchecked number | Q2 ($200.97B) | check text figures against XBRL |
| No verification | B: shared check | C: provenance | |
|---|---|---|---|
| 0.00 | 0.50 | 0.50 | 0.50 |
| 0.25 | 0.62 | 0.54 | 0.43 |
| 0.50 | 0.75 | 0.57 | 0.37 |
| 0.75 | 0.88 | 0.61 | 0.30 |
| 1.00 | 1.00 | 0.65 | 0.23 |
| Mistral-7B | DeepSeek-R1-8B | |
| Clean: SC agr. / accuracy | 0.55 / 0.50 | 0.26 / 0.35 |
| Corrupted: SC agr. / accuracy | 0.59 / 0.00 | 0.20 / 0.03 |
| B: shared-check detection | 0.35 [0.23, 0.47] | 0.68 [0.53, 0.80] |
| C: provenance detection | 0.77 [0.65, 0.87] | 0.90 [0.80, 0.97] |
| McNemar C vs. B ( / ) | 25 / 0 | 9 / 0 |
Financial QA
Financial question answering develops systems that answer questions about reports, markets, regulations, and other financial data with reliable evidence and reasoning.
See all 57 papers