When Self-Verification Hurts: Calibrated Answer Repair for Financial Question Answering
Organizations: National University of Singapore, Singapore · Nanyang Technological University, Singapore
Abstract
Financial question answering can fail even when a model extracts a plausible number from the supplied evidence: the answer may attach it to the wrong entity, period, unit, or accounting concept. Asking the model to verify and revise its draft can help, but open-ended revision may alter content that was already correct or inspect the wrong relation and reinforce the original error. We formulate repair as a decision that balances the information it may recover against the correct content it may damage. Based on this trade-off, we introduce Calibrated Answer Repair (CAR), which first estimates whether a draft is likely to benefit from repair, then selects a targeted check, localizes the relevant evidence or draft span, and limits the permitted edit. Across FinRank, Fin-RATE, and FinQA with three open-weight models, CAR outperforms generic verification in all dataset-model settings. At the system level, it recovers 14.6% of initially missed answer requirements while damaging 2.1% of initially correct ones. Its post-draft token cost is over 30 percent lower than the iterative revision alternatives. Ablations on two dataset-model settings show that evidence localization and bounded editing are the main contributors to CAR's gains.
Figures & tables
| Partition | FinRank | Fin-RATE | FinQA |
| Calibration | 910 | 5,250 | 6,251 |
| Development | 140 | 750 | 883 |
| Held-out test | 135 | 1,500 | 1,147 |
| Dataset / Metric | Model | Draft | Verify | SC-Verify | CoVe | RARR | CAR |
| FinRank / GAA | Qwen3-8B | 29.63 | |||||
| Qwen3-14B | 36.30 | 36.30 | |||||
| Mistral-24B | 34.81 | ||||||
| Fin-RATE / Macro CR | Qwen3-8B | 8.20 | |||||
| Qwen3-14B | 14.67 | ||||||
| Mistral-24B | 20.07 |
| Method | Recovery (%) | Damage (%) | Calls/q | Tokens per question | Rel. cost (%) |
| Draft | — | — | |||
| Verify | |||||
| SC-Verify | |||||
| CoVe | |||||
| RARR | |||||
| CAR | 14.6 | 2.1 |
| CAR metric | Rate (%) |
| Overall recovery | |
| Overall damage | |
| Gate-open recovery | |
| Gate-open damage | |
| Changed-answer damage | |
| Edit rate |
| Variant | GAA |
| Full CAR | |
| Always repair | |
| No pointer | |
| Fixed checklist | |
| Unbounded rewrite |
| Dataset | Always-repair | Always-repair | Gate-open rate (%) | |
| FinRank | ||||
| Fin-RATE | ||||
| FinQA |
| Miss-rate input | MAE | Pearson |
| Dataset mean only | — | |
| Text features only | ||
| Probe verdicts only | ||
| Text + probe | 0.146 | 0.512 |
| Model | DR-QA | EC-QA | LT-QA |
| Qwen3-8B | |||
| Qwen3-14B | |||
| Mistral-24B |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Family | Definition | Typical checks and localization |
| Omission | A requested entity, period, qualifier, comparison, or result is absent. | Retrieve the supporting source passage and append only the supported missing relation. |
| Binding | A stated value is attached to the wrong entity, period, unit, or accounting concept. | Trace the value to its row, column, and passage, then locally replace the incorrect binding. |
| Evidence | A claim is unsupported, weakly supported, or grounded in a distractor passage. | Compare the draft sentence with its strongest evidence and verify provenance. |
| Numerical | The selected operands, operation, scale, sign, or computed result is wrong. | Point to the calculation span and recompute from cited inputs. |
| Interpretation | The answer misstates a comparison, trend, aggregation, or requested relation. | Re-read the question relation and check the conclusion against the evidence. |
| Family | Leaf | Definition |
| Omission | f_omit_narrative | A requested fact appears in narrative evidence but is absent from the answer. |
| f_omit_absent | The requested fact is not supported by the supplied evidence and cannot be recovered by verification. | |
| f_omit_derivable | The required inputs are present, but the answer omits a value that must be derived. | |
| f_omit_table | A requested value appears in a supplied table but is not reported. | |
| f_omit_clause | A multi-part question is answered only in part. | |
| Binding | f_entity | A value is attributed to the wrong company or entity. |
| Setting | Metric | Pointer | Comparison |
| FinRank | Source Hit@3 | 72.8 | 46.9 random |
| Fin-RATE | Source Hit@3 | 52.4 | 20.1 random |
| Fin-RATE | Source Hit@3 | 52.4 relevance | 29.2 novelty |
| FinQA | Span overlap | 68.9 | 41.6 answer sent. |
| FinQA | All operands | 51.7 | 27.4 answer sent. |
| Dataset | Model | Always-repair | Always-repair | Gate-open rate | |
| FinRank | Qwen3-8B | 0.132 | 0.091 | 0.408 | 66.7% |
| FinRank | Qwen3-14B | 0.153 | 0.100 | 0.395 | 61.5% |
| FinRank | Mistral-24B | 0.141 | 0.094 | 0.400 | 59.8% |
| Fin-RATE | Qwen3-8B | 0.104 | 0.086 | 0.453 | 58.9% |
| Fin-RATE | Qwen3-14B | 0.117 | 0.089 | 0.432 | 54.7% |
| Fin-RATE | Mistral-24B | 0.125 | 0.093 | 0.427 | 52.6% |
| Model | Method | DR | EC | LT | Macro CR |
| Qwen3-8B | Draft | 12.8 | 0.4 | 3.4 | 5.53 |
| Verify | 13.2 | 0.4 | 3.6 | 5.73 | |
| SC-Verify | 13.8 | 0.6 | 4.0 | 6.13 | |
| CAR | 17.2 | 1.8 | 5.6 | 8.20 | |
| Qwen3-14B | Draft | 26.4 | 1.6 | 5.8 | 11.27 |
| Verify | 26.8 | 1.8 | 6.2 | 11.60 |
| Variant | NAA (%) | CAR advantage (pp) |
| Full CAR | 72.97 | — |
| Always repair | 70.53 | 2.44 |
| No pointer | 70.27 | 2.70 |
| Fixed checklist | 72.10 | 0.87 |
| Unbounded rewrite | 68.96 | 4.01 |
Question Answering
Question answering studies how AI systems interpret questions and produce accurate, evidence-grounded answers across text, dialogue, and multimodal data.
See all 270 papers