cs.AISep 21, 2026
SaveFinInteract: Benchmarking Clarification and Intent Integration in Ambiguous Financial Question Answering
Organizations: McGill University · University of California, Los Angeles · University of Toronto
Abstract
Large language model agents increasingly answer financial questions by searching regulatory filings. Such questions are often deceptively under-specified: Meta Platforms' "operating income" is 62.87B for the Family of Apps segment, and each reading is exactly verifiable against the filing. A capable agent should recognize the ambiguity and ask, rather than commit to a plausible but unintended reading. Existing financial benchmarks cannot measure this, because one gold answer per question cannot separate agents that resolve the ambiguity from those that guess the common reading, a blind spot we call the single-gold illusion. We release FinInteract, a bilingual (English/Chinese) benchmark of 173 instances that pairs each question with a default and an intended interpretation across a five-category ambiguity taxonomy, and grades whether an agent elicits the right clarification and then integrates it. Re-grading identical outputs against the default rather than the intended reading inflates GPT-4o's accuracy by 3.1 times, confirming the illusion. Beyond it, we find that models answer above 90% once the interpretation is supplied but at most 28.9% when they must elicit it themselves, that targeting is uneven across a taxonomy well powered for entity scope and metric definition and exploratory elsewhere, and that conditioning on the ambiguity category improves resolution at both inference and training time.
Figures & tables
Figure 1: Evaluating financial agents on ambiguous queries. (1-2) A real query (“What was Meta Platforms’ operating income?”) looks fully specified but hides ambiguity across five categories (entity, metric, period, filing, policy), here giving two defensible answers: a default 46.75B (consolidated GAAP). (3) A search-only agent returns the plausible default, which a single-gold benchmark scores as correct, the single-gold illusion that overstates competence by up to 3 times. (4) FinInteract instead grades whether the agent asks : the GPT-5 agent reaches 20.2% under free interaction versus 93-95% once the ambiguity is resolved, so the gap lies in interaction, not missing knowledge.
| Field | Value |
|---|---|
| What was Meta Platforms’ operating income? | |
| Total company (consolidated) results under U.S. GAAP for the year ended December 31, 2023. | |
| (intended) | $46.75B [consolidated GAAP] |
| (default) | $62.87B [Family of Apps segment] |
| Ambiguity category | Entity scope |
| 1.58 bits |
Table 1: This ambiguous financial question example admits three plausible entity scopes, i.e., the consolidated total and two reportable segments, giving an entropy of 1.58 bits. Without the disambiguating context , a model’s default interpretation yields the wrong answer.
| Category | Definition |
|---|---|
| Temporal scope | Ambiguity over which time period is intended: fiscal year ending September vs. calendar year; Q3 results vs. full year; TTM vs. point-in-time. |
| Metric definition | Ambiguity over accounting basis: GAAP net income vs. non-GAAP adjusted income; organic revenue vs. as-reported; operating EBITDA vs. net income. |
| Entity scope | Ambiguity over consolidation level: segment vs. consolidated total; parent-attributable vs. full consolidated; Class A vs. Class B shares; A-shares vs. H-shares. |
| Filing vintage | Ambiguity over filing version: original 10-K vs. amended 10-K/A; as-reported vs. restated; preliminary earnings release vs. final filing. |
| Recognition policy | Ambiguity over accounting policy: revenue recognized point-in-time vs. over time; gross vs. net revenue; capitalized vs. expensed R&D. |
Table 2: Five-category financial ambiguity taxonomy.
| Mode | Description |
|---|---|
| A | Pure parametric recall |
| A+S | Oracle retrieval augmented |
| A+S+I | Standard full interaction |
| Always-ask | follow-up question |
| Category-oracle | Provide primary category |
| Template-oracle | Fixed human-written question |
Table 3: We evaluate interaction modes including answer-only (A), answer with search (A+S) and answer with search and interaction (A+S+I).
Figure 2: FinInteract rewards actions that resolve the intended interpretation. (1) Each instance carries a primary ambiguity category and two defensible readings, a default (Family of Apps, 46.75B), separated by a hidden disambiguating context . (2) The agent either answers without clarifying (Part A, a plausible but wrong reading) or asks a question that a user simulator answers only from (Part B, correct elicitation). (3) FinInteract scores on final-answer correctness against the intended interpretation, and interaction quality (did it ask, target the right category, and resolve intent).
| Property | Value | Property | Value |
|---|---|---|---|
| Total instances | 173 | Category : entity | 94 |
| Language : EN | 53 | metric | 63 |
| ZH | 120 | recognition | 9 |
| Source : EDGAR | 50 | temporal | 7 |
| CSRC-CNINFO | 120 | filing | 0 |
| DocFinQA | 3 | # cat. : single | 128 |
Table 4: FinInteract dataset statistics. Disambiguation entropy over plausible interpretations.
| Model | A | A+I | Forced(4) | Oracle |
|---|---|---|---|---|
| GPT-5 | 0.6 | 20.2 | 1.5 | 95.4 |
| GPT-4o | 0.6 | 4.6 | 0.0 | 93.1 |
| GPT-5-mini | 0.6 | 0.0 | 0.6 | 95.4 |
| Qwen3-30B | 1.2 | 11.6 | – | 90.2 |
| Qwen3.5-35B | 0.6 | 28.9 | – | 91.9 |
Table 5: Forcing more clarification does not help accuracy, but resolving the ambiguity lifts every model to 93-95%. The gap is therefore the model’s inability to elicit for itself, not missing knowledge or grounding. Abbreviations follow Table 3 .
| Acc (%) | Answer+Search+Interact | ||||||
| Model | A | A+S | A+S | IR | AC@1 | ECE | |
| Proprietary models | |||||||
| GPT-5.6 | 10.0 | 36.0 | 30.0 | – | .98 | – | |
| GPT-5 | 0.6 | 6.4 | 20.2 | 98.8 | .86 | .53 | |
| GPT-4o | 0.6 | 12.1 | 4.6 | 99.4 | .94 | .82 | |
| GPT-5-mini | 0.6 | 0.0 | 0.0 | 96.0 | .94 | .00 | |
Table 6: (a) Main results, reporting accuracy (%) against the intended interpretation per mode, with the +Interact minus +Search difference. (b) Re-grading the same outputs against the intended versus the default reading a conventional benchmark would treat as gold (Same abbreviation as Table 3 ). (c) Per-category targeting (AC@1), where recognition policy is a shared blind spot ( for every model). Category sizes: entity (Ent.) 94, metric (Met.) 63, recognition (Rec.) 9, temporal (Tmp.) 7.
| Stage | AC@1 | Interaction | Accuracy | Reward |
|---|---|---|---|---|
| Base | 25.0 | 54.9 | 84.3 | 0.96 |
| + SFT | 68.6 | 100 | 88.2 | 1.28 |
| + KTO | 70.6 | 100 | 84.3 | 1.27 |
| + GRPO | 66.7 | 100 | 88.2 | 1.30 |
Table 7: RLVR ladder on FinInteract with Qwen3-4B, held-out n = 51. Category-guided supervision installs on-category clarification, with the leak-proof signals (AC@1 and interaction) rising almost entirely at the SFT stage. Accuracy is leak-confounded and is reported for completeness only.
Figure 3: Category-guided training installs on-category clarification. (a) The leak-proof signals, AC@1 and interaction rate, jump at the supervised stage and change little thereafter. (b) True multi-turn GRPO on the 4B policy: mean reward and clarifying turns rise together over training.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: The elicitation gap. Free-interaction accuracy (orange) versus the context-oracle ceiling (teal) per model. Once the ambiguity is resolved every model answers 90–95% correctly, so the 63–95 point gap is the cost of the model failing to elicit the disambiguation itself, not missing knowledge.
Figure 5: Per-category clarification-capability profile (touch AC@1 from Table 6 (c)). Every model targets entity, metric, and temporal ambiguity well (0.85–1.00) but collapses to zero on recognition_policy , a blind spot shared across the closed frontier and the strongest open model.
Figure 6: Per-model, per-category performance. (a) Targeting (AC@1 touch) versus (b) Resolution ( +interact accuracy) for every model and ambiguity category. The top five rows are the full-scale panel (n = 173, all categories) and the bottom seven are the proprietary pilot (n = 50), for which only entity and metric carry enough instances (n = 29 and 19), with recognition, temporal, and filing at at most 1 and shown n/a. filing_vintage is empty at full scale too (n = 0). Every model targets entity and metric almost perfectly, the proprietary models resolve them better than the older OpenAI ladder, and no model targets recognition policy.
| Model | Mode | Intended | Any-valid | Def.-cap | Ask |
|---|---|---|---|---|---|
| gpt 4o-mini | A | 31.5 | 72.5 | 55.0 | 0 |
| A+O | 49.5 | 68.5 | 31.0 | 0 | |
| A+I | 43.5 | 81.0 | 53.0 | 27.0 | |
| gpt 4o | A | 39.0 | 84.0 | 65.5 | 0 |
| A+O | 68.5 | 81.5 | 31.5 | 0 | |
| A+I | 43.0 | 86.0 | 64.0 | 2.5 |
Table 8: The single-gold illusion replicates on AmbigQA (general domain, n = 200, three models, same grading methodology; the grader retries with backoff so no answer is silently dropped). Intended : accuracy against a specified target reading. Any-valid : accuracy against any accepted reading, what a lenient single-gold benchmark credits. Def.-cap : fraction producing a valid but non-target reading. Ask : fraction that asks when permitted. gpt-5-mini interact omitted (it asks below 1%). Modes: A answer-only, A+O answer with context oracle, A+I answer with interaction.
Figure 7: Interaction is a capability, not a free lever. Added-interaction minus plain-search accuracy across the twelve full-scale and cross-vendor models. Interaction improves accuracy for only GPT-5 and Gemini and is flat or negative for the other ten, so the ability to convert clarification into a correct answer is rare.
Figure 8: Targeting does not imply resolution. Accuracy against AC@1 (touch) in the interact regime, for the five full-scale models and the human baseline (star). The relationship is inverted: models that ask on the correct category most often resolve the ambiguity least often, whereas the human asks less precisely yet resolves more. Asking the right question and answering it are separate abilities.
| Category | 4B | 14B | 32B |
|---|---|---|---|
| entity_scope | .00 | .00 | .68 |
| metric_definition | .05 | .00 | .17 |
| recognition_policy | .00 | .00 | .00 |
Table 9: Zero-shot central -targeting (central-form AC@1) by base-model scale (Qwen3, +interact , denominator = instances where the model asked). Under the validated touch form every scale already reaches about 0.9 on entity and metric, so scale buys the precision to center the question on the right category rather than detection. Entity central-targeting scales steeply and is largely solved at 32B, while recognition policy stays 0 at every scale, including the same 32B that targets entity well. Recognition is underpowered ( at most 9 asked-on) but flat throughout.
Figure 9: Four-model depth profile of the ambiguity signal (Qwen3 4Bto35B, dense and MoE). (a) Ambiguous-disambiguated separation rises to a peak in the final layers of every model (dotted verticals mark per-model peaks; normalized for depth and magnitude). (b) The ambiguity type (entity vs. metric) is linearly decodable at 0.80 on both MoE models, well above the 0.60 majority baseline, peaking at different depths.
| Type | Description | Improvement direction |
|---|---|---|
| E1 | Ambiguity blindness : no interaction, wrong answer | Ambiguity pre-check; train on detection |
| E2 | Wrong-category clarification : asked wrong dimension | Category classifier; financial taxonomy grounding |
| E3 | Generic clarification : vague ask, no category target | Category-conditioned clarification templates |
| E4 | Retrieval dominance : searched but skipped interaction | Force interpretation check after retrieval |
| E5 | Clarification misuse : right question, wrong answer | State-gated answering; stronger grounding |
| E6 | Evidence grounding : template-oracle correct, agent wrong | Improved source-grounded extraction |
Table 10: Error taxonomy E1–E7 with diagnostic signals and improvement directions.
Figure 10: Error-type distribution in +interact , attributing every instance to the taxonomy of Table 10 with the corrected category classifier. E5 misintegration (the agent asks, touches the correct category, yet answers wrong) dominates for every model, while ambiguity blindness (E1), wrong-category (E2), and generic (E3) asks stay comparatively rare.
| entity | metric | recognition | |
| abundant | abundant | scarce | |
| Frontier instances generable | 99 | 76 | 36 (ceiling) |
| Targeting, base evolved | |||
| on held-out generated | .40 | .88 | .00 |
| Interaction rate | |||
| De-leaked accuracy |
Table 11: Three co-evolution rounds (category-guided SFT on a 4B policy). Targeting (AC@1) is on a held-out human probe never seen in training. The loop closes on both abundant categories and fails only on the scarce one; whether asking helps accuracy flips on whether the base could already answer.
| Category | Primary | Salience | Slope | Decode | Co-evolve |
|---|---|---|---|---|---|
| entity_scope | 905 | 0.92 | 0.77 | GO | |
| metric_definition | 498 | 0.48 | 0.77 | GO ∗ | |
| temporal_scope | 150 | 0.15 | sat. | 0.92 | solved |
| filing_vintage | 264 | 0.33 | – | – | untestable |
| recognition_policy | 30 | 0.04 | 0.93 | weak ‡ |
Table 12: What predicts co-evolvability. The observed outcomes track abundance/salience , not the AC@1 scale-slope (metric: low slope, GO) and not decodability (recognition: highest AUROC, yet does not co-evolve). Salience = primary any-candidate passages; decodability = peak one-vs-rest AUROC (recognition on a size-matched augmented sample; raw n = 9 overfits to 1.0). ∗ metric is a GO but needs about 2 accumulated slices (about 50 instances); one 30-instance slice teaches it nothing (Finding 21). ‡ recognition is unreliable rather than a clean fail: held-out human AC@1 ranges 0–0.44 across training runs on the scarce n = 9 probe.
Figure 11: Multi-round co-evolution with a validation guard. Left : validation AC@1 (selection metric) per round. Right : held-out human -probe AC@1 (never used for selection). Entity learns targeting from a single 30-instance slice and transfers to humans (0.63 to 0.93); metric needs accumulation (one slice teaches nothing, brown, flat, while two accumulated slices reach 0.75, green); recognition stays at base (about 0.05) even with a fair 26-instance set. The guard preserves human transfer (it rolls back the round-3 candidate, which regresses on validation).
Table 13: Comparison of financial QA benchmarks. A check mark denotes a supported feature; – denotes not applicable or not evaluated.
| Quantity | Value |
|---|---|
| Constructor viability (viable / candidate) | |
| Verifier acceptance (accepted / viable) | |
| Overall acceptance (accepted / candidate) | |
| Cost per rejected candidate (constructor only) | $0.020 |
| Cost per viable passage (full pipeline) | $0.155 |
| Blended cost per candidate passage | $0.035 |
Table 14: Measured curation cost. Unit costs are from instrumented pipeline runs (constructor-only n = 20; full pipeline n = 6); the acceptance funnel is from the full construction run (n = 1127 candidate passages). A full pipeline pass is one GPT-5 constructor call plus ten verifier trials (five GPT-5, five GPT-5-mini) with GPT-4o-mini grading.
Figure 12: FinInteract distributions. (a) Primary ambiguity category: the corpus is entity/metric-dominant (54% / 36%), with the three subordinate categories sparse, mirroring how often each category is the primary ambiguity in public filings (Sec. 2 ). (b) Difficulty by disambiguation entropy : instances span one to 3.58 bits, most at one or two bits with a substantial multi-category tail at 3.58 bits (45 instances).
| Rule | Description |
|---|---|
| R1 | must not be answerable yes/no |
| R2 | must not contain disambiguating terms or inline dates |
| R3 | taken verbatim from the source candidate value (not invented) |
| R4 | must name the company |
| R5 | must ask about a substantive financial metric |
| R6 | Answer type consistent with question type |
Table 15: Pre-verifier sanity checks (14 rules).
More in this topic
Financial QA
Financial question answering develops systems that answer questions about reports, markets, regulations, and other financial data with reliable evidence and reasoning.
See all 51 papers