cs.AIOct 28, 2026
SaveRight Numbers, Wrong Questions: Evaluating LLMs on Underspecified Financial Queries
Organizations: TIFIN, India
Abstract
Financial assistants answer questions that sound factual but depend on what the user leaves unsaid: what a data figure measures, which product variant is meant, and the investor's situation and the rules that apply. An assistant that ignores these can still answer with a real number from its data, one that answers a different question. We call this frame collapse. We study it in Indian mutual funds, where product variants and tax rules are formally defined, so the correct answer can be computed for every reading. One study tests each challenge. Given an ambiguously labelled figure, models used it, often under the user's label. Given every alternative labelled, they listed them all, but the layout decided which came first. Missing a fact about the investor, weaker agents assumed a default or an outdated tax rule; requiring the fact did not guarantee its use. Because such answers match their data, checks against the data pass them. An AI judge caught mislabelled figures only when told what each field means, and still passed answers that silently chose another reading. We show how these failures arise, not how often, and propose controls for financial assistants.
Figures & tables
Figure 1. One real case for each challenge. Model answers are quoted from the study logs, shortened with ellipses (currency shown as Rs.). (1) Study 1, a lookup tool returning one ambiguously named record. (2) One Study 2 question asked twice, with the table’s plan columns in each order; each figure is that plan’s three-year annualised return. (3) A Study 3 run of a model chosen because it failed in exploratory runs: a profile tool holding the investor’s NRI status was available but never called. The panels illustrate each failure; Section 5 reports how often it occurred, and Appendix Figure 4 summarises every measurement. Three panels, one per challenge. Panel 1, ambiguous data: asked for August 2026's net inflow into equity funds, a lookup tool returns a record with a field named growth of 84,955; GPT-4.1 reports a net inflow of Rs. 84,955 crore, but that figure is the change in assets and the net inflow was Rs. 29,329 crore. Panel 2, product variants: asked for HDFC Flexi Cap Fund's annualised return, GPT-6.1 Sol leads with the Direct plan's 14.72 percent when the Direct column comes first and with the Regular plan's 13.97 percent when the Regular column comes first, labelling both plans each time; the right behaviour is to lead with the investor's plan or a fixed, stated order. Panel 3, the investor and the rules: asked how much a full redemption of HDFC Corporate Bond Fund will credit to the bank, Gemini 3.1 Pro, which never fetched the investor's profile, answers Rs. 5,52,834.69; the investor is an NRI, so 30 percent plus 4 percent cess is withheld on the gain and the credit is Rs. 5,36,350.
| Plan, option, mode | Measure, period | Return |
|---|---|---|
| Regular, Growth | 3-year annualised | 13.97% |
| Direct, Growth | 3-year annualised | 14.72% |
| Regular, IDCW payout | 3-year XIRR, before tax | 15.12% |
| Direct, Growth; Rs. 5,000 a month (SIP) | 5-year XIRR | 13.66% |
| Direct, Growth; one lump sum | 5-year annualised | 15.67% |
Table 1. Five defensible readings of “What are the returns of HDFC Flexi Cap Fund?”, computed from AMFI NAV records through 30 September 2026, net of fund expenses, before exit load and tax. XIRR is the annualised return over dated cash flows (glossary, Appendix A ). The IDCW (payout) figure is the investor’s own cash-flow return, not a fund return; it exceeds Growth even though that investor ends with less (Appendix B.1 ).
| Group | Dimensions | Example of an unsupported interpretation |
|---|---|---|
| A. Referent | A1 entity (fund, fund house, company, person, index; renamed schemes); A2 plan and option (Direct/Regular x Growth/IDCW); A3 unit and scale | “HDFC” read as HDFC Bank when HDFC’s fund house was meant; a Regular-plan NAV given for a Direct holding |
| B. Measurement | B1 metric (absolute, annualised, XIRR; asset change vs net flow); B2 horizon and as-of date; B3 mode (lump sum vs SIP); B4 gross or net basis (before or after exit load, transaction taxes and tax) | A 3-year total return reported as a 3-year annualised return; a redemption credit quoted before tax |
| C. Evaluative | C1 comparator (total-return vs price index, category); C2 objective; C3 risk measure | “Beat the Nifty” judged against the price index |
| D. Contextual | D1 jurisdiction; D2 tax status and regime (including treaty relief); D3 market regime; D4 data and rule vintage | A capital-gains rule superseded in July 2024 applied to a 2026 sale |
Table 2. Fourteen dimensions along which a financial question can be underspecified.
| Form | What the answer does | Example from our studies | Tested in | Scored by |
|---|---|---|---|---|
| (i) Mislabel | Attaches the user’s label to a figure that measures something else | A month’s change in assets reported as the net inflow | Study 1; judges | Codebook |
| (ii) Silent reading | Answers under another admissible reading without saying so | A lump-sum return given to an investor who invests monthly | Studies 1 and 2 | Codebook; extraction |
| (iii) Headline prioritisation | Shows and labels both readings, but the layout decides which leads | The first column’s plan headlined | Study 2 | Code (order) |
| (iv) Omitted condition | Leaves out a decisive investor fact, or uses a superseded rule | An NRI’s bank credit quoted before tax is withheld | Study 3 | Code, tool logs |
Table 3. The forms of failure, where each is tested and how it is scored. Forms (i), (ii) and (iv) are frame collapse; (iii) is a related, weaker presentation effect.
| Study 1: ambiguous data | Study 2: product variants | Study 3: the investor and the rules | |
|---|---|---|---|
| What changes | No data, a thin record or labelled records, returned by a lookup tool; precise or vague wording | Plan or mode open or stated; Direct or Regular column first (counterbalanced); table in the prompt or tools | Whether the agent’s interface mentions the decisive fact, demands it, or supplies the current rule |
| Models | GPT-4.1, Gemini 2.5 Flash, Qwen3.8 27B; replicated on five more | Eleven current models | Gemini 3.1 Pro and GPT-4.1, selected for failing; five further models as checks |
| Scale | 22 questions; 1,188 answers | 148 items; 3,256 answers | Two redemption cases; 350 planned runs, 80 check and final runs |
| Outcome | Wrong answer; mislabel | Silent on the open dimension; which plan leads | Fetched the fact; figure within tolerance |
| Scored by | An AI coder with a fixed codebook | Code, from two AI extractors’ readings | Code and tool logs |
| Read with care | Codebook stricter than the definition; illustrative figures | Rates over answers both extractors agree on; extractors are tested models | Two cases; models chosen for failing; mechanisms, not rates |
Table 4. Design of the three studies. Each changes one thing that reaches the model’s context. Cohorts and denominators: Appendix B.3 .
| Study 3 case | Investor and holding | Rule that decides it | Reference answer | If the fact is missed |
|---|---|---|---|---|
| Non-resident withholding | NRI; Rs. 5 lakh in a debt fund (HDFC Corporate Bond) bought January 2025; gain Rs. 52,835 | Debt funds (since April 2025, funds with more than 65% in debt and money-market instruments) bought on or after 1 April 2023 are taxed at slab rate, whatever the holding period; for an NRI, 30% plus 4% cess is withheld at source | Bank credit Rs. 5,36,350 | Rs. 5,52,835 (overstated by Rs. 16,484) |
| Booked gains | Resident; Rs. 5 lakh in an equity fund (Parag Parikh Flexi Cap) bought June 2023; Rs. 60,000 of long-term gains on listed shares already booked this tax year | Long-term gains on listed equity and equity-oriented fund units (held over 12 months) taxed at 12.5% plus cess above a Rs. 1.25 lakh annual exemption shared with other such gains (before 23 July 2024: 10% above Rs. 1 lakh) | Gain Rs. 2,66,118; tax Rs. 26,145 (tax year 2026-27) | Rs. 18,345 if the exemption is not shared; Rs. 23,516 under the superseded rule |
Table 5. The two Study 3 cases. Neither question mentions tax status or other gains. Both redeem at the 30 September 2026 NAV and assume no surcharge, a PAN on file, no treaty relief and no exit load. The NRI reference is the bank credit after TDS, not final tax; the resident reference is final tax, assuming other income above the basic exemption and no carried-forward losses. Rounding conventions: Appendix B.1 .
Figure 2. Study 2, per model, over agreed answers that stated a figure. (a, b) Silent selection (form ii, silent reading): answered without naming the open plan or mode, or asking. (c) Share of plan answers headlining the first column’s plan (order counterbalanced; 50% = no order effect); most of these answers show both plans. Open: NAV table in the prompt; filled: data fetched through tools; bars: Wilson 95% intervals; grey: fewer than 10 answers. *Gemini 3.8 Flash’s plan cases are side-by-side layouts that an audit coded as flagged. Three dot-and-interval panels, one row per model. Panels a and b show silent selection rates for the plan and mode questions: near zero for Claude Opus 5.5, GPT-6.1 Sol, GPT-6 Astra, Kimi K3 and DeepSeek V4.1 Flash, and high for some of Gemini 3.8 Flash, Grok 4.7, gpt-oss-120b, Qwen3.5-122B and Llama 4 Maverick. Panel c shows the share of plan answers headlining the first column, mostly between 60 and 100 percent, to the right of the 50 percent no-order-effect line.
Figure 3. Study 3 arms. (a) Gemini 3.1 Pro: share of runs that fetched the decisive fact. (b) GPT-4.1: share of runs with the correct figure. Filled circles: a demand or fix; open circles: a mention, no demand or a stale rule; grey squares: no fix; grey circle: field-named tools without a demand. Lines: Wilson 95% intervals; five runs per case, two cases per arm unless a case is named. Two dot-and-interval panels. Panel a: Gemini 3.1 Pro fetched the decisive fact in 2 of 10 runs at baseline, 1 of 10 with a mention and 0 of 10 with unnamed profile parts, but in 10 of 10 to 30 of 30 runs with any demand, and 9 of 10 with profile tools named for each field. Panel b: GPT-4.1 gave the correct figure in 0 of 10 runs at baseline, with a checklist or with facts preloaded, 1 of 10 with a rule tool alone, 10 of 10 with facts plus a rule tool or when the question asked for the after-tax amount, 0 of 5 with a superseded-rule tool, and 5 of 5 with the legal basis or the current rule as prompt text.
| Told reading | Sees | Mislabels named (of 44) | Wrong rated correct (of 96) | Acceptable rejected (of 61) |
|---|---|---|---|---|
| Yes | Question | 14 | 18 | 15 |
| Yes | + data | 17 | 43 | 15 |
| Yes | + data + dictionary | 40 | 0 | 23 |
| No | + data | 23 | 53 | 15 |
| No | + data + dictionary | 40 | 42 | 17 |
Table 6. GPT-6.1 Sol as a judge of Study 1’s 162 thin answers: 44 mislabels, 96 wrong answers (26 precise, 70 vague) and 61 acceptable (right, flagged or asked) answers. Both judges, all arms and tests: Appendix B.8 .
| # | Control | Form | Evidence here |
|---|---|---|---|
| 1 | Take each decisive investor fact as a required, typed input from the records | (iv) | Made both failing agents fetch it (GPT-4.1 via a checklist); fixed one; frontier agents did not need it |
| 2 | Supply the rule in force from a dated source; compute tax in code | (iv) | Current rule fixed GPT-4.1 (5 of 5 vs 0 of 5); a stale rule tool was relayed |
| 3 | State what each data field measures; check it against the question | (i) | Helped where the record named a different entity (exploratory); prompt check over-asked |
| 4 | Evaluate with the data’s meaning; add a reference for each reading | Judges | Dictionary tested; per-reading references untested |
| 5 | Label every figure with the interpretation used | All | Proposed |
| 6 | Order alternatives by the user’s own holding | (iii), (ii) | Proposed; order effect observed |
Table 7. Controls, the failure each addresses, and the evidence here. Rows 1–4 have evidence on few models and cases; rows 5–8 are proposed and untested. Full table with Indian implementations: Appendix B.11 .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Term | Meaning |
|---|---|
| Plan: Direct, Regular | Every scheme is offered in two plans that hold the same portfolio and differ only in where the investor bought it. A Direct plan is bought without a distributor (from the fund house, an execution-only platform or an adviser). A Regular plan is bought through a distributor, and its expense ratio includes the distributor’s commission. The two plans therefore have different NAVs and returns. |
| Option: Growth, IDCW | Within each plan, the Growth option pays nothing out, so gains stay in the NAV. The income distribution cum capital withdrawal (IDCW) option pays out distributions, which can include the investor’s own capital; its NAV falls with each payout. |
| Mode: lump sum, SIP | How the money went in: one single investment, or a systematic investment plan (SIP) that invests a fixed amount every month. |
| NAV | Net asset value per unit: the daily price at which units are bought and redeemed. |
| Expense ratio; exit load | The annual fee taken from the fund’s assets (already reflected in the NAV); a fee charged on redemption within a set period. |
| Annualised return; XIRR | Annualised return: the return per year between two dates for a single investment. XIRR: the annualised return computed over dated cash flows, as in an SIP or IDCW payouts (an internal rate of return). |
Table 8. Glossary of Indian mutual-fund and tax terms.
Figure 4. Evidence summary: what reached the model’s context, and what it did. Each block is one situation; each line is one measurement with its count and 95% interval. Red: failures (lower is better); green: appropriate behaviour (higher is better); open circles: share following the layout (dashed line at 50%: no order effect). [M] a manipulation; [C] a comparison across models or answers; [S] runs of a model chosen because it failed in exploratory runs. Lines are comparable within a block, not across blocks: the studies use different models and scoring rules (Section 4 ). Populations and interval methods: Appendix B.3 . A dot-and-interval chart in four blocks. Block one, one candidate in context: high failure rates with a thin record (96 of 126 wrong, 70 of 108 vague answers wrong) and 44 of 65 silent when only one plan was fetched, against 4 of 80 when both were. Block two, every candidate labelled: 64 of 66 precise answers right, 82 of 108 vague answers flagged or asked, 7 of 108 wrong; the first column's plan led in 130 of 145 answers and swapping columns flipped it in 61 of 84 pairs. Block three, a needed fact missing: 81 of 81 silent mode answers assumed a lump sum; a selected agent assumed residency in 20 of 20 runs; two frontier agents fetched the fact and were correct in 20 of 20. Block four, a demanded fact: fetched in 3 of 20 runs at baseline or with a mention, 10 of 10 when demanded; with the residency fact fetched but not passed to the compute tool, the figure was correct in 1 of 15.
| Model (samples per item) | Record | Precise mislabels (of 42) | Wrong, figure not the asked quantity (of 126) | Flagged or asked, same answers (of 126) | Asked (of 162) |
|---|---|---|---|---|---|
| Study 1: GPT-4.1, Gemini 2.5 Flash, Qwen3.8 27B (1) | As given | 26 | 96 | 18 | 1 |
| GPT-6.1 Sol (3) | As given | 10 | 38 | 88 | 43 |
| Gemini 3.8 Flash (3) | As given | 27 | 99 | 12 | 1 |
| Llama 3.1 8B (3) | As given / accurate | 34 / 21 | 109 / 91 | 12 / 25 | 0 / 0 |
| Mistral Nemo 12B (3) | As given / accurate | 35 / 25 | 113 / 100 | 4 / 20 | 1 / 0 |
| Qwen3 8B (3) | As given / accurate | 32 / 21 | 106 / 92 | 16 / 21 | 3 / 4 |
Table 9. Thin-data behaviour across models, and with accurate records (fields renamed, entity and plan named). The 42 are 14 questions x 3 models (Study 1) or x 3 samples of one model. The three small models’ rows come from one arm-hidden recode of both arms (their first coding differs by at most 3); replications used larger output limits.
| Judge | Told the intended reading | Sees | Mislabels named (of 44) | Wrong rated correct: precise (of 26) | Wrong rated correct: vague (of 70) | Acceptable rejected (of 61) |
|---|---|---|---|---|---|---|
| GPT-6.1 Sol | Yes | Question | 14 | 14 | 4 | 15 |
| GPT-6.1 Sol | Yes | + data | 17 | 15 | 28 | 15 |
| GPT-6.1 Sol | Yes | + data + dictionary | 40 | 0 | 0 | 23 |
| GPT-6.1 Sol | No | + data | 23 | 5 | 48 | 15 |
| GPT-6.1 Sol | No | + data + dictionary | 40 | 0 | 42 | 17 |
| gpt-oss-20b | Yes | Question | 5 | 17 | 9 | 37 |
Table 10. Judges on the 162 Study 1 thin answers. Mislabels named: codebook mislabels the judge labelled as such (the planned outcome); 44 mislabels cover all wordings (26 precise, 18 vague). Wrong rated correct: codebook-wrong answers the judge passed (96 = 26 precise + 70 vague). Acceptable rejected: right, flagged or asked answers called wrong.
| Claim | Evidence and design | Scoring | Status | Scope |
|---|---|---|---|---|
| F1. When a lookup tool returns one ambiguously named figure in reply to the question, models attach the user’s label to it or silently answer another reading: (i), (ii) | Study 1: thin vs labelled data; accurate-record arm; five replications | Codebook; blind, model-free and human checks | Design fixed in advance; codebook after collection; accurate-record arm planned (entity naming a deviation) | 7 of 8 models (GPT-6.1 Sol mostly flagged or asked; accurate-record arm: 3 small open-weight); 14 questions whose thin figure was not the asked quantity; illustrative figures |
| F2. With plan or mode open, several models answer without naming it (ii) | Study 2: open vs specified items (comparison across 11 models) | Code, two extractors | Planned | Two dimensions, 10 funds |
| F3. Layout sets which plan leads (iii) | Study 2: counterbalanced column order (manipulation) | Code | Planned; committed-answer effect exploratory | 9 of 10 models; committed answers in one |
| F4. Two agents selected for failing omitted conditions they could fetch (iv); a demand for the decisive fact made Gemini 3.1 Pro answer correctly; GPT-4.1 needed the current rule, or an explicit request for the bank credit after TDS | Study 3: one-element arms (manipulation) | Code and tool logs | Pre-specified threshold readings; six of nine Fisher contrasts survive a Holm correction added later | Two cases; models selected for failing; did not transfer to two frontier models (40 runs) |
| F5. A judge named most mislabels when given an accurate data dictionary, with or without the intended reading; without the reading it passed silent readings of vague questions | Judge arms on Study 1 answers (manipulation) | Codebook as reference | Planned outcome; the data arms with the reading added after a first review | Two judges, each with and without the reading (rubric reworded without it); 162 answers |
| F6. Checking data against the question cuts errors, mostly by asking | Data-match check vs baseline (paired) | Codebook; blind recode | Planned; scoring deviated | One open-weight model; always asking scores higher |
Table 11. Claims, evidence and status. A manipulation varies one element with everything else fixed; a comparison contrasts models or items.
| # | Control (any jurisdiction) | Failure addressed | Example implementation in India | Evidence here |
|---|---|---|---|---|
| 1 | Make each decisive fact a required, typed input taken from the investor’s records (or asked of the investor where only they hold it, such as gains booked elsewhere), enforced in code; never accept a model-supplied default | (iv) | Consolidated account statements; tax status on each folio | For Gemini 3.1 Pro a demand for the decisive fact fetched it and gave correct answers (two cases); GPT-4.1 fetched it under a checklist but stayed wrong unless the current rule was supplied or the user explicitly asked for the bank credit after TDS (a diagnostic, not a control); a model filled a required field with ‘resident’ when the fact was missing; frontier models gathered it unasked (20 of 20), so for them the control is a safeguard against regression (untested), not a fix shown to be needed |
| 2 | Supply rules from a dated source and compute the tax in code | (iv) | A rulebook re-tested when the Budget, tax notifications or SEBI circulars change | Rule as text 5 of 5 vs 0 of 5 on the version case; a tool with the superseded rule was relayed 5 of 5 (one model); legal-basis text not significant after Holm |
| 3 | Give data fields metadata saying what they measure; check them against the question, with a rule that answers when they match (the check tested here lacked one and over-asked) | (i) | Metadata separating total-return from price indices, net flows from asset changes | Accurate records cut mislabels where they named a different entity (exploratory, 3 small models), not for close measures; the data-match check (a prompt, not metadata) cut wrong answers 38 to 9 but asked back 33 times and lost all 11 plainly correct answers |
| 4 | Evaluate with the data’s meaning (tested); add code-checked reference answers for each reading (untested as a judge input) | Judges passing wrong answers | Pre-release tests of extracted figures against computed values | An accurate, author-written dictionary let a strong judge name 40 of 44 mislabels (i) with or without the intended reading (17 to 23 of 61 false alarms, against 15 with the data alone); without the reading, silent readings (ii) passed (42 of 52); a check that quoted percentages appear in the tool output passed 31 of 35 misleading answers (exploratory) |
| 5 | Label every figure with the interpretation used: variant, measure, period, as-of date, fees, tax basis | All four | State plan, option, return method and pre- or post-tax | Not tested (the central principle) |
| 6 | Order alternatives by the user’s own holding, not by layout or search rank | (iii), (ii) | The investor’s plan; else a fixed, disclosed order, never the plan paying the seller more | Not tested; effect observed (Study 2) |
Table 12. Controls for a financial assistant: jurisdiction-agnostic control, failure addressed, an Indian example, and the evidence our studies give for it. Rows 1 and 2 have planned evidence for the failing models only; row 4 has planned evidence for the data dictionary, not for reference answers; row 3 has planned evidence only for a prompt-level check that over-asked, and exploratory evidence for metadata; rows 5–8 are untested.