cs.DBNov 3, 2026
SaveText-to-SQL Models Make Choices Without Saying So
Organizations: University of California, Berkeley Berkeley, CA, USA · School of Data Science The Chinese University of Hong Kong, Shenzhen Shenzhen, China
Abstract
A text-to-SQL answer can run, return a plausible number and still compute something other than what the analyst meant. This can happen when the request leaves a calculation choice open, such as whether to count rows or distinct entities, and the model settles it without saying so. Standard benchmarks cannot detect this, because they score each question against a single reference interpretation. We introduce SpecBridge, which tests such a choice by varying it alone: it poses three otherwise identical requests, one leaving the choice open and one per interpretation, and scores each answer by running it. The databases are real, and the choices are ones that made models fail on existing benchmarks. On 204 hand-validated tests, four language models usually compute each interpretation when asked for it, yet for the open request return both in only 15-58% of the tests where they can, often without mentioning the choice. The gap lies in reporting, not in computing. Benchmarks even penalize the alternative: on BIRD questions where such a choice changes the answer, over two in five of the answers marked wrong are right under the other interpretation. Because the gap lies in reporting, a system can narrow it without a stronger model: it first decides whether a request leaves a choice open, then answers each interpretation in its own call. On our tests and on AMBROSIA, an existing benchmark of ambiguous questions, this approach raises the share of open requests that get every interpretation by 29-63 points over the single answer that production assistants give, without lowering precision on clear requests. It also needs fewer user turns than asking a clarifying question.
Figures & tables
Figure 1. One request, its two correct interpretations (shaded: the SQL that differs), its request set, and four models’ responses. A request on the film table leads to two interpretations, A (mean of daily prices) and B (ratio of means): the SQL expression that differs, and their different per-rating values. Below, the three requests of the request set, and a table of four models' recorded responses to the open request.
| AmbiQT ( Bhaskar et al., 2023 ) | AMBROSIA ( Saparina and Lapata, 2024 ) | SpecBridge | |
|---|---|---|---|
| Where the choices lie | schema | wording | calculation |
| Benchmark databases, unmodified | |||
| Scores the explanation | |||
| Separates computing from reporting | possible | ||
| Penalizes raising a settled choice |
Table 1. SpecBridge and the two closest benchmarks of ambiguous text-to-SQL. Schema: synonym or split columns and tables; wording: scope, attachment, vagueness; calculation: ties, counting units, missing values, aggregation level. Possible: AMBROSIA’s data allow it, but its scores do not (Section 5 ).
Figure 2. The SpecBridge pipeline. Two lanes of three boxes, Construct (catalogue, generate SQL, describe) and Mine (benchmark questions, diagnose, decision library), both feed a bottom lane: apply a choice, write three requests, evaluate.
Figure 3. A request set on insider , from the generated query to its checks. Shaded: the SQL that differs. Five stacked boxes: the generated query with its edited step, the two interpretations and their executed difference, the retrieved library example, the three requests, and the checks passed.
Figure 4. Choice types: an example open phrase with its two interpretations, choices found in benchmark failures (gray: open; white: settled), open BIRD and Spider questions whose reference takes the rarer interpretation, and accepted request sets. One row per choice type with an example phrase, its two interpretations in blue and orange, and three bar panels: choices found in failures, references taking the rarer interpretation, and request sets.
| Request, model | Main SQL | Alternative SQL | Names | Both | False alarm |
|---|---|---|---|---|---|
| Film, open, GPT-Sol | A: mean of daily prices | none | |||
| Film, open, GPT-Luna | A: mean of daily prices | none | |||
| Awards, open, Sonnet | A: COUNT(*) | B: COUNT(DISTINCT player_id) | |||
| Districts, specific, Sonnet | A, as asked | none |
Table 2. Four recorded responses and their scores.
| Use | Data | Sections |
|---|---|---|
| Request sets | 12 databases, four each from Spider (baseball, Sakila, customers-and-invoices, Chinook), BIRD (financial, student club, thrombosis prediction, Formula 1) and LiveSQLBench (archeology, insider, crypto, alien) | 2.4 , 5 , 7 |
| Mining choices | 481 BIRD development questions on five databases and 150 Spider development questions | 2.3 |
| Held-out test | 100 BIRD and 50 Spider development questions, never used for mining | 6 |
| Conventions | every open BIRD and Spider question, training and development | 2.3 , 6.1 |
| Answer changes | every open BIRD development question | 6.2 |
| Method | AMBROSIA test split, ambiguous and unambiguous questions | 5 , 7 |
Table 3. The data and where each is used.
Figure 5. What each construction stage keeps, from proposed choices to accepted request sets. Horizontal bars decreasing from proposed choices to accepted request sets.
Figure 6. Answering and communicating on 204 request sets, per model and measure (95% intervals). Dot plot of five measures for four models with 95 percent intervals.
Figure 7. Missed interpretations that the same model computes when asked for each of them alone, per model, on request sets with one call or after exploring the database, and on AMBROSIA (Wilson 95% intervals). Dot plot with one row per model and three markers per row; most markers lie between 50 and 95 percent.
Figure 8. What each model does with the 204 open requests (one call with explanation). Computable: the same call answers both specific requests correctly. Stacked horizontal bars, one per model, split into five outcomes; silent answers with both interpretations computable form the largest part for Haiku and GPT-Luna.
Figure 9. Open requests answered with both interpretations, by choice type and by source of the database, among the sets in which the same model answers both specific requests right (accepted sets in parentheses; Wilson 95% intervals). Dot plot with five choice types and four models; cutoffs are high for three models, ties and counting units are low for all.
Figure 10. Main answer to the open request on 61 tie and counting sets: the interpretation BIRD and Spider references take by convention, the other interpretation, or neither. Stacked horizontal bars per model; the conventional interpretation outnumbers the other interpretation by at least three to one for every model.
Figure 11. Held-out questions: original question versus the exact description of the reference SQL (95% intervals). Grouped bars for four models on BIRD and Spider held-out questions under three conditions.
Figure 12. Why the original question failed, on development questions answered right from the exact description and wrong as worded, one cause per question. Eight causes, one row each, with a bar for BIRD and one for Spider; open choices are shown in red.
Figure 13. Share of open questions whose reference takes the usual interpretation, by author group, for the two choices on which groups disagree, ties and counting. Two panels of horizontal bars. BIRD and Spider bars are at 91 to 99 percent; the imported Spider groups are at 9 percent (ties) and 0 percent (counting).
| Question | BIRD reference | Other interpretation |
|---|---|---|
| “Who is the tallest superhero?” | 1 hero | 2 tied heroes |
| “… cards in the set Coldsnap with the highest converted mana cost.” | 1 card | 12 tied cards |
| “What are the expenses of the budget with the lowest remaining?” | 1 expense | 2 tied expenses |
| “How many of the banned cards are white border?” | 258 rows | 89 cards |
| “… normal glutamic pylvic transaminase level, how many of them are male?” | 1,924 lab records | 45 patients |
| “… diagnosed with SJS, how many of them have a normal level of total protein?” | 780 lab records | 37 patients |
Table 4. Six of the 17 BIRD questions on which all four models return the other interpretation, confirmed by two authors; BIRD scores each of these answers wrong.
Figure 14. Open BIRD development questions whose answer changes under the other interpretation, by choice type and by how far a single answer value moves. Stacked horizontal bars for five choice types; inclusive bounds change 40 percent of answers and counting rows versus entities 29 percent, mostly with values moving by more than half.
Figure 15. Both phases on the film request. Blue and orange: interpretations A and B, as in Figure 1 . Four steps: decide whether a choice is open and list the interpretations, answer each in its own call, run both and compare the results, show one answer with the choice and carry the user's pick as one added clause.
Figure 16. Methods on 150 new AMBROSIA test questions per model; calls per question beside each point (95% intervals). Two scatter panels, Sonnet and GPT-Sol: decide-then-search, decide-then-answer, plain, three samples, DtP and single answer.
| AMBROSIA | Request sets | ||||||
|---|---|---|---|---|---|---|---|
| Model | Method | Cover | Prec. | Calls | Cover | Prec. | Calls |
| Sonnet | Plain | 26 | 49.1 | 1.0 | 64 | 94.6 | 1.0 |
| Three samples | 31 | 49.0 | 3.0 | 72 | 92.0 | 3.0 | |
| DtP | 30 | 47.1 | 8.0 | 10 | 18.6 | 8.0 | |
| Single answer | 9 | 65.0 | 1.0 | 0 | 98.2 | 1.0 | |
| Decide-then-answer | 32 | 62.6 | 3.1 | 78 | 96.1 | 2.6 | |
Table 5. Decide-then-answer , Decide-then-search and the baselines of Section 4.1 , measured the same way on both datasets: coverage (all interpretations returned, of 90 ambiguous AMBROSIA questions and 111 open requests), precision on clear or specific requests (%), and calls per request. Bold: best per model.
Figure 17. Editing versus rewriting a request (363 interpretations). Left: interpretations computed right from four request texts. Right: what the rewrite lost, by kind. Left: dot plot per model of accuracy for four request texts; the two rewrites lie below the edited request. Right: horizontal bars, formula largest.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1. Data and experiments: sources (top), data built from them (middle), and the experiments with their sample sizes (bottom). Arrows show what is built from what and what each experiment uses. Three rows of boxes. Sources: databases, models, BIRD and Spider questions, AMBROSIA. Built data: generated tasks, request sets, choice library. Experiments: generator, reporting, benchmarks, two-phase method, each with sample sizes.
| Library | References | |||||
|---|---|---|---|---|---|---|
| Choice type | The choice | Open | Settled | Usual | Other (%) | Sets |
| Aggregation level | Rows or group averages? | 6 | 2 | 244 | 15 (5.8) | 61 |
| Calculation | Which formula is meant? | 1 | 1 | 613 | 1 (0.2) | 0 |
| Quantifier | “Than any”: one or all? | 3 | 0 | 8 | 7 (46.7) | 0 |
| Condition linkage | Conditions on one row? | 1 | 0 | 260 | 0 (0.0) | 0 |
| Counting unit | Count rows or entities? | 7 | 27 | 2,521 | 237 (8.6) | 47 |
Table S1. All choice types (Figure 4 shows the major ones). Library: choices of meaning in benchmark failures, open (the wording fits both interpretations) or settled (it selects one). References: open BIRD and Spider questions, by the interpretation their reference SQL takes: the usual one, or the other with its share (%), after the less common interpretations were reviewed; –: not readable from the SQL. Sets: the 204 accepted request sets, of the types in which at least 1% of open questions have a reference that takes the other interpretation. The library’s value-representation type is split by its SQL edit: comparison strictness is a strict cutoff, any other edit value matching.
| Library choice type | Choices | DBs | Found in |
|---|---|---|---|
| Aggregation level | 8 | 4 | BIRD |
| Calculation | 2 | 2 | BIRD |
| Condition linkage | 1 | 1 | BIRD |
| Counting unit | 34 | 4 | BIRD, Spider |
| Missing values | 11 | 5 | BIRD, Spider |
| Negation | 1 | 1 | BIRD |
Table S2. Top: every library choice (Table S1 ), by the benchmark whose failures revealed it. Bottom: every accepted request set, by the source of its database. DBs: distinct databases.
| Domain | Case | Choice and consequence | Library type |
|---|---|---|---|
| Economics and policy | Public debt and growth ( Herndon et al., 2014 ) | Growth was averaged within each country before averaging across countries, so one year of New Zealand data weighed as much as decades of another country. With corrected data and weighting, high-debt countries grew 2.2% a year, not . | aggregation level |
| Health care | Risk algorithm for extra care ( Obermeyer et al., 2019 ) | “Need” was computed as health-care cost. The algorithm looked accurate by standard measures, yet fixing the proxy would raise the share of Black patients receiving extra help from 17.7% to 46.5%. | referent column |
| Health care | Hospital mortality rates ( Shahian et al., 2010 ) | Four methods, given the same 2.5 million discharges, included 28–95% of them. 12 of 28 hospitals with higher-than-expected mortality by one method had lower-than-expected mortality by another. | population scope |
| Public health | 1995 UK contraceptive pill warning ( Furedi, 1999 ) | A warning of higher thrombosis risk for two pill types was followed by pill use among under-16s falling from 40% to 27%, and by estimated NHS costs of £21 million for maternity care and £46 million for abortions. The author concludes that risk should be more carefully presented. | how a quantity is stated (not in our library) |
| Social science | Referees and skin tone ( Silberzahn et al., 2018 ) | 29 teams analysed the same data. Their odds ratios ranged from 0.89 to 2.93; 20 teams found a significant effect and 9 did not, using 21 combinations of covariates. | several analysis choices |
| Science | Brain imaging ( Botvinik-Nezer et al., 2020 ) | 70 teams analysed one dataset to test the same 9 hypotheses. No two chose the same workflow, and the results of the hypothesis tests varied substantially. | several analysis choices |
Table S3. Documented cases where one calculation choice changed a consequential number or decision, by domain, with the closest choice type in our library.
| Model | Both right | Both returned | Named | Right | Missed, computable |
|---|---|---|---|---|---|
| Haiku | 42 | 33 | 40 | 40 | 9 / 10 |
| Sonnet | 41 | 36 | 38 | 42 | 5 / 6 |
| GPT-Sol | 42 | 32 | 35 | 40 | 10 / 10 |
| GPT-Luna | 38 | 14 | 25 | 41 | 24 / 29 |
Table S4. Filter request sets (43 accepted), per model: both specific requests right (with exploration); among those, open requests answered with both interpretations; open requests that name the choice (Opus judge); specific requests right (single call); missed interpretations the model computes when asked, of all missed.
Figure S2. Generation outcomes with each construction rule removed (2,350 attempts). The labels give the share valid, on target and sound. Stacked bars of generation outcomes for nine configurations.
Figure S3. Exact description versus natural request on 382 generated tasks. Left: accuracy; numbers give the difference in points. Right: errors from the exact description by step depth. Bars for four models under two request forms, and lines of error rate by step depth.
Figure S4. Features the generator can build, as a share of 152 databases: possible by the schema, and generated and accepted. Paired horizontal bars for eleven features: possible by the schema versus generated and accepted.
| Source | DBs | Delivered | Constructions | Seconds |
|---|---|---|---|---|
| Spider | 4 | 398/400 | 289 | 127 |
| BIRD | 4 | 400/400 | 270 | 274 |
| LiveSQLBench | 4 | 400/400 | 309 | 37 |
| Total | 12 | 1198/1200 | 868 | 437 |
Table S5. Program construction on 12 databases, with no model calls after catalogue preparation. Every delivered SQL is distinct, has the requested depth, and passes both integrity checks.
More in this topic
Text-to-SQL
Text-to-SQL translates natural-language questions into executable database queries, linking user intent to schemas, values, and SQL logic.
See all 185 papers