Organizations: University of California, Berkeley Berkeley, CA, USA · School of Data Science The Chinese University of Hong Kong, Shenzhen Shenzhen, China
A text-to-SQL answer can run, return a plausible number and still compute something other than what the analyst meant. This can happen when the request leaves a calculation choice open, such as whether to count rows or distinct entities, and the model settles it without saying so. Standard benchmarks cannot detect this, because they score each question against a single reference interpretation. We introduce SpecBridge, which tests such a choice by varying it alone: it poses three otherwise identical requests, one leaving the choice open and one per interpretation, and scores each answer by running it. The databases are real, and the choices are ones that made models fail on existing benchmarks. On 204 hand-validated tests, four language models usually compute each interpretation when asked for it, yet for the open request return both in only 15-58% of the tests where they can, often without mentioning the choice. The gap lies in reporting, not in computing. Benchmarks even penalize the alternative: on BIRD questions where such a choice changes the answer, over two in five of the answers marked wrong are right under the other interpretation. Because the gap lies in reporting, a system can narrow it without a stronger model: it first decides whether a request leaves a choice open, then answers each interpretation in its own call. On our tests and on AMBROSIA, an existing benchmark of ambiguous questions, this approach raises the share of open requests that get every interpretation by 29-63 points over the single answer that production assistants give, without lowering precision on clear requests. It also needs fewer user turns than asking a clarifying question.
Figures & tables
Figure 1. One request, its two correct interpretations (shaded: the SQL that differs), its request set, and four models’ responses. A request on the film table leads to two interpretations, A (mean of daily prices) and B (ratio of means): the SQL expression that differs, and their different per-rating values. Below, the three requests of the request set, and a table of four models' recorded responses to the open request.
AmbiQT ( Bhaskar et al., 2023 )
AMBROSIA ( Saparina and Lapata, 2024 )
SpecBridge
Where the choices lie
schema
wording
calculation
Benchmark databases, unmodified
×
×
✓
Scores the explanation
×
×
✓
Separates computing from reporting
×
possible
✓
Penalizes raising a settled choice
×
✓
✓
Table 1. SpecBridge and the two closest benchmarks of ambiguous text-to-SQL. Schema: synonym or split columns and tables; wording: scope, attachment, vagueness; calculation: ties, counting units, missing values, aggregation level. Possible: AMBROSIA’s data allow it, but its scores do not (Section 5 ).
Figure 2. The SpecBridge pipeline. Two lanes of three boxes, Construct (catalogue, generate SQL, describe) and Mine (benchmark questions, diagnose, decision library), both feed a bottom lane: apply a choice, write three requests, evaluate.
Figure 3. A request set on insider , from the generated query to its checks. Shaded: the SQL that differs. Five stacked boxes: the generated query with its edited step, the two interpretations and their executed difference, the retrieved library example, the three requests, and the checks passed.
Figure 4. Choice types: an example open phrase with its two interpretations, choices found in benchmark failures (gray: open; white: settled), open BIRD and Spider questions whose reference takes the rarer interpretation, and accepted request sets. One row per choice type with an example phrase, its two interpretations in blue and orange, and three bar panels: choices found in failures, references taking the rarer interpretation, and request sets.
Request, model
Main SQL
Alternative SQL
Names
Both
False alarm
Film, open, GPT-Sol
A: mean of daily prices
none
×
×
Film, open, GPT-Luna
A: mean of daily prices
none
✓
×
Awards, open, Sonnet
A: COUNT(*)
B: COUNT(DISTINCT player_id)
✓
✓
Districts, specific, Sonnet
A, as asked
none
✓
Table 2. Four recorded responses and their scores.
Use
Data
Sections
Request sets
12 databases, four each from Spider (baseball, Sakila, customers-and-invoices, Chinook), BIRD (financial, student club, thrombosis prediction, Formula 1) and LiveSQLBench (archeology, insider, crypto, alien)
2.4 , 5 , 7
Mining choices
481 BIRD development questions on five databases and 150 Spider development questions
2.3
Held-out test
100 BIRD and 50 Spider development questions, never used for mining
6
Conventions
every open BIRD and Spider question, training and development
2.3 , 6.1
Answer changes
every open BIRD development question
6.2
Method
AMBROSIA test split, ambiguous and unambiguous questions
5 , 7
Table 3. The data and where each is used.
Figure 5. What each construction stage keeps, from proposed choices to accepted request sets. Horizontal bars decreasing from proposed choices to accepted request sets.
Figure 6. Answering and communicating on 204 request sets, per model and measure (95% intervals). Dot plot of five measures for four models with 95 percent intervals.
Figure 7. Missed interpretations that the same model computes when asked for each of them alone, per model, on request sets with one call or after exploring the database, and on AMBROSIA (Wilson 95% intervals). Dot plot with one row per model and three markers per row; most markers lie between 50 and 95 percent.
Figure 8. What each model does with the 204 open requests (one call with explanation). Computable: the same call answers both specific requests correctly. Stacked horizontal bars, one per model, split into five outcomes; silent answers with both interpretations computable form the largest part for Haiku and GPT-Luna.
Figure 9. Open requests answered with both interpretations, by choice type and by source of the database, among the sets in which the same model answers both specific requests right (accepted sets in parentheses; Wilson 95% intervals). Dot plot with five choice types and four models; cutoffs are high for three models, ties and counting units are low for all.
Figure 10. Main answer to the open request on 61 tie and counting sets: the interpretation BIRD and Spider references take by convention, the other interpretation, or neither. Stacked horizontal bars per model; the conventional interpretation outnumbers the other interpretation by at least three to one for every model.
Figure 11. Held-out questions: original question versus the exact description of the reference SQL (95% intervals). Grouped bars for four models on BIRD and Spider held-out questions under three conditions.
Figure 12. Why the original question failed, on development questions answered right from the exact description and wrong as worded, one cause per question. Eight causes, one row each, with a bar for BIRD and one for Spider; open choices are shown in red.
Figure 13. Share of open questions whose reference takes the usual interpretation, by author group, for the two choices on which groups disagree, ties and counting. Two panels of horizontal bars. BIRD and Spider bars are at 91 to 99 percent; the imported Spider groups are at 9 percent (ties) and 0 percent (counting).
Question
BIRD reference
Other interpretation
“Who is the tallest superhero?”
1 hero
2 tied heroes
“… cards in the set Coldsnap with the highest converted mana cost.”
1 card
12 tied cards
“What are the expenses of the budget with the lowest remaining?”
1 expense
2 tied expenses
“How many of the banned cards are white border?”
258 rows
89 cards
“… normal glutamic pylvic transaminase level, how many of them are male?”
1,924 lab records
45 patients
“… diagnosed with SJS, how many of them have a normal level of total protein?”
780 lab records
37 patients
Table 4. Six of the 17 BIRD questions on which all four models return the other interpretation, confirmed by two authors; BIRD scores each of these answers wrong.
Figure 14. Open BIRD development questions whose answer changes under the other interpretation, by choice type and by how far a single answer value moves. Stacked horizontal bars for five choice types; inclusive bounds change 40 percent of answers and counting rows versus entities 29 percent, mostly with values moving by more than half.
Figure 15. Both phases on the film request. Blue and orange: interpretations A and B, as in Figure 1 . Four steps: decide whether a choice is open and list the interpretations, answer each in its own call, run both and compare the results, show one answer with the choice and carry the user's pick as one added clause.
Figure 16. Methods on 150 new AMBROSIA test questions per model; calls per question beside each point (95% intervals). Two scatter panels, Sonnet and GPT-Sol: decide-then-search, decide-then-answer, plain, three samples, DtP and single answer.
AMBROSIA
Request sets
Model
Method
Cover
Prec.
Calls
Cover
Prec.
Calls
Sonnet
Plain
26
49.1
1.0
64
94.6
1.0
Three samples
31
49.0
3.0
72
92.0
3.0
DtP
30
47.1
8.0
10
18.6
8.0
Single answer
9
65.0
1.0
0
98.2
1.0
Decide-then-answer
32
62.6
3.1
78
96.1
2.6
Table 5. Decide-then-answer , Decide-then-search and the baselines of Section 4.1 , measured the same way on both datasets: coverage (all interpretations returned, of 90 ambiguous AMBROSIA questions and 111 open requests), precision on clear or specific requests (%), and calls per request. Bold: best per model.
Figure 17. Editing versus rewriting a request (363 interpretations). Left: interpretations computed right from four request texts. Right: what the rewrite lost, by kind. Left: dot plot per model of accuracy for four request texts; the two rewrites lie below the edited request. Right: horizontal bars, formula largest.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1. Data and experiments: sources (top), data built from them (middle), and the experiments with their sample sizes (bottom). Arrows show what is built from what and what each experiment uses. Three rows of boxes. Sources: databases, models, BIRD and Spider questions, AMBROSIA. Built data: generated tasks, request sets, choice library. Experiments: generator, reporting, benchmarks, two-phase method, each with sample sizes.
Library
References
Choice type
The choice
Open
Settled
Usual
Other (%)
Sets
Aggregation level
Rows or group averages?
6
2
244
15 (5.8)
61
Calculation
Which formula is meant?
1
1
613
1 (0.2)
0
Quantifier
“Than any”: one or all?
3
0
8
7 (46.7)
0
Condition linkage
Conditions on one row?
1
0
260
0 (0.0)
0
Counting unit
Count rows or entities?
7
27
2,521
237 (8.6)
47
Appendix
Table S1. All choice types (Figure 4 shows the major ones). Library: choices of meaning in benchmark failures, open (the wording fits both interpretations) or settled (it selects one). References: open BIRD and Spider questions, by the interpretation their reference SQL takes: the usual one, or the other with its share (%), after the less common interpretations were reviewed; –: not readable from the SQL. Sets: the 204 accepted request sets, of the types in which at least 1% of open questions have a reference that takes the other interpretation. The library’s value-representation type is split by its SQL edit: comparison strictness is a strict cutoff, any other edit value matching.
Library choice type
Choices
DBs
Found in
Aggregation level
8
4
BIRD
Calculation
2
2
BIRD
Condition linkage
1
1
BIRD
Counting unit
34
4
BIRD, Spider
Missing values
11
5
BIRD, Spider
Negation
1
1
BIRD
Appendix
Table S2. Top: every library choice (Table S1 ), by the benchmark whose failures revealed it. Bottom: every accepted request set, by the source of its database. DBs: distinct databases.
Domain
Case
Choice and consequence
Library type
Economics and policy
Public debt and growth ( Herndon et al., 2014 )
Growth was averaged within each country before averaging across countries, so one year of New Zealand data weighed as much as decades of another country. With corrected data and weighting, high-debt countries grew 2.2% a year, not −0.1% .
aggregation level
Health care
Risk algorithm for extra care ( Obermeyer et al., 2019 )
“Need” was computed as health-care cost. The algorithm looked accurate by standard measures, yet fixing the proxy would raise the share of Black patients receiving extra help from 17.7% to 46.5%.
referent column
Health care
Hospital mortality rates ( Shahian et al., 2010 )
Four methods, given the same 2.5 million discharges, included 28–95% of them. 12 of 28 hospitals with higher-than-expected mortality by one method had lower-than-expected mortality by another.
population scope
Public health
1995 UK contraceptive pill warning ( Furedi, 1999 )
A warning of higher thrombosis risk for two pill types was followed by pill use among under-16s falling from 40% to 27%, and by estimated NHS costs of £21 million for maternity care and £46 million for abortions. The author concludes that risk should be more carefully presented.
how a quantity is stated (not in our library)
Social science
Referees and skin tone ( Silberzahn et al., 2018 )
29 teams analysed the same data. Their odds ratios ranged from 0.89 to 2.93; 20 teams found a significant effect and 9 did not, using 21 combinations of covariates.
several analysis choices
Science
Brain imaging ( Botvinik-Nezer et al., 2020 )
70 teams analysed one dataset to test the same 9 hypotheses. No two chose the same workflow, and the results of the hypothesis tests varied substantially.
several analysis choices
Appendix
Table S3. Documented cases where one calculation choice changed a consequential number or decision, by domain, with the closest choice type in our library.
Model
Both right
Both returned
Named
Right
Missed, computable
Haiku
42
33
40
40
9 / 10
Sonnet
41
36
38
42
5 / 6
GPT-Sol
42
32
35
40
10 / 10
GPT-Luna
38
14
25
41
24 / 29
Appendix
Table S4. Filter request sets (43 accepted), per model: both specific requests right (with exploration); among those, open requests answered with both interpretations; open requests that name the choice (Opus judge); specific requests right (single call); missed interpretations the model computes when asked, of all missed.
Figure S2. Generation outcomes with each construction rule removed (2,350 attempts). The labels give the share valid, on target and sound. Stacked bars of generation outcomes for nine configurations.
Figure S3. Exact description versus natural request on 382 generated tasks. Left: accuracy; numbers give the difference in points. Right: errors from the exact description by step depth. Bars for four models under two request forms, and lines of error rate by step depth.
Figure S4. Features the generator can build, as a share of 152 databases: possible by the schema, and generated and accepted. Paired horizontal bars for eleven features: possible by the schema versus generated and accepted.
Source
DBs
Delivered
Constructions
Seconds
Spider
4
398/400
289
127
BIRD
4
400/400
270
274
LiveSQLBench
4
400/400
309
37
Total
12
1198/1200
868
437
Appendix
Table S5. Program construction on 12 databases, with no model calls after catalogue preparation. Every delivered SQL is distinct, has the requested depth, and passes both integrity checks.
Agentic Text-to-SQL systems can interact with users to clarify underspecified queries before generating SQL. However, a correct execution result does not necessarily imply that the agent has adequately resolved the underlying underspecification: the agent may silently make unverified assumptions that happen to match the intended answer. We show that this behavior is driven in part by premature clarification termination. Although forcing an agent to ask more questions improves execution accuracy, ambiguities are concentrated in earlier interactions, making brute-force questioning inefficient. More importantly, even when explicitly prompted to plan its clarification process, the agent frequently abandons questions that it has already identified as relevant. To address this failure mode, we introduce PlanPool, which externalizes the clarification plan as a mutable question pool. Every planned question must be explicitly asked or dropped before submission, while newly discovered ambiguities can be added during interaction. Across three benchmarks derived from BIRD-Interact and Spider, PlanPool consistently improves ambiguity coverage and reduces silent failures over unconstrained and prompt-based alternatives, while maintaining competitive execution accuracy. Our results highlight an important distinction in agentic reasoning: identifying missing information is not sufficient, and the agent must also reliably maintain and resolve it before committing to an answer.
Wen-Zhi Li, Yue Gong, Konstantinos Kanellis +1
Department of Computer Science Cornell University · Amazon Web Services
Organizations that cannot send data to a cloud API need to know how well self-hosted open-weight models perform on Text-to-SQL and which inexpensive techniques justify their compute. We benchmark Qwen2.5-Coder-Instruct (7B, 14B, 32B) and CodeLlama-Instruct (7B, 13B) self-hosted at 16-bit precision, Llama-3.1-8B and Llama-3.3-70B through an FP8 API, and Qwen2.5-Coder-32B through a second API on the BIRD development split (n = 1534), with one zero-shot prompt and the official execution-accuracy comparison on read-only databases. We ablate lexical and embedding schema linking, execution-guided self-correction and self-consistency, and test every within-model recipe difference with exact McNemar tests, Holm correction over 30 tests and a database-clustered bootstrap. Qwen2.5-Coder reaches 42.6%, 52.0% and 54.9%; CodeLlama-7B and -13B reach 23.7% and 26.3%. Self-correction is the only component with a consistent gain: within a run it adds 1.2 to 4.4 points at 1.1 to 1.9 times the tokens, and it is significant after Holm correction for every model except Llama-3.3-70B. Schema linking never helps significantly; lexical linking costs Llama-3.3-70B 4.2 points, and an embedding linker with higher table recall does not differ detectably from no linking. Self-consistency, run for one model, changed accuracy by at most 0.13 points at about five times the tokens. One API provider dropped spaces next to digits; after a whitespace repair the same model scored 54.5% through the API and 54.9% self-hosted, so an apparent three-point serving gap was mostly this defect. This version corrects six defects of v1, including writable databases, a stricter-than-official metric and a truncated API arm, and withdraws the v1 claim of a quantization penalty of about 13 points; all numbers are re-scored or re-run. Code, per-item predictions and analysis scripts are released.
Vladimir Beskorovainyi
Besk Tech | Moscow Institute of Physics and Technology (MIPT)
As text-to-SQL systems move beyond demonstrations toward real-world deployment, ambiguity in user questions becomes a primary source of errors. Such ambiguities are often subtle, domain- or data-specific, and can silently cause system outputs to deviate from the user's true intent. Ambiguity is traditionally addressed through conversational clarification, which is often inefficient, cognitively demanding, and poorly aligned with real-world user workflows. We propose structured disambiguation, a new paradigm in which ambiguity is resolved through explicit, constrained interactions rather than free-form dialogue. We construct ARCS (Ambiguity Resolution Corpus for SQL), the first text-to-SQL benchmark featuring naturally occurring, unconstrained ambiguities over real-world databases, with complete annotations of all valid ambiguity points, interpretations, and SQL queries. Experimental results show that text-to-SQL remains challenging in the presence of ambiguity: gpt-6-sol achieves only 51% end-to-end execution accuracy, and no open-source model exceeds 27%.