Beyond Correctness: Resolving Underspecification in Agentic Text-to-SQL

Wen-Zhi Li; Yue Gong; Konstantinos Kanellis; Balakrishnan Murali Narayanaswamy

cs.CLOct 2, 2026

Beyond Correctness: Resolving Underspecification in Agentic Text-to-SQL

Authors: Wen-Zhi Li, Yue Gong, Konstantinos Kanellis, Balakrishnan Murali Narayanaswamy

Organizations: Department of Computer Science Cornell University · Amazon Web Services

Abstract

Agentic Text-to-SQL systems can interact with users to clarify underspecified queries before generating SQL. However, a correct execution result does not necessarily imply that the agent has adequately resolved the underlying underspecification: the agent may silently make unverified assumptions that happen to match the intended answer. We show that this behavior is driven in part by premature clarification termination. Although forcing an agent to ask more questions improves execution accuracy, ambiguities are concentrated in earlier interactions, making brute-force questioning inefficient. More importantly, even when explicitly prompted to plan its clarification process, the agent frequently abandons questions that it has already identified as relevant. To address this failure mode, we introduce PlanPool, which externalizes the clarification plan as a mutable question pool. Every planned question must be explicitly asked or dropped before submission, while newly discovered ambiguities can be added during interaction. Across three benchmarks derived from BIRD-Interact and Spider, PlanPool consistently improves ambiguity coverage and reduces silent failures over unconstrained and prompt-based alternatives, while maintaining competitive execution accuracy. Our results highlight an important distinction in agentic reasoning: identifying missing information is not sufficient, and the agent must also reliably maintain and resolve it before committing to an answer.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Execution accuracy comparison on three datasets in single- and multiple-turn settings. (b) Ground-truth annotated ambiguity distributions versus ambiguities detected through multiple-turn interaction.

Panel of Figure 2 — Figure 2: (a) Execution accuracy scaling with the number of clarification questions $k$ . (b) Cumulative composition of asked clarification questions as $k$ increases. (c), (d) Planned versus actually asked question distributions for prompting-based plan mode on BIRD-Lite and BIRD-Full.

Statistic	BIRD-Lite	BIRD-Full	Spider
# Instances	171	225	297
# Distinct Databases	18	22	19
# Tokens / User Query	31.8	24.0	14.3
# Tokens / SQL	149.5	78.5	18.1
# Ambiguities / Task	3.56	2.77	2.08

Table 1: Dataset statistics.

Dataset	Method	EA $\uparrow$	Recall $\uparrow$	GS $\uparrow$	US $\downarrow$	SF $\downarrow$	#Asks $\downarrow$
BIRD-Lite	Naive	74.3	73.9	33.9	40.4	17.0	2.3
	Read-All	69.6	70.8	29.8	39.8	18.7	2.0
	Regenerate	74.9	73.7	31.6	43.3	15.2	2.3
	Draft	75.4	71.6	28.7	46.8	19.9	2.5
	Self-Reflection	77.2	75.5	35.7	41.5	16.4	2.7
	Plan-Prompt	79.5	80.8	42.1	37.4	10.5	3.3

Table 2: Method comparison across datasets.

Statistic	BIRD-Lite	BIRD-Full	Spider
Initial Pool Size	3.9	3.3	2.3
Actual Asks	4.2	3.6	2.5
Unmodified Pool	65.5%	61.8%	81.5%
Adding question usage	30.4%	32.9%	18.5%
Dropping question usage	4.7%	10.7%	0.7%

Table 3: PlanPool tool usage across datasets.

Panel of Figure 3 — Figure 3: Annotated versus detected ambiguity distributions produced by different methods, together with the KL divergence with respect to the annotated ground-truth distribution: (a) BIRD-Lite, (b) BIRD-Full, and (c) Spider.

Dataset	Method	Input Tok. $\downarrow$	Output Tok. $\downarrow$	Cost ( $)$ \downarrow$
BIRD-Lite	Naive	136k	2.4k	0.36
	Read-All	122k	1.8k	0.30
	Regenerate	169k	3.1k	0.35
	Draft	139k	2.7k	0.29
	Plan-Prompt	148k	2.6k	0.29
	PlanPool	201k	2.5k	0.32

Table 4: Computational cost across datasets. Token counts are averaged per query.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Construction pipeline for the multi-ambiguity Spider benchmark. Each applicable operator introduces one type of underspecification, after which the modified instance is validated and annotated.

Action	Observation
execute	SQL execution result
get_schema	Database schema with example data
get_all_column_meanings	Meanings of all columns
get_column_meaning	Meaning of the specified column
get_all_external_knowledge_names	Names of available external knowledge
get_knowledge_definition	Definition of the specified knowledge

Appendix

Table 5: Datasource interaction tools adopted from BIRD-Interact. We use the original tool interfaces but ignore the action-level costs defined in BIRD-Interact.

Step	Pool Action	Outcome
1	pool_next(Q0)	The user specifies that artifacts should be ranked directly by CPI rather than AVS: “Just use the Conservation Priority Index (CPI) itself—rank by CPI, no AVS needed.”
2	pool_drop(Q1)	As AVS is no longer needed, ERF becomes irrelevant. The agent explicitly removes Q1 instead of asking an unnecessary question, reasoning that CPI alone is sufficient.
3	pool_next(Q2)	The user clarifies that CPI should not be taken directly from ConservePriorityScore : “No, don’t use that column. CPI needs to be calculated from the artifact’s historical significance, research value, cultural score, and conservation status.”
4	pool_add(Q5)	The previous answer reveals a new clarification need. The agent adds the following question with highest priority: “What is the exact CPI formula? How are the ratings combined, and what is the numerical mapping for ConserveStatus?”
5	pool_next(Q5)	The user provides the specific formula to calculate CPI.
6	pool_next(Q3)	The user confirms that historical era corresponds to ArtDynasty .

Appendix

Table 6: Evolution of the clarification pool of PlanPool in the case study.

Panel of Figure 5 — Figure 5: Base system prompt shared by all agentic methods (Part 1/2).

Panel of Figure 6 — Figure 6: Base system prompt shared by all agentic methods (Part 2/2).

Panel of Figure 7 — Figure 7: Additional system instruction used by the Read-All baseline.

Panel of Figure 8 — Figure 8: Runtime observation injected by the Regenerate baseline after two independently generated SQL candidates diverge.

Panel of Figure 9 — Figure 9: Additional system instruction used by the Draft baseline.

Panel of Figure 10 — Figure 10: Additional system instruction used by the Self-Reflection baseline.

Panel of Figure 11 — Figure 11: Additional system instruction used by the Plan-Prompt baseline.

Panel of Figure 12 — Figure 12: Additional system instruction used by PlanPool .

Text-to-SQL

Text-to-SQL translates natural-language questions into executable database queries, linking user intent to schemas, values, and SQL logic.

See all 173 papers

Probing Ambiguity: Adversarial Intent Resolution for Robust Text-to-SQL
Oct 16, 2026Youbing Yin, Xujun Peng, Muhammad S. Uddin +3Text-to-SQL Execution-Guided Text-to-SQL
Capital One
ARCS: Towards Precise Text-to-SQL via Structured Disambiguation
Oct 7, 2026Yihao Hu, Yanlin Feng, Naoki Otani +1Text-to-SQL
αDuke University · βMegagon Labs
PV-SQL: Synergizing Database Probing and Rule-based Verification for Text-to-SQL Agents
Apr 19, 2026Yuan Tian, Tianyi ZhangText-to-SQL
Purdue University

Oct 16, 2026J/K move · Enter open · S save

Probing Ambiguity: Adversarial Intent Resolution for Robust Text-to-SQL

Youbing Yin, Xujun Peng, Muhammad S. Uddin, Erin Babinsky+2

Capital One

Text-to-SQL Execution-Guided Text-to-SQL Clarification Question Generation

Open paper AskPDF arXiv

Text-to-SQL systems typically assume a one-to-one mapping between natural-language queries and correct SQL translations. In practice, however, many queries are inherently ambiguous, admitting multiple valid interpretations that yield distinct outputs. We introduce Adversarial Intent Resolution (AIR), a multi-agent framework that actively probes and resolves such ambiguity without any task-specific training. AIR first constructs competing interpretations, verifies through SQL execution that they produce genuinely different results, and generates targeted clarification questions grounded in the observed data differences. Evaluating on leading benchmarks for ambiguous questions, AIR demonstrates strong performance, efficiency, and robust cross-benchmark transferability. On the AMBROSIA benchmark under schema-only context, a realistic production setting where exposing the full database is infeasible, AIR outperforms IntentRL, the state-of-the-art reinforcement learning model, across all metrics (e.g., F1 0.679 vs. 0.618). Crucially, AIR exhibits only ~7% performance degradation without the full database, compared to ~20% for IntentRL. On the AmbiQT benchmark, where ambiguity arises from schema design rather than linguistic structure, AIR achieves higher F1 under both context settings without any benchmark-specific adaptation.

Abstract

Text-to-SQL

Explore similar work

Probing Ambiguity: Adversarial Intent Resolution for Robust Text-to-SQL

ARCS: Towards Precise Text-to-SQL via Structured Disambiguation

PV-SQL: Synergizing Database Probing and Rule-based Verification for Text-to-SQL Agents