Evaluating LLMs on Conversational Text-to-SQL under Chain Ambiguity and Intent Drift

Yujia Liu; Jiayan Lin; Zijin Hong; Zheng Yuan; Shengyuan Chen; Hao Chen; Qinggang Zhang; Xiao Huang

cs.CLAug 30, 2026

Evaluating LLMs on Conversational Text-to-SQL under Chain Ambiguity and Intent Drift

Authors: Yujia Liu, Jiayan Lin, Zijin Hong, Zheng Yuan, Shengyuan Chen, Hao Chen, Qinggang Zhang, Xiao Huang,

Organizations: Jinan University · The Hong Kong Polytechnic University · City University of Macau · Jilin University ·

Abstract

Recent advances in large language models (LLMs) have established conversational text-to-SQL as a practical interface between users and databases, often involving multiple turns of clarification and revision. However, existing benchmarks primarily evaluate execution accuracy, leaving the unfolding and shifting of user intent across turns largely uncovered. To address this, we introduce TIDE-Bench, a benchmark for conversational text-to-SQL under chain ambiguity and intent drift evaluation, targeting two recurring patterns: chain ambiguity, where an underspecified question triggers layered clarification with conditional dependencies, and intent drift, where the user retracts and replaces a previously committed request element. Built on 514 anchor SQLs from BIRD, TIDE-Bench comprises 1,542 samples and introduces dedicated metrics for chain identification and drift recognition-resolution beyond execution accuracy. Evaluating 12 advanced LLMs reveals a persistent chain identification bottleneck unaffected by clarification frequency, a wide drift recognition-resolution gap, and overlap between failure modes when jointly activated. The corresponding code of TIDE-Bench is released for further research.

Figures & tables

Figure 1: (A) Two conversational patterns that challenge text-to-SQL. (B) Neither is covered by existing text-to-SQL benchmarks or standard evaluation protocols.

Figure 2: Left: motivating dialogue illustrating chain ambiguity (L1/L2 clarification with conditional dependency) and user intent drift (non-monotonic retraction of a prior commitment). Right: overview of the TIDE-Bench four-stage construction pipeline (anchor selection, chain injection, drift injection, and triplet assembly; see § 3 ).

Require: Queries $y_{\mathrm{pre}}$ and $y_{\mathrm{post}}$
Ensure: Drift status or subtype
Notation:
	$\mathcal{W}(y)$ : filtering commitments in $y$
	$\mathcal{O}(y)=S(y)\setminus\mathcal{W}(y)$ : output-side commitments in $y$
	$\operatorname{LitReplace}(\mathcal{W}_{\mathrm{pre}},\mathcal{W}_{\mathrm{post}})$ : one predicate literal changes; its structure and all other filters remain fixed
	% Extract commitments

Algorithm 1 User Intent Drift Operationalization

Split	Majority Acc.	Strict Acc.	Fleiss’ $\boldsymbol{\kappa}$
Chain-only	93.2	84.8	0.82
Drift-only	95.6	87.6	0.80
Joint	88.4	76.8	0.77
Overall	92.4	83.1	0.81

Table 1: Human validation on 250 stratified triplets. Majority requires

\geq 3/5

annotators per item; strict requires unanimity. Fleiss’

\kappa

uses binary accept/reject verdicts.

Metric	Full Name	Definition	Set
DEA	Dialogue Execution Accuracy	$\frac{1}{\|\mathcal{S}\|}\sum_{\tau\in\mathcal{S}}\prod_{i=1}^{n_{\tau}}\mathrm{EX}(y_{i},y_{i}^{*})$	all
AR	Ask Rate	$\frac{1}{\|\mathcal{S}\|}\sum_{\tau\in\mathcal{S}}\mathds{1}\!\left[\tau\text{ contains a clarification}\right]$	all
L1-IR	L1 Identification Rate	$\frac{1}{\|\mathcal{S}\|}\sum_{\tau\in\mathcal{S}}\mathds{1}\!\left[\exists\,t:\rho_{t}=\mathrm{L1}\right]$	$\mathcal{C},\mathcal{J}$
L2-IR p	L2 Identification Rate	$\frac{1}{\|\mathcal{S}\|}\sum_{\tau\in\mathcal{S}}\mathds{1}\!\left[\exists\,t:\rho_{t}=\mathrm{L2}\right]$	$\mathcal{C},\mathcal{J}$
JIR	Joint Identification Rate	$\frac{1}{\|\mathcal{S}\|}\sum_{\tau\in\mathcal{S}}\mathds{1}\!\left[\exists\,t_{1},t_{2}:\rho_{t_{1}}=\mathrm{L1}\wedge\rho_{t_{2}}=\mathrm{L2}\right]$	$\mathcal{C},\mathcal{J}$
DRR	Drift Recognition Rate	$\frac{1}{\|\mathcal{S}\|}\sum_{\tau\in\mathcal{S}}\mathds{1}\!\left[\mathrm{Rec}(\tau)\right]$	$\mathcal{D},\mathcal{J}$

Table 2: TIDE-Bench metrics spanning four behavioral facets: execution accuracy (DEA), clarification behavior (AR), chain-layer identification (L1-IR, L2-IR p , JIR), and drift handling (DRR, DRA).

\mathcal{C},\mathcal{D},\mathcal{J}

denote Chain-only, Drift-only, and Joint sample sets. L2-IR p uses the

p

-protocol (§ 4.1 ); all other metrics use the

f

-protocol instead.

#	Model	Chain-only	Drift-only				Joint
#	Model	Chain-only	Value	Constraint	Projection	All	Value	Constraint	Projection	All
1	Claude-Sonnet-4.6	35.9	47.7	44.2	43.4	45.1	31.2	20.2	31.4	27.8
2	Qwen2.5-72B	32.4	44.3	41.1	44.0	43.2	30.1	21.5	28.6	26.8
3	o4-mini	32.9	42.6	41.1	40.0	41.2	31.2	21.5	27.4	26.8
4	GPT-4o	30.9	46.0	38.7	44.6	43.2	27.8	22.1	29.7	26.7
5	GPT-5.4	32.5	43.2	36.8	42.3	40.9	27.8	20.9	29.7	26.3

Table 3: DEA (%) under the paired factorial design. The best and second-best results per column are in bold and underlined, respectively. Chain-only reports strict execution accuracy; Drift-only and Joint report end-to-end accuracy broken down by drift subtype; All denotes aggregate performance, and higher metric values are better.

Figure 3: Chain-layer identification on Chain-only samples. (a) L1-IR (

f

-protocol) vs. L2-IR p (

p

-protocol) per model: even with L1 pre-resolved, L2 identification stays low. (b) JIR against ask rate: identification remains low regardless of how much the agent asks. Model names are abbreviated; full names appear in Table 3 for reference.

Figure 4: Drift handling from Drift-only to Joint across DRR, DRA, and DEA, with one line per model. DRR remains high and flat, while DRA and DEA start lower and decline further in Joint. The dashed DRA reference marks DRR, highlighting the recognition–resolution gap that persists for all models across both interaction settings. The orange line highlights Claude-Sonnet-4.6, which achieves the highest DEA in both Drift-only and Joint.

Figure 5: Drift-only DRA by drift subtype. Points denote models, and the bar marks the corresponding subtype mean. Orange points mark per-subtype maxima.

[Context] Schema: coaches(coachID, year, tmID, won, lost) ; teams(year, tmID, won, lost) . Question: “Which coaches had disappointing seasons in 2000–2010? Show their coach ID, team, and year.”
[Chain Ambiguity] Gold path: $L1$ : overall losing record $\rightarrow$ $L2$ : team’s full-season record. Alternatives: $L1$ : away-game record; $L2$ : coach-stint record.
Gold SQL: SELECT DISTINCT c.coachID, t.tmID, c.year FROM coaches c JOIN teams t ON c.tmID=t.tmID AND c.year=t.year WHERE c.year BETWEEN 2000 AND 2010 AND t.lost>t.won
Claude-Sonnet-4.6: Clarifies that “disappointing” means a losing record, but uses coaches . lost > coaches . won without asking whose record should define the season. $L1$ recognized $L2$ missed preEX=False ✗
Llama-3.3-70B: Also resolves the losing-record interpretation, but applies coaches . lost > coaches . won despite joining teams . $L1$ recognized $L2$ missed preEX=False ✗
[User Intent Drift] (constraint-remove) Drift utterance: “Do not limit it to 2000–2010; show those coaches for all years.”
Gold post-drift SQL: SELECT DISTINCT c.coachID, t.tmID, c.year FROM coaches c JOIN teams t ON c.tmID=t.tmID AND c.year=t.year WHERE t.lost>t.won

Table 4: A TIDE-Bench Joint example. Both models resolve

L1

but miss the dependent record scope at

L2

; the resulting error persists after a correctly handled drift. SQL and model turns are abridged for space and clarity.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Scope	Emission rule	Example
WHERE	Emit each conjunct after top-level AND decomposition; keep predicates under OR or NOT intact.	WHERE a =1 AND b =2 $\to$ { a =1 , b =2 }
SELECT	Emit each SELECT item; SELECT * is one marker.	SELECT c , AVG ( v ) $\to$ { c , AVG ( v ) }
GROUP BY	Emit each grouping expression.	GROUP BY c , d $\to$ { c , d }
ORDER BY	Emit each ordering expression with its direction.	ORDER BY v DESC $\to$ { v DESC }
HAVING	Emit each conjunct after top-level AND decomposition.	HAVING COUNT ()>5 $\to$ { COUNT ()>5 }
LIMIT / OFFSET	Emit the clause with its numeric argument.	LIMIT 10 $\to$ { LIMIT 10 }

Appendix

Table 5: Scope-specific emission rules for constructing the commitment set

S(y)

. Each emitted unit is paired with its query-scope path and canonicalized into value-preserving and structural forms before insertion for drift testing.

Require: Candidate SQL query $y$
Ensure: Commitment set $S(y)$
Notation:
	$W$ : worklist of query-scope pairs $(q,\pi)$
	$\epsilon$ : root scope
	$\pi\oplus\ell$ : extend path $\pi$ with location $\ell$
	$\textsc{Branch}(q,j)$ : path label for branch $b_{j}$

Appendix

Algorithm 2 Commitment Extraction

Criterion	Requirement
Plausibility	L1 and L2 options are plausible and distinct, with no strawmen or rewordings.
Re-ask	Each layer adds a dimension absent from the initial question.
Counterfactuality	L1 must eliminate or transform the core of L2, rather than leave it invariant.

Appendix

Table 6: LLM judging rubric for chain injection. Both judges must pass three criteria for benchmark inclusion.

Step	Check	Pass condition	Failure signal
1	Output schema	Required JSON fields are present.	json_format
2	SQL parsing	post_drift_sql parses and yields extractable commitments.	sql_parse
3	SQL execution	post_drift_sql executes on the target database.	sql_execute
4	Non-empty result	The post-drift query returns at least one row.	empty_result
5	Observable effect	Pre- and post-drift result sets differ.	identical_result
6	Non-monotonicity	The commitment diff satisfies $S_{\mathrm{pre}}\nsubseteq S_{\mathrm{post}}$ .	no_change ; monotonic

Appendix

Table 7: Sequential AST-level checks for validating drift candidates. Checks are applied in order; the final column reports the rejection signal emitted upon failure, and only candidates passing all checks undergo LLM judging.

Cond.	Evidence	Question	Dialogue	Acc.	$\Delta$
A	orig.	orig.	—	51.2	—
B	rew.	ambiguous	✓	50.4	$-0.8$
C	rew.	ambiguous	—	31.2	$-20.0$

Appendix

Table 8: Reconstructability of rewritten evidence under three input settings (

n=695

). “orig.” / “rew.” denote original / rewritten BIRD evidence; ✓marks dialogue input;

\Delta

is the overall accuracy change (pp) from A.

Item	Pass Rate	Fleiss’ $\boldsymbol{\kappa}$
Counterfactual dependency	93.4%	0.79
Non-monotonic retraction	94.8%	0.83
Joint transition coherence	91.6%	0.71

Appendix

Table 9: Majority pass rates and annotator agreement for three core validation criteria. A criterion passes when at least three of five annotators mark it as satisfied; Fleiss’

\kappa

uses binary criterion-level verdicts across annotators.

Display name	Model configuration
Claude Sonnet 4.6	claude-sonnet-4.6
Qwen2.5-72B	qwen2.5-72b-instruct
o4-mini	o4-mini
GPT-4o	gpt-4o-2024-11-20
GPT-5.4	gpt-5.4
Llama-3.3-70B	llama-3.3-70b-instruct

Appendix

Table 10: Display names and API model configurations for the 12 evaluated LLMs. The configuration strings identify exact model variants and, where applicable, their reasoning modes used in all reported evaluations.

Split	Accuracy (%)
Una (refusal)	97.4
Amb (labeled)	94.3
Loc (unlabeled)	93.0

Appendix

Table 11: Stage 1 router accuracy on the three UserSim-Guard splits. Una tests refusal decisions, Amb tests labeled ambiguity routing, and Loc tests unlabeled SQL-detail localization during interaction.

	Value	Constraint	Projection
	( $n{=}175$ )	( $n{=}161$ )	( $n{=}178$ )
SQL Tokens	$+0.00$	$-7.47$	$-5.87$
WHERE Predicates	$+0.00$	$-1.11$	$-0.03$
JOIN Count	$+0.00$	$+0.00$	$-0.03$
Aggregation Count	$+0.00$	$+0.00$	$-0.26$
Subquery Depth	$+0.00$	$+0.00$	$-0.01$

Appendix

Table 12: Mean change in structural complexity from

y_{\mathrm{pre}}^{*}

y_{\mathrm{post}}^{*}

, by drift subtype. Values are means over the corresponding subtype subset. Negative values indicate reduced post-drift complexity along the corresponding dimension, with zero denoting no change on average.

#	Model	Chain-only				Joint				Drift-only
#	Model	L1-IR	L2-IR p	JIR	AR	L1-IR	L2-IR p	JIR	AR	AR
1	Claude-Sonnet-4.6	5.6	13.8	0.6	9.7	6.0	11.1	1.4	10.5	0.4
2	Qwen2.5-72B	36.2	3.3	1.2	60.1	35.8	3.9	1.8	60.7	20.0
3	o4-mini	27.8	11.5	1.4	54.9	29.2	12.6	2.1	55.4	17.5
4	GPT-4o	31.3	5.3	1.4	48.2	30.0	6.2	4.1	47.3	8.9
5	GPT-5.4	19.5	11.7	2.5	30.0	20.0	10.9	3.1	31.7	2.5

Appendix

Table 13: Chain-layer identification and ask rates (%) on Chain-only, Joint, and Drift-only samples. L1-IR and L2-IR p measure whether the agent identifies the corresponding clarification layer; JIR requires both layers to be identified within one trajectory, while AR records whether the agent issues any clarification request. L2-IR p is measured under the

p

-protocol with L1 pre-injected; all other metrics use the

f

-protocol throughout all evaluations.

#	Model	Drift-only DRA				Joint DRA				DRR
#	Model	Value	Constraint	Projection	All	Value	Constraint	Projection	All	Drift	Joint
1	Claude-Sonnet-4.6	53.4	48.5	64.0	55.4	38.6	34.4	55.4	43.0	98.2	96.9
2	Qwen2.5-72B	47.2	46.0	58.3	50.6	38.3	35.6	45.7	40.0	98.2	95.8
3	o4-mini	46.6	44.2	59.4	50.2	39.7	32.5	49.7	40.8	94.9	93.7
4	GPT-4o	49.4	45.1	62.3	52.4	34.1	35.0	51.4	40.3	97.3	96.9
5	GPT-5.4	48.3	42.3	61.7	51.0	35.8	34.4	52.6	41.1	97.3	94.7

Appendix

Table 14: Drift handling metrics (%) under the paired factorial design. The best and second-best results in each column are highlighted in bold and underlined, respectively. Drift-only DRA and Joint DRA are reported overall (All) and by drift subtype. The DRR columns report aggregate Drift Recognition Rate on Drift-only and Joint samples. Higher values indicate more reliable recognition and resolution of injected drift across interaction settings.

Panel of Figure 6 — Figure 6: Compressed version of the prompt used to generate chain ambiguity samples. The full prompt will be released with the code upon publication.

Panel of Figure 7 — Figure 7: Compressed version of the prompt used to generate intent drift samples. The full prompt will be released with the code upon publication.

Panel of Figure 8 — Figure 8: Agent system prompt template used uniformly across all 12 evaluated models. {schema_ddl} and {evidence} are populated per sample.

Text-to-SQL

Text-to-SQL translates natural-language questions into executable database queries, linking user intent to schemas, values, and SQL logic.

See all 169 papers

Probing Ambiguity: Adversarial Intent Resolution for Robust Text-to-SQL
Oct 16, 2026Youbing Yin, Xujun Peng, Muhammad S. Uddin +3Text-to-SQL Execution-Guided Text-to-SQL
Capital One
SQLStructEval: Structural Evaluation of LLM Text-to-SQL Generation
Apr 8, 2026Yixi Zhou, Fan Zhang, Zhiqiao Guo +4Text-to-SQL Evaluation Text-to-SQL
Hong Kong Baptist University · ShanghaiTech University · The University of Tokyo +2
ARCS: Towards Precise Text-to-SQL via Structured Disambiguation
Oct 7, 2026Yihao Hu, Yanlin Feng, Naoki Otani +1Text-to-SQL
αDuke University · βMegagon Labs

Oct 16, 2026J/K move · Enter open · S save

Probing Ambiguity: Adversarial Intent Resolution for Robust Text-to-SQL

Youbing Yin, Xujun Peng, Muhammad S. Uddin, Erin Babinsky+2

Capital One

Text-to-SQL Execution-Guided Text-to-SQL Clarification Question Generation

Open paper AskPDF arXiv

Text-to-SQL systems typically assume a one-to-one mapping between natural-language queries and correct SQL translations. In practice, however, many queries are inherently ambiguous, admitting multiple valid interpretations that yield distinct outputs. We introduce Adversarial Intent Resolution (AIR), a multi-agent framework that actively probes and resolves such ambiguity without any task-specific training. AIR first constructs competing interpretations, verifies through SQL execution that they produce genuinely different results, and generates targeted clarification questions grounded in the observed data differences. Evaluating on leading benchmarks for ambiguous questions, AIR demonstrates strong performance, efficiency, and robust cross-benchmark transferability. On the AMBROSIA benchmark under schema-only context, a realistic production setting where exposing the full database is infeasible, AIR outperforms IntentRL, the state-of-the-art reinforcement learning model, across all metrics (e.g., F1 0.679 vs. 0.618). Crucially, AIR exhibits only ~7% performance degradation without the full database, compared to ~20% for IntentRL. On the AmbiQT benchmark, where ambiguity arises from schema design rather than linguistic structure, AIR achieves higher F1 under both context settings without any benchmark-specific adaptation.

Abstract

Text-to-SQL

Explore similar work

Probing Ambiguity: Adversarial Intent Resolution for Robust Text-to-SQL

SQLStructEval: Structural Evaluation of LLM Text-to-SQL Generation

ARCS: Towards Precise Text-to-SQL via Structured Disambiguation