Recent advances in large language models (LLMs) have established conversational text-to-SQL as a practical interface between users and databases, often involving multiple turns of clarification and revision. However, existing benchmarks primarily evaluate execution accuracy, leaving the unfolding and shifting of user intent across turns largely uncovered. To address this, we introduce TIDE-Bench, a benchmark for conversational text-to-SQL under chain ambiguity and intent drift evaluation, targeting two recurring patterns: chain ambiguity, where an underspecified question triggers layered clarification with conditional dependencies, and intent drift, where the user retracts and replaces a previously committed request element. Built on 514 anchor SQLs from BIRD, TIDE-Bench comprises 1,542 samples and introduces dedicated metrics for chain identification and drift recognition-resolution beyond execution accuracy. Evaluating 12 advanced LLMs reveals a persistent chain identification bottleneck unaffected by clarification frequency, a wide drift recognition-resolution gap, and overlap between failure modes when jointly activated. The corresponding code of TIDE-Bench is released for further research.
Figures & tables
Figure 1: (A) Two conversational patterns that challenge text-to-SQL. (B) Neither is covered by existing text-to-SQL benchmarks or standard evaluation protocols.
Figure 2: Left: motivating dialogue illustrating chain ambiguity (L1/L2 clarification with conditional dependency) and user intent drift (non-monotonic retraction of a prior commitment). Right: overview of the TIDE-Bench four-stage construction pipeline (anchor selection, chain injection, drift injection, and triplet assembly; see § 3 ).
Require: Queries ypre and ypost
Ensure: Drift status or subtype
Notation:
W(y) : filtering commitments in y
O(y)=S(y)∖W(y) : output-side commitments in y
LitReplace(Wpre,Wpost) : one predicate literal changes; its structure and all other filters remain fixed
% Extract commitments
Algorithm 1 User Intent Drift Operationalization
Split
Majority Acc.
Strict Acc.
Fleiss’ κ
Chain-only
93.2
84.8
0.82
Drift-only
95.6
87.6
0.80
Joint
88.4
76.8
0.77
Overall
92.4
83.1
0.81
Table 1: Human validation on 250 stratified triplets. Majority requires ≥3/5 annotators per item; strict requires unanimity. Fleiss’ κ uses binary accept/reject verdicts.
Metric
Full Name
Definition
Set
DEA
Dialogue Execution Accuracy
∣S∣1∑τ∈S∏i=1nτEX(yi,yi∗)
all
AR
Ask Rate
∣S∣1∑τ∈S\mathds1[τ contains a clarification]
all
L1-IR
L1 Identification Rate
∣S∣1∑τ∈S\mathds1[∃t:ρt=L1]
C,J
L2-IR p
L2 Identification Rate
∣S∣1∑τ∈S\mathds1[∃t:ρt=L2]
C,J
JIR
Joint Identification Rate
∣S∣1∑τ∈S\mathds1[∃t1,t2:ρt1=L1∧ρt2=L2]
C,J
DRR
Drift Recognition Rate
∣S∣1∑τ∈S\mathds1[Rec(τ)]
D,J
Table 2: TIDE-Bench metrics spanning four behavioral facets: execution accuracy (DEA), clarification behavior (AR), chain-layer identification (L1-IR, L2-IR p , JIR), and drift handling (DRR, DRA). C,D,J denote Chain-only, Drift-only, and Joint sample sets. L2-IR p uses the p -protocol (§ 4.1 ); all other metrics use the f -protocol instead.
#
Model
Chain-only
Drift-only
Joint
Value
Constraint
Projection
All
Value
Constraint
Projection
All
1
Claude-Sonnet-4.6
35.9
47.7
44.2
43.4
45.1
31.2
20.2
31.4
27.8
2
Qwen2.5-72B
32.4
44.3
41.1
44.0
43.2
30.1
21.5
28.6
26.8
3
o4-mini
32.9
42.6
41.1
40.0
41.2
31.2
21.5
27.4
26.8
4
GPT-4o
30.9
46.0
38.7
44.6
43.2
27.8
22.1
29.7
26.7
5
GPT-5.4
32.5
43.2
36.8
42.3
40.9
27.8
20.9
29.7
26.3
Table 3: DEA (%) under the paired factorial design. The best and second-best results per column are in bold and underlined, respectively. Chain-only reports strict execution accuracy; Drift-only and Joint report end-to-end accuracy broken down by drift subtype; All denotes aggregate performance, and higher metric values are better.
Figure 3: Chain-layer identification on Chain-only samples. (a) L1-IR ( f -protocol) vs. L2-IR p ( p -protocol) per model: even with L1 pre-resolved, L2 identification stays low. (b) JIR against ask rate: identification remains low regardless of how much the agent asks. Model names are abbreviated; full names appear in Table 3 for reference.
Figure 4: Drift handling from Drift-only to Joint across DRR, DRA, and DEA, with one line per model. DRR remains high and flat, while DRA and DEA start lower and decline further in Joint. The dashed DRA reference marks DRR, highlighting the recognition–resolution gap that persists for all models across both interaction settings. The orange line highlights Claude-Sonnet-4.6, which achieves the highest DEA in both Drift-only and Joint.
Figure 5: Drift-only DRA by drift subtype. Points denote models, and the bar marks the corresponding subtype mean. Orange points mark per-subtype maxima.
[Context] Schema: coaches(coachID, year, tmID, won, lost) ; teams(year, tmID, won, lost) . Question: “Which coaches had disappointing seasons in 2000–2010? Show their coach ID, team, and year.”
Gold SQL: SELECT DISTINCT c.coachID, t.tmID, c.year FROM coaches c JOIN teams t ON c.tmID=t.tmID AND c.year=t.year WHERE c.year BETWEEN 2000 AND 2010 AND t.lost>t.won
Claude-Sonnet-4.6: Clarifies that “disappointing” means a losing record, but uses coaches . lost > coaches . won without asking whose record should define the season. L1 recognized L2 missed preEX=False ✗
Llama-3.3-70B: Also resolves the losing-record interpretation, but applies coaches . lost > coaches . won despite joining teams . L1 recognized L2 missed preEX=False ✗
[User Intent Drift] (constraint-remove) Drift utterance: “Do not limit it to 2000–2010; show those coaches for all years.”
Gold post-drift SQL: SELECT DISTINCT c.coachID, t.tmID, c.year FROM coaches c JOIN teams t ON c.tmID=t.tmID AND c.year=t.year WHERE t.lost>t.won
Table 4: A TIDE-Bench Joint example. Both models resolve L1 but miss the dependent record scope at L2 ; the resulting error persists after a correctly handled drift. SQL and model turns are abridged for space and clarity.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Scope
Emission rule
Example
WHERE
Emit each conjunct after top-level AND decomposition; keep predicates under OR or NOT intact.
WHERE a =1 AND b =2 → { a =1 , b =2 }
SELECT
Emit each SELECT item; SELECT * is one marker.
SELECT c , AVG ( v ) → { c , AVG ( v ) }
GROUP BY
Emit each grouping expression.
GROUP BY c , d → { c , d }
ORDER BY
Emit each ordering expression with its direction.
ORDER BY v DESC → { v DESC }
HAVING
Emit each conjunct after top-level AND decomposition.
HAVING COUNT ()>5 → { COUNT ()>5 }
LIMIT / OFFSET
Emit the clause with its numeric argument.
LIMIT 10 → { LIMIT 10 }
Appendix
Table 5: Scope-specific emission rules for constructing the commitment set S(y) . Each emitted unit is paired with its query-scope path and canonicalized into value-preserving and structural forms before insertion for drift testing.
Require: Candidate SQL query y
Ensure: Commitment set S(y)
Notation:
W : worklist of query-scope pairs (q,π)
ϵ : root scope
π⊕ℓ : extend path π with location ℓ
\textscBranch(q,j) : path label for branch bj
Appendix
Algorithm 2 Commitment Extraction
Criterion
Requirement
Plausibility
L1 and L2 options are plausible and distinct, with no strawmen or rewordings.
Re-ask
Each layer adds a dimension absent from the initial question.
Counterfactuality
L1 must eliminate or transform the core of L2, rather than leave it invariant.
Appendix
Table 6: LLM judging rubric for chain injection. Both judges must pass three criteria for benchmark inclusion.
Step
Check
Pass condition
Failure signal
1
Output schema
Required JSON fields are present.
json_format
2
SQL parsing
post_drift_sql parses and yields extractable commitments.
sql_parse
3
SQL execution
post_drift_sql executes on the target database.
sql_execute
4
Non-empty result
The post-drift query returns at least one row.
empty_result
5
Observable effect
Pre- and post-drift result sets differ.
identical_result
6
Non-monotonicity
The commitment diff satisfies Spre⊈Spost .
no_change ; monotonic
Appendix
Table 7: Sequential AST-level checks for validating drift candidates. Checks are applied in order; the final column reports the rejection signal emitted upon failure, and only candidates passing all checks undergo LLM judging.
Cond.
Evidence
Question
Dialogue
Acc.
Δ
A
orig.
orig.
—
51.2
—
B
rew.
ambiguous
✓
50.4
−0.8
C
rew.
ambiguous
—
31.2
−20.0
Appendix
Table 8: Reconstructability of rewritten evidence under three input settings ( n=695 ). “orig.” / “rew.” denote original / rewritten BIRD evidence; ✓marks dialogue input; Δ is the overall accuracy change (pp) from A.
Item
Pass Rate
Fleiss’ κ
Counterfactual dependency
93.4%
0.79
Non-monotonic retraction
94.8%
0.83
Joint transition coherence
91.6%
0.71
Appendix
Table 9: Majority pass rates and annotator agreement for three core validation criteria. A criterion passes when at least three of five annotators mark it as satisfied; Fleiss’ κ uses binary criterion-level verdicts across annotators.
Display name
Model configuration
Claude Sonnet 4.6
claude-sonnet-4.6
Qwen2.5-72B
qwen2.5-72b-instruct
o4-mini
o4-mini
GPT-4o
gpt-4o-2024-11-20
GPT-5.4
gpt-5.4
Llama-3.3-70B
llama-3.3-70b-instruct
Appendix
Table 10: Display names and API model configurations for the 12 evaluated LLMs. The configuration strings identify exact model variants and, where applicable, their reasoning modes used in all reported evaluations.
Split
Accuracy (%)
Una (refusal)
97.4
Amb (labeled)
94.3
Loc (unlabeled)
93.0
Appendix
Table 11: Stage 1 router accuracy on the three UserSim-Guard splits. Una tests refusal decisions, Amb tests labeled ambiguity routing, and Loc tests unlabeled SQL-detail localization during interaction.
Value
Constraint
Projection
( n=175 )
( n=161 )
( n=178 )
SQL Tokens
+0.00
−7.47
−5.87
WHERE Predicates
+0.00
−1.11
−0.03
JOIN Count
+0.00
+0.00
−0.03
Aggregation Count
+0.00
+0.00
−0.26
Subquery Depth
+0.00
+0.00
−0.01
Appendix
Table 12: Mean change in structural complexity from ypre∗ to ypost∗ , by drift subtype. Values are means over the corresponding subtype subset. Negative values indicate reduced post-drift complexity along the corresponding dimension, with zero denoting no change on average.
#
Model
Chain-only
Joint
Drift-only
L1-IR
L2-IR p
JIR
AR
L1-IR
L2-IR p
JIR
AR
AR
1
Claude-Sonnet-4.6
5.6
13.8
0.6
9.7
6.0
11.1
1.4
10.5
0.4
2
Qwen2.5-72B
36.2
3.3
1.2
60.1
35.8
3.9
1.8
60.7
20.0
3
o4-mini
27.8
11.5
1.4
54.9
29.2
12.6
2.1
55.4
17.5
4
GPT-4o
31.3
5.3
1.4
48.2
30.0
6.2
4.1
47.3
8.9
5
GPT-5.4
19.5
11.7
2.5
30.0
20.0
10.9
3.1
31.7
2.5
Appendix
Table 13: Chain-layer identification and ask rates (%) on Chain-only, Joint, and Drift-only samples. L1-IR and L2-IR p measure whether the agent identifies the corresponding clarification layer; JIR requires both layers to be identified within one trajectory, while AR records whether the agent issues any clarification request. L2-IR p is measured under the p -protocol with L1 pre-injected; all other metrics use the f -protocol throughout all evaluations.
#
Model
Drift-only DRA
Joint DRA
DRR
Value
Constraint
Projection
All
Value
Constraint
Projection
All
Drift
Joint
1
Claude-Sonnet-4.6
53.4
48.5
64.0
55.4
38.6
34.4
55.4
43.0
98.2
96.9
2
Qwen2.5-72B
47.2
46.0
58.3
50.6
38.3
35.6
45.7
40.0
98.2
95.8
3
o4-mini
46.6
44.2
59.4
50.2
39.7
32.5
49.7
40.8
94.9
93.7
4
GPT-4o
49.4
45.1
62.3
52.4
34.1
35.0
51.4
40.3
97.3
96.9
5
GPT-5.4
48.3
42.3
61.7
51.0
35.8
34.4
52.6
41.1
97.3
94.7
Appendix
Table 14: Drift handling metrics (%) under the paired factorial design. The best and second-best results in each column are highlighted in bold and underlined, respectively. Drift-only DRA and Joint DRA are reported overall (All) and by drift subtype. The DRR columns report aggregate Drift Recognition Rate on Drift-only and Joint samples. Higher values indicate more reliable recognition and resolution of injected drift across interaction settings.
Figure 6: Compressed version of the prompt used to generate chain ambiguity samples. The full prompt will be released with the code upon publication.
Figure 7: Compressed version of the prompt used to generate intent drift samples. The full prompt will be released with the code upon publication.
Figure 8: Agent system prompt template used uniformly across all 12 evaluated models. {schema_ddl} and {evidence} are populated per sample.
Text-to-SQL systems typically assume a one-to-one mapping between natural-language queries and correct SQL translations. In practice, however, many queries are inherently ambiguous, admitting multiple valid interpretations that yield distinct outputs. We introduce Adversarial Intent Resolution (AIR), a multi-agent framework that actively probes and resolves such ambiguity without any task-specific training. AIR first constructs competing interpretations, verifies through SQL execution that they produce genuinely different results, and generates targeted clarification questions grounded in the observed data differences. Evaluating on leading benchmarks for ambiguous questions, AIR demonstrates strong performance, efficiency, and robust cross-benchmark transferability. On the AMBROSIA benchmark under schema-only context, a realistic production setting where exposing the full database is infeasible, AIR outperforms IntentRL, the state-of-the-art reinforcement learning model, across all metrics (e.g., F1 0.679 vs. 0.618). Crucially, AIR exhibits only ~7% performance degradation without the full database, compared to ~20% for IntentRL. On the AmbiQT benchmark, where ambiguity arises from schema design rather than linguistic structure, AIR achieves higher F1 under both context settings without any benchmark-specific adaptation.
In Text-to-SQL tasks, large language models can generate structurally different SQL queries that return correct answers for the same intent. We call this phenomenon execution-correct structural divergence (ECSD). We introduce SQLStructEval, a framework that analyzes this behavior through canonical abstract syntax tree representations. We quantify structural diversity and agreement across generations. Experiments with different LLMs on multiple Text-to-SQL datasets, including Spider, document ECSD across models and datasets. Furthermore, our experiments demonstrate that generated queries are sensitive to question paraphrases and schema presentation. To address ECSD, we adopt a pipeline that first generates structured intermediate representations and then deterministically compiles them into SQL, improving execution accuracy and structural agreement among correct outputs. Structural analysis thus provides an additional diagnostic perspective that complements execution-based evaluation. Code is available at https://xanderzhou2022.github.io/AACL2026-SQLSTRUCTEVAL/.
Yixi Zhou, Fan Zhang, Zhiqiao Guo +4
Hong Kong Baptist University · ShanghaiTech University · The University of Tokyo +2
As text-to-SQL systems move beyond demonstrations toward real-world deployment, ambiguity in user questions becomes a primary source of errors. Such ambiguities are often subtle, domain- or data-specific, and can silently cause system outputs to deviate from the user's true intent. Ambiguity is traditionally addressed through conversational clarification, which is often inefficient, cognitively demanding, and poorly aligned with real-world user workflows. We propose structured disambiguation, a new paradigm in which ambiguity is resolved through explicit, constrained interactions rather than free-form dialogue. We construct ARCS (Ambiguity Resolution Corpus for SQL), the first text-to-SQL benchmark featuring naturally occurring, unconstrained ambiguities over real-world databases, with complete annotations of all valid ambiguity points, interpretations, and SQL queries. Experimental results show that text-to-SQL remains challenging in the presence of ambiguity: gpt-6-sol achieves only 51% end-to-end execution accuracy, and no open-source model exceeds 27%.