Large language models can achieve strong Text-to-SQL benchmark performance while still producing schema and query-logic errors. Most evaluations characterize such failures only in the final generated query, making it difficult to determine whether a related error was also observed in an earlier, simpler step or appears only at the final stage. We present a drill-down evaluation framework that decomposes Text-to-SQL queries into progressively more complex executable sub-queries and corresponding natural-language questions, allowing rule-based hallucination categories to be examined across a generation path. We distinguish recurrent hallucinations, in which the same rule-based category is observed in an earlier step and again in the final query, from emergent hallucinations, in which the category is observed at the final step without an earlier detection in the same path. Across five LLMs evaluated on an expanded BIRD-mini benchmark, several categories show appreciable recurrence probabilities, while clause abuse shows comparatively higher conditional emergence. Under the conversational-history condition, recurrence probabilities are generally higher and emergence probabilities are generally lower. These observations indicate that the two conditional measures do not move uniformly in the same direction under additional context and motivate trajectory-level analysis in addition to final-output evaluation. Code and evaluation artifacts are available in the project repository. https://github.com/HaydenMM/bird-mini-drilldown
Figures & tables
Figure 1: Drill-down framework. Gold SQL is decomposed into progressively more complex executable sub-queries, which are paired with natural-language sub-questions. LLM outputs are evaluated at each step and annotated to trace recurrent and emergent hallucination patterns.
Figure 2: Distribution of rule-based hallucination categories across the evaluated outputs, averaged across models. Schema contradiction and clause abuse are the most common categories in this analysis.
Figure 3: Conditional probabilities for recurrent (a) and emergent (b) rule-based hallucination categories. For most categories, P(In Final Step∣Occurs Earlier) is higher than P(In Final Step∣Does Not Occur Earlier) ; clause abuse shows a different pattern.
Figure 4: Probability of recurrent hallucinations with and without conversational history. Across most evaluated model–category pairs, the history condition is associated with higher recurrence probabilities, with some differences close to zero.
Figure 5: Probability of emergent hallucinations with and without conversational history. Across most evaluated model–category pairs, the history condition is associated with lower emergence probabilities, with some differences close to zero.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Description
Schema-Based: Schema Contradiction
The rule flags predicted tables, columns, or aliases not supported by the database schema, together with selected wildcard or backtick patterns handled by the implementation.
Schema-Based: Attribute Overanalysis
The rule flags additional valid tables or columns that appear in the prediction but not in the reference SQL, serving as an operational indicator of possible over-specification.
Schema-Based: Value Misrepresentation
The rule flags selected differences in literal values or type-casting behavior between the prediction and reference SQL.
Logic-Based: Join Redundancy
The rule flags predictions containing more JOIN operations than the reference SQL, indicating additional relational structure under this operational definition.
Logic-Based: Clause Abuse
The rule flags selected structural SQL clauses (e.g., GROUP BY , LIMIT , ORDER BY ) or logical operators (e.g., AND , OR ) that appear in the prediction but not in the reference SQL.
Logic-Based: Mathematical Delusion
The rule flags selected mathematical or operator patterns, including division without casting, use of % , selected uses of BETWEEN , or parser-detected malformed arithmetic structures.
Appendix
Table 1: Operational taxonomy of hallucination types, adapted from Qu et al. (2024) .
Category
Description
Recurrent Hallucination
A rule-based hallucination category detected in at least one earlier step and again in the final step.
Emergent Hallucination
A rule-based hallucination category detected in the final step with no earlier detection of that category in the same path.
Appendix
Table 2: Recurrent and emergent hallucination definitions used in the drill-down analysis.
Model
Simple
Moderate
Challenging
Total
Count
148
250
102
500
Claude-3.5-Sonnet
56.08
35.20
23.53
39.00 (EX)
59.39
38.61
31.15
43.24 (F1)
Claude-3.5-Sonnet (+ History)
50.00
34.40
20.59
36.20 (EX)
56.93
37.84
29.01
41.69 (F1)
Claude-3.7-Sonnet
51.35
38.40
21.57
38.80 (EX)
Appendix
Table 3: BIRD-mini execution accuracy (EX, %) and Soft-F1 scores across difficulty levels.
Recent advances in large language models (LLMs) have substantially improved Text-to-SQL performance, yet existing NL2SQL systems largely assume that user queries are valid and answerable. As a result, models often generate executable SQL even for queries with flawed premises. We introduce premise critique ability in NL2SQL, namely the ability to recognize flawed queries and abstain from generating misleading SQL. To evaluate this capability, we propose PremiseSQL, a benchmark containing 1,800 flawed NL2SQL queries across four categories of premise failures, along with their 1,800 original queries. We further design an evaluation framework to measure intrinsic and prompt-induced premise-aware behavior across three prompting settings. Experiments on frontier LLMs show that current models are strongly premise-trusting by default and rarely critique flawed premises. Stronger critique-oriented prompting improves detection and abstention on flawed queries, but also increases false invalid judgments and suppresses SQL generation on originals. The code is available at https://github.com/vegeet/PremiseSQL.
Shuo Lin, Jinzhe Li, Chenlu Guo +3
School of Artificial Intelligence, Jilin University · International Center of Future Science, Jilin University · Bilibili Inc. +2
Recent advances in large language models (LLMs) have established conversational text-to-SQL as a practical interface between users and databases, often involving multiple turns of clarification and revision. However, existing benchmarks primarily evaluate execution accuracy, leaving the unfolding and shifting of user intent across turns largely uncovered. To address this, we introduce TIDE-Bench, a benchmark for conversational text-to-SQL under chain ambiguity and intent drift evaluation, targeting two recurring patterns: chain ambiguity, where an underspecified question triggers layered clarification with conditional dependencies, and intent drift, where the user retracts and replaces a previously committed request element. Built on 514 anchor SQLs from BIRD, TIDE-Bench comprises 1,542 samples and introduces dedicated metrics for chain identification and drift recognition-resolution beyond execution accuracy. Evaluating 12 advanced LLMs reveals a persistent chain identification bottleneck unaffected by clarification frequency, a wide drift recognition-resolution gap, and overlap between failure modes when jointly activated. The corresponding code of TIDE-Bench is released for further research.
Yujia Liu, Jiayan Lin, Zijin Hong +6
Jinan University · The Hong Kong Polytechnic University · City University of Macau +2
In Text-to-SQL tasks, large language models can generate structurally different SQL queries that return correct answers for the same intent. We call this phenomenon execution-correct structural divergence (ECSD). We introduce SQLStructEval, a framework that analyzes this behavior through canonical abstract syntax tree representations. We quantify structural diversity and agreement across generations. Experiments with different LLMs on multiple Text-to-SQL datasets, including Spider, document ECSD across models and datasets. Furthermore, our experiments demonstrate that generated queries are sensitive to question paraphrases and schema presentation. To address ECSD, we adopt a pipeline that first generates structured intermediate representations and then deterministically compiles them into SQL, improving execution accuracy and structural agreement among correct outputs. Structural analysis thus provides an additional diagnostic perspective that complements execution-based evaluation. Code is available at https://xanderzhou2022.github.io/AACL2026-SQLSTRUCTEVAL/.
Yixi Zhou, Fan Zhang, Zhiqiao Guo +4
Hong Kong Baptist University · ShanghaiTech University · The University of Tokyo +2