Tracing Recurrent and Emergent Hallucination Patterns in Decomposed Text-to-SQL Generation

Hayden Moore; Abhinav Verma; Anshuman Dutt

cs.CLOct 20, 2026

Tracing Recurrent and Emergent Hallucination Patterns in Decomposed Text-to-SQL Generation

Authors: Hayden Moore, Abhinav Verma, Anshuman Dutt

Organizations: The Pennsylvania State University · Microsoft

Abstract

Large language models can achieve strong Text-to-SQL benchmark performance while still producing schema and query-logic errors. Most evaluations characterize such failures only in the final generated query, making it difficult to determine whether a related error was also observed in an earlier, simpler step or appears only at the final stage. We present a drill-down evaluation framework that decomposes Text-to-SQL queries into progressively more complex executable sub-queries and corresponding natural-language questions, allowing rule-based hallucination categories to be examined across a generation path. We distinguish recurrent hallucinations, in which the same rule-based category is observed in an earlier step and again in the final query, from emergent hallucinations, in which the category is observed at the final step without an earlier detection in the same path. Across five LLMs evaluated on an expanded BIRD-mini benchmark, several categories show appreciable recurrence probabilities, while clause abuse shows comparatively higher conditional emergence. Under the conversational-history condition, recurrence probabilities are generally higher and emergence probabilities are generally lower. These observations indicate that the two conditional measures do not move uniformly in the same direction under additional context and motivate trajectory-level analysis in addition to final-output evaluation. Code and evaluation artifacts are available in the project repository. https://github.com/HaydenMM/bird-mini-drilldown

Figures & tables

Figure 1: Drill-down framework. Gold SQL is decomposed into progressively more complex executable sub-queries, which are paired with natural-language sub-questions. LLM outputs are evaluated at each step and annotated to trace recurrent and emergent hallucination patterns.

Figure 2: Distribution of rule-based hallucination categories across the evaluated outputs, averaged across models. Schema contradiction and clause abuse are the most common categories in this analysis.

Panel of Figure 3 — Figure 3: Conditional probabilities for recurrent (a) and emergent (b) rule-based hallucination categories. For most categories, $P(\text{In Final Step}\mid\text{Occurs Earlier})$ is higher than $P(\text{In Final Step}\mid\text{Does Not Occur Earlier})$ ; clause abuse shows a different pattern.

Figure 4: Probability of recurrent hallucinations with and without conversational history. Across most evaluated model–category pairs, the history condition is associated with higher recurrence probabilities, with some differences close to zero.

Figure 5: Probability of emergent hallucinations with and without conversational history. Across most evaluated model–category pairs, the history condition is associated with lower emergence probabilities, with some differences close to zero.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Description
Schema-Based: Schema Contradiction	The rule flags predicted tables, columns, or aliases not supported by the database schema, together with selected wildcard or backtick patterns handled by the implementation.
Schema-Based: Attribute Overanalysis	The rule flags additional valid tables or columns that appear in the prediction but not in the reference SQL, serving as an operational indicator of possible over-specification.
Schema-Based: Value Misrepresentation	The rule flags selected differences in literal values or type-casting behavior between the prediction and reference SQL.
Logic-Based: Join Redundancy	The rule flags predictions containing more JOIN operations than the reference SQL, indicating additional relational structure under this operational definition.
Logic-Based: Clause Abuse	The rule flags selected structural SQL clauses (e.g., GROUP BY , LIMIT , ORDER BY ) or logical operators (e.g., AND , OR ) that appear in the prediction but not in the reference SQL.
Logic-Based: Mathematical Delusion	The rule flags selected mathematical or operator patterns, including division without casting, use of % , selected uses of BETWEEN , or parser-detected malformed arithmetic structures.

Appendix

Table 1: Operational taxonomy of hallucination types, adapted from Qu et al. (2024) .

Category	Description
Recurrent Hallucination	A rule-based hallucination category detected in at least one earlier step and again in the final step.
Emergent Hallucination	A rule-based hallucination category detected in the final step with no earlier detection of that category in the same path.

Appendix

Table 2: Recurrent and emergent hallucination definitions used in the drill-down analysis.

Model	Simple	Moderate	Challenging	Total
Count	148	250	102	500
Claude-3.5-Sonnet	56.08	35.20	23.53	39.00 (EX)
	59.39	38.61	31.15	43.24 (F1)
Claude-3.5-Sonnet (+ History)	50.00	34.40	20.59	36.20 (EX)
	56.93	37.84	29.01	41.69 (F1)
Claude-3.7-Sonnet	51.35	38.40	21.57	38.80 (EX)

Appendix

Table 3: BIRD-mini execution accuracy (EX, %) and Soft-F1 scores across difficulty levels.

Hallucination in Language Models

Hallucination in language models studies how systems produce unsupported or false claims and how to detect, explain, and reduce them.

See all 253 papers

PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems
Oct 15, 2026Shuo Lin, Jinzhe Li, Chenlu Guo +3Text-to-SQL Text-to-SQL Evaluation
School of Artificial Intelligence, Jilin University · International Center of Future Science, Jilin University · Bilibili Inc. +2
Evaluating LLMs on Conversational Text-to-SQL under Chain Ambiguity and Intent Drift
Aug 30, 2026Yujia Liu, Jiayan Lin, Zijin Hong +6Text-to-SQL LLM Evaluation
Jinan University · The Hong Kong Polytechnic University · City University of Macau +2
SQLStructEval: Structural Evaluation of LLM Text-to-SQL Generation
Apr 8, 2026Yixi Zhou, Fan Zhang, Zhiqiao Guo +4Text-to-SQL Evaluation Text-to-SQL
Hong Kong Baptist University · ShanghaiTech University · The University of Tokyo +2

Oct 15, 2026J/K move · Enter open · S save

PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems

Shuo Lin, Jinzhe Li, Chenlu Guo, Yanhan Li+2

School of Artificial Intelligence, Jilin University · International Center of Future Science, Jilin University · Bilibili Inc.+2

Text-to-SQL Text-to-SQL Evaluation LLM Evaluation LLM Prompting

Open paper AskPDF arXiv

Recent advances in large language models (LLMs) have substantially improved Text-to-SQL performance, yet existing NL2SQL systems largely assume that user queries are valid and answerable. As a result, models often generate executable SQL even for queries with flawed premises. We introduce premise critique ability in NL2SQL, namely the ability to recognize flawed queries and abstain from generating misleading SQL. To evaluate this capability, we propose PremiseSQL, a benchmark containing 1,800 flawed NL2SQL queries across four categories of premise failures, along with their 1,800 original queries. We further design an evaluation framework to measure intrinsic and prompt-induced premise-aware behavior across three prompting settings. Experiments on frontier LLMs show that current models are strongly premise-trusting by default and rarely critique flawed premises. Stronger critique-oriented prompting improves detection and abstention on flawed queries, but also increases false invalid judgments and suppresses SQL generation on originals. The code is available at https://github.com/vegeet/PremiseSQL.

Abstract

Hallucination in Language Models

Explore similar work

PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems

Evaluating LLMs on Conversational Text-to-SQL under Chain Ambiguity and Intent Drift

SQLStructEval: Structural Evaluation of LLM Text-to-SQL Generation