PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems

Shuo Lin; Jinzhe Li; Chenlu Guo; Yanhan Li; Yuan Wu; Yi Chang

cs.CLOct 15, 2026

PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems

Authors: Shuo Lin, Jinzhe Li, Chenlu Guo, Yanhan Li, Yuan Wu, Yi Chang

Organizations: School of Artificial Intelligence, Jilin University · International Center of Future Science, Jilin University · Bilibili Inc. · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd ·

Abstract

Recent advances in large language models (LLMs) have substantially improved Text-to-SQL performance, yet existing NL2SQL systems largely assume that user queries are valid and answerable. As a result, models often generate executable SQL even for queries with flawed premises. We introduce premise critique ability in NL2SQL, namely the ability to recognize flawed queries and abstain from generating misleading SQL. To evaluate this capability, we propose PremiseSQL, a benchmark containing 1,800 flawed NL2SQL queries across four categories of premise failures, along with their 1,800 original queries. We further design an evaluation framework to measure intrinsic and prompt-induced premise-aware behavior across three prompting settings. Experiments on frontier LLMs show that current models are strongly premise-trusting by default and rarely critique flawed premises. Stronger critique-oriented prompting improves detection and abstention on flawed queries, but also increases false invalid judgments and suppresses SQL generation on originals. The code is available at https://github.com/vegeet/PremiseSQL.

Figures & tables

Figure 1: Illustration of how LLMs handle an NL2SQL query with an ambiguous premise. The example contrasts two model behaviors: one generates executable SQL from an unsupported interpretation directly; the other identifies the ambiguity, requests clarification, and abstains from SQL generation. This contrast demonstrates the necessity of premise critique ability , namely the ability to recognize flawed queries and abstain from generating misleading SQL.

Code	Category	Subcategory	Description
E1	Expression Error	E1.1 Ambiguity	Query contains ambiguous references, scopes, or targets that admit multiple plausible SQL interpretations.
		E1.2 Unclear Intent	Query intent or operation type is underspecified, making the SQL structure indeterminate.
E2	Premise Contradiction	E2.1 Internal Text Contradiction	Multiple query conditions cannot logically hold simultaneously.
		E2.2 Question–Schema Contradiction	Required schema elements or relations are absent or incompatible with the database schema.
E3	Missing Premise	E3.1 Missing Constraint	Essential filtering conditions or target specifications are omitted.
		E3.2 Missing Schema Premise	Required tables, columns, or relations are not provided in the schema.

Table 1: Taxonomy of flawed premises in Text-to-SQL.

Method	FP	Abst.	SQL	Fine	Prompt
TriageSQL	$\checkmark$	$\times$	$\checkmark$	$\triangle$	$\times$
KWIDK	$\triangle$	$\checkmark$	$\checkmark$	$\triangle$	$\times$
TrustSQL	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\times$
NL2SQL-BUGs	$\times$	$\times$	$\checkmark$	$\checkmark$	$\times$
SQuADRUN	$\checkmark$	$\checkmark$	$\times$	$\times$	$\times$
AbstentionBench	$\checkmark$	$\checkmark$	$\times$	$\triangle$	$\triangle$

Table 2: Comparison with related work on flawed-premise and abstention evaluation. FP denotes explicit flawed-premise reasoning before SQL generation, Abst. denotes abstention evaluation, SQL indicates whether the benchmark is formulated around SQL generation tasks, Fine denotes fine-grained error taxonomy, and Prompt denotes prompt-aware evaluation settings. Symbols:

\checkmark

Fully supported,

\triangle

Partially supported,

\times

Not supported.

Figure 2: Overview of PremiseSQL. Starting from standard Text-to-SQL queries, we construct flawed premise variants under a unified taxonomy, followed by multi-stage verification and premise-aware evaluation under Base, Implicit, and Explicit settings.

Figure 3: DET–GEN shifts from Implicit to Explicit prompting across 16 models. Teal and sage bars denote flaw detection (DET) on flawed queries and SQL generation (GEN) on paired originals, respectively. Black caps mark Implicit scores, while bar endpoints and arrowheads mark Explicit scores; outward/inward arrows indicate increases/decreases. Explicit prompting increases DET but decreases GEN for every model, showing that stronger flaw cues improve detection at the cost of more false rejection of valid queries.

Code	Error Type	SQL	DET	TYPE
E1.1	Ambiguity	0.210	0.242	0.138
E1.2	Unclear Intent	0.202	0.193	0.131
E2.1	Internal Text Contrad.	0.920	0.972	0.966
E2.2	Question–Schema Contrad.	0.881	0.923	0.915
E3.1	Missing Constraint	0.537	0.611	0.543
E3.2	Missing Schema Premise	0.568	0.586	0.474

Table 4: Performance by taxonomy error type under Implicit prompting, aggregated across all 16 models and both datasets with labels averaged over two judges.

Figure 4: Failure-mechanism incidence across prompting settings. Lines show the percentage of eligible failures assigned each mechanism by both judges. The M01–M03 reversal shows that stronger critique instructions shift residual failures from silent assumption injection to overinterpreting limited evidence rather than eliminating them, while M02 and M04 remain rare.

Appendix figures & tables38 assets

Supplementary material from the paper’s appendix.

Appendix

Error Category / Subcategory	Spider	BIRD
Expression Error (400)
– Ambiguity	90	110
– Unclear Intent	107	93
Premise Contradiction (400)
– Internal Text Contradiction	103	97
– Question–Schema Contradiction	156	44

Appendix

Table 5: Statistics of PremiseSQL across fine-grained premise error categories and source benchmarks.

Subcat.	Cand.	LLM Pass	Human Appr.	Final
E1.1	4,280	708	200	200
E1.2	1,108	673	382	200
E2.1	1,108	886	457	200
E2.2	1,108	933	350	200
E3.1	2,758	692	252	200
E3.2	–	–	–	200

Appendix

Table 6: Candidate generation and filtering statistics. Cand. denotes generated candidates, LLM Pass denotes candidates passing LLM review, Human Appr. denotes those retained after manual review, and Final denotes the benchmark allocation. E3.2 is constructed separately.

Target Model	Setting	Reported Two-Judge Average			Cross-Family Judge Only
Target Model	Setting	SQL	DET	TYPE	SQL	DET	TYPE
GPT-5.4	Base	0.117	0.261	0.258	0.109	0.240	0.239
	Implicit	0.682	0.702	0.648	0.681	0.697	0.671
	Explicit	0.960	0.981	0.855	0.959	0.982	0.907
Claude Opus 4.6	Base	0.204	0.281	0.277	0.205	0.296	0.288
	Implicit	0.510	0.579	0.536	0.513	0.579	0.523

Appendix

Table 7: Leave-self-out evaluation for models that also serve as automatic evaluators. The reported scores average judgments from GPT-5.4 and Claude Opus 4.6. The cross-family scores retain only Claude Opus 4.6 judgments for GPT-5.4 outputs and only GPT-5.4 judgments for Claude Opus 4.6 outputs.

Setting	GEN	EX	$\mathrm{EX}\mid\mathrm{GEN}$
Base	0.987	0.713	0.722
Implicit	0.902	0.652	0.722
Explicit	0.524	0.399	0.762

Appendix

Table 8: Execution-based validation on the 1,726 original queries with reliably executable gold SQL. EX measures execution accuracy over these queries, treating abstention as incorrect, while

\mathrm{EX}\mid\mathrm{GEN}

measures execution accuracy among outputs for which SQL is generated.

Model	$\Delta$ DET	$\Delta$ GEN
ChatGLM-5.1	0.358 [0.336, 0.380]	$-0.465$ [ $-0.488$ , $-0.442$ ]
Claude Opus 4.6	0.319 [0.297, 0.341]	$-0.183$ [ $-0.203$ , $-0.164$ ]
Claude Opus 5.5	0.180 [0.162, 0.199]	$-0.181$ [ $-0.199$ , $-0.163$ ]
Claude Sonnet 4.6	0.411 [0.389, 0.434]	$-0.064$ [ $-0.078$ , $-0.050$ ]
DeepSeek V3.2	0.079 [0.066, 0.092]	$-0.306$ [ $-0.327$ , $-0.284$ ]
DeepSeek V4.1 Flash	0.307 [0.286, 0.329]	$-0.615$ [ $-0.638$ , $-0.592$ ]

Appendix

Table 9: Paired bootstrap estimates for the change from Implicit to Explicit prompting. Each entry reports the point estimate and 95% confidence interval, with

\Delta=\text{Explicit}-\text{Implicit}

Code	Error Type	SQL	DET	TYPE
E1.1	Ambiguity	0.210 [0.181, 0.241]	0.242 [0.209, 0.276]	0.138 [0.115, 0.163]
E1.2	Unclear Intent	0.202 [0.177, 0.228]	0.193 [0.170, 0.218]	0.131 [0.111, 0.151]
E2.1	Internal Text Contradiction	0.920 [0.908, 0.932]	0.972 [0.963, 0.980]	0.966 [0.956, 0.976]
E2.2	Question–Schema Contradiction	0.881 [0.855, 0.906]	0.923 [0.896, 0.947]	0.915 [0.887, 0.941]
E3.1	Missing Constraint	0.537 [0.494, 0.580]	0.611 [0.564, 0.657]	0.543 [0.494, 0.593]
E3.2	Missing Schema Premise	0.568 [0.542, 0.594]	0.586 [0.561, 0.612]	0.474 [0.449, 0.498]

Appendix

Table 10: Implicit-prompting performance by error type with query-level bootstrap 95% confidence intervals. Models and judges are retained as fixed panels within each resampled query.

Stage	BIRD	Spider	Total
Strict wash retained	18,701	17,983	36,684
Eligible for attribution	13,709	13,864	27,573
Final sampled responses	4,000	4,000	8,000

Appendix

Table 11: Failure-mechanism attribution pool after the strict two-evaluator wash, context-based eligibility filtering, and sampling.

Mechanism	Cohen’s $\kappa$	Raw agreement
M01	0.8862	96.33%
M02	1.0000	100.00%
M03	0.9800	99.67%
M04	0.9074	99.67%
Macro average	0.9434	98.92%

Appendix

Table 12: Agreement between the two independent human annotators on the 300-example mechanism-attribution sample.

Comparison	M01	M02	M03	M04	Macro
Human 1–GPT	0.8656	1.0000	0.9579	0.8299	0.9133
Human 2–GPT	0.8705	1.0000	0.9379	0.9074	0.9290
Mean over humans	0.8681	1.0000	0.9479	0.8687	0.9212

Appendix

Table 13: Cohen’s

\kappa

between each human annotator and GPT 6.1 Sol.

Comparison	M01	M02	M03	M04	Macro
Human 1–Claude	0.8984	0.8872	0.8890	0.7967	0.8678
Human 2–Claude	0.8832	0.8872	0.8693	0.8872	0.8817
Mean over humans	0.8908	0.8872	0.8792	0.8420	0.8748

Appendix

Table 14: Cohen’s

\kappa

between each human annotator and Claude Opus 5.5.

Code	Source error category	$N$	M01	M02	M03	M04
E1.1	Ambiguity	1,999	70.14	0.00	14.46	0.35
E1.2	Unclear Intent	1,926	83.54	0.00	12.51	0.47
E2.1	Internal Text Contradiction	217	1.84	50.69	6.45	3.23
E2.2	Question–Schema Contradiction	355	76.34	0.00	6.48	4.51
E3.1	Missing Constraint	1,228	38.60	0.00	12.46	0.57
E3.3	Semantic Gap	926	89.20	0.00	2.70	0.00

Appendix

Table 15: Observed strict-consensus mechanism incidence (%) by source error category in the analyzed sample. E3.2 is excluded because its original reduced-schema exposure is unavailable.

Panel of Figure 5 — Figure 5: Representative prompt for ambiguity-type instance generation.

Panel of Figure 6 — Figure 6: Representative prompt for ambiguity-type instance review.

Panel of Figure 7 — Figure 7: Representative prompt for ambiguity-type automatic evaluation.

ID	Mechanism	supported when	Boundary and exclusion rules
M01	Assumption Injection	The response makes a material, answer-changing commitment not established by the current question or model-visible context, such as an invented value, threshold, metric, proxy, entity, schema role, relationship, or semantic mapping. A concrete response commitment and a material alternative or direct counterevidence must be identifiable.	Ordinary literal grounding, declared foreign-key use, and surface value normalization are not M01. Nonappearance in bounded rows alone is assessed as M03. Merely executing SQL, leaving an underspecified request broad, or making an arithmetic/grouping error does not establish M01.
M02	Contradiction Reinterpretation	The question contains explicit incompatible requirements on the same target and the response makes them executable by changing their logical relation, entity, time, scope, quantifier, population, or granularity (e.g., AND to OR or application to separate rows). Both original conditions and the transformed satisfaction relation must remain observable.	Preserving the contradiction literally is not M02. Removing one contradictory branch without replacement is M04. A change of field or scope in a question without an explicit internal contradiction is considered under M01 or left unlabeled.
M03	Context/Evidence Misuse	The response makes an incorrect availability or answerability claim by either treating bounded example rows as exhaustive database contents or denying a table, column, or declared relation that verified model-visible schema context contains. The claim and contextual counterevidence are both required.	Unknown data coverage is uncertain , not supported . A complete reference schema is insufficient unless exposure to the tested model is verified. Unsupported mappings used to construct SQL are M01; M03 is reserved for misuse of supplied context to infer absence, impossibility, or unanswerability.
M04	Explicit Constraint Omission	A semantically complete, explicit atomic requirement in the current question disappears from a response that still implements part of the request, and no value, predicate, proxy, field, or other operation replaces that requirement.	Incomplete referents, missing values or thresholds, and vague evaluative terms are not eligible targets. Complete abstention is not omission from a remaining implementation. An attempted but incorrect implementation blocks M04; a concrete substitute is M01, and a changed contradiction relation is M02. Requirements appearing only in the paired valid item cannot be imported into the flawed question.

Appendix

Table 16: Final decision rubric for failure-mechanism attribution. Each mechanism is evaluated independently, subject to the same-target exclusivity rules described in the text.

Panel of Figure 8 — Figure 8: Final prompt used for failure-mechanism attribution. The instance-specific fields listed in the final paragraph are appended as structured input.

Error Type	Question	Error Reason
Ambiguity	Which product had the highest sales in 2018?	The phrase “highest sales” admits multiple schema-supported interpretations, including the number of orders, total quantity sold, gross revenue, and discounted revenue. Because these alternatives can produce different product rankings, the query does not determine a unique SQL formulation.
Unclear Intent	For the lists created by the user who rated the movie “Pavee Lackeen: The Traveller Girl” on 3/27/2011 at 2:06:34 AM, what is the situation with their followers?	The phrase “what is the situation with their followers” does not specify whether to retrieve follower counts, aggregate them, compare them, or rank the lists. The intended SQL operation is therefore indeterminate.
Internal Text Contradiction	List the movies that were released in 1945 and also released after the year 2000.	The question contains mutually contradictory constraints that cannot be simultaneously satisfied under the schema. A single movie record has only one release date, making it impossible for a movie to be released in both 1945 and after 2000.
Question–Schema Contradiction	Who played the character named “Chanice Kobolowski,” and what is that actor’s email address?	The schema supports identifying the actor through the characters and actor tables, but the actor table contains no email-address column and no related table provides this attribute. The requested email address therefore cannot be retrieved from the database.
Missing Constraint	What’s the percentage of coins that is higher than the price 1 hour ago in that day? List the names of these coins.	The expression “that day” has no specified antecedent and leaves the required date unspecified. Because the historical table contains observations for multiple dates, the relevant time point cannot be determined.
Missing Schema Premise	List the powers of Hunter Zolomon.	We intentionally removed 30% of the gold schema beforehand. Therefore, the schema does not provide a way to identify Hunter Zolomon or to link a character entity to their associated powers. There is no table or relationship connecting characters to superpowers, making the query unanswerable under the given schema.

Appendix

Table 17: Examples of Expression Error, Premise Contradiction, Missing Premise, and Beyond Ability types.

Figure 9: E1.1 Ambiguity, LLaMA 4 Maverick. The model silently selects Songs.Title , misses the unresolved reference, and generates SQL (DET=0, TYPE=0, SQL=0).

Figure 10: E1.1 Ambiguity, Claude Opus 5.5. The model recognizes the ambiguity but combines both interpretations without user support (DET=1, TYPE=1, SQL=0).

Figure 11: E1.1 Ambiguity, DeepSeek V3.2. The model identifies the unresolved reference and abstains from generating unsupported SQL (DET=1, TYPE=1, SQL=2).

Figure 12: E2.1 Internal Text Contradiction, GPT 6.1 Sol. The model treats mutually exclusive winner and loser conditions as ordinary predicates and misses the flaw (DET=0, TYPE=0, SQL=0).

Figure 13: E2.1 Internal Text Contradiction, Claude Sonnet 4.6. The model states the contradiction but still encodes the incompatible requirements in SQL (DET=1, TYPE=1, SQL=0).

Figure 14: E2.1 Internal Text Contradiction, Claude Opus 5.5. The model recognizes the incompatible requirements and abstains (DET=1, TYPE=1, SQL=2).

Figure 15: E3.3 Semantic Gap, Claude Sonnet 4.6. The model invents a keyword-based proxy for clinical seriousness and misses the flaw (DET=0, TYPE=0, SQL=0).

Figure 16: E3.3 Semantic Gap, GPT-5.4. The model acknowledges the missing severity attribute but substitutes a count of all allergies (DET=1, TYPE=1, SQL=0).

Figure 17: E3.3 Semantic Gap, ChatGLM-5.1. The model preserves the unsupported distinction and abstains (DET=1, TYPE=1, SQL=2).

Figure 18: E4.1 Inexpressible in SQL, LLaMA 4 Maverick. The model turns a forecasting request into a historical count and misses the flaw (DET=0, TYPE=0, SQL=0).

Figure 19: E4.1 Inexpressible in SQL, DeepSeek V4.1 Flash. The model recognizes the forecasting limitation but returns a historical count as a proxy (DET=1, TYPE=1, SQL=0).

Figure 20: E4.1 Inexpressible in SQL, ChatGLM-5.1. The model detects the unsupported prediction request and abstains (DET=1, TYPE=1, SQL=2).

Figure 21: Original-query control under Base prompting, ChatGLM-5.1. The model constructs the valid schema-supported join and generates SQL (GEN=1).

Figure 22: Original-query control under Base prompting, Doubao Seed 2.0 Pro. The model notes that the supplied rows contain no matching city but correctly separates answer existence from query validity and generates SQL (GEN=1).

Figure 23: Original-query control under Base prompting, Claude Opus 5.5. The model treats the absence of a matching city in the supplied rows as grounds for withholding otherwise valid SQL (GEN=0).

Figure 24: Original-query control under Implicit prompting, Claude Opus 5.5. The model accepts the valid request and generates a workable query (VAL=1, GEN=1).

Figure 25: Original-query control under Implicit prompting, LLaMA 4 Maverick. The model incorrectly calls the request ambiguous but still generates SQL (VAL=0, GEN=1).

Figure 26: Original-query control under Implicit prompting, DeepSeek V3.2. The model mistakes the absence of a dedicated first-name column for invalidity and abstains (VAL=0, GEN=0).

Figure 27: Original-query control under Explicit prompting, DeepSeek V3.2. The model rejects the prompt’s unsupported claim of a flaw and generates SQL (VAL=1, GEN=1).

Figure 28: Original-query control under Explicit prompting, Doubao Seed 2.0 Pro. The model incorrectly judges the request invalid but nevertheless generates the correct filter (VAL=0, GEN=1).

Figure 29: Original-query control under Explicit prompting, Qwen3.5-397B. The model accepts the false-invalid framing and abstains (VAL=0, GEN=0).

Text-to-SQL

Text-to-SQL translates natural-language questions into executable database queries, linking user intent to schemas, values, and SQL logic.

See all 169 papers

SQLStructEval: Structural Evaluation of LLM Text-to-SQL Generation
Apr 8, 2026Yixi Zhou, Fan Zhang, Zhiqiao Guo +4Text-to-SQL Evaluation Text-to-SQL
Hong Kong Baptist University · ShanghaiTech University · The University of Tokyo +2
DexterSQL: Deep Schema Exploration and Rule-based Correction for Text-to-SQL Generation
Aug 12, 2026Anik Pramanik, Murat Kantarcioglu, Vincent Oria +1LLM Prompting Schema Linking
New Jersey Institute of Technology, USA. · Virginia Tech, USA.
Reliable Text-to-SQL in Resource-Constrained Settings with Small LLMs via Iterative Retry Workflows
Oct 12, 2026Bilel Abderrahmane Benziane, Xavier ScheplerText-to-SQL Small Language Models
Data Lab, Generix Group, Paris, France

Apr 8, 2026J/K move · Enter open · S save

SQLStructEval: Structural Evaluation of LLM Text-to-SQL Generation

Yixi Zhou, Fan Zhang, Zhiqiao Guo, Yu Chen+3

Hong Kong Baptist University · ShanghaiTech University · The University of Tokyo+2

Text-to-SQL Evaluation Text-to-SQL Language Model Generation Evaluation

Open paper AskPDF arXiv

In Text-to-SQL tasks, large language models can generate structurally different SQL queries that return correct answers for the same intent. We call this phenomenon execution-correct structural divergence (ECSD). We introduce SQLStructEval, a framework that analyzes this behavior through canonical abstract syntax tree representations. We quantify structural diversity and agreement across generations. Experiments with different LLMs on multiple Text-to-SQL datasets, including Spider, document ECSD across models and datasets. Furthermore, our experiments demonstrate that generated queries are sensitive to question paraphrases and schema presentation. To address ECSD, we adopt a pipeline that first generates structured intermediate representations and then deterministically compiles them into SQL, improving execution accuracy and structural agreement among correct outputs. Structural analysis thus provides an additional diagnostic perspective that complements execution-based evaluation. Code is available at https://xanderzhou2022.github.io/AACL2026-SQLSTRUCTEVAL/.

Abstract

Text-to-SQL

Explore similar work

SQLStructEval: Structural Evaluation of LLM Text-to-SQL Generation

DexterSQL: Deep Schema Exploration and Rule-based Correction for Text-to-SQL Generation

Reliable Text-to-SQL in Resource-Constrained Settings with Small LLMs via Iterative Retry Workflows