In Text-to-SQL tasks, large language models can generate structurally different SQL queries that return correct answers for the same intent. We call this phenomenon execution-correct structural divergence (ECSD). We introduce SQLStructEval, a framework that analyzes this behavior through canonical abstract syntax tree representations. We quantify structural diversity and agreement across generations. Experiments with different LLMs on multiple Text-to-SQL datasets, including Spider, document ECSD across models and datasets. Furthermore, our experiments demonstrate that generated queries are sensitive to question paraphrases and schema presentation. To address ECSD, we adopt a pipeline that first generates structured intermediate representations and then deterministically compiles them into SQL, improving execution accuracy and structural agreement among correct outputs. Structural analysis thus provides an additional diagnostic perspective that complements execution-based evaluation. Code is available at https://xanderzhou2022.github.io/AACL2026-SQLSTRUCTEVAL/.
Figures & tables
Figure 1: Motivation for SQLStructEval . Repeated Text-to-SQL generation can produce different query constructions even when they yield the same execution result. An AST-based view makes these structural differences explicit, supporting comparison of their diversity, stability, and sensitivity to input variations alongside conventional execution accuracy.
Table 1: Repeated SQL generation on Spider yields different structural distributions. GPT-5-mini has the largest D and NE, while GPT-4.1-mini has the highest MR.
Table 2: Execution accuracy and structural agreement differ across models on Spider. Gemini-3-Pro has the highest EA, whereas GPT-4.1-mini has the highest CA; positive HDR values reveal ECSD at high accuracy.
Table 3: ECSD is observed on the three additional subsets. HDR is positive for all evaluated settings except Claude-4.5-Sonnet on Spider-Syn.
Measure
Direct SQL
DIN-SQL
Compile-style
EA ↑
0.742
0.736
0.785
CA ↑
0.552
0.579
0.632
D
1.908
1.553
2.527
JV ↑
–
–
0.998
CR ↑
–
–
0.971
PR ↑
–
–
0.902
Table 4: On Spider with GPT-5-mini, compile-style generation improves EA and CA over both baselines while producing more distinct structures overall.
Table 5: Question paraphrases change dominant SQL structures across models. GPT-5-mini has the highest SF, while Gemini-3-Pro has the highest VA.
Table 6: Schema reordering yields higher VA than question paraphrasing for every evaluated model. Each model exhibits majority-structure changes on some questions.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Table 7: Parsing coverage varies substantially across the additional subsets. GPT-5-mini has the greatest overall structural diversity in each subset, while Claude-4.5-Sonnet has no observed correct-output diversity on Spider-Syn.
Category
Direct SQL
Compile-style
Δ
Schema linking
0.20
0.13
-0.07
Logical form
0.18
0.15
-0.03
Join path
0.14
0.09
-0.05
Predicate
0.16
0.10
-0.06
Structural variation
0.28
0.40
+0.12
Compilation
0.04
0.13
+0.09
Appendix
Table 8: The inspected compile-style sample has smaller shares of schema, logic, join, and predicate errors, but larger shares of structural variation and compilation errors.
Figure 4: AST comparison for the minimum-horsepower example in Appendix F . The reference uses ordering and a row limit; the generated query uses an aggregate subquery. Both return the same result on the evaluated database.
LLM-based agents for text-to-SQL often struggle with latency-performance trade-off, where performance improvements come at the cost of latency or vice versa. We reformulate text-to-SQL generation within the lens of software test coverage where the original query is prepared with a suite of test cases with simpler, atomic SQLs that are executed in parallel and together ensure semantic coverage of the original query. After iterating on test case coverage, the final SQL is generated only when enough information is gathered, leveraging the explored test case SQLs to ground the final generation. We validated our framework on a state-of-the-art benchmark for text-to-SQL, Spider 2.0, achieving a new state-of-the-art with 70.2% execution accuracy.
Recent advances in large language models (LLMs) have substantially improved Text-to-SQL performance, yet existing NL2SQL systems largely assume that user queries are valid and answerable. As a result, models often generate executable SQL even for queries with flawed premises. We introduce premise critique ability in NL2SQL, namely the ability to recognize flawed queries and abstain from generating misleading SQL. To evaluate this capability, we propose PremiseSQL, a benchmark containing 1,800 flawed NL2SQL queries across four categories of premise failures, along with their 1,800 original queries. We further design an evaluation framework to measure intrinsic and prompt-induced premise-aware behavior across three prompting settings. Experiments on frontier LLMs show that current models are strongly premise-trusting by default and rarely critique flawed premises. Stronger critique-oriented prompting improves detection and abstention on flawed queries, but also increases false invalid judgments and suppresses SQL generation on originals. The code is available at https://github.com/vegeet/PremiseSQL.
Shuo Lin, Jinzhe Li, Chenlu Guo +3
School of Artificial Intelligence, Jilin University · International Center of Future Science, Jilin University · Bilibili Inc. +2
Improving the reliability of large language models (LLMs) at inference time is a central challenge in structured reasoning tasks such as Text-to-SQL. Common test-time inference strategies, including Best-of-N sampling and Majority Voting, rely on heuristic signals such as execution success or output frequency, which provide limited semantic discrimination across candidate outputs. In this work, we study Outcome Reward Models (ORMs) as learned semantic scoring functions for test-time verification in Text-to-SQL. While ORMs have been previously explored for test-time scaling and alignment, their application to structured query generation remains underexplored. We introduce GradeSQL, a scalable framework for training task-specific ORMs via automated candidate generation and execution-based labeling, enabling verifier training without manual annotation. We integrate ORMs into a verification-driven Best-of-N pipeline and evaluate our approach on the BIRD and Spider benchmarks across multiple open-source LLM families. ORM-based selection consistently outperforms execution-based Best-of-N and Majority Voting, with gains of up to +4.33% on BIRD and +2.10% on Spider. We further show that ORMs scale effectively with larger candidate sets and yield stronger improvements on complex queries. Overall, our results demonstrate that ORM-based verification provides a simple, effective, and scalable alternative to heuristic test-time selection strategies for Text-to-SQL. Code datasets and models are publicly available.
Mattia Tritto, Giuseppe Farano, Dario Di Palma +4
Polytechnic University of Bari, Bari, Italy · IBM T.J. Watson Research Center, Yorktown Heights, NY, USA