SQLStructEval: Structural Evaluation of LLM Text-to-SQL Generation

Yixi Zhou; Fan Zhang; Zhiqiao Guo; Yu Chen; Haipeng Zhang; Preslav Nakov; Zhuohan Xie

cs.CLApr 8, 2026

SQLStructEval: Structural Evaluation of LLM Text-to-SQL Generation

Authors: Yixi Zhou, Fan Zhang, Zhiqiao Guo, Yu Chen, Haipeng Zhang, Preslav Nakov, Zhuohan Xie

Organizations: Hong Kong Baptist University · ShanghaiTech University · The University of Tokyo · University of Michigan ·

Abstract

In Text-to-SQL tasks, large language models can generate structurally different SQL queries that return correct answers for the same intent. We call this phenomenon execution-correct structural divergence (ECSD). We introduce SQLStructEval, a framework that analyzes this behavior through canonical abstract syntax tree representations. We quantify structural diversity and agreement across generations. Experiments with different LLMs on multiple Text-to-SQL datasets, including Spider, document ECSD across models and datasets. Furthermore, our experiments demonstrate that generated queries are sensitive to question paraphrases and schema presentation. To address ECSD, we adopt a pipeline that first generates structured intermediate representations and then deterministically compiles them into SQL, improving execution accuracy and structural agreement among correct outputs. Structural analysis thus provides an additional diagnostic perspective that complements execution-based evaluation. Code is available at https://xanderzhou2022.github.io/AACL2026-SQLSTRUCTEVAL/.

Figures & tables

Figure 1: Motivation for SQLStructEval . Repeated Text-to-SQL generation can produce different query constructions even when they yield the same execution result. An AST-based view makes these structural differences explicit, supporting comparison of their diversity, stability, and sensitivity to input variations alongside conventional execution accuracy.

Panel of Table 1 — Table 1: Repeated SQL generation on Spider yields different structural distributions. GPT-5-mini has the largest D and NE, while GPT-4.1-mini has the highest MR.

Panel of Table 2 — Table 2: Execution accuracy and structural agreement differ across models on Spider. Gemini-3-Pro has the highest EA, whereas GPT-4.1-mini has the highest CA; positive HDR values reveal ECSD at high accuracy.

Panel of Table 3 — Table 3: ECSD is observed on the three additional subsets. HDR is positive for all evaluated settings except Claude-4.5-Sonnet on Spider-Syn.

Measure	Direct SQL	DIN-SQL	Compile-style
EA $\uparrow$	0.742	0.736	0.785
CA $\uparrow$	0.552	0.579	0.632
D	1.908	1.553	2.527
JV $\uparrow$	–	–	0.998
CR $\uparrow$	–	–	0.971
PR $\uparrow$	–	–	0.902

Table 4: On Spider with GPT-5-mini, compile-style generation improves EA and CA over both baselines while producing more distinct structures overall.

Panel of Table 5 — Table 5: Question paraphrases change dominant SQL structures across models. GPT-5-mini has the highest SF, while Gemini-3-Pro has the highest VA.

Panel of Table 6 — Table 6: Schema reordering yields higher VA than question paraphrasing for every evaluated model. Each model exhibits majority-structure changes on some questions.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Table 7 — Table 7: Parsing coverage varies substantially across the additional subsets. GPT-5-mini has the greatest overall structural diversity in each subset, while Claude-4.5-Sonnet has no observed correct-output diversity on Spider-Syn.

Category	Direct SQL	Compile-style	$\Delta$
Schema linking	0.20	0.13	-0.07
Logical form	0.18	0.15	-0.03
Join path	0.14	0.09	-0.05
Predicate	0.16	0.10	-0.06
Structural variation	0.28	0.40	+0.12
Compilation	0.04	0.13	+0.09

Appendix

Table 8: The inspected compile-style sample has smaller shares of schema, logic, join, and predicate errors, but larger shares of structural variation and compilation errors.

Panel of Figure 4 — Figure 4: AST comparison for the minimum-horsepower example in Appendix F . The reference uses ordering and a row limit; the generated query uses an aggregate subquery. Both return the same result on the evaluated database.

Text-to-SQL Evaluation

See all 23 papers

PExA: Parallel Exploration Agent for Complex Text-to-SQL
Apr 24, 2026Tanmay Parekh, Ella Hofmann-Coyle, Shuyi Wang +3LLM Inference Efficiency Text-to-SQL
University of California Los Angeles · Bloomberg
PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems
Oct 15, 2026Shuo Lin, Jinzhe Li, Chenlu Guo +3Text-to-SQL Text-to-SQL Evaluation
School of Artificial Intelligence, Jilin University · International Center of Future Science, Jilin University · Bilibili Inc. +2
Test-Time Verification for Text-to-SQL via Outcome Reward Models
Jun 29, 2026Mattia Tritto, Giuseppe Farano, Dario Di Palma +4Reward Modeling LLM Answer Verification
Polytechnic University of Bari, Bari, Italy · IBM T.J. Watson Research Center, Yorktown Heights, NY, USA

Apr 24, 2026J/K move · Enter open · S save

PExA: Parallel Exploration Agent for Complex Text-to-SQL

Tanmay Parekh, Ella Hofmann-Coyle, Shuyi Wang, Sachith Sri Ram Kothur+2

University of California Los Angeles · Bloomberg

LLM Inference Efficiency Text-to-SQL LLM Agents

Open paper AskPDF arXiv

LLM-based agents for text-to-SQL often struggle with latency-performance trade-off, where performance improvements come at the cost of latency or vice versa. We reformulate text-to-SQL generation within the lens of software test coverage where the original query is prepared with a suite of test cases with simpler, atomic SQLs that are executed in parallel and together ensure semantic coverage of the original query. After iterating on test case coverage, the final SQL is generated only when enough information is gathered, leveraging the explored test case SQLs to ground the final generation. We validated our framework on a state-of-the-art benchmark for text-to-SQL, Spider 2.0, achieving a new state-of-the-art with 70.2% execution accuracy.

Abstract

Text-to-SQL Evaluation

Explore similar work

PExA: Parallel Exploration Agent for Complex Text-to-SQL

PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems

Test-Time Verification for Text-to-SQL via Outcome Reward Models