We present SpotIt+, an open-source tool for evaluating Text-to-SQL systems via bounded equivalence verification. Given a generated SQL query and the ground truth, SpotIt+ actively searches for database instances that differentiate the two queries. To ensure that the generated counterexamples reflect practically relevant discrepancies, we introduce a best-effort constraint-mining pipeline that combines rule-based specification mining with LLM-based validation over example databases. Experimental results on the BIRD dataset show that the mined constraints enable SpotIt+ to generate more realistic differentiating databases, while preserving its ability to efficiently uncover numerous discrepancies between generated and gold SQL queries that are missed by standard test-based evaluation.
Figures & tables
Fig. 2 : The workflow of SpotIt + . The ‘Equivalent’ outcome denotes bounded equivalence up to K rows per table.
Database
Domain
Tables
Columns
Rows
Questions
california_schools
Education
3
89
9,980
89
card_games
Entertainment
6
115
133,908
191
codebase_community
Technology
8
71
92,581
186
debit_card_specializing
Finance
5
21
84,610
64
european_football_2
Sports
7
103
31,828
129
financial
Finance
8
55
134,960
106
TABLE I: Statistics of the BIRD development set [ 6 ] .
Entry
Acronym
Alpha-SQL + Qwen2.5-Coder-32B [ 17 ]
Alpha
CSC-SQL + Qwen2.5-Coder-7B [ 18 ]
csc-7b
CSC-SQL + XiYanSQL [ 18 ]
csc-32b
GenaSQL-1 [ 19 ]
gena-1
GenaSQL-2 [ 19 ]
gena-2
RSL-SQL + GPT-4o [ 20 ]
rsl
TABLE II : The considered Text-to-SQL methods.
EX-test
EX-SpotIt
EX-SpotIt + -noV
EX- SpotIt +
Acc. (%)
Rnk
Acc. (%)
Rnk
Acc. (%)
Rnk
Acc. (%)
Rnk
csc-32b
71.32
1
60.10
3
63.56
2
63.43
2
gena-2
70.53
2
60.89
1
63.75
1
63.56
1
Alpha
69.36
3
56.91
6
61.21
5
60.56
5
gena-1
69.23
4
60.50
2
63.23
3
63.17
3
csc-7b
69.17
5
59.91
4
62.58
4
62.26
4
TABLE III: Performance of Text-to-SQL methods evaluated under EX-test , SpotIt , SpotIt + -noV , and SpotIt + on the 1533 BIRD-dev benchmarks.
SpotIt
SpotIt + -noV
SpotIt +
Method
Avg.
Med.
Avg.
Med.
Avg.
Med.
csc-32b
2.87
0.33
0.72
0.46
0.85
0.47
gena-2
1.06
0.30
2.26
0.36
0.98
0.34
Alpha
3.67
0.36
0.99
0.58
1.57
0.55
gena-1
0.82
0.39
0.66
0.40
0.57
0.37
csc-7b
0.58
0.29
3.08
0.36
1.37
0.34
TABLE IV : Average and median runtime (in seconds) for counterexample generation across Text-to-SQL methods.
Text-to-SQL systems often struggle with deep contextual understanding, particularly for complex queries with subtle requirements. We present PV-SQL, an agentic framework that addresses these failures through two complementary components: Probe and Verify. The Probe component iteratively generates probing queries to retrieve concrete records from the database, resolving ambiguities in value formats, column semantics, and inter-table relationships to build richer contextual understanding. The Verify component employs a rule-based method to extract verifiable conditions and construct an executable checklist, enabling iterative SQL refinement that effectively reduces missing constraints. Experiments on the BIRD benchmarks show that PV-SQL outperforms the best text-to-SQL baseline by 5% in execution accuracy and 20.8% in valid efficiency score while consuming fewer tokens.
In Text-to-SQL tasks, large language models can generate structurally different SQL queries that return correct answers for the same intent. We call this phenomenon execution-correct structural divergence (ECSD). We introduce SQLStructEval, a framework that analyzes this behavior through canonical abstract syntax tree representations. We quantify structural diversity and agreement across generations. Experiments with different LLMs on multiple Text-to-SQL datasets, including Spider, document ECSD across models and datasets. Furthermore, our experiments demonstrate that generated queries are sensitive to question paraphrases and schema presentation. To address ECSD, we adopt a pipeline that first generates structured intermediate representations and then deterministically compiles them into SQL, improving execution accuracy and structural agreement among correct outputs. Structural analysis thus provides an additional diagnostic perspective that complements execution-based evaluation. Code is available at https://xanderzhou2022.github.io/AACL2026-SQLSTRUCTEVAL/.
Yixi Zhou, Fan Zhang, Zhiqiao Guo +4
Hong Kong Baptist University · ShanghaiTech University · The University of Tokyo +2
Text-to-SQL systems are commonly evaluated using ground-truth SQL queries or reference execution results, but such supervision is unavailable at inference time in real-world deployments. This creates a critical verification problem: given only a user question, database context, and generated SQL, can a system estimate whether the generated query is likely to correctly answer the question? Recent approaches use LLMs as judge or specialized agents to inspect generated SQL, but their decisions can be difficult to trace. Outcome Reward Models (ORMs) address this by learning from execution-labeled candidate SQLs and assigning correctness scores to unseen queries, yet they still provide limited visibility into the signals behind each verification. To address this limitation, we propose TraceSQL, a lightweight and traceable verification model built on explicit diagnostic features. TraceSQL combines 67 features capturing question ambiguity, question requirements, question-schema-SQL consistency, SQL structure, and intent alignment. These signals remain available for examining which factors influence each prediction and for tracing decisions back to diagnostic evidence. On BIRD development databases, TraceSQL achieves 66.47% F1 and 64.48% ROC-AUC, compared with 61.87% F1 and 58.26% ROC-AUC for the GradeSQL-7B ORM baseline on the same generated-SQL evaluation. Feature attribution further shows that the model relies on both semantic grounding and deterministic SQL-structure signals. These results show that SQL verification can be performed with a lightweight learned model while retaining feature-level evidence for inspecting and diagnosing its predictions.