We present SpotIt+, an open-source tool for evaluating Text-to-SQL systems via bounded equivalence verification. Given a generated SQL query and the ground truth, SpotIt+ actively searches for database instances that differentiate the two queries. To ensure that the generated counterexamples reflect practically relevant discrepancies, we introduce a best-effort constraint-mining pipeline that combines rule-based specification mining with LLM-based validation over example databases. Experimental results on the BIRD dataset show that the mined constraints enable SpotIt+ to generate more realistic differentiating databases, while preserving its ability to efficiently uncover numerous discrepancies between generated and gold SQL queries that are missed by standard test-based evaluation.
Figures & tables
Fig. 2 : The workflow of SpotIt + . The ‘Equivalent’ outcome denotes bounded equivalence up to K rows per table.
Database
Domain
Tables
Columns
Rows
Questions
california_schools
Education
3
89
9,980
89
card_games
Entertainment
6
115
133,908
191
codebase_community
Technology
8
71
92,581
186
debit_card_specializing
Finance
5
21
84,610
64
european_football_2
Sports
7
103
31,828
129
financial
Finance
8
55
134,960
106
TABLE I: Statistics of the BIRD development set [ 6 ] .
Entry
Acronym
Alpha-SQL + Qwen2.5-Coder-32B [ 17 ]
Alpha
CSC-SQL + Qwen2.5-Coder-7B [ 18 ]
csc-7b
CSC-SQL + XiYanSQL [ 18 ]
csc-32b
GenaSQL-1 [ 19 ]
gena-1
GenaSQL-2 [ 19 ]
gena-2
RSL-SQL + GPT-4o [ 20 ]
rsl
TABLE II : The considered Text-to-SQL methods.
EX-test
EX-SpotIt
EX-SpotIt + -noV
EX- SpotIt +
Acc. (%)
Rnk
Acc. (%)
Rnk
Acc. (%)
Rnk
Acc. (%)
Rnk
csc-32b
71.32
1
60.10
3
63.56
2
63.43
2
gena-2
70.53
2
60.89
1
63.75
1
63.56
1
Alpha
69.36
3
56.91
6
61.21
5
60.56
5
gena-1
69.23
4
60.50
2
63.23
3
63.17
3
csc-7b
69.17
5
59.91
4
62.58
4
62.26
4
TABLE III: Performance of Text-to-SQL methods evaluated under EX-test , SpotIt , SpotIt + -noV , and SpotIt + on the 1533 BIRD-dev benchmarks.
SpotIt
SpotIt + -noV
SpotIt +
Method
Avg.
Med.
Avg.
Med.
Avg.
Med.
csc-32b
2.87
0.33
0.72
0.46
0.85
0.47
gena-2
1.06
0.30
2.26
0.36
0.98
0.34
Alpha
3.67
0.36
0.99
0.58
1.57
0.55
gena-1
0.82
0.39
0.66
0.40
0.57
0.37
csc-7b
0.58
0.29
3.08
0.36
1.37
0.34
TABLE IV : Average and median runtime (in seconds) for counterexample generation across Text-to-SQL methods.