Natural Language to SQL (NL2SQL) technology empowers non-expert users to query relational databases without requiring SQL expertise. While large language models (LLMs) have greatly improved NL2SQL algorithms, their rapid development outpaces systematic evaluation, leaving a critical gap in understanding their effectiveness, efficiency, and limitations. To this end, we present NL2SQLBench, the first modular evaluation and benchmarking framework for LLM-enabled NL2SQL approaches. Specifically, we dissect NL2SQL systems into three core modules: Schema Selection, Candidate Generation, and Query Revision. For each module, we comprehensively review existing strategies and propose novel fine-grained metrics that systematically quantify module-level effectiveness and efficiency. We further implement these metrics in a flexible multi-agent framework, allowing configurable benchmarking across diverse NL2SQL approaches. Leveraging NL2SQLBench, we rigorously evaluate ten representative open-source methods on two datasets, the BIRD development set and the ScienceBenchmark development set, using two LLMs, DeepSeek-V3 and GPT-4o mini. We systematically assess each approach across the three core modules and evaluate multiple critical performance dimensions. Our evaluation reveals significant gaps in existing NL2SQL methods, highlighting not only substantial room for accuracy improvements but also the significant computational inefficiency, which severely hampers real-world adoption. Furthermore, our analysis identifies critical shortcomings in current benchmark datasets and evaluation rules, emphasizing issues such as inaccurate gold SQL annotations and limitations in existing evaluation rules. By synthesizing these insights into a unified benchmarking, our study establishes a clear reference point for fair comparison and serves as essential guidance for future targeted innovation in NL2SQL technology.
Figures & tables
Figure 1. Overview of LLM-enabled NL2SQL approaches and our benchmarking framework.
Approach
Few-shot CoT
Multi-Stage Pruning
Preliminary SQL
C3-SQL ( Dong et al., 2023 )
✗
✓
✗
DIN-SQL ( Pourreza and Rafiei, 2023a )
✓
✗
✗
MAC-SQL ( Wang et al., 2025 )
✓
✗
✗
CHESS (IR,SS,CG) ( Talaei et al., 2024 )
✓
✓
✗
TA-SQL ( Qu et al., 2024 )
✗
✗
✓
OpenSearch-SQL ( Xie et al., 2025 )
✓
✓
✗
Table 1. Classification of Schema Selection strategies.
Approach
Few-shot CoT
Classifi cation
Decom position
Inter Represent
Multi Candidate
C3-SQL ( Dong et al., 2023 )
✗
✗
✗
✗
✓
DIN-SQL ( Pourreza and Rafiei, 2023a )
✓
✓
✗
NatSQL
✗
MAC-SQL ( Wang et al., 2025 )
✓
✗
✓
✗
✗
CHESS (IR,SS,CG) ( Talaei et al., 2024 )
Zero-Shot
✗
✗
✗
✗
CHESS (IR,CG,UT) ( Talaei et al., 2024 )
Zero-Shot
✗
✗
✗
✓
TA-SQL ( Qu et al., 2024 )
✗
✗
✗
Pandas-Like
✗
Table 2. Classification of Candidate Generation strategies.
Approach
LLM Based
Execution Guided
Consistency Based
Unit Test
Rank Select
Question Rewrite
C3-SQL ( Dong et al., 2023 )
✗
Single-Turn
✓
✗
✗
✗
DIN-SQL ( Pourreza and Rafiei, 2023a )
✓
✗
✗
✗
✗
✗
MAC-SQL ( Wang et al., 2025 )
✗
Up to 3
✗
✗
✗
✗
CHESS (IR,SS,CG) ( Talaei et al., 2024 )
✗
Single-Turn
✓
✗
✗
✗
CHESS (IR,CG,UT) ( Talaei et al., 2024 )
✗
Single-Turn
✗
✓
✗
✗
OpenSearch-SQL ( Xie et al., 2025 )
✗
Single-Turn
✓
✗
✗
✗
Table 3. Classification of Query Revision strategies.
Approach
Table Selection
Column Selection
Efficiency
Precis- ion (%)
Recall (%)
F1-score (%)
Precis- ion (%)
Recall (%)
F1-score (%)
#Tok- ens
#LLM calls
C3-SQL
50.03
98.80
64.63
27.88
95.03
41.76
15886
20
DIN-SQL
93.27
95.47
93.41
89.10
88.76
88.01
7360
1
MAC-SQL
32.74
99.84
46.70
15.84
98.70
26.12
3179
1
CHESS (IR,SS,CG)
94.50
95.85
94.35
89.78
88.00
87.66
307894
78.56
TA-SQL
93.66
95.73
93.77
82.46
90.77
84.89
4249
1
Table 4. Analysis results of Schema Selection on BIRD using DeepSeek-V3. The best results are in bold and underlined while the second-best results are underlined .
Figure 2. Execution accuracy using different schemas.
Approach
DeepSeek-V3
GPT-4o-mini
CR (%)
IR (%)
ER (%)
#Tok.
#Calls
CR (%)
IR (%)
ER (%)
#Tok.
#Calls
C3-SQL
38.14
50.07
11.80
11786
20
35.66
51.83
12.52
1353
20
DIN-SQL
59.58
35.52
4.89
6680
2
56.65
34.49
8.87
6523
2
MAC-SQL
58.74
33.83
7.43
2829
1
52.74
35.92
11.34
3031
1
CHESS (IR,SS,CG)
59.78
33.12
7.11
1017
1
53.42
41.03
5.54
934
1
CHESS (IR,CG,UT)
62.91
33.12
3.98
234394
20
55.08
32.01
12.91
228177
20
Table 5. Results of Candidate Generation on BIRD dev with two LLMs. The best results inside each block are bold+underlined , the second-best are underlined . We use Pass@1 for methods producing multiple candidates.
Approach
DeepSeek-V3
GPT-4o-mini
CR (%)
IR (%)
ER (%)
#Tok.
#Calls
CR (%)
IR (%)
ER (%)
#Tok.
#Calls
C3-SQL
54.18
42.81
3.01
12110
20
50.84
42.81
6.35
1350
20
DIN-SQL
52.84
44.82
2.34
6149
2
42.81
48.49
8.70
5932
2
MAC-SQL
47.16
42.81
10.03
3514
1
44.15
41.81
14.05
3734
1
CHESS (IR,SS,CG)
37.92
55.37
6.71
978
1
53.42
41.03
5.54
934
1
CHESS (IR,CG,UT)
54.85
39.46
5.69
259569
20
40.80
46.82
12.37
247969
20
Table 6. Results of Candidate Generation on ScienceBenchmark. The best results are bold+underlined and the second-best are underlined . We report Pass@1 for methods producing multiple candidates.
Figure 3. Pass@k results for multiple-candidate approaches.
Figure 4. Analysis of the Query Revision module on BIRD.
Figure 5. Analysis of the Query Revision module on ScienceBenchmark.
Figure 6. The coefficient heatmap of different solutions on Incorrect cases.
Figure 7. Breakdown of questions by number of approaches producing Correct queries (for different difficulty levels).
Approach
DeepSeek-V3
GPT-4o-mini
CI (%)
I2C (%)
E2C (%)
#Tok.
#Calls
CI (%)
I2C (%)
E2C (%)
#Tok.
#Calls
C3-SQL
-0.51
1.43
4.97
–
–
2.56
2.52
8.33
–
–
DIN-SQL
0.66
2.02
21.33
4948
1
-0.11
4.16
19.12
4791
1
MAC-SQL
3.55
3.66
13.16
1566
2
4.45
1.45
21.84
1729
2
CHESS (IR,SS,CG)
6.11
6.89
29.36
1844
1
6.72
10.49
35.29
1683
1
CHESS (IR,CG,UT)
2.18
15.16
47.54
106126
21
14.04
16.09
38.38
115055
23.19
Table 7. Effectiveness and efficiency on Query Revision on BIRD dev with DeepSeek-V3 vs GPT-4o-mini. We report CI/I2C/E2C and #Tokens, #LLM Calls. The best/second-best results are bold+underlined / underlined within each LLM.
Approach
DeepSeek-V3
GPT-4o-mini
CI (%)
I2C (%)
E2C (%)
#Tok.
#Calls
CI (%)
I2C (%)
E2C (%)
#Tok.
#Calls
C3-SQL
0
0
0
–
–
0
0
0
–
–
DIN-SQL
1.27
2.99
14.29
4349
1
2.34
1.38
15.38
4156
1
MAC-SQL
1.42
2.34
10
2395
2
0.76
3.20
9.52
2676
2
CHESS (IR,SS,CG)
19.47
14.55
0
2319
1
26.32
13.30
14.29
1683
1
CHESS (IR,CG,UT)
1.83
9.32
52.94
201372
23.42
20.49
12.14
43.24
115055
23.19
Table 8. Effectiveness and efficiency on Query Revision on ScienceBenchmark dev with DeepSeek-V3 vs GPT-4o-mini. We report CI/I2C/E2C and #Tokens, #LLM Calls. The best/second-best results are bold+underlined / underlined .
Approach
DeepSeek-V3
GPT-4o-mini
Correct Rate (%)
#Tok- ens
#LLM Calls
Cost ($)
Correct Rate (%)
#Tok- ens
#LLM Calls
Cost ($)
C3-SQL
37.94
27673
40
0.0126
36.57
12332
40
0.0067
DIN-SQL
59.97
18988
3
0.0038
56.58
18511
3
0.0024
MAC-SQL
60.82
7574
4
0.0019
55.08
7867
4
0.0013
CHESS (IR,SS,CG)
63.43
310756
80.6
0.0604
57.01
300061
80.5
0.0364
CHESS (IR,CG,UT)
64.28
340601
41.8
0.0816
62.82
343232
43.2
0.0525
Table 9. End-to-end results on BIRD dev with two LLMs. We report Correct Rate , efficiency (#Tokens, #LLM Calls) and Cost . The best/second-best results are bold+underlined / underlined within each LLM block.
Approach
DeepSeek-V3
GPT-4o-mini
Correct Rate (%)
#Tok- ens
#LLM Calls
Cost ($)
Correct Rate (%)
#Tok- ens
#LLM Calls
Cost ($)
C3-SQL
54.18
30979
40
0.0152
50.84
14179
40
0.0078
DIN-SQL
53.51
17299
3
0.0035
43.81
16660
3
0.0022
MAC-SQL
47.83
9290
4
0.0023
44.48
9577
4
0.0015
CHESS (IR,SS,CG)
45.30
355983
92.59
0.0677
57.01
300061
80.5
0.0364
CHESS (IR,CG,UT)
55.85
460941
43.42
0.1018
49.16
393208
47.6
0.0428
Table 10. End-to-end results on ScienceBenchmark dev with two LLMs. We report Correct Rate , efficiency (#Tokens, #LLM Calls) and Cost . The best/second-best results are bold+underlined / underlined within each LLM block.
Benchmarking Framework
Evaluation Objectives
E2E Eval
Modular Eval
Fine-grain- ed Metrics
Error Analysis
Practical Guides
Text2SQL Benchmarks ( Mitsopoulou and Koutrika, 2025 )
Datasets
–
–
–
Yes
–
SQLDriller ( Yang et al., 2025 )
Datasets
–
–
–
Yes
–
NL2SQL-BUGs ( Liu et al., 2025b )
Datasets
–
–
–
Yes
–
DR-Spider ( Chang et al., 2023 )
Systems
Yes
No
No
Yes
Partial
Cross-Domain Text2SQL ( Pourreza and Rafiei, 2023b )
Systems
Yes
No
No
Yes
Partial
Text2SQL by LLMs ( Gao et al., 2024a )
Systems
Yes
No
No
No
Partial
Table 11. Comparison of evaluation frameworks for NL2SQL datasets and systems.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Approach
Table Selection
Column Selection
Efficiency
Precis- ion (%)
Recall (%)
F1-score (%)
Precis- ion (%)
Recall (%)
F1-score (%)
#Tok- ens
#LLM calls
C3-SQL
49.79
98.46
64.35
26.80
93.75
40.30
10979
20
DIN-SQL
92.08
95.89
92.83
87.49
85.51
85.47
7197
1
MAC-SQL
32.74
99.84
46.70
14.83
96.63
24.49
3107
1
CHESS (IR,SS,CG)
90.96
95.80
92.17
88.74
74.14
78.96
297443
78.53
TA-SQL
92.56
95.38
92.88
81.78
88.44
83.25
4119
1
Appendix
Table 12. Analysis results of Schema Selection on BIRD dev set using GPT-4o-mini. The best results are in bold and underlined while the second-best results are underlined
Approach
Table Selection
Column Selection
Efficiency
Precis- ion (%)
Recall (%)
F1-score (%)
Precis- ion (%)
Recall (%)
F1-score (%)
#Tok- ens
#LLM calls
C3-SQL
49.50
97.06
62.98
30.69
94.30
43.64
18869
20
DIN-SQL
89.58
81.15
83.48
79.80
73.20
73.92
6802
1
MAC-SQL
18.56
100
28.93
11.15
96.39
18.42
3380
1
CHESS (IR,SS,CG)
91.23
82.44
84.86
76.48
73.51
72.42
352686
92.59
TA-SQL
92.72
90.20
90.13
74.16
85.27
76.39
5155
1
Appendix
Table 13. Analysis results of Schema Selection on ScienceBenchmark dev set using DeepSeek-V3. The best results are in bold and underlined while the second-best results are underlined
Approach
Table Selection
Column Selection
Efficiency
Precis- ion (%)
Recall (%)
F1-score (%)
Precis- ion (%)
Recall (%)
F1-score (%)
#Tok- ens
#LLM calls
C3-SQL
49.25
95.39
62.19
29.17
91.50
41.51
12829
20
DIN-SQL
90.18
81.71
83.65
77.23
68.77
70.65
6571
1
MAC-SQL
18.56
100
28.93
10.48
96.22
17.26
3167
1
CHESS (IR,SS,CG)
90.96
95.80
92.17
88.74
74.14
78.96
341059
92.59
TA-SQL
90.17
86.36
86.38
67.98
79.63
70.19
5049
1
Appendix
Table 14. Analysis results of Schema Selection on ScienceBenchmark dev set using GPT-4o-mini. The best results are in bold and underlined while the second-best results are underlined
Approach
Error Rates (%)
Total (%)
No Tabl/Col
No Func
Syntax Err
Timeout
Others
C3-SQL
3.32
0
0.65
0.52
7.30
11.80
DIN-SQL
0.98
0.13
3.12
0.26
0.39
4.89
MAC-SQL
1.11
0.07
1.83
0.26
4.17
7.43
CHESS (IR,SS,CG)
0.46
1.43
1.69
3.32
0.20
7.11
CHESS (IR,CG,UT)
1.30
0
2.28
0.20
0.20
3.98
Appendix
Table 15. Error analysis on BIRD dev set with DeepSeek-V3 for Candidate Generation. The best results are in bold and underlined while the second-best results are underlined . We use Pass@1 results for approaches producing multiple candidates.
Approach
Error Rates (%)
Total (%)
No Tabl/Col
No Func
Syntax Err
Timeout
Others
C3-SQL
9.19
0
1.96
0.07
1.3
12.52
DIN-SQL
5.48
0.46
2.41
0.2
0.33
8.87
MAC-SQL
6.26
1.43
0.85
0.33
2.48
11.34
CHESS (IR,SS,CG)
2.54
0.72
1.11
0.91
0.26
5.54
CHESS (IR,CG,UT)
9.32
1.76
1.63
0
0.2
12.91
Appendix
Table 16. Error analysis on BIRD Dev using GPT-4o-mini for Candidate Generation . The best results are in bold and underlined while the second-best results are underlined . We use Pass@1 results for approaches producing multiple candidates.
Approach
Error Rates (%)
Total (%)
No Tabl/Col
No Func
Syntax Err
Timeout
Others
C3-SQL
3.01
0
0
0
0
3.01
DIN-SQL
2.01
0
0
0
0.33
2.34
MAC-SQL
4.35
0
0.33
0
5.35
10.03
CHESS (IR,SS,CG)
0.67
0
0
6.02
0
6.69
CHESS (IR,CG,UT)
3.01
0
0
0
2.68
5.69
Appendix
Table 17. Error analysis on ScienceBenchmark Dev using DeepSeek-V3 for Candidate Generation. The best results are in bold and underlined while the second-best results are underlined . We use Pass@1 results for approaches producing multiple candidates.
Approach
Error Rates (%)
Total (%)
No Tabl/Col
No Func
Syntax Err
Timeout
Others
C3-SQL
4.68
0
0
0
1.67
6.35
DIN-SQL
8.36
0
0
0.33
0
8.70
MAC-SQL
8.70
0
0
0.33
5.02
14.05
CHESS (IR,SS,CG)
2.01
0
0
0.33
0
2.34
CHESS (IR,CG,UT)
12.37
0
0
0
0
12.37
Appendix
Table 18. Error analysis on ScienceBenchmark Dev using GPT-4o-mini for Candidate Generation. The best results are in bold and underlined while the second-best results are underlined . We use Pass@1 results for approaches producing multiple candidates.
Figure 8. Correct Rates, Incorrect Rates, and Error Rates on BIRD dev set using DeepSeek-V3 for the Candidate Generation module across difficulty levels.
Figure 9. Correct Rates, Incorrect Rates, and Error Rates on BIRD dev set using GPT-4o-mini for the Candidate Generation module across difficulty levels.
Figure 10. Correct Rates, Incorrect Rates, and Error Rates on ScienceBenchmark dev set using DeepSeek-V3 for the Candidate Generation module across difficulty levels.
Figure 11. Correct Rates, Incorrect Rates, and Error Rates on ScienceBenchmark dev set using GPT-4o-mini for the Candidate Generation module across difficulty levels.
Figure 12. Correct Rates, Incorrect Rates, and Error Rates on BIRD dev set using DeepSeek-V3 for Query Revision module across difficulty levels.
Figure 13. Correct Rates, Incorrect Rates, and Error Rates on BIRD dev set using GPT-4o-mini for Query Revision module across difficulty levels.
Figure 14. Correct Rates, Incorrect Rates, and Error Rates on ScienceBenchmark dev set using DeepSeek-V3 for Query Revision module across difficulty levels.
Figure 15. Correct Rates, Incorrect Rates, and Error Rates on ScienceBenchmark dev set using GPT-4o-mini for Query Revision module across difficulty levels.
Figure 16. The coefficient heatmap of different solutions on Incorrect cases (using GPT-4o-mini).
Approach
CI (%)
I2C (%)
E2C (%)
C2I (%)
C2E (%)
#Tok- ens
#LLM Calls
C3-SQL
-0.51
1.43
4.97
3.93
0.00
–
–
DIN-SQL
0.66
2.02
21.33
0.98
1.31
4948
1
MAC-SQL
3.55
3.66
13.16
0.22
0.00
1566
2
CHESS (IR,SS,CG)
6.11
6.89
29.36
0.65
0.55
1844
1
CHESS (IR,CG,UT)
2.18
15.16
47.54
8.81
0.00
106126
21
GSR
7.27
9.49
38.10
2.16
0.34
2220
2
Appendix
Table 19. Effectiveness and efficiency on Query Revision on BIRD dev with DeepSeek-V3. The best results are in bold and underlined while the second-best results are underlined .
Approach
CI (%)
I2C (%)
E2C (%)
C2I (%)
C2E (%)
#Tok- ens
#LLM Calls
C3-SQL
2.56
2.52
8.33
4.02
0
–
–
DIN-SQL
-0.11
4.16
19.12
4.37
1.27
4791.23
1
MAC-SQL
4.45
1.45
21.84
1.24
0
1729.12
2
CHESS (IR,SS,CG)
6.72
10.49
35.29
4.64
0.37
1683.28
1
CHESS (IR,CG,UT)
14.04
16.09
38.38
4.26
0.12
115055.19
23.19
GSR
2.6
4.85
21.3
3.71
0.25
2448.84
2
Appendix
Table 20. Effectiveness and efficiency on Query Revision on BIRD dev with GPT-4o-mini. The best results are in bold and underlined while the second-best results are underlined .
Approach
CI (%)
I2C (%)
E2C (%)
C2I (%)
C2E (%)
#Tok- ens
#LLM Calls
C3-SQL
0
0
0
0
0
–
–
DIN-SQL
1.27
2.99
14.29
0.63
1.27
4349.08
1
MAC-SQL
1.42
2.34
10
2.84
0
2395.41
2
CHESS (IR,SS,CG)
19.47
14.55
0
1.77
0
2319.17
1
CHESS (IR,CG,UT)
1.83
9.32
52.94
10.37
0
201372.44
23.42
GSR
-24.8
14.08
33.33
23.14
28.1
1756.85
2
Appendix
Table 21. Effectiveness and efficiency on Query Revision on ScienceBenchmark dev with DeepSeek-V3. The best results are in bold and underlined while the second-best results are underlined .
Approach
CI (%)
I2C (%)
E2C (%)
C2I (%)
C2E (%)
#Tok- ens
#LLM Calls
C3-SQL
0
0
0
0
0
–
–
DIN-SQL
2.34
1.38
15.38
1.56
0.78
4156.48
1
MAC-SQL
0.76
3.20
9.52
5.30
0
2675.69
2
CHESS (IR,SS,CG)
26.32
13.30
14.29
2.11
0
1683.28
1
CHESS (IR,CG,UT)
20.49
12.14
43.24
6.56
0
115055.19
23.19
GSR
4.96
5.13
18.18
4.13
0.83
1498.20
2
Appendix
Table 22. Effectiveness and efficiency on Query Revision on ScienceBenchmark dev with GPT-4o-mini. The best results are in bold and underlined while the second-best results are underlined .
Approach
Candidate Generation Correct Rate (%)
Query Revision Correct Rate (%)
Recall = 1
Recall <1
Recall = 1
Recall <1
C3 - SQL (Pass@1)
42.51%
15.10%
42.28%
15.10%
C3 - SQL (Pass@5)
45.31%
16.33%
42.28%
15.10%
C3 - SQL (Pass@10)
46.16%
16.33%
42.28%
15.10%
C3 - SQL (Pass@15)
46.47%
17.55%
42.28%
15.10%
C3 - SQL (Pass@20)
46.86%
18.78%
42.28%
15.10%
Appendix
Table 23. Correct-rate analysis on BIRD Dev + DeepSeek-V3 conditioned on schema recall. We report percentages (higher is better).
Approach
Candidate Generation
Query Revision
Correct Recall
Wrong Recall
Correct Recall
Wrong Recall
C3 - SQL (Pass@1)
0.9848
0.9291
0.9837
0.9300
C3 - SQL (Pass@5)
0.9834
0.9277
0.9837
0.9300
C3 - SQL (Pass@10)
0.9837
0.9268
0.9837
0.9300
C3 - SQL (Pass@15)
0.9829
0.9269
0.9837
0.9300
C3 - SQL (Pass@20)
0.9817
0.9273
0.9837
0.9300
Appendix
Table 24. Schema selection recall (higher is better) under four scenarios: Candidate Generation module generates correct SQL, wrong SQL, and Query Revision module generates correct SQL, wrong SQL. For methods producing multiple candidates, we list results at different Pass@k .
Robustness evaluation for Natural Language to SQL (NL2SQL) systems is essential because real-world database environments are dynamic, noisy, and continuously evolving, whereas conventional benchmark evaluations typically assume static schemas and well-formed user inputs. In this work, we introduce a robustness evaluation benchmark containing approximately ten types of perturbations and conduct evaluations under both traditional and agentic settings. We assess multiple state-of-the-art large language models (LLMs), including Grok-4.1, Gemini-3-Pro, Claude-Opus-4.6, and GPT-5.2. Our results show that these models generally maintain strong performance under several perturbations; however, notable performance degradation is observed for surface-level noise (e.g., character-level corruption) and linguistic variation that preserves semantics while altering lexical or syntactic forms. Furthermore, we observe that surface-level noise causes larger performance drops in traditional pipelines, whereas linguistic variation presents greater challenges in agentic settings. These findings highlight the remaining challenges in achieving robust NL2SQL systems, particularly in handling linguistic variability.
The natural language to SQL (NL2SQL) task plays a pivotal role in democratizing data access by enabling non-expert users to interact with relational databases through intuitive language. While recent frameworks have enhanced translation accuracy via task specialization, their reliance on Large Language Models (LLMs) raises significant concerns regarding computational overhead, data privacy, and real-world deployability in resource-constrained environments. To address these challenges, we propose a schema based agentic system that strategically employs Small Language Models (SLMs) as primary agents, complemented by a selective LLM fallback mechanism. The LLM is invoked only upon detection of errors in SLM-generated output, the proposed system significantly minimizes computational expenditure. Experimental results on the BIRD benchmark demonstrate that our system achieves an execution accuracy of 47.78% and a validation efficiency score of 51.05%, achieving over 90% cost reduction compared to LLM-centric baselines as approximately 67% of queries are resolved using local SLMs. The system achieves an average cost per query of 0.0085 compared to 0.094 for LLM-only systems, achieving near-zero operational costs for locally executed queries. [Github repository: https://github.com/mindslab25/CESMA.]
David Onyango, Naseef Mansoor
Project Path, Boston, MA, USA · Minnesota State University Mankato, MN, USA
Natural language to SQL (NL-to-SQL) benchmarks are foundational to progress in data analysis research, yet recent work has shown that widely-used benchmarks contain significant annotation errors. These errors silently corrupt evaluation metrics, penalize correct model output, and distort the field's understanding of state-of-the-art performance. We present SALUS, a system that automatically detects annotation errors in NL-to-SQL benchmarks. SALUS frames benchmark auditing as a weakly supervised error detection: SQL generated by multiple LLM agents drive a suite of complementary weak-labeling functions. By passing this noisy vote matrix through a generative label model, we extract high-confidence training samples without requiring human ground truth. These samples train a decision plane that maps gold SQL query features to per-agent trustworthiness, allowing SALUS to intelligently fuse reliability estimates with raw verdicts for rigorous benchmark error detection. We evaluate on BIRD-Clean-xs, a benchmark of 298 BIRD development tasks with manually verified correctness labels. SALUS achieves F1 = 0.9194, significantly outperforming the state-of-the-art baselines. Applying SALUS to the full development sets, we estimate annotation error rates of approximately 37% on BIRD and 27% on Spider.