Open-source LLMs provide a flexible alternative to closed-source models for Text-to-SQL but still struggle with complex queries. Finetuning with chain-of-thought (CoT) data improves reasoning, yet standard datasets lack CoT annotations, requiring synthetic generation. Existing synthesis methods only verify the final answer, allowing flawed reasoning steps to corrupt the training data. We propose VCoT-SQL, a framework that improves Text-to-SQL generation through intermediate-step validated CoT supervision and a dedicated SQL validator, which enables small open-source models to outperform frontier closed-source models. VCoT-SQL trains two complementary models on curated reasoning traces: an SQL generator using clause-level CoT templates where reasoning steps are verified via execution and schema alignment checks, and an SQL validator using diagnostic CoT templates to detect syntactic, semantic, and type level errors. We further design a hierarchical schema filtering pipeline that integrates and refines complementary retrieval strategies to distill relevant database elements. VCoT-SQL achieves 66.82% execution accuracy on challenging dataset BIRD dev and 86.8% on Spider test, improving the base 7B model by 7% on BIRD. Notably, our CoT synthesis relies on 7B models yet outperforms methods that use 72B-scale or even proprietary closed source teachers for data generation, demonstrating that rigorous step-level validation can be more effective than scaling up teacher model size. Our dataset and code are available at https://github.com/INTREBID/VCoT-SQL.
Figures & tables
Figure 1: Overview of the proposed VCoT-SQL framework. (1) Model Training : two pipelines collect CoT supervision data to train the CoT-enhanced SQL Generator and Validator. (2) Schema Filtering : the question and full database are processed through Pre-SQL, Column, and Value Retrieval to obtain a filtered schema (see Figure 2 ). (3) Inference : the Generator produces an initial SQL from the filtered schema; the SQL is executed, and any syntax errors along with the initial SQL are passed to the Validator to yield the final SQL.
Method
Model
Training Data Size
Spider Dev
Spider Test
BIRD Dev
Spider- DK
Spider- Syn
Spider- Realistic
Prompting with Closed-Source LLMs
GPT-4o OpenAI (2024c)
-
-
81.9
83.0
61.93
70.8
74.2
82.9
GPT-4o-mini OpenAI (2024b)
-
-
82.9
82.8
60.43
73.6
74.2
81.7
GPT-4-turbo OpenAI (2024c)
-
-
83.8
85.2
62.06
71.8
77.5
83.2
Prompting with Open-Source LLMs
Llama3 Touvron et al. (2023)
Llama3-8B
-
69.3
69.1
39.05
57.2
62.6
72.8
Table 1: Performance of Text-to-SQL Models and Methods. To ensure fairness in comparison, we used the results from the greedy decoding released officially by OmniSQL. SQL Generator denotes single-pass generation; VCoT-SQL further applies our SQL Validator. For Training Data Size, prompting-based methods require no Text-to-SQL data (marked as "-"). Among fine-tuning methods, 18K refers to the combined Spider and BIRD training sets, 2.5M to OmniSQL’s SynSQL-2.5M, and our 9K/10K to CoT data constructed from the Spider and BIRD training splits.
Base Model
Spider Dev
Spider Test
BIRD Dev
Spider- DK
Spider- Syn
Spider- Realistic
EX
Δ
EX
Δ
EX
Δ
EX
Δ
EX
Δ
EX
Δ
Only SQL Generator
DeepSeek-Coder-6.7B-Instruct
84.9
-
86.9
-
60.63
-
74.6
-
77.5
-
80.1
-
Qwen2.5-Coder-7B-Instruct
85.3
-
86.7
-
64.86
-
76.8
-
77.6
-
82.7
-
XiYanSQL-QwenCoder-7B
87.2
-
86.4
-
65.38
-
76.6
-
80.0
-
84.3
-
w/ Base Model Validation
Table 2: Ablation study on SQL validation strategies across three backbones. EX denotes execution accuracy (%) and Δ indicates absolute improvement over the generation-only baseline.
Method
Spider Dev
Spider Test
BIRD Dev
Spider- DK
Spider- Syn
Spider- Realistic
Base Model (Qwen2.5-Coder-7B-Instruct)
82.3
81.0
58.87
70.9
71.3
80.9
FT w/ original Spider and BIRD
82.6
83.1
60.04
73.6
74.5
79.5
FT w/ unvalidated CoT-enhanced (Syntax-Order) Spider and BIRD
83.8
84.1
60.69
73.3
74.9
80.0
FT w/ unvalidated CoT-enhanced (Logic-Order) Spider and BIRD
84.4
84.9
61.60
73.5
75.0
79.9
FT w/ validated CoT-enhanced (Syntax-Order) Spider and BIRD
85.0
86.8
62.58
76.3
77.2
82.3
FT w/ validated CoT-enhanced (Logic-Order) Spider and BIRD
85.3
86.7
64.86
76.8
77.6
82.7
Table 3: Ablation Results for Validated SQL Generation on different data construction settings (EX).
Method
Spider Dev
Spider Test
BIRD Dev
Precision
Recall
EX
Precision
Recall
EX
Precision
Recall
EX
Full schema
-
-
83.6
-
-
86.6
-
-
59.71
CodeS schema filter
18.02
99.83
85.4
20.22
99.87
85.1
16.44
97.76
62.63
VCoT-SQL schema filter
30.85
99.79
85.3
34.51
99.89
86.7
31.36
98.09
64.86
w/o Pre-SQL retriever
30.77
98.94
83.8
33.92
97.74
84.4
31.39
94.91
61.15
w/o column retriever
67.35
97.02
83.6
70.02
96.26
84.5
65.48
92.30
61.08
Table 4: Comparison and ablation results of the three-path schema retrieval framework. Precision and Recall assess the discrepancy between the retrieved schema and the gold schema (only the tables and columns appearing in the gold SQL). EX evaluates the downstream generation accuracy of our Generator (Qwen2.5-Coder-7B-Instruct).
Qwen2.5-Coder -7B-Instruct
Full Schema
Schema Filter
Δ Schema Filter
Base Model
53.92
58.87
+4.95
Dual-Validation
61.34
66.49
+5.15
Δ Dual-Validation
+7.42
+7.62
—
Table 5: 2×2 ablation of Dual-Validation and Schema Filter on BIRD Dev (EX).
Method
Data Synthesis Model
Spider
BIRD
Dev
Test
Dev
EXCoT
GPT-4o
-
81.2
59.65
OmniSQL
Qwen2.5-Coder- 32B-Instruct Qwen2.5- 72B-Instruct
80.3
86.6
59.60
VCoT-SQL
Qwen2.5-Coder- 7B-Instruct Qwen2.5- 7B-Instruct
85.3
86.7
64.86
VCoT-SQL
Qwen2.5-Coder- 32B-Instruct Qwen2.5- 72B-Instruct
86.2
87.3
65.06
Table 6: Data synthesis setting and downstream performance (EX). ExCoT and OmniSQL report results trained on their synthesized CoT data from the Spider and BIRD training sets, VCoT-SQL follows the same setting. For the last row, the downstream Generator is fixed to Qwen2.5-Coder-7B-Instruct; only the data synthesis models are scaled.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: An example of the serialized schema representation used in our framework
Method
Spider Dev
Spider Test
BIRD Dev
Spider- DK
Spider- Syn
Spider- Realistic
only SQL Generator
85.3
86.7
64.86
76.8
77.6
82.7
Validator FT w/o multi-stage verification
85.6
86.7
65.78
77.0
77.9
83.1
Validator FT w/ multi-stage verification
85.7
86.9
66.49
77.2
78.5
83.3
Appendix
Table 7: Effect of multi-stage verification on CoT training data quality.
After Validation
Correct
Incorrect
Total
Before
Correct
64.67
0.20
64.86
Incorrect
1.83
33.31
35.14
Validation
Total
66.49
33.51
100
Appendix
Table 8: Transition matrix of execution correctness (%) before and after applying the validator on BIRD Dev.
Unval.
Val.
Val. Gain
Syntax-order
76.13
78.36
+2.23
Logic-order
76.55
78.99
+2.44
Logic-order Gain
+0.42
+0.63
+0.21
Appendix
Table 9: 2 × 2 factorial analysis of template ordering and CoT validation.
Method
Spider Dev
Spider Test
BIRD Dev
Spider- DK
Spider- Syn
Spider- Realistic
Generator only
85.3
86.7
64.86
76.8
77.6
82.7
w/ fine-tuned 7B Validator
85.7
86.9
66.49
77.2
78.5
83.3
w/ GPT-5.4 Validator
86.3
87.4
67.99
80.9
79.3
82.7
Appendix
Table 10: Upper-bound comparison of validators using outputs from the Qwen2.5-Coder-7B-Instruct Generator.
Backbone
DB perturbation
NLQ perturbation
SQL perturbation
Average
DeepSeek-Coder-6.7B-Instruct
86.2–69.6
85.8–74.2
84.3–82.3
85.4–75.8
Qwen2.5-Coder-7B-Instruct
85.0–72.5
84.8–74.9
85.7–81.7
85.1–76.5
XiYanSQL-QwenCoder-7B
87.0–73.5
87.3–76.6
86.6–84.2
87.1–78.3
Appendix
Table 11: VCoT-SQL robustness on Dr.Spider (pre-/post-perturbation EX).
Figure 3: The prompt template for synthesizing clause-level CoT rationales in SQL generation.
Figure 4: The example 1 for synthesizing clause-level CoT rationales in SQL generation.
Figure 5: The example 2 for synthesizing clause-level CoT rationales in SQL generation.
Figure 6: The example 3 for synthesizing clause-level CoT rationales in SQL generation.
Figure 7: The prompt template for synthesizing multi-type CoT rationales in SQL validation.
Figure 8: The example 1 for synthesizing multi-type CoT rationales in SQL validation.
Figure 9: The example 2 for synthesizing multi-type CoT rationales in SQL validation.
Figure 10: The example 3 for synthesizing multi-type CoT rationales in SQL validation.
Figure 11: Inference prompt template for the SQL generation stage.
Figure 12: Inference prompt template for the SQL validation stage.
Harbin Engineering University, Harbin, China · Harbin Institute of Technology, Harbin, China · Beijing University of Posts and Telecommunications, Beijing, China