Relational databases are among the most widely deployed forms of structured knowledge, and natural language access to them requires grounding language onto schema entities and relations while handling the ambiguity inherent in how people phrase requests. Existing synthetic NL-to-SQL data generation methods largely ignore this ambiguity and produce oversimplified queries that fail to prepare models for the complexity of real-world structured knowledge access. We present FALCON, a framework that generates realistic, ambiguity-aware NL-to-SQL data matching the complexity of challenging real-world benchmarks, at low cost using compact open models. Our approach combines reserved-word SQL seeding and persona-based prompting to generate structurally complex queries, while alignment-based filtering preserves difficulty by distinguishing genuinely incorrect examples from complex but valid queries. Human evaluation confirms consistent high quality across model sizes, and our generated data exceeds existing benchmarks in both SQL complexity and natural language richness. Difficulty-stratified analysis shows models trained on FALCON data increasingly outperform baseline-trained models as query complexity increases, validating our pipeline's success in generating challenging training data. When combined with a small proportion of existing benchmark data, mixed training recovers performance on simpler queries while preserving these advantages on complex ones. The model- and database-agnostic design enables organizations to generate high-complexity NL-to-SQL training data locally without external APIs.
Figures & tables
Figure 1: Simplified architecture of the FALCON pipeline. It integrates systematic seeding, SQL and NL generation with CoT, and a multistage filtering process (Ambiguity, Reward, and Prompt-based), to produce high-quality NL-to-SQL pairs. To see a more detailed diagram of our pipeline, see Figure 6
Figure 2: Persona seeding example included in SQL generation prompts to introduce linguistic variety.
Configuration
SQL Comp.
NL Rich.
Generation Order
SQL → NL
0.42
0.31
NL → SQL
0.40
0.21
Seeding & Filtering (SQL → NL)
Baseline
0.42
0.31
+ Reserved Words
0.48
0.31
Table 1: Ablation results for generation order, seeding methods, and NL filtering strategies on SQL complexity and NL richness (n=1000) using Qwen-2.5-3B.
Method
F1
Specificity
Recall
Reward Model
93.73
71.43
92.81
Prompt Filter
91.54
28.57
95.42
LLaMA Teacher
17.65
92.86
9.80
Source2Synth
7.55
100.00
3.92
Table 2: Filtering results across different methods. F1 measures overall filtering performance. Specificity shows the percentage of low-quality examples correctly filtered out. Recall shows the percentage of high-quality examples retained. Results are shown at the best threshold for each model.
Accuracy
Precision
Recall
F1
Specificity
94.0%
66.7%
28.6%
40.0%
98.9%
Table 3: End-to-end ambiguity pipeline performance on human-annotated samples from final synthetic dataset (n=100), evaluating both detection accuracy and validity of generated alternative queries.
Model
SQL
NL
Alignment
Qwen-3B
0.96
0.95
0.94
Llama-8B
0.92
0.94
0.95
Qwen-32B
0.98
0.97
0.93
Llama-70B
0.93
0.97
0.94
Table 4: Human evaluation scores across model sizes for the FALCON-generated data that was seeded with WikiSQL and filtered. All models were evaluated according to our rubric. We used the following models: Qwen-2.5-3B, Llama-3.1-8B, Qwen-3-32B, and Llama-3.3-70B.
Dataset
SQL Comp.
NL Rich.
Spider Seeded FALCON
0.43
0.39
WikiSQL Seeded FALCON
0.28
0.39
BIRD
0.42
0.31
Spider
0.38
0.29
WikiSQL
0.20
0.28
Table 5: Comparison of the SQL complexity and NL richness of multiple datasets with the most complex scores in bold. Spider seeded is FALCON seeded on spider data. WikiSQL seeded is FALCON seeded on WikiSQL.
Error Type
Count
%
Acceptable variation
33
33.0
Spurious predicate hallucination
20
20.0
Result boundary error
18
18.0
Schema entity confusion
10
10.0
Logical negation failure
4
4.0
Degenerate join structure
4
4.0
Table 6: Distribution of error types across 100 sampled incorrect predictions from the model finetuned on FALCON-generated data, evaluated on the Spider test set.
Train → Test
EM (%)
FS
Soft-EX
Spider on Spider
35.95
0.89
0.42
Spider on Pipe
0.05
0.69
0.18
FALCON on Spider
0.80 †
0.68
0.32
FALCON on Pipe
13.80
0.83
0.48
Mixed on Spider
27.10
0.84
–
Mixed on Pipe
13.60
0.83
–
Table 7: Cross-dataset evaluation on Qwen 2.5-3B. In-distribution rows serve as baselines contextualizing the cross-dataset results. Mixed is the intended deployment configuration. EM = Exact Match, FS = Fuzzy Score, Soft-EX = Soft Execution Accuracy. † Raw EM underestimates performance due to acceptable variations undetected by string match (Appendix A.2 ).
Figure 4: A comparison of soft execution accuracy for a model finetuned on Spider and the FALCON-generated data for different complexities.
Figure 5: The fuzzy scores for a model finetuned on FALCON, Spider, and a mix of both.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: FALCON pipeline flowchart. Yellow diamonds indicate decision points. Orange boxes indicate ambiguity handling steps. Blue/purple boxes indicate alignment filtering. Green indicates the final dataset output. Red indicates discarded examples.
State/Territory
Text/bg Color
Format
Current Slogan
Current Series
Notes
Australian Capital Territory
blue/white
Yaa ⋅ nna
ACT ⋅ CELEBRATION OF A CENTURY 2013
YIL ⋅ 00A
Slogan screenprinted on plate
New South Wales
black/yellow
aa ⋅ nn ⋅ aa
NEW SOUTH WALES
BX ⋅ 99 ⋅ HI
No slogan on current series
New South Wales
black/white
aaa ⋅ nna
NSW
CPX ⋅ 12A
Optional white slimline series
Northern Territory
ochre/white
Ca ⋅ nn ⋅ aa
NT ⋅ OUTBACK AUSTRALIA
CB ⋅ 06 ⋅ ZZ
New series began in June 2011
Appendix
Table 8: Example table used in the pipeline walkthrough, drawn from WikiSQL. Used for demonstration purposes only.
Figure 7: Step-by-step pipeline walkthrough using a persona-seeded ambiguous example. Intermediate outputs are simplified for illustration purposes.
Figure 8: Step-by-step pipeline walkthrough using a reserved-word-seeded non-ambiguous example. Intermediate outputs are simplified for illustration purposes.
Model
SQL Comp.
NL Rich.
NF Qwen-3B
0.38
0.40
NF Llama-8B
0.43
0.37
NF Qwen-32B
0.37
0.37
NF Llama-70B
0.41
0.40
F Qwen-3B
0.38
0.41
F Llama-8B
0.39
0.43
Appendix
Table 9: SQL complexity and NL richness across different models (seeded with WikiSQL) before and after filtering, showing no significant differences between complexity. Before filtering is indicated by NF (No-Filtering) and after filtering is indicated by F (Filtering). All metrics were calculated on a sample size of 100 for each model. We used the following models: Qwen-2.5-3B, Llama-3.1-8B, Qwen-3-32B, and Llama-3.3-70B.
Figure 9: SQL complexity distributions for Spider train (n=7,000), Spider test (n=2,147), and FALCON-generated data (n=8,308). FALCON data exhibits higher complexity with bimodal peaks and extends substantially into harder query ranges (0.4-0.8) underrepresented in Spider.
Figure 10: Similarity scoring example. Despite high sentence similarity (0.67), entity penalties and corruption penalty reduce the final score to 0.506.
Metric
Value
Correlation Metrics
Pearson’s r
0.701
Spearman’s ρ
0.700
Appendix
Table 10: Correlation between human evaluation scores and alignment metric on 101 SQL-to-NL pairs ( p<0.001 ). Human evaluators scored examples as 0, 0.5, or 1.0 according to our rubric (Table A.9 ). Alignment scores were computed using our hybrid similarity metric combining semantic and schema-entity matching.
Model
MSE
MAE
RLVR w/ Qwen
0.0341
0.1506
FT + CodeBERT
0.0268
0.1262
FT + ModernBERT
0.0203
0.1129
Appendix
Table 11: Alignment model performance on held-out test set.
Model
SQL
NL
Alignment
WikiSQL
0.92
0.94
0.95
Spider
0.99
0.95
0.96
Appendix
Table 12: Human evaluation scores across databases on Llama-3.1-70B.
Metric
Score
Description
SQL-NL Alignment
1.0
Correct Match: Covers all NL requirements. Minor relevant additions (e.g., adding a "points" column when asking for "who scored most", setting a limit value, etc.) are permitted.
0.5
Minor Issues: Mostly correct but adds unrelated details (e.g., groups by category when not asked to) or misses 1-2 small parts of a complex request.
0.0
Incorrect: Major omissions or fundamental misunderstanding of the NL intent.
SQL Quality
1.0
Good: Efficient structure and standard formatting.
0.5
Functional: Query works but is slightly inefficient or unnecessarily complex.
0.0
Poor: Query is nonsense, invalid, or extremely convoluted.
Appendix
Table 13: The three-point scoring rubric used for human evaluation of synthetic NL-SQL pairs.
Model
Test Set
Total
EM %
FM %
FS
Exec %
Spider (5.5k)
Spider
2147
31.21
52.77
0.87
49.65
Spider (5.5k)
Pipeline
2000
0.1
3.2
0.66
26.72
Pipeline (5.5k)
Spider
2147
1.07
8.52
0.70
72.71
Pipeline (5.5k)
Pipeline
2000
9.1
42.55
0.83
84.11
Mixed 5% (5.5k)
Spider
2147
22.82
38.94
0.81
–
Mixed 5% (5.5k)
Pipeline
2000
9.2
42.8
0.83
–
Appendix
Table 14: Performance of Qwen 2.5-1.5B LoRA variants (2 epochs) on Spider and FALCON test sets. Training data sizes shown in parentheses. Metrics: EM = Exact Match (string equivalence), FM = Fuzzy Match at 90%, FS = Fuzzy Score (average string similarity), Exec = Execution Accuracy (result set overlap). Training data: Spider = real Spider data, Pipeline = FALCON-generated data, Mixed = combined data (5% or 10% real Spider + remaining pipeline). Models trained on FALCON data excel on FALCON tests, while Spider-trained models excel on Spider tests, demonstrating training distribution effects.
Figure 15: SQL generation prompt with reserved-word seeding.
Figure 16: SQL generation prompt with persona seeding.
Figure 17: Creative persona generation prompt.
Figure 18: Common persona generation prompt.
Figure 19: NL question generation prompt with chain-of-thought.
Figure 20: NL question generation with persona-specific voice and tone.
Figure 21: LLM-as-a-judge prompt for NL candidate selection. (Compare)
Figure 22: LLM-as-a-judge prompt for NL candidate selection. (Score)
Natural language interfaces to databases aim to translate user questions into executable SQL, yet remain brittle in real-world settings where questions are underspecified and schemas are large and ambiguous. Ambiguity across user questions, database schemas, and model interpretations are central failure modes in NL2SQL, leading to misaligned intent, incorrect schema grounding, and erroneous SQL generation. Existing approaches rely on human clarification or treat ambiguity as a schema representation problem, but these do not scale nor resolve ambiguity autonomously. We propose SOMA-SQL to automatically resolve ambiguity via targeted synthetic query log and ambiguity-driven probing. SOMA-SQL constructs synthetic query log to ground schema interpretation and guide candidate SQL generation; it then executes targeted probing queries, driven by a structured ambiguity taxonomy and candidate disagreements, to produce disambiguation evidence for final SQL selection and repair. This active approach to ambiguity discovery and resolution generalizes across unseen schemas and query distributions without human-in-the-loop. Experiments on six public benchmarks demonstrate that SOMA-SQL improves execution accuracy by 13.0% on average over state-of-the-art baselines, with gains of up to 16.7% on ambiguous questions.
Sai Ashish Somayajula, Marianne Menglin Liu, Chuan Lei +9
Natural language to SQL (NL2SQL) conversion is an important problem for researchers and enterprises due to the ubiquitous importance of relational databases in broad-ranging practical problems. Despite the rapid advancements in the capabilities of LLMs, NL2SQL has not reached parity in accuracy with human expert SQL writers, hence needing additional improvements in NL2SQL algorithms. This study presents a new multi-agent method for NL2SQL that achieves 78.1% semantic accuracy on the BIg Bench for LaRge-scale Database (BIRD) benchmark. Our method leverages a semantically enriched representation of user-provided schema, adds user-provided business rules, and produces accurate SQL queries. The main contributions of this study are (a) We designed an optimized new orchestrator in a multi-agent solution that uses LLMs to plan, orchestrate, reflect, and self-correct to generate accurate SQL queries, (b) We developed an advanced schema enrichment method that creates context-aware metadata to improve accuracy, and (c) We demonstrated the accuracy and generalizability of the method across different domains and datasets by evaluating it on the BIRD-SQL benchmark.
Translating Natural Language to SQL (NL2SQL) remains a critical bottleneck for democratization of data in enterprises. Although Large Language Models (LLMs) like Gemini 2.5 and other LLMs have demonstrated impressive zero-shot capabilities, their high inference costs limit deployment at scale. This paper explores the efficacy of fine-tuning both large and small language models on NL2SQL tasks. Our research reveals a counter-intuitive scaling phenomenon. Fine-tuning large models (Gemini 2.5 Flash/Lite) on standard datasets yields negligible returns, often leading to overfitting on complex queries. Conversely, small models (Qwen) show significant gains. Fine-tuning improved the small model baseline from 36% to 45%, and further enriching the dataset with explicit Chain-of-Thought (CoT) reasoning surged accuracy to 54.5%(Fig 2). While this is still lower than the accuracy of large models like Gemini 2.5 , it does serve the business goal of significant cost reduction, latency in inference time and also meeting the business critical performance accuracy threshold.This paper demonstrates that transferring reasoning patterns enables compute-efficient smaller models to approach production-grade performance.