EvoSchema: Towards Text-to-SQL Robustness Against Schema Evolution
Authors: Tianshu Zhang, Kun Qian, Siddhartha Sahai, Yuan Tian, Shaddy Garg, Huan Sun, Yunyao Li
Organizations: The Ohio State University, Columbus, OH · Adobe Inc., Seattle, WA · Purdue University, West Lafayette, IN · Adobe Inc., Bangalore · Adobe Inc., San Jose, CA
Neural text-to-SQL models, which translate natural language questions (NLQs) into SQL queries given a database schema, have achieved remarkable performance. However, database schemas frequently evolve to meet new requirements. Such schema evolution often leads to performance degradation for models trained on static schemas. Existing work either mainly focuses on simply paraphrasing some syntactic or semantic mappings among NLQ, DB and SQL, or lacks a comprehensive and controllable way to investigate the model robustness issue under the schema evolution, which is insufficient when facing the increasingly complex and rich database schema changes in reality, especially in the LLM era. To address the challenges posed by schema evolution, we present EvoSchema, a comprehensive benchmark designed to assess and enhance the robustness of text-to-SQL systems under real-world schema changes. EvoSchema introduces a novel schema evolution taxonomy, encompassing ten perturbation types across columnlevel and table-level modifications, systematically simulating the dynamic nature of database schemas. Through EvoSchema, we conduct an in-depth evaluation spanning different open source and closed-source LLMs, revealing that table-level perturbations have a significantly greater impact on model performance compared to column-level changes. Furthermore, EvoSchema inspires the development of more resilient text-to-SQL systems, in terms of both model training and database design. The models trained on EvoSchema's diverse schema designs can force the model to distinguish the schema difference for the same questions to avoid learning spurious patterns, which demonstrate remarkable robustness compared to those trained on unperturbed data on average. This benchmark offers valuable insights into model behavior and a path forward for designing systems capable of thriving in dynamic, real-world environments.
Figures & tables
Figure 1. The left (a) is the overview of the framework to collect EvoSchema dataset. The top right (b) is a column-level schema evolution example; the bottom right (c) is a table-level schema evolution example.
Figure 2. An overview of different perturbation types of EvoSchema . The top is an unperturbed example in BIRD ( Li et al., 2024d ) ; the middle is the column-level perturbation; the bottom is the table-level perturbation. “Remove Col in SQL": remove columns that appear in gold SQL; “Remove Tables": the relevant tables appear in gold SQL are removed. Thus there is no gold SQL for these two cases. Note we don’t illustrate “Merge Columns” in the figure as this example is not suitable for applying merging column changes.
Figure 3. This figure shows two examples of our data collection procedure of EvoSchema . The top (a) is a “rename columns" data collection procedure; the bottom (b) is a “split tables" data collection procedure. The blue box indicates prompting GPT models for the generation. “</>" means programmatically processing the data.
Table 2. Data statistics of EvoSchema . “Original” refers to the original BIRD dataset; “Column Manipulation” refers to applying the column-level operations on the columns of the original BIRD data; “Table Manipulation” refers to applying the table-level operations on the tables of the original BIRD data. “*”: the evaluation data for calculating execution accuracy. We synthesize values to reconstruct the database after schema evolution, and filter out those not executable by gold SQL, which results in the smaller size of the evaluation data for calculating execution accuracy.
Perturbation Type
Code Llama
Mistral
Llama 3
SQLCoder
GPT-3.5
GPT-4
w/o
w/
w/o
w/
w/o
w/
w/o
w/
Table Match F1
Original
89.77
90.42
89.58
90.62
89.96
89.53
89.69
90.64
87.28
88.98
Add Columns
89.73
90.27
89.65
90.03
89.08
89.70
89.30
90.52
86.35
88.12
Remove Columns
89.82
90.24
89.89
90.66
90.09
89.82
89.81
90.54
87.18
88.87
Rename Columns
85.28
85.07
84.32
84.27
83.74
82.92
85.32
84.93
81.73
83.20
Table 3. Evaluation on EvoSchema . “w/”: the model is trained by merging the original data and all the perturbation training types together; “w/o”: the model is only trained on the original training data. The best performance for each model is in bold, and red shows a larger gain. “-”: some of the relevant tables are removed so there should be no gold SQL used to calculate the metrics here.
Human Evaluation on EvoSchema
Perturbation Type
GPT-4
Code Llama
CHESS SS+CG
ZS
ICL
w/o
w/
Original
62
58
65
64
63
Add Columns
59
55
62
61
66
Remove Columns
65
61
66
63
64
Rename Columns
57
56
57
57
62
Table 4. Human Evaluation on EvoSchema . “ZS” refers to zero-shot, which prompts models without any examples. “ICL” refers to in-context learning, which prompts models with three demonstration examples. “w/o” means fine-tuning model without perturbation training data; “w/” means fine-tuning model with perturbation training data. Bold color indicates the best performance among each row.
Exec Acc on EvoSchema
Perturbation Type
Code Llama
Mistral
Llama 3
SQLCoder
GPT-3.5
GPT-4
w/o
w/
w/o
w/
w/o
w/
w/o
w/
Original
58
57
59
58
55
51
58
58
44
47
Add Columns
57
55
56
56
52
49
55
57
43
46
Remove Columns
59
57
60
58
56
53
60
58
45
47
Rename Columns
54
52
55
54
49
47
56
55
43
45
Table 5. Execution Accuracy on EvoSchema . “w/”: the model is trained with all the perturbation types; “w/o”: the model is only trained on the original training data.
Perturbation Type Ablation
Perturbation Type
Table Match F1
Column Match F1
both
w/o table-p
w/o column-p
both
w/o table-p
w/o column-p
Original
90.73
90.80 (+0.07)
90.04 (-0.69)
81.09
82.15 (+1.06)
80.49 (-0.60)
Add Columns
90.86
90.80 (-0.06)
89.75 (-1.11)
79.63
80.81 (+1.18)
77.29 (-2.34)
Remove Columns
90.72
90.83 (+0.11)
90.48 (-0.24)
83.28
83.85 (+0.57)
82.61 (-0.67)
Rename Columns
85.35
85.38 (+0.03)
84.57 (-0.78)
76.49
77.53 (+1.04)
75.17 (-1.32)
Table 6. Perturbation type ablation on EvoSchema . The base model is Code Llama. “both": the model is trained with both column-level perturbation and table-level perturbation types; “w/o table-p": the model is trained without table-level perturbation types; “w/o column-p": the model is trained without column-level perturbation types.
Out of Scope Effect
Perturbation Type
Table Match F1
Column Match F1
+ OOS FP
+ OOS TP
w/o
w/
+ OOS
w/o
w/
+ OOS
Tab
Col
Tab
Col
Original
89.77
90.42
82.98 (-7.44)
80.66
81.64
75.43 (-6.21)
7.11
0.65
-
-
Add Columns
89.73
90.27
86.07 (-4.20)
78.26
80.27
77.00 (-3.27)
4.25
0.40
-
-
Remove Columns
89.82
90.24
82.24 (-8.00)
82.67
82.75
75.90 (-6.85)
7.56
0.72
-
-
Remove Col in SQL
-
-
-
-
-
-
5.02
-
-
84.03
Table 7. Out of Scope Effect on EvoSchema . The base model is Code Llama. “w/o": the model is trained without perturbation types; “w/": the model is trained on the original data and all the perturbation types; “+ OOS": the model is trained on the original data, perturbation types and two out-of-scope (OOS) perturbation types; “+ OOS FP": The model trained with two OOS perturbation types makes an incorrect prediction on the original data and in-scope perturbation data; “+ OOS TP": The model trained with two OOS perturbation types makes the correct prediction on the two OOS perturbation data; “Tab": the model refuses to predict SQL due to the lack of table information; “Col": the model refuses to predict SQL due to the lack of column information.
Add Irrelevant Tables Effect
Perturbation Type
Table Match F1
Column Match F1
w/o
w/o+
w/
w/o
w/o+
w/
Original
89.77
87.65
90.42
80.66
79.24
81.64
Add Columns
89.73
86.35
90.27
78.26
75.31
80.27
Remove Columns
89.82
87.30
90.24
82.67
80.74
82.75
Rename Columns
85.28
81.90
85.07
76.50
73.28
76.94
Table 8. Irrelevant tables effect. “w/”: the model is trained with all the perturbation types; “w/o”: the model is only trained on the original training data; “w/o+": the model is only trained on the original training data, but for the input table schema, we also add irrelevant tables.
Intra-database Effect
Perturbation Type
Table Match F1
Column Match F1
w/o
w/
w/o
w/
Original
87.24
87.43
79.54
80.89
Add Columns
87.14
87.43
76.36
78.92
Remove Columns
87.29
87.27
81.14
81.29
Rename Columns
85.71
86.43
77.45
79.09
Table 9. Intra-database Effect. This experiment emphasizes that the training and evaluation occur within the same database, instead of across databases.
Text-to-SQL enables non-expert users to query databases in natural language, yet real-world schemas often suffer from ambiguous, abbreviated, or inconsistent naming conventions that degrade model accuracy. Existing approaches treat schemas as fixed and address errors downstream. In this paper, we frame schema refinement as a constrained optimization problem: find a renaming function that maximizes downstream Text-to-SQL execution accuracy while preserving query equivalence through database views. We analyze the computational hardness of this problem, which motivates a column-wise greedy decomposition, and instantiate it as EGRefine: a four-phase pipeline that screens ambiguous columns, generates context-aware candidate names, verifies them through execution-grounded feedback, and materializes the result as non-destructive SQL views. The pipeline carries two structural properties: column-local non-degradation, ensured by the conservative selection rule in the verification phase, and database-level query equivalence, ensured by the view-based materialization phase. Together they make the resulting refinement safe by construction at the column level, with cross-column and prompt-level interactions handled empirically rather than analytically. Across controlled schema-degradation, real-world, and enterprise benchmarks, EGRefine recovers accuracy lost to schema naming noise where applicable and correctly abstains where the underlying task exceeds current Text-to-SQL capabilities, with refined schemas transferring across model families to enable refine-once, serve-many-models deployment. Code and data are publicly available at https://github.com/ai-jiaqian/EGRefine.
Text-to-SQL systems translate natural language queries into executable SQL, democratizing access to structured data. Despite recent advances driven by large language models (LLMs), evaluation remains a major bottleneck: public benchmarks fail to capture the complexity of enterprise schema, while building private evaluation sets is costly and nondeterministic, making evaluation results difficult to reproduce. To address this issue, we present SQLMorph, a framework for Text-to-SQL evaluation via query mutation. SQLMorph introduces two techniques to automatically generate and expand evaluation sets: Join Query Expansion (JQE), which systematically increases structural complexity through valid join additions, and Textual Query Augmentation (TQA), which generates controlled natural language perturbations to assess robustness to linguistic variation. JQE and TQA create targeted choke points to challenge specific system components. When applied to state-of-the-art systems, JQE increases query coverage and reveals accuracy degradation as the number of joins grows. Meanwhile, TQA shows that linguistic brittleness induced by heavy abbreviation can reduce accuracy by up to 17%. Beyond evaluation sets, SQLMorph introduces a family of execution-level metrics that address the limitations of current binary measures, such as Execution Accuracy. We define Execution Precision (EXP) and Execution Recall (EXR) to quantify the fraction of correct and recovered results, respectively, and combine them via F1 for unified scoring. Our experiments show that these relaxed metrics enable fine-grained analysis of over- and under-prediction, revealing differences across systems that binary metrics obscure. Together, SQLMorph's query mutation and fine-grained metrics support debugging and better align Text-to-SQL evaluation practices with real-world deployments.
Mohammadhossein Malekpour, Mohamed Riahi, Maxime Lamothe +1
Large Language Models (LLMs) are transforming database interaction paradigms, evolving from simple query translators to autonomous database administrators (DBAs). However, current evaluation benchmarks remain disproportionately fixated on Text-to-SQL tasks, neglecting the holistic Database Lifecycle-from initial schema design to post-deployment maintenance. This narrow focus fails to capture the diverse capabilities required for real-world database management. To bridge this gap, we introduce DBLifeBench, the first benchmark to evaluate LLMs across five critical lifecycle phases: Design, Implementation, Operation, Debugging, and Maintenance. Furthermore, addressing the cognitive mismatch between ambiguous natural language and complex SQL logic, we propose Progressive-Text2SQL, a novel task utilizing structured reasoning graphs to mimic human iterative problem-solving. Our extensive evaluation reveals a critical insight: while general-purpose models demonstrate balanced performance, specialized Text-to-SQL models suffer from ``catastrophic forgetting'' in non-coding phases like design and maintenance. DBLifeBench serves as a foundational step toward evaluating and building true full-stack database intelligence.
Shunfan Zheng, Dongsheng Shi, Yue Li +3
1East China Normal University · 2Hasso Plattner Institute/University of Potsdam