EvoSchema: Towards Text-to-SQL Robustness Against Schema Evolution

Tianshu Zhang; Kun Qian; Siddhartha Sahai; Yuan Tian; Shaddy Garg; Huan Sun; Yunyao Li

cs.DBMar 11, 2026

EvoSchema: Towards Text-to-SQL Robustness Against Schema Evolution

Authors: Tianshu Zhang, Kun Qian, Siddhartha Sahai, Yuan Tian, Shaddy Garg, Huan Sun, Yunyao Li

Organizations: The Ohio State University, Columbus, OH · Adobe Inc., Seattle, WA · Purdue University, West Lafayette, IN · Adobe Inc., Bangalore ·

Abstract

Neural text-to-SQL models, which translate natural language questions (NLQs) into SQL queries given a database schema, have achieved remarkable performance. However, database schemas frequently evolve to meet new requirements. Such schema evolution often leads to performance degradation for models trained on static schemas. Existing work either mainly focuses on simply paraphrasing some syntactic or semantic mappings among NLQ, DB and SQL, or lacks a comprehensive and controllable way to investigate the model robustness issue under the schema evolution, which is insufficient when facing the increasingly complex and rich database schema changes in reality, especially in the LLM era. To address the challenges posed by schema evolution, we present EvoSchema, a comprehensive benchmark designed to assess and enhance the robustness of text-to-SQL systems under real-world schema changes. EvoSchema introduces a novel schema evolution taxonomy, encompassing ten perturbation types across columnlevel and table-level modifications, systematically simulating the dynamic nature of database schemas. Through EvoSchema, we conduct an in-depth evaluation spanning different open source and closed-source LLMs, revealing that table-level perturbations have a significantly greater impact on model performance compared to column-level changes. Furthermore, EvoSchema inspires the development of more resilient text-to-SQL systems, in terms of both model training and database design. The models trained on EvoSchema's diverse schema designs can force the model to distinguish the schema difference for the same questions to avoid learning spurious patterns, which demonstrate remarkable robustness compared to those trained on unperturbed data on average. This benchmark offers valuable insights into model behavior and a path forward for designing systems capable of thriving in dynamic, real-world environments.

Figures & tables

Figure 1. The left (a) is the overview of the framework to collect EvoSchema dataset. The top right (b) is a column-level schema evolution example; the bottom right (c) is a table-level schema evolution example.

Figure 2. An overview of different perturbation types of EvoSchema . The top is an unperturbed example in BIRD ( Li et al., 2024d ) ; the middle is the column-level perturbation; the bottom is the table-level perturbation. “Remove Col in SQL": remove columns that appear in gold SQL; “Remove Tables": the relevant tables appear in gold SQL are removed. Thus there is no gold SQL for these two cases. Note we don’t illustrate “Merge Columns” in the figure as this example is not suitable for applying merging column changes.

Figure 3. This figure shows two examples of our data collection procedure of EvoSchema . The top (a) is a “rename columns" data collection procedure; the bottom (b) is a “split tables" data collection procedure. The blue box indicates prompting GPT models for the generation. “</>" means programmatically processing the data.

Perturbation Data	Column-level	Table-level	Schema Evolution	Multiple DB	Seed Data	Features of Seed Data (Average)
Perturbation Data	Column-level	Table-level	Affects SQL	Multiple DB	Seed Data	Tab/DB	Col/DB	Col/Tab	PK/DB	FK/DB
FootballDB ( Fürst et al., 2025 )	-	reduce PK/FK references, reduce JOIN paths	✓	✗	FIFA World Cup ( Becklas, 2018 )	15	107	7.1	-	16
Dr.Spider ( Chang et al., 2023 )	Rename	✗	✓	✓	Spider ( Chang et al., 2023 )	5.1	22.1	5.4	3.7	3.2
ADVETA ( Pi et al., 2022 )	Add; Rename	✗	✓	✓	Spider ( Chang et al., 2023 )	5.1	22.1	5.4	3.7	3.2
MT-TEQL ( Ma and Wang, 2021 )	Add; Remove; Shuffle; Rename	Split; Merge; Shuffle	✗	✓	Spider ( Chang et al., 2023 )	5.1	22.1	5.4	3.7	3.2
EvoSchema (Ours)	Add; Remove; Rename; Split; Merge	Split; Merge; Rename; Add; Remove	✓	✓	BIRD ( Li et al., 2024d )	7.3	72.5	10.6	6.5	9.3

Table 1. Statistics of EvoSchema compared with existing benchmarks. “Tab”: tables; “DB”: database; “Col”: columns; “PK”: primary keys; “FK”: foreign keys.

Data Statistics
Perturbation Type	Train	Eval	Eval*	Manipulated Items/Table				Manipulated Items/Query
Perturbation Type	Train	Eval	Eval*	Min	Mean	Median	Max	Min	Mean	Median	Max
Original	9426	1534	1068	-	-	-	-	-	-	-	-
				Column Manipulation
Add Columns	9219	1506	846	1	5.7	3	83	1	5.9	4	43
Remove Columns	9426	1534	1076	1	6.2	2	87	1	6.9	3	70

Table 2. Data statistics of EvoSchema . “Original” refers to the original BIRD dataset; “Column Manipulation” refers to applying the column-level operations on the columns of the original BIRD data; “Table Manipulation” refers to applying the table-level operations on the tables of the original BIRD data. “*”: the evaluation data for calculating execution accuracy. We synthesize values to reconstruct the database after schema evolution, and filter out those not executable by gold SQL, which results in the smaller size of the evaluation data for calculating execution accuracy.

Table Match F1
Perturbation Type	Code Llama		Mistral		Llama 3		SQLCoder		GPT-3.5	GPT-4
Perturbation Type	w/o	w/	w/o	w/	w/o	w/	w/o	w/
Original	89.77	90.42	89.58	90.62	89.96	89.53	89.69	90.64	87.28	88.98
Add Columns	89.73	90.27	89.65	90.03	89.08	89.70	89.30	90.52	86.35	88.12
Remove Columns	89.82	90.24	89.89	90.66	90.09	89.82	89.81	90.54	87.18	88.87
Rename Columns	85.28	85.07	84.32	84.27	83.74	82.92	85.32	84.93	81.73	83.20

Table 3. Evaluation on EvoSchema . “w/”: the model is trained by merging the original data and all the perturbation training types together; “w/o”: the model is only trained on the original training data. The best performance for each model is in bold, and red shows a larger gain. “-”: some of the relevant tables are removed so there should be no gold SQL used to calculate the metrics here.

Human Evaluation on EvoSchema
Perturbation Type	GPT-4		Code Llama		CHESS SS+CG
Perturbation Type	ZS	ICL	w/o	w/
Original	62	58	65	64	63
Add Columns	59	55	62	61	66
Remove Columns	65	61	66	63	64
Rename Columns	57	56	57	57	62

Table 4. Human Evaluation on EvoSchema . “ZS” refers to zero-shot, which prompts models without any examples. “ICL” refers to in-context learning, which prompts models with three demonstration examples. “w/o” means fine-tuning model without perturbation training data; “w/” means fine-tuning model with perturbation training data. Bold color indicates the best performance among each row.

Exec Acc on EvoSchema
Perturbation Type	Code Llama		Mistral		Llama 3		SQLCoder		GPT-3.5	GPT-4
Perturbation Type	w/o	w/	w/o	w/	w/o	w/	w/o	w/
Original	58	57	59	58	55	51	58	58	44	47
Add Columns	57	55	56	56	52	49	55	57	43	46
Remove Columns	59	57	60	58	56	53	60	58	45	47
Rename Columns	54	52	55	54	49	47	56	55	43	45

Table 5. Execution Accuracy on EvoSchema . “w/”: the model is trained with all the perturbation types; “w/o”: the model is only trained on the original training data.

Perturbation Type Ablation
Perturbation Type	Table Match F1			Column Match F1
Perturbation Type	both	w/o table-p	w/o column-p	both	w/o table-p	w/o column-p
Original	90.73	90.80 (+0.07)	90.04 (-0.69)	81.09	82.15 (+1.06)	80.49 (-0.60)
Add Columns	90.86	90.80 (-0.06)	89.75 (-1.11)	79.63	80.81 (+1.18)	77.29 (-2.34)
Remove Columns	90.72	90.83 (+0.11)	90.48 (-0.24)	83.28	83.85 (+0.57)	82.61 (-0.67)
Rename Columns	85.35	85.38 (+0.03)	84.57 (-0.78)	76.49	77.53 (+1.04)	75.17 (-1.32)

Table 6. Perturbation type ablation on EvoSchema . The base model is Code Llama. “both": the model is trained with both column-level perturbation and table-level perturbation types; “w/o table-p": the model is trained without table-level perturbation types; “w/o column-p": the model is trained without column-level perturbation types.

Out of Scope Effect
Perturbation Type	Table Match F1			Column Match F1			+ OOS FP		+ OOS TP
Perturbation Type	w/o	w/	+ OOS	w/o	w/	+ OOS	Tab	Col	Tab	Col
Original	89.77	90.42	82.98 (-7.44)	80.66	81.64	75.43 (-6.21)	7.11	0.65	-	-
Add Columns	89.73	90.27	86.07 (-4.20)	78.26	80.27	77.00 (-3.27)	4.25	0.40	-	-
Remove Columns	89.82	90.24	82.24 (-8.00)	82.67	82.75	75.90 (-6.85)	7.56	0.72	-	-
Remove Col in SQL	-	-	-	-	-	-	5.02	-	-	84.03

Table 7. Out of Scope Effect on EvoSchema . The base model is Code Llama. “w/o": the model is trained without perturbation types; “w/": the model is trained on the original data and all the perturbation types; “+ OOS": the model is trained on the original data, perturbation types and two out-of-scope (OOS) perturbation types; “+ OOS FP": The model trained with two OOS perturbation types makes an incorrect prediction on the original data and in-scope perturbation data; “+ OOS TP": The model trained with two OOS perturbation types makes the correct prediction on the two OOS perturbation data; “Tab": the model refuses to predict SQL due to the lack of table information; “Col": the model refuses to predict SQL due to the lack of column information.

Add Irrelevant Tables Effect
Perturbation Type	Table Match F1			Column Match F1
Perturbation Type	w/o	w/o+	w/	w/o	w/o+	w/
Original	89.77	87.65	90.42	80.66	79.24	81.64
Add Columns	89.73	86.35	90.27	78.26	75.31	80.27
Remove Columns	89.82	87.30	90.24	82.67	80.74	82.75
Rename Columns	85.28	81.90	85.07	76.50	73.28	76.94

Table 8. Irrelevant tables effect. “w/”: the model is trained with all the perturbation types; “w/o”: the model is only trained on the original training data; “w/o+": the model is only trained on the original training data, but for the input table schema, we also add irrelevant tables.

Intra-database Effect
Perturbation Type	Table Match F1		Column Match F1
Perturbation Type	w/o	w/	w/o	w/
Original	87.24	87.43	79.54	80.89
Add Columns	87.14	87.43	76.36	78.92
Remove Columns	87.29	87.27	81.14	81.29
Rename Columns	85.71	86.43	77.45	79.09

Table 9. Intra-database Effect. This experiment emphasizes that the training and evaluation occur within the same database, instead of across databases.

Text-to-SQL

Text-to-SQL translates natural-language questions into executable database queries, linking user intent to schemas, values, and SQL logic.

See all 173 papers

EGREFINE: An Execution-Grounded Optimization Framework for Text-to-SQL Schema Refinement
May 1, 2026Jiaqian Wang, Yutao Qi, Wenjin Hou +2Text-to-SQL Relational Databases
Xidian University, Xi’an, Shaanxi 710071, China
SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation
Sep 8, 2026Mohammadhossein Malekpour, Mohamed Riahi, Maxime Lamothe +1Mutation Testing Text-to-SQL
Polytechnique Montréal
Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Tasks
Aug 4, 2026Shunfan Zheng, Dongsheng Shi, Yue Li +3LLM Evaluation Text-to-SQL
1East China Normal University · 2Hasso Plattner Institute/University of Potsdam

May 1, 2026J/K move · Enter open · S save

EGREFINE: An Execution-Grounded Optimization Framework for Text-to-SQL Schema Refinement

Jiaqian Wang, Yutao Qi, Wenjin Hou, Yu Pang+1

Xidian University, Xi’an, Shaanxi 710071, China

Text-to-SQL Relational Databases

Open paper AskPDF arXiv

Text-to-SQL enables non-expert users to query databases in natural language, yet real-world schemas often suffer from ambiguous, abbreviated, or inconsistent naming conventions that degrade model accuracy. Existing approaches treat schemas as fixed and address errors downstream. In this paper, we frame schema refinement as a constrained optimization problem: find a renaming function that maximizes downstream Text-to-SQL execution accuracy while preserving query equivalence through database views. We analyze the computational hardness of this problem, which motivates a column-wise greedy decomposition, and instantiate it as EGRefine: a four-phase pipeline that screens ambiguous columns, generates context-aware candidate names, verifies them through execution-grounded feedback, and materializes the result as non-destructive SQL views. The pipeline carries two structural properties: column-local non-degradation, ensured by the conservative selection rule in the verification phase, and database-level query equivalence, ensured by the view-based materialization phase. Together they make the resulting refinement safe by construction at the column level, with cross-column and prompt-level interactions handled empirically rather than analytically. Across controlled schema-degradation, real-world, and enterprise benchmarks, EGRefine recovers accuracy lost to schema naming noise where applicable and correctly abstains where the underlying task exceeds current Text-to-SQL capabilities, with refined schemas transferring across model families to enable refine-once, serve-many-models deployment. Code and data are publicly available at https://github.com/ai-jiaqian/EGRefine.

Abstract

Text-to-SQL

Explore similar work

EGREFINE: An Execution-Grounded Optimization Framework for Text-to-SQL Schema Refinement

SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation

Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Tasks