An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Data

Trinh Pham; Thanh Tam Nguyen; Viet Huynh; Hongzhi Yin; Quoc Viet Hung Nguyen

cs.CLMar 8, 2026

An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Data

Authors: Trinh Pham, Thanh Tam Nguyen, Viet Huynh, Hongzhi Yin, Quoc Viet Hung Nguyen

Organizations: Griffith University (Australia) · Edith Cowan University (Australia) · The University of Queensland (Australia)

Abstract

Recent advances in large language models have strengthened Text2SQL systems that translate natural language questions into database queries. A persistent deployment challenge is to assess a newly trained Text2SQL system on an unseen and unlabeled dataset when no verified answers are available. This situation arises frequently because database content and structure evolve, privacy policies slow manual review, and carefully written SQL labels are costly and time-consuming. Without timely evaluation, organizations cannot approve releases or detect failures early. FusionSQL addresses this gap by working with any Text2SQL models and estimating accuracy without reference labels, allowing teams to measure quality on unseen and unlabeled datasets. It analyzes patterns in the system's own outputs to characterize how the target dataset differs from the material used during training. FusionSQL supports pre-release checks, continuous monitoring of new databases, and detection of quality decline. Experiments across diverse application settings and question types show that FusionSQL closely follows actual accuracy and reliably signals emerging issues. Our code is available at https://github.com/phkhanhtrinh23/FusionSQL.

Figures & tables

Fig. 1 : Top : Existing Text2SQL evaluations rely on ground-truth labels, which are often unavailable as databases evolve. Bottom : FusionSQL estimates model accuracy directly from unlabeled inputs without requiring ground-truth SQL labels.

Fig. 2 : FusionSQL framework. Training : A frozen Text2SQL model encodes training and FusionDataset samples into embeddings to compute shift descriptors ( $\textit{SD}_{F}$ , $\textit{SD}_{M}$ , $\textit{SD}_{SW}$ ) for training the FusionSQL evaluator. Inference : For unseen, unlabeled workloads, the same descriptors are computed to estimate accuracy without labels or retraining.

Dataset	Source	# Examples	# Unique SQL	# DB
WikiSQL [ 2 ]	Human+Template	80,654	80,257	26,531 ⋆
Spider [ 32 ]	Human	10,181	5,693	200
SParC [ 33 ]	Human	12,726	—	200
CoSQL [ 34 ]	Human	10,000+	—	200
BIRD [ 35 ]	Human	12,751	— $\dagger$	95
ScienceBenchmark [ 56 ]	LLM-Gen+Human+Template	5,031	3,652	3

TABLE I : Comparison of Text2SQL datasets. # Examples count question–SQL pairs, while # Unique SQL counts distinct canonicalized SQL programs. ⋆ Each WikiSQL “DB” contains a single table;

\dagger

BIRD’s Unique SQL is unavailable due to its inaccessible test set. FusionDataset offers greater structural and linguistic diversity than prior datasets.

Model	Simple (30%)		Moderate (35%)		Complex (25%)		Highly Complex (10%)		Overall
Model	EM	EX	EM	EX	EM	EX	EM	EX	EM	EX
Qwen2.5-72B-Instruct	82.1	86.5	74.5	80.1	63.2	70.5	51.4	59.8	71.6	77.2
Llama-3.1-70B-Instruct	80.4	84.9	72.3	78.6	61.1	68.3	49.5	56.7	69.7	74.6
DeepSeek-Coder-33B	75.2	80.8	67.8	73.9	56.0	62.5	44.2	52.0	64.7	70.3
CSC-SQL-7B	68.8	74.1	60.1	66.5	48.7	55.2	37.3	44.0	57.6	63.0
XiYanSQL-14B	71.0	76.4	63.2	69.8	52.0	58.6	40.5	47.2	60.5	66.1

TABLE II : Difficulty-stratified results. EM and EX (%) decrease consistently with increasing complexity, confirming coverage of a realistic reasoning spectrum.

Fig. 3 : t-SNE coverage. Comparing 50K samples, FusionDataset bridges clusters of existing benchmarks in both domain (a) and question space (b), reflecting broader semantic and structural diversity of real-world Text2SQL variability.

Fig. 4 : Structural coverage. Radar chart over 16 schemas and SQL complexity dimensions shows that FusionDataset (red) consistently achieves higher normalized coverage than existing datasets, reflecting diverse schema and query structures.

Fig. 5 : Sample-set size. Impact of sample-set size $|S_{i}|$ used to compute distribution shifts for an instance $(\mathcal{D}_{\mathrm{train}},S_{i})$ . Error decreases as $|S_{i}|$ grows.

Transfer ( $A\rightarrow B$ )	Methods	Qwen2.5-72B	Llama-3.1-70B	DeepSeek-33B	XiYanSQL-14B	CSC-SQL-7B	Avg.
Spider $\rightarrow$ BIRD	ATC-MC [ 42 ]	13.9 $\pm$ 1.1	14.6 $\pm$ 1.2	15.2 $\pm$ 1.2	17.4 $\pm$ 1.4	18.3 $\pm$ 1.5	15.9 $\pm$ 1.3
	ATC-NE [ 42 ]	15.0 $\pm$ 1.2	15.7 $\pm$ 1.3	16.5 $\pm$ 1.3	18.6 $\pm$ 1.5	19.8 $\pm$ 1.6	17.1 $\pm$ 1.4
	DoC ( $\tau{=}0.8$ ) [ 43 ]	15.5 $\pm$ 1.3	16.0 $\pm$ 1.3	17.3 $\pm$ 1.4	19.2 $\pm$ 1.6	20.5 $\pm$ 1.6	17.7 $\pm$ 1.4
	DoC ( $\tau{=}0.9$ ) [ 43 ]	16.7 $\pm$ 1.4	17.3 $\pm$ 1.4	18.6 $\pm$ 1.5	20.3 $\pm$ 1.7	21.7 $\pm$ 1.7	18.9 $\pm$ 1.5
	PseAutoEval [ 40 ]	11.6 $\pm$ 0.9	12.2 $\pm$ 1.0	13.1 $\pm$ 1.0	15.1 $\pm$ 1.2	16.3 $\pm$ 1.3	13.7 $\pm$ 1.1
	BugJudge [ 44 ]	14.8 $\pm$ 1.2	15.4 $\pm$ 1.2	16.2 $\pm$ 1.3	18.1 $\pm$ 1.4	19.0 $\pm$ 1.5	16.7 $\pm$ 1.3

TABLE III : MAE (

\downarrow

) of dataset-level accuracy estimation for various source-target transfers. Each cell reports mean

\pm

95% CI (percentage points). Best in bold , second-best underlined .

Fig. 6 : Meta-set size. FusionSQL’s MLP attains the lowest error and benefits most from larger meta-sets, while costs rise sharply beyond $30{,}000$ with marginal gains.

Fig. 7 : EX across transfers. Across six source–target transfers and five base models, FusionSQL closely follows the ground-truth (GT) trend, while FusionSQL-TL exhibits mild bias on harder targets due to limited labeled adaptation.

Fig. 8 : Latency–accuracy trade-off. Bars report average evaluation latency, while the red curve shows MAE (lower is better). FusionSQL offers the best trade-off, whereas judge-based methods incur high latency without much MAE gains.

Fig. 9 : Latency–accuracy trade-off on unseen Text2SQL models. Bars denote relative latency, while lines show MAE. FusionSQL-ML attains the lowest MAE with low latency.

Transfer ( $A\rightarrow B$ )	Methods	CodeLlama-34B	StarCoder2-15B	Mistral-7B	DeepSeek-Coder-6.7B	Phi-3-mini	Avg.
Spider $\rightarrow$ BIRD	BugJudge [ 44 ]	13.8 $\pm$ 1.0	13.5 $\pm$ 1.1	14.0 $\pm$ 1.0	13.9 $\pm$ 0.9	13.6 $\pm$ 1.0	13.8 $\pm$ 1.0
	ArenaCmp [ 45 ]	11.1 $\pm$ 0.8	10.8 $\pm$ 0.9	11.4 $\pm$ 1.0	11.2 $\pm$ 0.9	10.9 $\pm$ 0.8	11.1 $\pm$ 0.9
	FusionSQL-ML (Ours)	6.7 $\pm$ 0.5	6.5 $\pm$ 0.6	6.8 $\pm$ 0.7	6.7 $\pm$ 0.6	6.6 $\pm$ 0.5	6.7 $\pm$ 0.6
WikiSQL $\rightarrow$ Spider	BugJudge [ 44 ]	12.7 $\pm$ 1.0	12.4 $\pm$ 1.1	12.9 $\pm$ 1.0	12.8 $\pm$ 0.9	12.5 $\pm$ 1.0	12.7 $\pm$ 1.0
	ArenaCmp [ 45 ]	10.4 $\pm$ 0.8	10.1 $\pm$ 0.9	10.6 $\pm$ 1.0	10.4 $\pm$ 0.9	10.2 $\pm$ 0.8	10.3 $\pm$ 0.9
	FusionSQL-ML (Ours)	6.0 $\pm$ 0.4	5.8 $\pm$ 0.5	6.1 $\pm$ 0.6	6.0 $\pm$ 0.5	5.9 $\pm$ 0.4	6.0 $\pm$ 0.5

TABLE IV : MAE (

\downarrow

) for generalizing FusionSQL to new Text2SQL models (mean

\pm

95% CI, percentage points). Columns are the unseen model pool. Best in bold .

Panel of Figure 14 — Fig. 10 : FusionSQL reliability and shift sensitivity. (a) FusionSQL accurately predicts EX. (b) Shift impacts EX on hard targets but only mildly on distant-but-easy transfers, demonstrating robustness.

Fig. 11 : Latency + MAE vs. number of slices $L$ . Latency scales linearly with $L$ , while MAE stabilizes beyond $L{\approx}32$ . Hybrid SWD (red points) achieves low MAE with reduced $L$ , optimizing efficiency without sacrificing fidelity.

Config.	Lat. (s)	Mem. (GB)	$\Delta$ MAE (%) [CI]
AR-SWD (64)	3.42	8.9	$0.00{\pm}0.12$
HSWD (4,12)	0.96	2.8	$+0.60{\pm}0.15$
HSWD (6,16)	1.12	3.0	$+0.28{\pm}0.14$
HSWD (8,16)	1.27	3.2	$+0.19{\pm}0.13$
HSWD (10,24)	1.48	3.6	$\mathbf{-0.02}{\pm}0.12$
HSWD (12,32)	1.94	4.5	$\mathbf{-0.05}{\pm}0.11$

TABLE V : Efficiency of SWD variants. Comparison of all-random SWD (AR,

L{=}64

) and hybrid SWD (HSWD). Latency, memory, 95% CI, and

\Delta

MAE (%) relative to AR (negative values = improved accuracy) are reported.

Spider
Dataset / Method	ATHENA	ATHENA++	SQLizer	Avg.
BugJudge [ 44 ]	12.0 $\pm$ 1.0	11.8 $\pm$ 0.9	12.1 $\pm$ 1.0	12.0 $\pm$ 1.0
ArenaCmp [ 45 ]	10.8 $\pm$ 0.9	10.6 $\pm$ 0.8	10.9 $\pm$ 0.9	10.8 $\pm$ 0.9
FusionSQL-TI	14.3 $\pm$ 1.1	14.1 $\pm$ 1.1	14.4 $\pm$ 1.2	14.3 $\pm$ 1.2
FusionSQL-LLM	12.8 $\pm$ 1.1	12.6 $\pm$ 1.0	12.9 $\pm$ 1.1	12.8 $\pm$ 1.1
FusionSQL	8.3 $\pm$ 0.6	8.2 $\pm$ 0.7	8.4 $\pm$ 0.8	8.3 $\pm$ 0.7

TABLE VI : MAE (

\downarrow

) of classic models (e.g., ATHENA++) across datasets. Each cell reports mean

\pm

95% CI (percentage points). Best in bold ; second-best underlined .

	A	B	$\mathbf{s}(A\!\rightarrow\!B)$
A: Q = “What is the population of Paris?” ( WikiSQL style ) Sch = City(City, Country, Population) B: Q = “List singer names for concerts in 2014.” ( Spider style ) Sch = singer(Singer_ID, Name), concert(Concert_ID, Year), singer_in_concert( Singer_ID, Concert_ID)	$\mu_{A}=\begin{bmatrix}0.03\\ -0.01\\ 0.00\\ 0.02\end{bmatrix},\ \mathrm{diag}(\Sigma_{A})=\begin{bmatrix}0.35\\ 0.10\\ 0.06\\ 0.05\end{bmatrix}$	$\mu_{B}=\begin{bmatrix}0.12\\ 0.18\\ 0.05\\ 0.04\end{bmatrix},\ \mathrm{diag}(\Sigma_{B})=\begin{bmatrix}1.80\\ 0.42\\ 0.30\\ 0.22\end{bmatrix}$	$\begin{bmatrix}1.95\\ 2.60\\ 0.90\\ 0.22\end{bmatrix}$
A: Q = “For each department, return the average salary.” Sch = emp(Emp_ID, Dept_ID, Salary) B: SQL = SELECT Dept_ID, Salary FROM emp; ( missing AVG and GROUP BY )	$\mu_{A}=\begin{bmatrix}0.24\\ 0.08\\ -0.02\\ 0.04\end{bmatrix},\ \mathrm{diag}(\Sigma_{A})=\begin{bmatrix}1.10\\ 0.32\\ 0.14\\ 0.11\end{bmatrix}$	$\mu_{B}=\begin{bmatrix}0.03\\ 0.22\\ 0.00\\ 0.02\end{bmatrix},\ \mathrm{diag}(\Sigma_{B})=\begin{bmatrix}0.62\\ 0.40\\ 0.16\\ 0.12\end{bmatrix}$	$\begin{bmatrix}2.10\\ 3.85\\ 1.05\\ 0.25\end{bmatrix}$
Spider $\rightarrow$ BIRD	$\mu_{A}=\begin{bmatrix}0.12\\ 0.02\\ -0.05\\ 0.01\end{bmatrix},\ \mathrm{diag}(\Sigma_{A})=\begin{bmatrix}4.90\\ 0.95\\ 0.62\\ 0.41\end{bmatrix}$	$\mu_{B}=\begin{bmatrix}0.05\\ 0.34\\ -0.03\\ 0.06\end{bmatrix},\ \mathrm{diag}(\Sigma_{B})=\begin{bmatrix}4.65\\ 2.70\\ 0.76\\ 0.52\end{bmatrix}$	$\begin{bmatrix}0.88\\ 3.15\\ 0.31\\ 0.33\end{bmatrix}$
BIRD $\rightarrow$ WikiSQL	$\mu_{A}=\begin{bmatrix}0.06\\ 0.31\\ -0.02\\ 0.05\end{bmatrix},\ \mathrm{diag}(\Sigma_{A})=\begin{bmatrix}4.70\\ 2.60\\ 0.74\\ 0.50\end{bmatrix}$	$\mu_{B}=\begin{bmatrix}-0.02\\ 0.04\\ 0.01\\ -0.02\end{bmatrix},\ \mathrm{diag}(\Sigma_{B})=\begin{bmatrix}1.05\\ 0.20\\ 0.14\\ 0.11\end{bmatrix}$	$\begin{bmatrix}2.10\\ 2.40\\ 1.70\\ 0.29\end{bmatrix}$
WikiSQL $\rightarrow$ Spider	$\mu_{A}=\begin{bmatrix}-0.01\\ 0.02\\ 0.00\\ -0.01\end{bmatrix},\ \mathrm{diag}(\Sigma_{A})=\begin{bmatrix}1.10\\ 0.22\\ 0.15\\ 0.10\end{bmatrix}$	$\mu_{B}=\begin{bmatrix}-0.06\\ 0.30\\ 0.12\\ 0.08\end{bmatrix},\ \mathrm{diag}(\Sigma_{B})=\begin{bmatrix}8.05\\ 0.86\\ 0.95\\ 0.72\end{bmatrix}$	$\begin{bmatrix}4.92\\ 5.41\\ 1.25\\ 0.32\end{bmatrix}$
SParC $\rightarrow$ CoSQL	$\mu_{A}=\begin{bmatrix}0.00\\ 0.01\\ 0.02\\ -0.01\end{bmatrix},\ \mathrm{diag}(\Sigma_{A})=\begin{bmatrix}2.35\\ 0.58\\ 0.44\\ 0.30\end{bmatrix}$	$\mu_{B}=\begin{bmatrix}-0.08\\ 0.41\\ 0.05\\ 0.12\end{bmatrix},\ \mathrm{diag}(\Sigma_{B})=\begin{bmatrix}2.40\\ 0.62\\ 0.55\\ 0.39\end{bmatrix}$	$\begin{bmatrix}0.79\\ 6.05\\ 0.27\\ 0.43\end{bmatrix}$

TABLE VII: Embedding shifts. Qwen2.5-72B-Instruct embeddings (

8192

-D) are reduced to

4

-D by PCA. We report the mean

\mu

, diagonal variance

\mathrm{diag}(\Sigma)

, and shift vector

\mathbf{s}{=}[{\textit{SD}_{F}},{\textit{SD}_{M}},{\textit{SD}_{SW}},\|\mu_{A}-\mu_{B}\|_{2}]^{\top}

Automated Evaluation

Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.

See all 396 papers

TraceSQL: Traceable Answerability Estimation for Reference-Free Text-to-SQL Verification
Aug 18, 2026Neelesh Kumar Shukla, Debasmita Panda, Srutanik Bhaduri +3Text-to-SQL Interpretable ML
Oracle Corporation India · Oracle Corporation United States
A Demonstration of SQLyzr: A Platform for Fine-Grained Text-to-SQL Evaluation and Analysis
Apr 23, 2026Sepideh Abedini, M. Tamer ÖzsuLLM Evaluation Text-to-SQL
University of Waterloo
SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation
Sep 8, 2026Mohammadhossein Malekpour, Mohamed Riahi, Maxime Lamothe +1Mutation Testing Text-to-SQL
Polytechnique Montréal

Aug 18, 2026J/K move · Enter open · S save

TraceSQL: Traceable Answerability Estimation for Reference-Free Text-to-SQL Verification

Neelesh Kumar Shukla, Debasmita Panda, Srutanik Bhaduri, Aditya Banerjee+2

Oracle Corporation India · Oracle Corporation United States

Text-to-SQL Interpretable ML

Open paper AskPDF arXiv

Text-to-SQL systems are commonly evaluated using ground-truth SQL queries or reference execution results, but such supervision is unavailable at inference time in real-world deployments. This creates a critical verification problem: given only a user question, database context, and generated SQL, can a system estimate whether the generated query is likely to correctly answer the question? Recent approaches use LLMs as judge or specialized agents to inspect generated SQL, but their decisions can be difficult to trace. Outcome Reward Models (ORMs) address this by learning from execution-labeled candidate SQLs and assigning correctness scores to unseen queries, yet they still provide limited visibility into the signals behind each verification. To address this limitation, we propose TraceSQL, a lightweight and traceable verification model built on explicit diagnostic features. TraceSQL combines 67 features capturing question ambiguity, question requirements, question-schema-SQL consistency, SQL structure, and intent alignment. These signals remain available for examining which factors influence each prediction and for tracing decisions back to diagnostic evidence. On BIRD development databases, TraceSQL achieves 66.47% F1 and 64.48% ROC-AUC, compared with 61.87% F1 and 58.26% ROC-AUC for the GradeSQL-7B ORM baseline on the same generated-SQL evaluation. Feature attribution further shows that the model relies on both semantic grounding and deterministic SQL-structure signals. These results show that SQL verification can be performed with a lightweight learned model while retaining feature-level evidence for inspecting and diagnosing its predictions.

Abstract

Automated Evaluation

Explore similar work

TraceSQL: Traceable Answerability Estimation for Reference-Free Text-to-SQL Verification

A Demonstration of SQLyzr: A Platform for Fine-Grained Text-to-SQL Evaluation and Analysis

SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation