ModularSQL: A Runtime Guardrail for the Multiplicity Blind Spot in Text-to-SQL

Tianxin Zhou; Ruixi Lin

cs.CLAug 26, 2026

ModularSQL: A Runtime Guardrail for the Multiplicity Blind Spot in Text-to-SQL

Authors: Tianxin Zhou, Ruixi Lin

Organizations: University of Southern California · Northeastern University

Abstract

Text-to-SQL systems are increasingly deployed on production databases, where queries that pass benchmark evaluation can still produce results that distort downstream workflows. Standard set-based execution accuracy (Set-EX) collapses duplicate rows and can therefore miss multiplicity errors, including missing DISTINCT, inflated aggregates, and Cartesian-style join explosions. We call this the Multiplicity Blind Spot (MBS) and introduce Multiset-EX, a multiplicity-preserving evaluation criterion that exposes such failures. Across released DeepEye-SQL artifacts from three backbones (Qwen2.5-Coder-32B, Qwen3-Coder-30B-A3B, and Gemma-3-27B) on executable BIRD-Dev N=1532, we find a consistent 5.81--6.79 pp gap between Set-EX and Multiset-EX. The gap is not specific to DeepEye-SQL: it persists on released DAIL-SQL+GPT-4 (5.22 pp) and BIRD GPT-3.5-turbo (3.39 pp) predictions. We further introduce ModularSQL, a lightweight post-selection runtime guardrail that probes executed results for multiplicity anomalies and applies deterministic patches or low-cost LLM rescue only to flagged queries. Integrated with DeepEye-SQL using Qwen3-Coder, ModularSQL preserves Set-EX at 72.06% while improving Multiset-EX from 65.86% to 67.75% (+1.89 pp). It flags 77 high-risk anomalies, while adding only $0.0076 in total LLM cost and 120 ms amortized latency per query. Cross-pipeline evaluation shows that the candidate-free detector and deterministic patches also transfer to independently released prediction sets. Overall, these results show that benchmark accuracy does not necessarily imply execution-safe SQL, and that lightweight, multiplicity-aware runtime guardrails can narrow this gap with modest computational overhead.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of ModularSQL. ModularSQL operates as a post-selection runtime guardrail layer atop DeepEye-SQL. After S7 selects an executable SQL query, ModularSQL probes the execution behavior for multiplicity-related anomalies such as high duplication ratio, empty results, or execution errors. Unaffected queries pass through unchanged, while anomalous queries are routed through targeted interventions, including deterministic patches or a low-cost LLM rescue.

Backbone	Set	Multi	Gap
Qwen2.5-Coder-32B	69.91	64.10	5.81
Qwen3-Coder-30B-A3B	72.85	66.51	6.33
Gemma-3-27B	70.50	63.71	6.79
Qwen3-Coder-30B-A3B (ours)	72.06	65.86	6.20

Table 1: Set-EX and Multiset-EX on executable BIRD-Dev (

N=1{,}532

). Scores are percentages; gaps are percentage points.

System	Set-EX	Mset-EX	Gap	Flagged	Patched Mset
DAIL-SQL+GPT-4	53.59	48.37	5.22	270	49.02 ( $+10$ )
BIRD GPT-3.5 (kg)	35.90	32.51	3.39	555	33.03 ( $+8$ )
DeepEye-SQL (ours)	72.06	65.86	6.20	77	—

Table 2: Candidate-free guardrail components applied unchanged to two released BIRD-Dev prediction sets (

N{=}1{,}532

). Set-EX is unchanged by the patch on both systems.

System	Set-EX (%)	Multiset-EX (%)	$\Delta$ Multiset-EX
DeepEye reproduction	72.06	65.86	—
+ P1	72.06	67.23	$+1.37$
+ P1 + P2	72.06	67.43	$+1.57$
v1: P1+P2 $\rightarrow$ Rescue	72.06	67.17	$+1.31$
v2: Rescue $\rightarrow$ P1+P2	72.06	67.75	$\mathbf{+1.89}$
v2 variant: dup-ratio selector	72.19	67.30	$+1.44$

Table 3: Main results on the executable BIRD-Dev subset (

N=1{,}532

\Delta

is measured relative to the DeepEye reproduction. Rows 1 to 3 give the cumulative effect of the deterministic patches; rows 4 to 5 contrast the two intervention orderings. The highlight v2 row is the canonical ModularSQL configuration used throughout the paper. The dup-ratio selector replaces the LLM rescue with a zero-cost rule that selects the healthy candidate with the lowest duplication ratio; unlike the LLM rescue, it does not fully preserve Set-EX, yielding a net gain of 2 Set-EX queries.

Difficulty	$n$	Base	ModularSQL v2	$\Delta$
Simple	925	71.57	73.19	$+1.62$
Moderate	463	60.48	62.20	$+1.73$
Challenging	144	46.53	50.69	$\mathbf{+4.17}$

Table 4: Multiset-EX gain by BIRD difficulty stratum (

N=1{,}532

). Scores are percentages;

\Delta

is the ModularSQL v2 gain over the DeepEye reproduction. The challenging stratum shows the largest relative gain.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

$\tau$	Fix	Break	Net
0.10	45	34	$+11$
0.30	44	26	$+18$
0.60	36	15	$+21$
0.80	30	8	$\mathbf{+22}$
0.90	20	3	$+17$

Appendix

Table 5: P1 (DISTINCT injection) threshold sensitivity in our DeepEye-SQL reproduction. Fix and Break count the rule’s effect on the 87 P1-fire cases at each

\tau

; Net is

\text{Fix}-\text{Break}

. The deployed operating point

\tau=0.80

at the top of the plateau is highlighted.

$\tau$	Fires	Fix	Break	Net
0.10	5	3	0	$\mathbf{+3}$
0.20	8	4	1	$+3$
0.30	12	4	4	$0$
0.50	25	9	9	$0$
0.80	45	15	18	$-3$

Appendix

Table 6: P2 (DISTINCT removal) threshold sensitivity in our DeepEye-SQL reproduction. Higher

\tau

raises the firing rate but does not improve net rescue; the fix-to-break ratio collapses to

1{:}1

\tau=0.30

and inverts above

\tau=0.50

Pipeline stage	Multiset-EX	$\Delta$ vs P1+P2
DeepEye reproduction	1009	—
+ P1	1030	—
+ P1 + P2	1033	$0$
v1 (P1+P2 $\rightarrow$ Rescue)	1029	$-4$
v2 (Rescue $\rightarrow$ P1+P2)	1038	$+5$

Appendix

Table 7: Per-stage Multiset-EX counts (N=1,532). The marginal

\Delta

column reports the gain or loss relative to the +P1+P2 configuration. In v1 the rescue acts on patched SQL and results in a net loss of 4 Multiset-EX-correct queries; in v2 the rescue acts on the original S7 selection before patches refine the result, yielding 5 additional queries beyond patches alone.

Difficulty	$n$	Base	+ P1	+ P1+P2	v1	v2
Simple	925	662	671	673	671	677
Moderate	463	280	287	288	286	288
Challenging	144	67	72	72	72	73
Total	1532	1009	1030	1033	1029	1038

Appendix

Table 8: Multiset-EX counts per BIRD-Dev difficulty stratum at each pipeline stage. "v1" denotes the P1+P2

\rightarrow

Rescue ordering and "v2" the canonical Rescue

\rightarrow

P1+P2 ordering, matching Table 3 .

	Flagged by reason					P1 patch (Mset-EX)
System	Exec	Empty	Cart	Total	P1	Fix	Brk	Net
DeepEye-SQL	1	6	70	77	56	30	8	$+22$
DAIL-SQL+GPT-4	96	100	74	270	49	22	12	$+10$
GPT-3.5-turbo	354	160	41	555	11	8	0	$+8$

Appendix

Table 9: Per-system flagged counts by trigger reason and P1 patch outcomes on BIRD-Dev (

N=1{,}532

). “Exec” = execution error, “Empty” = empty result set, “Cart” = cartesian explosion (dup-ratio

\geq\tau

). P1 fires only on cartesian-flagged queries that do not contain a LIMIT clause. Fix and Brk count Multiset-EX transitions (

0\!\to\!1

and

1\!\to\!0

, respectively) among the P1-fired subset.

System	Base Set-EX	New Set-EX	Base Mset-EX	New Mset-EX
DAIL-SQL+GPT-4	53.59%	53.59%	48.37%	49.02%
GPT-3.5-turbo	35.90%	35.90%	32.51%	33.03%

Appendix

Table 10: Set-EX and Multiset-EX accuracy before and after the candidate-free guardrail (P1 only) on the two cross-pipeline systems. Set-EX is preserved exactly; Multiset-EX improves by 0.65 pp and 0.52 pp, respectively.

System	Flagged	TP	FP	$P$	MBS	Flagged	$R$
		Precision			MBS recall
DeepEye-SQL	77	65	12	0.844	95	36	0.379
DAIL-SQL+GPT-4	271	256	15	0.945	80	34	0.425
GPT-3.5-turbo	555	550	5	0.991	52	18	0.346

Appendix

Table 11: Detector precision (

P

) and MBS recall (

R

) on BIRD-Dev (

N=1{,}532

). TP denotes flagged queries that are Multiset-EX failures; FP denotes flagged queries that are Multiset-EX correct. MBS failures pass Set-EX but fail Multiset-EX; the detector prioritizes precision rather than maximizing MBS recall.

Trigger reason	DeepEye-SQL	DAIL-SQL+GPT-4	GPT-3.5-turbo
exec_error	1.000 (1/1)	1.000 (97/97)	1.000 (354/354)
empty_result	1.000 (6/6)	1.000 (100/100)	1.000 (160/160)
cartesian_expl	0.829 (58/70)	0.797 (59/74)	0.878 (36/41)

Appendix

Table 12: Per-trigger-reason precision. Each cell shows precision followed by (TP/total) for that trigger reason. All false positives originate from the cartesian-explosion trigger.

Text-to-SQL

Text-to-SQL translates natural-language questions into executable database queries, linking user intent to schemas, values, and SQL logic.

See all 169 papers

SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation
Sep 8, 2026Mohammadhossein Malekpour, Mohamed Riahi, Maxime Lamothe +1Mutation Testing Text-to-SQL
Polytechnique Montréal
SIRIUS-SQL: Anchoring Multi-Candidate Text-to-SQL in Execution Feedback
May 31, 2026Leo Luo, Haining Xie, Siqi Shen +8Execution-Guided Text-to-SQL LLM Reranking
TEG, Tencent Inc., China · 2Peking University, China
SpotIt+: Verification-based Text-to-SQL Evaluation with Database Constraints
Mar 4, 2026Andrew Tremante, Yang He, Rocky Klopfenstein +3Text-to-SQL Evaluation Text-to-SQL
∗Amherst College · †Simon Fraser University · ‡VMware Research by Broadcom

Sep 8, 2026J/K move · Enter open · S save

SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation

Mohammadhossein Malekpour, Mohamed Riahi, Maxime Lamothe, Amine Mhedhbi

Polytechnique Montréal

Mutation Testing Text-to-SQL Automated Evaluation Text-to-SQL Evaluation

Open paper AskPDF arXiv

Text-to-SQL systems translate natural language queries into executable SQL, democratizing access to structured data. Despite recent advances driven by large language models (LLMs), evaluation remains a major bottleneck: public benchmarks fail to capture the complexity of enterprise schema, while building private evaluation sets is costly and nondeterministic, making evaluation results difficult to reproduce. To address this issue, we present SQLMorph, a framework for Text-to-SQL evaluation via query mutation. SQLMorph introduces two techniques to automatically generate and expand evaluation sets: Join Query Expansion (JQE), which systematically increases structural complexity through valid join additions, and Textual Query Augmentation (TQA), which generates controlled natural language perturbations to assess robustness to linguistic variation. JQE and TQA create targeted choke points to challenge specific system components. When applied to state-of-the-art systems, JQE increases query coverage and reveals accuracy degradation as the number of joins grows. Meanwhile, TQA shows that linguistic brittleness induced by heavy abbreviation can reduce accuracy by up to 17%. Beyond evaluation sets, SQLMorph introduces a family of execution-level metrics that address the limitations of current binary measures, such as Execution Accuracy. We define Execution Precision (EXP) and Execution Recall (EXR) to quantify the fraction of correct and recovered results, respectively, and combine them via F1 for unified scoring. Our experiments show that these relaxed metrics enable fine-grained analysis of over- and under-prediction, revealing differences across systems that binary metrics obscure. Together, SQLMorph's query mutation and fine-grained metrics support debugging and better align Text-to-SQL evaluation practices with real-world deployments.

Abstract

Text-to-SQL

Explore similar work

SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation

SIRIUS-SQL: Anchoring Multi-Candidate Text-to-SQL in Execution Feedback

SpotIt+: Verification-based Text-to-SQL Evaluation with Database Constraints