BudgetSchemaBench: A Budget-Swept Diagnostic for Schema Context in Text-to-SQL

Chen Shen

cs.CLSep 8, 2026

BudgetSchemaBench: A Budget-Swept Diagnostic for Schema Context in Text-to-SQL

Authors: Chen Shen

Organizations: Megagon Labs

Abstract

Data agents over structured sources must fit database schema into the model's context window. Large catalogs can span many databases and thousands of columns, so cost constraints may require choosing between table coverage and serialization detail well before the context window is full. We introduce BudgetSchemaBench, an execution-grounded diagnostic for this setting. Its construction derives relevance labels mechanically from gold SQL, without human- or LLM-authored ground truth. Using a pooled 80-database catalog, we sweep four schema-context budgets and compare three representations while keeping each retriever's table ranking fixed. A source-namespace check rejects queries that obtain the correct result from the wrong database. The evaluation covers three conditions: end-to-end retrieval; frozen-gold, in which the required tables are guaranteed; and a probe that removes those tables. For the primary solver with raw serialization, raising the budget from 2.5% to 50% of the catalog improves execution accuracy on 1,279 held-out questions by 18 percentage points under lexical retrieval but only 3 under dense retrieval; the dense retriever already finds most required tables at the smallest budget. When the required tables are removed, 94.6% of correct predictions name one of them exactly, consistent with reconstruction of absent schema from parametric knowledge. For the two main solvers in the frozen-gold condition, the three representations differ by at most 2 percentage points, and the widest paired 95% confidence interval bounds the difference within +/-4 points. We observe the same qualitative patterns with one reasoning model from a different family. When retrieval is coverage-limited, execution accuracy is more sensitive to the schema budget than to the tested serializations. The diagnostic and the code used to construct and evaluate it are publicly available.

Figures & tables

	large catalog	budget-swept axis	repr.-isolated EX
CRUSH4SQL	✓	recall@ $K$	—
MURRE	✓	EX, unswept	—
RULER	prose	✓	prose
Death of Schema Linking?	—	—	noise, not budget
Synthetic descriptions (Wretblad)	—	—	✓(fits-window)
LinkAlign / RASL / CRED-SQL	✓	—	retriever, not repr.

Table 1: Where prior work stops. To our knowledge, BudgetSchemaBench is the first to combine a fixed large catalog, the budget as a controlled swept axis, and execution-grounded representation isolation (plus a source-namespace validity contract and a remove-needed-table probe).

databases (BIRD snapshot)	$80$
tables	$597$
tokens, full catalog ( cl100k ; raw $97{,}144$ )	$127{,}560$
colliding base-name groups / tables in collision	$50\,/\,146$ ( $24.5\%$ )
budgets $K$ (% of catalog)	$2.5\,/\,10\,/\,25\,/\,50$
as cl100k tokens, to the nearest $100$	$3200\,/\,12800\,/\,31900\,/\,63800$
parser coverage	$100\%$ ( $1534/1534$ )

Table 2: Union catalog at a glance (the pilot instantiation). The full catalog never fits any budget

K

; the namespace prefix and the cross-database collisions are reported because they make a pooled catalog easier or harder than a real enterprise one. The bottom rows are the construction-validation gates (§ 4 ), computed over the released artifact.

	gpt-5.4-mini EX		gpt-4o-mini EX		recall
budget $K$	lexical	dense	lexical	dense	lex.	dense
2.5%	$0.38$ $[.35,.40]$	$0.57$ $[.54,.59]$	$0.30$ $[.28,.33]$	$0.48$ $[.45,.51]$	$0.60$	$0.92$
10%	$0.48$ $[.45,.51]$	$0.59$ $[.57,.62]$	$0.37$ $[.34,.40]$	$0.49$ $[.46,.52]$	$0.81$	$0.98$
25%	$0.54$ $[.51,.57]$	$0.59$ $[.57,.62]$	$0.40$ $[.37,.43]$	$0.48$ $[.45,.51]$	$0.89$	$1.00$
50%	$0.55$ $[.53,.58]$	$0.60$ $[.57,.63]$	$0.39$ $[.37,.42]$	$0.47$ $[.44,.50]$	$0.90$	$1.00$

Table 3: End-to-end budget sweep (full held-out complement,

N{=}1{,}279

, raw serialization): EX

(K)

(BIRD set-match

+

lineage) with bootstrap

95\%

CIs, plus table-recall

(K)

, for two solvers under lexical vs. dense retrieval. Under lexical retrieval, the large-minus-tiny gains have

95\%

confidence intervals that exclude zero (

\Delta{=}{+}0.18\,[0.15,0.20]

for gpt-5.4-mini ,

{+}0.09\,[0.07,0.12]

for gpt-4o-mini ; both

p{<}10^{-3}

). The corresponding gains are much smaller under dense retrieval (

\Delta{=}{+}0.03\,[0.02,0.05]

for gpt-5.4-mini ;

{-}0.01\,[{-}0.03,0.01]

, not separable, for gpt-4o-mini ).

gpt-5.4-mini
budget $K$	raw	enriched	hybrid
2.5%	$0.65$ $[.62,.67]$	$0.65$ $[.62,.68]$	$0.65$ $[.63,.68]$
10%	$0.64$ $[.62,.67]$	$0.66$ $[.64,.69]$	$0.65$ $[.62,.67]$
25%	$0.63$ $[.60,.65]$	$0.64$ $[.62,.67]$	$0.64$ $[.61,.67]$
50%	$0.63$ $[.61,.66]$	$0.64$ $[.61,.67]$	$0.63$ $[.60,.66]$
gpt-4o-mini

Table 4: Frozen-gold representation contrast (full held-out complement,

N{=}1{,}279

): with the gold tables guaranteed, only the per-table serialization varies. EX

(K)

with bootstrap

95\%

CIs for raw , enriched , and hybrid . Across all

24

paired contrasts (the two main solvers) the largest point difference is

|\Delta|{=}0.02

, no contrast survives Holm correction (every

p_{\mathrm{Holm}}{\geq}0.27

), and the widest paired

95\%

CI reaches

\pm 0.04

EX: a bounded null at this scale, not an established equivalence (the frontier slice gpt-5.5 ,

N{=}30

, is wider and illustrative only).

Figure 1: Execution accuracy increases with the schema budget under coverage-limited retrieval. End-to-end EX

(K)

on the full held-out complement (

N{=}1{,}279

, raw , gpt-5.4-mini ; bootstrap

95\%

CIs; the increase holds across the strict/relaxed bracket). The lexical retriever (solid) rises with

K

, tracking table-recall

0.60\!\to\!0.90

; the dense retriever (dashed) keeps recall high (

0.92\!\to\!1.0

) even at the tightest budget and stays near the gold-guaranteed ceiling, flattening the curve. The dotted reference line is EX with the gold tables guaranteed; the dash-dotted line is the lexical schema-memorization floor with the gold tables removed (

0.10

at the

25\%

budget, a likely leakage level). The dense floor is higher and is not drawn (

0.18

at the same budget); across

K

the two span

0.04

–

0.19

. The conditions differ in retrieval mechanism, so this is a sensitivity contrast, not a causal decomposition.

diagnostic pattern	DeepSeek-V4-Flash	( gpt-5.4-mini )
budget axis, lexical (large-minus-tiny $\Delta$ )	$+0.18$	$(+0.18)$
budget axis, dense (large-minus-tiny $\Delta$ )	$+0.06$	$(+0.03)$
frozen representation (max $\|\Delta\|$ )	$0.05$	$(0.02)$
memorization floor, lexical (EX across $K$ )	$0.01$ – $0.05$	$(0.04$ – $0.10)$
memorization floor, dense (EX across $K$ )	$0.05$ – $0.13$	$(0.16$ – $0.19)$

Table 5: Cross-family replication. An off-family reasoning model ( DeepSeek-V4-Flash , reasoning-enabled) reproduces all three qualitative patterns on the full held-out complement (EX

=

BIRD set-match

+

lineage); the gpt-5.4-mini value is shown in brackets for reference. This is one off-family replication, not a broad-generality claim.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	lexical EX		dense EX
budget $K$	unprefixed	prefixed	unprefixed	prefixed
2.5%	$0.348$ $[.32,.37]$	$0.375$ $[.35,.40]$	$0.533$ $[.51,.56]$	$0.567$ $[.54,.59]$
10%	$0.450$ $[.42,.48]$	$0.481$ $[.45,.51]$	$0.554$ $[.53,.58]$	$0.593$ $[.57,.62]$
25%	$0.507$ $[.48,.53]$	$0.539$ $[.51,.57]$	$0.561$ $[.54,.59]$	$0.594$ $[.57,.62]$
50%	$0.505$ $[.48,.53]$	$0.553$ $[.53,.58]$	$0.558$ $[.53,.59]$	$0.601$ $[.57,.63]$
large-minus-tiny $\Delta$	${+}0.157$ vs. ${+}0.178$		${+}0.025$ vs. ${+}0.034$

Appendix

Table 6: Unprefixed vs. prefixed catalog (end-to-end, raw , gpt-5.4-mini ,

N{=}1{,}279

): EX

(K)

with bootstrap

95\%

CIs over questions. Removing the <db> signal lowers EX by a small uniform margin but preserves the budget-dependent pattern: lexical EX increases with

K

, and its large-minus-tiny

95\%

confidence interval excludes zero; dense EX remains near saturation.

Text-to-SQL

Text-to-SQL translates natural-language questions into executable database queries, linking user intent to schemas, values, and SQL logic.

See all 169 papers

Jun 23, 2026

Schema-First Retrieval: Embedding Catalogs for Natural Language Analytics

Enterprise text-to-SQL systems often fail before SQL is generated: the model receives the wrong schema context. Modern warehouses contain thousands of tables, abbreviated columns, informal metrics, hidden join conventions, and permission boundaries that are not captured by raw table names. We introduce Schema-First Retrieval, a retrieval layer that embeds catalog metadata rather than warehouse rows. The system indexes five typed catalog objects, tables, columns, metrics, relationships, and query history, using object-specific text templates. At query time, it combines parallel vector search, lineage expansion, cross-encoder reranking, workload memory, and deterministic access-control gates before SQL generation. On CRUSH4SQL (1,534 questions), Schema-First Retrieval reaches 96.4% table recall@20 and cross-encoder reranking adds +11.1 points at column recall@10; against an equally-templated BM25 baseline, semantic retrieval is +32.8 points at table recall@5. On SEDE (857 questions), query history raises table recall@5 from 52.1% to 92.3%. On BIRD (96 questions), schema-first context reduces SQL execution errors from 15.6% to 6.2%, a 2.5x reduction. These results show that catalog selection is a first-class retrieval problem for natural language analytics, not a prompt formatting detail.

Adarsh Agrawal, Shashank Indukuri

Text-to-SQL Information Retrieval Table Retrieval Hybrid Retrieval Dense Retrieval Database Systems

Ask PDF

Jul 14, 2026

Finding the Right Tables and Columns: A Benchmark and Corpus-Adaptive Embeddings for SQL Schema Retrieval

Retrieval in the SQL setting has largely been studied as the task of finding, within a large collection of SQL statements, the statement that answers a natural-language question. At scale, however, a more fundamental retrieval problem precedes generation: schema retrieval, identifying the tables and columns a question requires in a database that may contain thousands of them, far more than fit in a model's context. We argue that this step warrants first-class evaluation. To this end, we recast five text-to-SQL datasets (Spider, BIRD, BEAVER, and two LiveSQLBench variants) as retrieval tasks at both table and column granularity, covering realistic and enterprise-scale schemas under two document representations, and we show that off-the-shelf text and code embedders transfer poorly to this setting. We then propose corpus-adaptive fine-tuning: natural-language queries are synthesized directly from the target schema corpus, granularity-aware hard negatives are mined, and a 305M-parameter embedder is fine-tuned contrastively. This procedure raises average recall@10 from 60.4 to 75.6 (nDCG@10 from 51.9 to 68.0), making the 305M model the strongest retriever under one billion parameters and competitive with state-of-the-art embedders of 4-8B parameters, more than an order of magnitude larger. The same recipe improves an 8B state-of-the-art embedder from 77.8 to 78.4 recall@10, matching the best result on the benchmark and indicating that the adaptation is backbone-agnostic. Leave-one-corpus-out experiments and a leakage audit show that these gains reflect a transferable schema-retrieval ability rather than memorization of the evaluation data. Our results establish schema linking as a standalone retrieval task and lightweight, label-free corpus adaptation as a practical route to deploying it at enterprise scale.

Qingcheng Zeng, Puxuan Yu, Aman Mehta +2

Northwestern University · Snowflake Inc.

Schema Linking Text-to-SQL Table Retrieval IR Evaluation Relational Databases Dense Retrieval

Ask PDF

Aug 10, 2026

MDB-Link: Hierarchical Schema Linking for Multi-Database Text-to-SQL

Traditional Text-to-SQL research and benchmarks assume a known target database, overlooking settings in which a query must be routed within a large, heterogeneous database collection. We therefore study schema linking in a multi-database setting, where the system must first locate the target database and then construct a compact, SQL-relevant schema for generation. We propose MDB-Link, a hierarchical schema-linking framework that retrieves question-relevant columns from a global index, aggregates retrieval evidence to shortlist databases, and uses a budget-aware large language model (LLM) for database reranking, table selection, and column grounding. With Qwen2.5-14B, MDB-Link outperforms LinkAlign on MMQA, Spider2-Snow, and BIRD-dev in database localization and column selection while producing schema subsets close in size to the gold schemas. Exact match improves from 16.88 to 51.41 on MMQA, 2.50 to 9.17 on Spider2-Snow, and 12.52 to 38.01 on BIRD-dev. MDB-Link also runs faster than LinkAlign and AutoLink, demonstrating the effectiveness of hierarchical schema reduction for downstream SQL generation.

Beiyu Xu, Zhenyu Wu, Jiaoyan Chen +1

University of Manchester Faculty of Science and Engineering Department of Computer Science

Schema Linking Text-to-SQL Database Systems

Ask PDF

Abstract

Text-to-SQL

Explore similar work

Schema-First Retrieval: Embedding Catalogs for Natural Language Analytics

Finding the Right Tables and Columns: A Benchmark and Corpus-Adaptive Embeddings for SQL Schema Retrieval

MDB-Link: Hierarchical Schema Linking for Multi-Database Text-to-SQL