BudgetSchemaBench: A Budget-Swept Diagnostic for Schema Context in Text-to-SQL

Chen Shen

cs.CLSep 8, 2026

BudgetSchemaBench: A Budget-Swept Diagnostic for Schema Context in Text-to-SQL

Authors: Chen Shen

Organizations: Megagon Labs

Abstract

Data agents over structured sources must fit database schema into the model's context window. Large catalogs can span many databases and thousands of columns, so cost constraints may require choosing between table coverage and serialization detail well before the context window is full. We introduce BudgetSchemaBench, an execution-grounded diagnostic for this setting. Its construction derives relevance labels mechanically from gold SQL, without human- or LLM-authored ground truth. Using a pooled 80-database catalog, we sweep four schema-context budgets and compare three representations while keeping each retriever's table ranking fixed. A source-namespace check rejects queries that obtain the correct result from the wrong database. The evaluation covers three conditions: end-to-end retrieval; frozen-gold, in which the required tables are guaranteed; and a probe that removes those tables. For the primary solver with raw serialization, raising the budget from 2.5% to 50% of the catalog improves execution accuracy on 1,279 held-out questions by 18 percentage points under lexical retrieval but only 3 under dense retrieval; the dense retriever already finds most required tables at the smallest budget. When the required tables are removed, 94.6% of correct predictions name one of them exactly, consistent with reconstruction of absent schema from parametric knowledge. For the two main solvers in the frozen-gold condition, the three representations differ by at most 2 percentage points, and the widest paired 95% confidence interval bounds the difference within +/-4 points. We observe the same qualitative patterns with one reasoning model from a different family. When retrieval is coverage-limited, execution accuracy is more sensitive to the schema budget than to the tested serializations. The diagnostic and the code used to construct and evaluate it are publicly available.

Figures & tables

	large catalog	budget-swept axis	repr.-isolated EX
CRUSH4SQL	✓	recall@ $K$	—
MURRE	✓	EX, unswept	—
RULER	prose	✓	prose
Death of Schema Linking?	—	—	noise, not budget
Synthetic descriptions (Wretblad)	—	—	✓(fits-window)
LinkAlign / RASL / CRED-SQL	✓	—	retriever, not repr.

Table 1: Where prior work stops. To our knowledge, BudgetSchemaBench is the first to combine a fixed large catalog, the budget as a controlled swept axis, and execution-grounded representation isolation (plus a source-namespace validity contract and a remove-needed-table probe).

databases (BIRD snapshot)	$80$
tables	$597$
tokens, full catalog ( cl100k ; raw $97{,}144$ )	$127{,}560$
colliding base-name groups / tables in collision	$50\,/\,146$ ( $24.5\%$ )
budgets $K$ (% of catalog)	$2.5\,/\,10\,/\,25\,/\,50$
as cl100k tokens, to the nearest $100$	$3200\,/\,12800\,/\,31900\,/\,63800$
parser coverage	$100\%$ ( $1534/1534$ )

Table 2: Union catalog at a glance (the pilot instantiation). The full catalog never fits any budget

K

; the namespace prefix and the cross-database collisions are reported because they make a pooled catalog easier or harder than a real enterprise one. The bottom rows are the construction-validation gates (§ 4 ), computed over the released artifact.

	gpt-5.4-mini EX		gpt-4o-mini EX		recall
budget $K$	lexical	dense	lexical	dense	lex.	dense
2.5%	$0.38$ $[.35,.40]$	$0.57$ $[.54,.59]$	$0.30$ $[.28,.33]$	$0.48$ $[.45,.51]$	$0.60$	$0.92$
10%	$0.48$ $[.45,.51]$	$0.59$ $[.57,.62]$	$0.37$ $[.34,.40]$	$0.49$ $[.46,.52]$	$0.81$	$0.98$
25%	$0.54$ $[.51,.57]$	$0.59$ $[.57,.62]$	$0.40$ $[.37,.43]$	$0.48$ $[.45,.51]$	$0.89$	$1.00$
50%	$0.55$ $[.53,.58]$	$0.60$ $[.57,.63]$	$0.39$ $[.37,.42]$	$0.47$ $[.44,.50]$	$0.90$	$1.00$

Table 3: End-to-end budget sweep (full held-out complement,

N{=}1{,}279

, raw serialization): EX

(K)

(BIRD set-match

+

lineage) with bootstrap

95\%

CIs, plus table-recall

(K)

, for two solvers under lexical vs. dense retrieval. Under lexical retrieval, the large-minus-tiny gains have

95\%

confidence intervals that exclude zero (

\Delta{=}{+}0.18\,[0.15,0.20]

for gpt-5.4-mini ,

{+}0.09\,[0.07,0.12]

for gpt-4o-mini ; both

p{<}10^{-3}

). The corresponding gains are much smaller under dense retrieval (

\Delta{=}{+}0.03\,[0.02,0.05]

for gpt-5.4-mini ;

{-}0.01\,[{-}0.03,0.01]

, not separable, for gpt-4o-mini ).

gpt-5.4-mini
budget $K$	raw	enriched	hybrid
2.5%	$0.65$ $[.62,.67]$	$0.65$ $[.62,.68]$	$0.65$ $[.63,.68]$
10%	$0.64$ $[.62,.67]$	$0.66$ $[.64,.69]$	$0.65$ $[.62,.67]$
25%	$0.63$ $[.60,.65]$	$0.64$ $[.62,.67]$	$0.64$ $[.61,.67]$
50%	$0.63$ $[.61,.66]$	$0.64$ $[.61,.67]$	$0.63$ $[.60,.66]$
gpt-4o-mini

Table 4: Frozen-gold representation contrast (full held-out complement,

N{=}1{,}279

): with the gold tables guaranteed, only the per-table serialization varies. EX

(K)

with bootstrap

95\%

CIs for raw , enriched , and hybrid . Across all

24

paired contrasts (the two main solvers) the largest point difference is

|\Delta|{=}0.02

, no contrast survives Holm correction (every

p_{\mathrm{Holm}}{\geq}0.27

), and the widest paired

95\%

CI reaches

\pm 0.04

EX: a bounded null at this scale, not an established equivalence (the frontier slice gpt-5.5 ,

N{=}30

, is wider and illustrative only).

Figure 1: Execution accuracy increases with the schema budget under coverage-limited retrieval. End-to-end EX

(K)

on the full held-out complement (

N{=}1{,}279

, raw , gpt-5.4-mini ; bootstrap

95\%

CIs; the increase holds across the strict/relaxed bracket). The lexical retriever (solid) rises with

K

, tracking table-recall

0.60\!\to\!0.90

; the dense retriever (dashed) keeps recall high (

0.92\!\to\!1.0

) even at the tightest budget and stays near the gold-guaranteed ceiling, flattening the curve. The dotted reference line is EX with the gold tables guaranteed; the dash-dotted line is the lexical schema-memorization floor with the gold tables removed (

0.10

at the

25\%

budget, a likely leakage level). The dense floor is higher and is not drawn (

0.18

at the same budget); across

K

the two span

0.04

–

0.19

. The conditions differ in retrieval mechanism, so this is a sensitivity contrast, not a causal decomposition.

diagnostic pattern	DeepSeek-V4-Flash	( gpt-5.4-mini )
budget axis, lexical (large-minus-tiny $\Delta$ )	$+0.18$	$(+0.18)$
budget axis, dense (large-minus-tiny $\Delta$ )	$+0.06$	$(+0.03)$
frozen representation (max $\|\Delta\|$ )	$0.05$	$(0.02)$
memorization floor, lexical (EX across $K$ )	$0.01$ – $0.05$	$(0.04$ – $0.10)$
memorization floor, dense (EX across $K$ )	$0.05$ – $0.13$	$(0.16$ – $0.19)$

Table 5: Cross-family replication. An off-family reasoning model ( DeepSeek-V4-Flash , reasoning-enabled) reproduces all three qualitative patterns on the full held-out complement (EX

=

BIRD set-match

+

lineage); the gpt-5.4-mini value is shown in brackets for reference. This is one off-family replication, not a broad-generality claim.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	lexical EX		dense EX
budget $K$	unprefixed	prefixed	unprefixed	prefixed
2.5%	$0.348$ $[.32,.37]$	$0.375$ $[.35,.40]$	$0.533$ $[.51,.56]$	$0.567$ $[.54,.59]$
10%	$0.450$ $[.42,.48]$	$0.481$ $[.45,.51]$	$0.554$ $[.53,.58]$	$0.593$ $[.57,.62]$
25%	$0.507$ $[.48,.53]$	$0.539$ $[.51,.57]$	$0.561$ $[.54,.59]$	$0.594$ $[.57,.62]$
50%	$0.505$ $[.48,.53]$	$0.553$ $[.53,.58]$	$0.558$ $[.53,.59]$	$0.601$ $[.57,.63]$
large-minus-tiny $\Delta$	${+}0.157$ vs. ${+}0.178$		${+}0.025$ vs. ${+}0.034$

Appendix

Table 6: Unprefixed vs. prefixed catalog (end-to-end, raw , gpt-5.4-mini ,

N{=}1{,}279

): EX

(K)

with bootstrap

95\%

CIs over questions. Removing the <db> signal lowers EX by a small uniform margin but preserves the budget-dependent pattern: lexical EX increases with

K

, and its large-minus-tiny

95\%

confidence interval excludes zero; dense EX remains near saturation.

Text-to-SQL

Text-to-SQL translates natural-language questions into executable database queries, linking user intent to schemas, values, and SQL logic.

See all 169 papers

Schema-First Retrieval: Embedding Catalogs for Natural Language Analytics
Jun 23, 2026Adarsh Agrawal, Shashank IndukuriText-to-SQL Information Retrieval
Finding the Right Tables and Columns: A Benchmark and Corpus-Adaptive Embeddings for SQL Schema Retrieval
Jul 14, 2026Qingcheng Zeng, Puxuan Yu, Aman Mehta +2Schema Linking Text-to-SQL
Northwestern University · Snowflake Inc.
MDB-Link: Hierarchical Schema Linking for Multi-Database Text-to-SQL
Aug 10, 2026Beiyu Xu, Zhenyu Wu, Jiaoyan Chen +1Schema Linking Text-to-SQL
University of Manchester Faculty of Science and Engineering Department of Computer Science

Jun 23, 2026J/K move · Enter open · S save

Schema-First Retrieval: Embedding Catalogs for Natural Language Analytics

Adarsh Agrawal, Shashank Indukuri

Text-to-SQL Information Retrieval Table Retrieval Hybrid Retrieval Dense Retrieval Database Systems

Open paper AskPDF arXiv

Enterprise text-to-SQL systems often fail before SQL is generated: the model receives the wrong schema context. Modern warehouses contain thousands of tables, abbreviated columns, informal metrics, hidden join conventions, and permission boundaries that are not captured by raw table names. We introduce Schema-First Retrieval, a retrieval layer that embeds catalog metadata rather than warehouse rows. The system indexes five typed catalog objects, tables, columns, metrics, relationships, and query history, using object-specific text templates. At query time, it combines parallel vector search, lineage expansion, cross-encoder reranking, workload memory, and deterministic access-control gates before SQL generation. On CRUSH4SQL (1,534 questions), Schema-First Retrieval reaches 96.4% table recall@20 and cross-encoder reranking adds +11.1 points at column recall@10; against an equally-templated BM25 baseline, semantic retrieval is +32.8 points at table recall@5. On SEDE (857 questions), query history raises table recall@5 from 52.1% to 92.3%. On BIRD (96 questions), schema-first context reduces SQL execution errors from 15.6% to 6.2%, a 2.5x reduction. These results show that catalog selection is a first-class retrieval problem for natural language analytics, not a prompt formatting detail.

Abstract

Text-to-SQL

Explore similar work

Schema-First Retrieval: Embedding Catalogs for Natural Language Analytics

Finding the Right Tables and Columns: A Benchmark and Corpus-Adaptive Embeddings for SQL Schema Retrieval

MDB-Link: Hierarchical Schema Linking for Multi-Database Text-to-SQL