Beyond the Harness: End-to-End Optimization of Context Artifacts for Enterprise Text-to-SQL

Kate Gwimm; Carson Eisenach

cs.AIAug 24, 2026

Beyond the Harness: End-to-End Optimization of Context Artifacts for Enterprise Text-to-SQL

Authors: Kate Gwimm, Carson Eisenach

Organizations: Amazon

Abstract

Deploying LLMs for enterprise Text-to-SQL is bottlenecked less by the model than by what context reaches it: business logic spans thousands of tables, and no model can ingest a full catalog at once. We argue that the most effective place to intervene is therefore the knowledge-base context the model consumes, and that this context should be constructed from historical usage rather than tuned for as a fixed input. Using a query-DAG decomposition--the same family of intermediates that enterprise benchmarks like BEAVER annotate, here recovered from production SQL--we compare the value of oracle query graphs versus retrieved knowledge-base context. In this ablation, retrieved knowledge-base context provides the largest marginal improvement when added to the full oracle graph. Building on this, we optimize a distillation procedure that turns historical query profiles into reusable SQL reference cards. On a benchmark of 5176 production queries from a major online retailer, optimizing these context artifacts yields larger gains ( ${\sim}12$ -- $25\%$ AST similarity) than optimizing the retrieval harness ( ${\sim}3$ -- $12\%$ ). On the public BEAVER benchmark, which lacks the production-usage signals available in our internal setting, the picture is more mixed: table cards alone perform about the same as raw historical SQL. The best optimized variant retrieves both cards and raw SQL, scoring $9.00\%$ versus $6.33\%$ (p-value $0.12$ ) for the comparable baseline on a held-out $N{=}300$ subset, using retrieved context and harness changes but no agentic loop.

Figures & tables

		Similarity
Model	Input	AST	String	Linkage
Claude Sonnet 4.5	$\mathcal{I}_{1}$ : NL	0.100	0.303	–
Qwen Coder 3-30B	$\mathcal{I}_{1}$ : NL	0.082	0.281	–
Claude Sonnet 4.5	$\mathcal{I}_{1}$ : NL + RAG	0.278	0.447	–
Qwen Coder 3-30B	$\mathcal{I}_{1}$ : NL + RAG	0.184	0.346	–
Claude Sonnet 4.5	$\mathcal{I}_{2}$ : NL + Linkage	0.248	0.468	0.735

Table 1: Synthesis quality versus oracle graph fidelity. Rows are ordered by increasing input; linkage similarity applies only to graph-level rows.

Panel of Figure 1 — Figure 1: Context-artifact optimization loop. An outer LLM mutates the selector $f$ or summarizer prompt $P_{\text{sum}}$ , regenerates per-table artifacts $K_{t}$ , scores held-out queries, and accepts or reverts by metric delta. Failure feedback attributes errors to $f$ , $P_{\text{sum}}$ , or harness $H$ to target later mutations.

			Table Selection		End-to-End			Exec.
Model	Harness	Context	Recall	Prec.	AST	String	LLM-J	EX
Claude Sonnet 4.6	Baseline	Baseline	0.714	0.658	0.490	0.591	0.553	0.255
	Optimized	Baseline	0.741	0.679	0.503	0.600	0.546	0.275
	Baseline	Optimized	0.805	0.777	0.550	0.639	0.597	0.333
	Optimized	Optimized	0.766	0.717	0.570	0.655	0.600	0.304
Qwen Coder 3-30B	Baseline	Baseline	0.660	0.577	0.407	0.508	0.503	0.176

Table 2: Internal benchmark results: harness optimization versus context-artifact optimization, evaluated end-to-end with real retrieval. Table-selection and similarity metrics use the

n{=}517

sample; EX uses the executable cohort with

n{=}102

. “Context” = table schema information vs. optimized SQL reference cards.

Method	Context injected	Tools / calls	Exec. acc.	95% CI
Few-shot (ours)	schemas + demos	–	6.33%	[4.1, 9.7]
Few-shot †	schemas + demos	–	8.8%	–
ReFoRCE †	self-explored schema	explore, vote, fix	10.8%	–
Raw queries (Optimized)	retrieved gold SQL	–	6.33%	[4.1, 9.7]
Table cards (Optimized)	aggregated per-table	–	6.67%	[4.4, 10.1]
Both (Optimized)	cards + raw queries	–	9.00%	[6.3, 12.8]

Table 3: External validation on BEAVER (execution accuracy, held-out

N{=}300

; no oracle table linkage). Exec. acc. is reported with a 95% CI over the 300 binary outcomes. Published rows (

\dagger

) are point estimates from the benchmark authors’ harness ( Chen et al. 2024 ) . Pairwise differences among the optimized rows are not significant under paired t-tests: Both vs. raw

p{=}0.12

, Both vs. cards

p{=}0.14

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

			Table Selection		End-to-End			Exec.
Model	Harness	Context	Recall	Prec.	AST	String	LLM-J	EX
Claude Sonnet 4.6	Baseline	Baseline	[0.682, 0.745]	[0.628, 0.689]	[0.461, 0.520]	[0.564, 0.618]	[0.525, 0.580]	[0.180, 0.347]
	Optimized	Baseline	[0.712, 0.770]	[0.650, 0.709]	[0.473, 0.533]	[0.574, 0.628]	[0.518, 0.574]	[0.197, 0.368]
	Baseline	Optimized	[0.778, 0.830]	[0.749, 0.802]	[0.521, 0.579]	[0.613, 0.665]	[0.570, 0.625]	[0.249, 0.429]
	Optimized	Optimized	[0.738, 0.793]	[0.689, 0.745]	[0.540, 0.599]	[0.627, 0.681]	[0.571, 0.629]	[0.223, 0.399]
Qwen Coder 3-30B	Baseline	Baseline	[0.628, 0.690]	[0.545, 0.607]	[0.381, 0.435]	[0.484, 0.534]	[0.478, 0.531]	[0.115, 0.262]

Appendix

Table 4:

95\%

confidence intervals for every cell of Table 2 . Effective per-cell

n

after evaluation failures is

488

–

517

for table selection and

509

–

517

end-to-end; EX is

n{=}102

in every cell. “Context” baseline = handcrafted documentation, optimized = SQL reference cards.

Figure 2: LLM-judge semantic similarity versus AST similarity on the internal benchmark. The positive association supports using the judge as a complementary semantic signal to the structural metrics.

Figure 3: BEAVER-taxonomy failure mix before versus after optimization on the internal execution cohort ( $n{=}102$ ; failures defined by execution accuracy $=0$ ). Bars show the percentage of analyzed failures exhibiting each error category (multi-label; bars do not sum to $100\%$ ). Retrieval-phase categories (R1–R3) appear above the dotted line and generation-phase categories (G1–G5) below. Panel titles report binary execution accuracy and the analyzed-failure count for each condition.

Query structure	Exec. acc.
Single-level ( base )	$48\%$
Deeply compositional (nested-CTE)	${\sim}1.5\%$

Appendix

Table 5: BEAVER execution accuracy by query compositionality (

N{=}300

, dw split). The pattern holds uniformly across all context conditions of Table 3 ; difficulty is governed by query structure, not by the injected context.

Text-to-SQL

Text-to-SQL translates natural-language questions into executable database queries, linking user intent to schemas, values, and SQL logic.

See all 173 papers

Jun 2, 2026

EntSQL: A Benchmark for Grounding Text-to-SQL in Long-Context Enterprise Knowledge

Text-to-SQL enables natural language access to databases, and recent LLMs have substantially advanced its capabilities. Existing benchmarks such as Spider, BIRD, and Spider~2.0 evaluate schema generalization, large-scale databases, and realistic workflows, but largely overlook enterprise scenarios where SQL generation depends on private business knowledge, such as internal metrics, reporting conventions, and organizational rules. We introduce EntSQL, an enterprise-oriented Text-to-SQL benchmark for evaluating long-context grounding over proprietary business documents. EntSQL contains 1,066 aligned Chinese-English semantic examples across five business domains, with most examples requiring domain knowledge beyond the question and schema and involving complex SQL structures. On English inputs, the best evaluated system reaches only 15.9% when long-form documents are provided, highlighting the difficulty of grounding SQL generation in enterprise knowledge.

Chengxi Liao, Tao Xu, Zulong Chen +7

1HKUST (GZ) · 2Alibaba Group · 4Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) +1

LLM Grounding Text-to-SQL Text-to-SQL Evaluation Database Systems

Ask PDF

Oct 24, 2026

Schema-Free Text-to-SQL: Learning Enterprise Database Structure from Query Logs

Text-to-SQL systems achieve remarkable accuracy on academic benchmarks, yet performance drops dramatically in enterprise deployments. This gap stems from a fundamental assumption: existing approaches require complete schema documentation, but enterprise databases typically lack comprehensive documentation, featuring cryptic naming and implicit relationships. We present the first text-to-SQL framework that operates without schema documentation, instead learning database structure from historical query logs. Query logs encode implicit structural knowledge: which tables are queried together, how they join, and what columns map to business concepts. We construct a knowledge graph from parsed query patterns and introduce entity-driven GraphRAG retrieval that grounds user questions in domain terminology rather than relying solely on embedding similarity. Validated on both an industrial deployment with expert evaluation and the Beaver benchmark, our approach substantially outperforms similarity-based retrieval methods, the dominant industry approach, in both executability and answer quality. On Beaver, we achieve 95.7% execution validity compared to 60.8% for embedding-based retrieval, with expert data engineers preferring our system 70% to 10% on internal deployment. Our work demonstrates that text-to-SQL can operate effectively in schema-ambiguous enterprise environments, enabling natural language database interfaces "in the wild".

Ömer Faruk Akgül, Nikhil Kapahi, Prajit Reddy Muppidi

University of Southern California · Amazon

Text-to-SQL Graph Retrieval-Augmented Generation

Ask PDF

Aug 29, 2026

BIRD-History: A Benchmark for History-Driven Text-to-SQL with Fine-Grained Knowledge Annotations

While recent Large Language Model (LLM)-based text-to-SQL systems achieve impressive performance on standard benchmarks, they struggle when user queries implicitly rely on domain-specific knowledge, such as business logic, data conventions, and analytical practices, that is neither captured by the schema nor explicitly stated in the natural language question. Historical SQL query logs offer a valuable source of such knowledge, yet existing benchmarks do not adequately support evaluation of history-driven approaches. To address this gap, we introduce BIRD-History, a benchmark consisting of 1,393 tasks across 11 databases, designed to evaluate text-to-SQL systems' ability to ground underspecified natural language questions using historical SQL scripts. Each task is annotated with ground-truth labels specifying which historical queries contain relevant knowledge and which SQL clauses encode it, enabling systematic evaluation of both retrieval effectiveness and knowledge utilization. Alongside the benchmark, we propose a plug-in retriever that extracts five types of external knowledge from historical SQL scripts, then retrieves and reranks relevant fragments for query generation. The retriever integrates seamlessly into existing few-shot text-to-SQL pipelines without requiring prompt modifications. Experiments demonstrate consistent improvements across four text-to-SQL systems, highlighting the value of leveraging historical query logs for handling underspecified queries. Dataset and code are open-sourced on https://github.com/zjuidg/BIRD-History.

Yunfan Zhou, Qiming Shi, Yizhou Yang +2

State Key Lab of CAD&CG, Zhejiang University · School of Software Technology, Zhejiang University

Text-to-SQL Text-to-SQL Evaluation LLM Grounding

Ask PDF

Abstract

Text-to-SQL

Explore similar work

EntSQL: A Benchmark for Grounding Text-to-SQL in Long-Context Enterprise Knowledge

Schema-Free Text-to-SQL: Learning Enterprise Database Structure from Query Logs

BIRD-History: A Benchmark for History-Driven Text-to-SQL with Fine-Grained Knowledge Annotations