Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL

Jiayan Lin; Yujia Liu; Zijin Hong; Zheng Yuan; Yilin Xiao; Hao Chen; Qinggang Zhang; Xiao Huang

cs.CLAug 28, 2026

Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL

Authors: Jiayan Lin, Yujia Liu, Zijin Hong, Zheng Yuan, Yilin Xiao, Hao Chen, Qinggang Zhang, Xiao Huang,

Organizations: Jinan University Guangzhou, China · The Hong Kong Polytechnic University Kowloon, Hong Kong · City University of Macau Taipa, Macau · Jilin University Changchun, China ·

Abstract

Recent advances in in-context learning (ICL) text-to-SQL have substantially improved execution accuracy on public benchmarks by assembling increasingly elaborate pipelines around the base generator, yet existing studies typically report aggregate end-to-end accuracy, without quantifying the marginal accuracy-cost contribution of individual design choices. Consequently, providing a unified, paradigm-level cost-accuracy quantification remains a critical challenge for understanding and configuring modern text-to-SQL. To address this, we instantiate 17 paradigm-level configurations across five recurring modules of the ICL text-to-SQL pipeline under a single controlled implementation, and attribute each paradigm's marginal contribution and incurred cost across all four backbones spanning diverse capability levels and reasoning styles. Our analysis reveals that execution-feedback refinement is the only paradigm whose benefit holds universally at consistently low cost, while most other modules help only under backbone-dependent conditions. Token accounting shows that input demand is more closely tied to pipeline structure, whereas output demand is more sensitive to backbone generation behavior. Cross-module analysis further shows that stacking improves accuracy on most backbones, although how the gains compose varies with backbone capability. We also find that a fixed budget is often better spent engineering a more elaborate pipeline over a mid-tier backbone than upgrading to a frontier model with a lean pipeline. These findings distill into an actionable, cost-aware tiered guideline that transfers to five additional backbones without per-paradigm search.

Figures & tables

Figure 1. Overview of a modular ICL Text-to-SQL pipeline and its accuracy–cost trade-offs. Five recurring modules span pre-processing, SQL generation, and post-processing, but their benefits and inference costs vary across backbones, yielding different cost-efficient configurations under fixed budgets. A schematic comparison of rising Text-to-SQL leaderboard accuracy and increasing token and dollar costs as additional pipeline modules are added.

Figure 2. Taxonomy of ICL Text-to-SQL pipelines, organizing five modules across three pipeline stages into 17 paradigm-level configurations and 24 representative methods, including methods that naturally span multiple paradigms within the pipeline. Repeated method names indicate systems that combine design paradigms across modules or use multiple strategies within the same module, providing a unified view of how modern ICL Text-to-SQL systems assemble pipeline components in practice. A circular taxonomy diagram organizing five ICL text-to-SQL modules across preprocessing, SQL generation, and post-processing stages, with branches for 17 paradigm-level configurations and representative methods.

Paradigm	GPT-4o-mini					DeepSeek-V4-Flash					Gemini-2.5-Flash					GPT-5.4
Paradigm	EX $\boldsymbol{\uparrow}$	Cost $\boldsymbol{\downarrow}$	CPP $\boldsymbol{\downarrow}$	IN $\boldsymbol{\downarrow}$	OUT $\boldsymbol{\downarrow}$	EX $\boldsymbol{\uparrow}$	Cost $\boldsymbol{\downarrow}$	CPP $\boldsymbol{\downarrow}$	IN $\boldsymbol{\downarrow}$	OUT $\boldsymbol{\downarrow}$	EX $\boldsymbol{\uparrow}$	Cost $\boldsymbol{\downarrow}$	CPP $\boldsymbol{\downarrow}$	IN $\boldsymbol{\downarrow}$	OUT $\boldsymbol{\downarrow}$	EX $\boldsymbol{\uparrow}$	Cost $\boldsymbol{\downarrow}$	CPP $\boldsymbol{\downarrow}$	IN $\boldsymbol{\downarrow}$	OUT $\boldsymbol{\downarrow}$
Baselines
$N{=}1$	49.74	0.394	–	1510	51	56.32	0.547	–	1557	494	59.84	1.035	–	1642	73	61.28	7.305	–	1509	66
$N{=}5$	49.67	1.972	–	7550	255	56.45	2.756	–	7787	2523	60.04	5.219	–	8222	374	61.41	35.864	–	7545	301
$\mathrm{OCR}@5$	56.91	1.972	–	7550	255	60.56	2.756	–	7787	2523	64.54	5.219	–	8222	374	66.82	35.864	–	7545	301
Example Retrieval

Table 1. Module-level accuracy–cost results on BIRD dev across four LLMs. EX : execution accuracy (%); Cost : total API spend (USD); CPP : incremental USD per EX point over the matched-budget baseline (

N{=}5

for Candidate Selection,

N{=}1

otherwise; § 3.2 ). IN/OUT : mean per-question tokens (DeepSeek-V4-Flash reasoning tokens count as OUT). Significance uses McNemar’s test (

\alpha{=}0.05

; full CIs in the supplement). In CPP, numbers denote significant gains; “ ∘ ”, nonsignificant gains; “–”, no EX improvement; and “ ∗ ”, a significant gain at no higher cost. / mark the best/second-best results; OCR@5 is the oracle upper bound for

N{=}5

Figure 4.

\Delta\mathrm{EX}

(pp) relative to the matched-budget reference baseline for 17 paradigm configurations on BIRD dev. Shaded bands show 95% CIs for paired differences, and the dashed line marks

\Delta\mathrm{EX}=0

. Statistical significance is assessed using McNemar’s test. Full numerical CIs are provided in the supplementary material for every paradigm on each backbone. A line chart showing execution-accuracy changes for 17 paradigm configurations across GPT-4o-mini, DeepSeek-V4-Flash, Gemini-2.5-Flash, and GPT-5.4, with 95 percent confidence bands and a zero-change baseline.

Backbone	Paradigm	EX $\uparrow$	Cost $\downarrow$	CPP $\downarrow$	IN $\downarrow$	OUT $\downarrow$
Qwen3-235B (Thinking)	$G_{1}$	57.82	4.55	–	1555	1134
	$G_{2}$	58.02 ( $+0.20$ )	5.22	3.426	1529	1327
	$G_{3}$	57.30 ( $-0.52$ )	5.84	–	1596	1495
o3-mini	$G_{1}$	54.11	5.67	–	1509	463
	$G_{2}$	53.46 ( $-0.65$ )	5.83	–	1546	477
	$G_{3}$	53.06 ( $-1.04$ )	7.08	–	1538	665

Table 2. Generation strategy results on BIRD dev for two additional reasoning backbones. Qwen3-235B uses thinking mode. Parentheses report

\Delta\mathrm{EX}

over

G_{1}

(pp). Units: Cost in USD, CPP in USD/pp, and IN/OUT in tokens per question.

Figure 5. Per-question input (IN) and output (OUT) token consumption across the 17 paradigm configurations for two backbones, GPT-4o-mini and DeepSeek-V4-Flash, on BIRD. Dashed horizontal lines mark the corresponding baseline token levels for direct comparison across configurations.

	GPT-4o-mini			DeepSeek-V4-Flash			Gemini-2.5-Flash			GPT-5.4
Combination	EX $\uparrow$	$\boldsymbol{\mathcal{I}}$	Total Cost $\downarrow$	EX $\uparrow$	$\boldsymbol{\mathcal{I}}$	Total Cost $\downarrow$	EX $\uparrow$	$\boldsymbol{\mathcal{I}}$	Total Cost $\downarrow$	EX $\uparrow$	$\boldsymbol{\mathcal{I}}$	Total Cost $\downarrow$
$G_{2}\times R_{1}$	56.84	$-2.02$	0.612	57.82	$+0.07$	0.743	65.12	$-0.07$	1.971	64.60	$-0.26$	10.310
$G_{2}\times R_{2}$	55.87	$-0.20$	1.143	55.22	$-0.97$	1.457	63.62	$-0.20$	3.856	64.73	$+1.11$	24.751
$G_{2}\times R_{3}$	57.17	$-0.33$	1.525	59.84	$+0.26$	2.004	66.04	$+0.13$	4.748	65.19	$-0.39$	32.632
$G_{2}\times E_{2}$	54.50	$-0.39$	0.685	56.39	$-0.07$	0.793	64.34	$+0.20$	2.217	65.32	$+0.07$	12.190
$G_{3}\times R_{1}$	55.74	$-0.39$	0.657	57.89	$-0.07$	0.733	63.43	$+0.19$	1.956	65.38	$+0.26$	12.114

Table 3. Cross-module interactions across four backbones. EX is combined execution accuracy (%), with gray values marking combinations that do not outperform the better of the two individual paradigms. Green and red

\mathcal{I}

denote super- and sub-additive interactions. The

G_{1}\times R_{j}

rows are omitted because they reduce to

R_{j}

and have

\mathcal{I}=0

by construction. Cost is reported in USD. All interaction values are measured in percentage points relative to the matched-budget reference baseline for each backbone.

Backbone	Stack	$\mathrm{EX}$ (%) $\uparrow$	Cost $\downarrow$
GPT-4o-mini	$E_{2}{+}S_{2}{+}G_{2}{+}R_{1}$	58.02 ( $+8.28$ )	0.759
DeepSeek-V4-Flash	$E_{2}{+}S_{2}{+}G_{1}{+}R_{1}$	61.15 ( $+4.82$ )	1.584
Gemini-2.5-Flash	$E_{2}{+}S_{2}{+}G_{2}{+}R_{1}$	67.14 ( $+7.30$ )	2.916
GPT-5.4	$E_{2}{+}S_{2}{+}G_{2}{+}R_{1}$	69.43 ( $+8.15$ )	16.237

Table 4. Module-best pipelines on BIRD dev. Parentheses show

\Delta\mathrm{EX}

(pp) over the

N{=}1

baseline; API costs are in USD. Stacks follow the cost-aware rule in § 5.2 and favor paradigms with low

\mathrm{CPP}

and broadly positive gains across tested models.

Figure 6. Cost–EX tradeoffs on Spider dev. Hollow circles denote

N{=}1

baselines, filled stars denote the module-best pipelines selected on BIRD and applied unchanged to Spider. The horizontal axis reports total cost on a logarithmic scale.

	$\mathrm{EX}$ (%)			Cost
Backbone	T1	T2	T3	T1	T2	T3
Llama-3.1-8B-Instruct	37.61	40.22 $(+2.61)$	44.07 $(+6.46)$	0.05	0.06	0.09
Qwen3-32B	50.59	52.80 $(+2.21)$	55.48 $(+4.89)$	0.43	0.52	0.93
o3-mini	54.11	55.87 $(+1.76)$	59.97 $(+5.87)$	5.67	6.40	12.67
Qwen3-235B	57.82	59.06 $(+1.24)$	61.80 $(+3.98)$	4.55	5.76	11.36
Claude Sonnet 4.5	62.71	65.25 $(+2.54)$	70.34 $(+7.63)$	8.62	9.65	18.94

Table 5. Guideline validation on additional backbones.

\mathrm{EX}

(%) and total API cost in USD are measured on BIRD; parenthesized values report

\Delta\mathrm{EX}

in percentage points over Tier 1. T1–T3 denote three progressively richer configurations, ranging from the lean baseline to the full cost-aware pipeline.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Table 11

Panel of Figure 8 — Figure 8. Controlled implementation of pre-processing modules: Example Retrieval and Schema Linking.

Panel of Figure 9 — Figure 9. Controlled implementation of the generation stage: Generation Strategy.

Panel of Figure 10 — Figure 10. Controlled implementation of post-processing modules: Candidate Selection and Refinement.

Model	Input	Output
GPT-4o-mini	0.15	0.60
Gemini-2.5-Flash	0.30	2.50
DeepSeek-V4-Flash	0.14	0.28
GPT-5.4	2.50	15.00
Llama-3.1-8B-Instruct	0.02	0.04
Qwen3-32B	0.08	0.28

Appendix

Table 8. API prices in USD per million tokens, recorded on May 2, 2026. For reasoning backbones, output charges include all billed reasoning tokens used in evaluation.

Execution-Guided Text-to-SQL

See all 17 papers

LIMIT: Less Is More for Instruction Tuning in Text-to-SQL
Sep 21, 2026Haoyuan Ma, Hengwei Liu, Linjuan Wu +2Text-to-SQL Instruction-Tuning Data Selection
Zhejiang University
FINER-SQL: Boosting Small Language Models for Text-to-SQL
May 5, 2026Thanh Dat Hoang, Thanh Trung Huynh, Matthias Weidlich +4Reward Modeling Small Language Models
Griffith University (Australia) · VinUniversity (Vietnam) · Humboldt-Universitat zu Berlin (Germany) +1
ReToolSQL: Agentic Reinforcement Learning for Robust Text-to-SQL
Aug 28, 2026Pratik Kakkar, Chandra Dhir, Ravi Shankar +2Text-to-SQL Execution-Guided Text-to-SQL
JPMorganChase

Sep 21, 2026J/K move · Enter open · S save

LIMIT: Less Is More for Instruction Tuning in Text-to-SQL

Haoyuan Ma, Hengwei Liu, Linjuan Wu, Yongliang Shen+1

Zhejiang University

Text-to-SQL Instruction-Tuning Data Selection LLM Fine-Tuning

Open paper AskPDF arXiv

Large language models have achieved remarkable progress on Text-to-SQL through reasoning-enhanced fine-tuning, yet existing approaches predominantly rely on massive instruction corpora under the assumption that scale drives performance. We challenge this paradigm by investigating a fundamental question: what is the minimal data requirement for effective Text-to-SQL instruction tuning? We propose LIMIT(Less Is More for Instruction Tuning in Text-to-SQL), a data-centric framework that demonstrates strong database reasoning can emerge from an extremely compact training set when examples are strategically selected. LIMIT operates through four stages: difficulty-aware filtering that identifies samples within the model's learning frontier, chain-of-thought synthesis with consistency-based selection, multi-dimensional quality scoring via LLM-as-judge, and genetic algorithm optimization that jointly maximizes schema coverage and sample quality. On the BIRD and Spider benchmark, LIMIT selects only 796 and 863 samples while achieving 100% table coverage, enabling Qwen3-8B to reach 69.1% and 88.9% execution accuracy.This result surpasses methods trained on 20 times more data and establishes a new state-of-the-art among open-source approaches. Our findings suggest that careful data curation, rather than scale, is the key to efficient Text-to-SQL learning.

Abstract

Execution-Guided Text-to-SQL

Explore similar work

LIMIT: Less Is More for Instruction Tuning in Text-to-SQL

FINER-SQL: Boosting Small Language Models for Text-to-SQL

ReToolSQL: Agentic Reinforcement Learning for Robust Text-to-SQL