Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL

Jiayan Lin; Yujia Liu; Zijin Hong; Zheng Yuan; Yilin Xiao; Hao Chen; Qinggang Zhang; Xiao Huang

cs.CLAug 28, 2026

Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL

Authors: Jiayan Lin, Yujia Liu, Zijin Hong, Zheng Yuan, Yilin Xiao, Hao Chen, Qinggang Zhang, Xiao Huang,

Organizations: Jinan University Guangzhou, China · The Hong Kong Polytechnic University Kowloon, Hong Kong · City University of Macau Taipa, Macau · Jilin University Changchun, China ·

Abstract

Recent advances in in-context learning (ICL) text-to-SQL have substantially improved execution accuracy on public benchmarks by assembling increasingly elaborate pipelines around the base generator, yet existing studies typically report aggregate end-to-end accuracy, without quantifying the marginal accuracy-cost contribution of individual design choices. Consequently, providing a unified, paradigm-level cost-accuracy quantification remains a critical challenge for understanding and configuring modern text-to-SQL. To address this, we instantiate 17 paradigm-level configurations across five recurring modules of the ICL text-to-SQL pipeline under a single controlled implementation, and attribute each paradigm's marginal contribution and incurred cost across all four backbones spanning diverse capability levels and reasoning styles. Our analysis reveals that execution-feedback refinement is the only paradigm whose benefit holds universally at consistently low cost, while most other modules help only under backbone-dependent conditions. Token accounting shows that input demand is more closely tied to pipeline structure, whereas output demand is more sensitive to backbone generation behavior. Cross-module analysis further shows that stacking improves accuracy on most backbones, although how the gains compose varies with backbone capability. We also find that a fixed budget is often better spent engineering a more elaborate pipeline over a mid-tier backbone than upgrading to a frontier model with a lean pipeline. These findings distill into an actionable, cost-aware tiered guideline that transfers to five additional backbones without per-paradigm search.

Figures & tables

Figure 1. Overview of a modular ICL Text-to-SQL pipeline and its accuracy–cost trade-offs. Five recurring modules span pre-processing, SQL generation, and post-processing, but their benefits and inference costs vary across backbones, yielding different cost-efficient configurations under fixed budgets. A schematic comparison of rising Text-to-SQL leaderboard accuracy and increasing token and dollar costs as additional pipeline modules are added.

Figure 2. Taxonomy of ICL Text-to-SQL pipelines, organizing five modules across three pipeline stages into 17 paradigm-level configurations and 24 representative methods, including methods that naturally span multiple paradigms within the pipeline. Repeated method names indicate systems that combine design paradigms across modules or use multiple strategies within the same module, providing a unified view of how modern ICL Text-to-SQL systems assemble pipeline components in practice. A circular taxonomy diagram organizing five ICL text-to-SQL modules across preprocessing, SQL generation, and post-processing stages, with branches for 17 paradigm-level configurations and representative methods.

Paradigm	GPT-4o-mini					DeepSeek-V4-Flash					Gemini-2.5-Flash					GPT-5.4
Paradigm	EX $\boldsymbol{\uparrow}$	Cost $\boldsymbol{\downarrow}$	CPP $\boldsymbol{\downarrow}$	IN $\boldsymbol{\downarrow}$	OUT $\boldsymbol{\downarrow}$	EX $\boldsymbol{\uparrow}$	Cost $\boldsymbol{\downarrow}$	CPP $\boldsymbol{\downarrow}$	IN $\boldsymbol{\downarrow}$	OUT $\boldsymbol{\downarrow}$	EX $\boldsymbol{\uparrow}$	Cost $\boldsymbol{\downarrow}$	CPP $\boldsymbol{\downarrow}$	IN $\boldsymbol{\downarrow}$	OUT $\boldsymbol{\downarrow}$	EX $\boldsymbol{\uparrow}$	Cost $\boldsymbol{\downarrow}$	CPP $\boldsymbol{\downarrow}$	IN $\boldsymbol{\downarrow}$	OUT $\boldsymbol{\downarrow}$
Baselines
$N{=}1$	49.74	0.394	–	1510	51	56.32	0.547	–	1557	494	59.84	1.035	–	1642	73	61.28	7.305	–	1509	66
$N{=}5$	49.67	1.972	–	7550	255	56.45	2.756	–	7787	2523	60.04	5.219	–	8222	374	61.41	35.864	–	7545	301
$\mathrm{OCR}@5$	56.91	1.972	–	7550	255	60.56	2.756	–	7787	2523	64.54	5.219	–	8222	374	66.82	35.864	–	7545	301
Example Retrieval

Table 1. Module-level accuracy–cost results on BIRD dev across four LLMs. EX : execution accuracy (%); Cost : total API spend (USD); CPP : incremental USD per EX point over the matched-budget baseline (

N{=}5

for Candidate Selection,

N{=}1

otherwise; § 3.2 ). IN/OUT : mean per-question tokens (DeepSeek-V4-Flash reasoning tokens count as OUT). Significance uses McNemar’s test (

\alpha{=}0.05

; full CIs in the supplement). In CPP, numbers denote significant gains; “ ∘ ”, nonsignificant gains; “–”, no EX improvement; and “ ∗ ”, a significant gain at no higher cost. / mark the best/second-best results; OCR@5 is the oracle upper bound for

N{=}5

Figure 4.

\Delta\mathrm{EX}

(pp) relative to the matched-budget reference baseline for 17 paradigm configurations on BIRD dev. Shaded bands show 95% CIs for paired differences, and the dashed line marks

\Delta\mathrm{EX}=0

. Statistical significance is assessed using McNemar’s test. Full numerical CIs are provided in the supplementary material for every paradigm on each backbone. A line chart showing execution-accuracy changes for 17 paradigm configurations across GPT-4o-mini, DeepSeek-V4-Flash, Gemini-2.5-Flash, and GPT-5.4, with 95 percent confidence bands and a zero-change baseline.

Backbone	Paradigm	EX $\uparrow$	Cost $\downarrow$	CPP $\downarrow$	IN $\downarrow$	OUT $\downarrow$
Qwen3-235B (Thinking)	$G_{1}$	57.82	4.55	–	1555	1134
	$G_{2}$	58.02 ( $+0.20$ )	5.22	3.426	1529	1327
	$G_{3}$	57.30 ( $-0.52$ )	5.84	–	1596	1495
o3-mini	$G_{1}$	54.11	5.67	–	1509	463
	$G_{2}$	53.46 ( $-0.65$ )	5.83	–	1546	477
	$G_{3}$	53.06 ( $-1.04$ )	7.08	–	1538	665

Table 2. Generation strategy results on BIRD dev for two additional reasoning backbones. Qwen3-235B uses thinking mode. Parentheses report

\Delta\mathrm{EX}

over

G_{1}

(pp). Units: Cost in USD, CPP in USD/pp, and IN/OUT in tokens per question.

Figure 5. Per-question input (IN) and output (OUT) token consumption across the 17 paradigm configurations for two backbones, GPT-4o-mini and DeepSeek-V4-Flash, on BIRD. Dashed horizontal lines mark the corresponding baseline token levels for direct comparison across configurations.

	GPT-4o-mini			DeepSeek-V4-Flash			Gemini-2.5-Flash			GPT-5.4
Combination	EX $\uparrow$	$\boldsymbol{\mathcal{I}}$	Total Cost $\downarrow$	EX $\uparrow$	$\boldsymbol{\mathcal{I}}$	Total Cost $\downarrow$	EX $\uparrow$	$\boldsymbol{\mathcal{I}}$	Total Cost $\downarrow$	EX $\uparrow$	$\boldsymbol{\mathcal{I}}$	Total Cost $\downarrow$
$G_{2}\times R_{1}$	56.84	$-2.02$	0.612	57.82	$+0.07$	0.743	65.12	$-0.07$	1.971	64.60	$-0.26$	10.310
$G_{2}\times R_{2}$	55.87	$-0.20$	1.143	55.22	$-0.97$	1.457	63.62	$-0.20$	3.856	64.73	$+1.11$	24.751
$G_{2}\times R_{3}$	57.17	$-0.33$	1.525	59.84	$+0.26$	2.004	66.04	$+0.13$	4.748	65.19	$-0.39$	32.632
$G_{2}\times E_{2}$	54.50	$-0.39$	0.685	56.39	$-0.07$	0.793	64.34	$+0.20$	2.217	65.32	$+0.07$	12.190
$G_{3}\times R_{1}$	55.74	$-0.39$	0.657	57.89	$-0.07$	0.733	63.43	$+0.19$	1.956	65.38	$+0.26$	12.114

Table 3. Cross-module interactions across four backbones. EX is combined execution accuracy (%), with gray values marking combinations that do not outperform the better of the two individual paradigms. Green and red

\mathcal{I}

denote super- and sub-additive interactions. The

G_{1}\times R_{j}

rows are omitted because they reduce to

R_{j}

and have

\mathcal{I}=0

by construction. Cost is reported in USD. All interaction values are measured in percentage points relative to the matched-budget reference baseline for each backbone.

Backbone	Stack	$\mathrm{EX}$ (%) $\uparrow$	Cost $\downarrow$
GPT-4o-mini	$E_{2}{+}S_{2}{+}G_{2}{+}R_{1}$	58.02 ( $+8.28$ )	0.759
DeepSeek-V4-Flash	$E_{2}{+}S_{2}{+}G_{1}{+}R_{1}$	61.15 ( $+4.82$ )	1.584
Gemini-2.5-Flash	$E_{2}{+}S_{2}{+}G_{2}{+}R_{1}$	67.14 ( $+7.30$ )	2.916
GPT-5.4	$E_{2}{+}S_{2}{+}G_{2}{+}R_{1}$	69.43 ( $+8.15$ )	16.237

Table 4. Module-best pipelines on BIRD dev. Parentheses show

\Delta\mathrm{EX}

(pp) over the

N{=}1

baseline; API costs are in USD. Stacks follow the cost-aware rule in § 5.2 and favor paradigms with low

\mathrm{CPP}

and broadly positive gains across tested models.

Figure 6. Cost–EX tradeoffs on Spider dev. Hollow circles denote

N{=}1

baselines, filled stars denote the module-best pipelines selected on BIRD and applied unchanged to Spider. The horizontal axis reports total cost on a logarithmic scale.

	$\mathrm{EX}$ (%)			Cost
Backbone	T1	T2	T3	T1	T2	T3
Llama-3.1-8B-Instruct	37.61	40.22 $(+2.61)$	44.07 $(+6.46)$	0.05	0.06	0.09
Qwen3-32B	50.59	52.80 $(+2.21)$	55.48 $(+4.89)$	0.43	0.52	0.93
o3-mini	54.11	55.87 $(+1.76)$	59.97 $(+5.87)$	5.67	6.40	12.67
Qwen3-235B	57.82	59.06 $(+1.24)$	61.80 $(+3.98)$	4.55	5.76	11.36
Claude Sonnet 4.5	62.71	65.25 $(+2.54)$	70.34 $(+7.63)$	8.62	9.65	18.94

Table 5. Guideline validation on additional backbones.

\mathrm{EX}

(%) and total API cost in USD are measured on BIRD; parenthesized values report

\Delta\mathrm{EX}

in percentage points over Tier 1. T1–T3 denote three progressively richer configurations, ranging from the lean baseline to the full cost-aware pipeline.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Table 11

Panel of Figure 8 — Figure 8. Controlled implementation of pre-processing modules: Example Retrieval and Schema Linking.

Panel of Figure 9 — Figure 9. Controlled implementation of the generation stage: Generation Strategy.

Panel of Figure 10 — Figure 10. Controlled implementation of post-processing modules: Candidate Selection and Refinement.

Model	Input	Output
GPT-4o-mini	0.15	0.60
Gemini-2.5-Flash	0.30	2.50
DeepSeek-V4-Flash	0.14	0.28
GPT-5.4	2.50	15.00
Llama-3.1-8B-Instruct	0.02	0.04
Qwen3-32B	0.08	0.28

Appendix

Table 8. API prices in USD per million tokens, recorded on May 2, 2026. For reasoning backbones, output charges include all billed reasoning tokens used in evaluation.

Execution-Guided Text-to-SQL

See all 17 papers

Sep 21, 2026

LIMIT: Less Is More for Instruction Tuning in Text-to-SQL

Large language models have achieved remarkable progress on Text-to-SQL through reasoning-enhanced fine-tuning, yet existing approaches predominantly rely on massive instruction corpora under the assumption that scale drives performance. We challenge this paradigm by investigating a fundamental question: what is the minimal data requirement for effective Text-to-SQL instruction tuning? We propose LIMIT(Less Is More for Instruction Tuning in Text-to-SQL), a data-centric framework that demonstrates strong database reasoning can emerge from an extremely compact training set when examples are strategically selected. LIMIT operates through four stages: difficulty-aware filtering that identifies samples within the model's learning frontier, chain-of-thought synthesis with consistency-based selection, multi-dimensional quality scoring via LLM-as-judge, and genetic algorithm optimization that jointly maximizes schema coverage and sample quality. On the BIRD and Spider benchmark, LIMIT selects only 796 and 863 samples while achieving 100% table coverage, enabling Qwen3-8B to reach 69.1% and 88.9% execution accuracy.This result surpasses methods trained on 20 times more data and establishes a new state-of-the-art among open-source approaches. Our findings suggest that careful data curation, rather than scale, is the key to efficient Text-to-SQL learning.

Haoyuan Ma, Hengwei Liu, Linjuan Wu +2

Zhejiang University

Text-to-SQL Instruction-Tuning Data Selection LLM Fine-Tuning

Ask PDF

May 5, 2026

FINER-SQL: Boosting Small Language Models for Text-to-SQL

Large language models have driven major advances in Text-to-SQL generation. However, they suffer from high computational cost, long latency, and data privacy concerns, which make them impractical for many real-world applications. A natural alternative is to use small language models (SLMs), which enable efficient and private on-premise deployment. Yet, SLMs often struggle with weak reasoning and poor instruction following. Conventional reinforcement learning methods based on sparse binary rewards (0/1) provide little learning signal when the generated SQLs are incorrect, leading to unstable or collapsed training. To overcome these issues, we propose FINER-SQL, a scalable and reusable reinforcement learning framework that enhances SLMs through fine-grained execution feedback. Built on group relative policy optimization, FINER-SQL replaces sparse supervision with dense and interpretable rewards that offer continuous feedback even for incorrect SQLs. It introduces two key reward functions: a memory reward, which aligns reasoning with verified traces for semantic stability, and an atomic reward, which measures operation-level overlap to grant partial credit for structurally correct but incomplete SQLs. This approach transforms discrete correctness into continuous learning, enabling stable, critic-free optimization. Experiments on the BIRD and Spider benchmarks show that FINER-SQL achieves up to 67.73% and 85% execution accuracy with a 3B model -- matching much larger LLMs while reducing inference latency to 5.57~s/sample. These results highlight a cost-efficient and privacy-preserving path toward high-performance Text-to-SQL generation. Our code is available at https://github.com/thanhdath/finer-sql.

Thanh Dat Hoang, Thanh Trung Huynh, Matthias Weidlich +4

Griffith University (Australia) · VinUniversity (Vietnam) · Humboldt-Universitat zu Berlin (Germany) +1

Reward Modeling Small Language Models Text-to-SQL

Ask PDF

Aug 28, 2026

ReToolSQL: Agentic Reinforcement Learning for Robust Text-to-SQL

Recent work has shown that reinforcement learning from execution feedback can substantially improve text-to-SQL performance, often enabling smaller models to match or exceed much larger systems. However, most existing approaches treat SQL generation as a single-turn task, limiting the model's ability to recover from errors through iterative refinement. We present ReToolSQL, a two-stage training framework for text-to-SQL that combines (i) a supervised warm-start on rejection-sampled reasoning traces with (ii) agentic reinforcement fine-tuning (RFT) over multi-turn tool-use trajectories. The key insight is that the two stages act on complementary axes, the supervised fine-tuning (SFT) on verified privileged-teacher traces expands the set of solvable questions (raising pass@k coverage on the hardest cases), while RFT converts that expanded capability into higher single-pass accuracy by teaching the model when to verify, what evidence to retrieve, and how to repair faulty SQL from execution feedback. Applied to Gemma 4 instruction-tuned (31B), RFT alone achieves 73.66% execution accuracy (EX) on the BIRD-SQL development benchmark (74.12% EX with self-consistency). Initializing RFT from the SFT checkpoint (SFT

\to

RFT) yields our strongest model at 74.32% EX single-pass and 74.77% EX with self-consistency. At the time of writing, this ranked first on the BIRD single-model development-set leaderboard. The approach uses composite rewards anchored on execution correctness, requires no human annotation beyond the benchmark itself, and operates within a single dense 31B model, showing that a properly designed SFT

\to

RFT pipeline over tool-use trajectories is a practical path toward robust enterprise-grade text-to-SQL.

Pratik Kakkar, Chandra Dhir, Ravi Shankar +2

JPMorganChase

Text-to-SQL Execution-Guided Text-to-SQL RL Fine-Tuning RL for Tool Use

Ask PDF

Abstract

Execution-Guided Text-to-SQL

Explore similar work

LIMIT: Less Is More for Instruction Tuning in Text-to-SQL

FINER-SQL: Boosting Small Language Models for Text-to-SQL

ReToolSQL: Agentic Reinforcement Learning for Robust Text-to-SQL