ReToolSQL: Agentic Reinforcement Learning for Robust Text-to-SQL

Pratik Kakkar; Chandra Dhir; Ravi Shankar; Pareekshit Reddy Gaddam; Anup Shirgaonkar

cs.AIAug 28, 2026

ReToolSQL: Agentic Reinforcement Learning for Robust Text-to-SQL

Authors: Pratik Kakkar, Chandra Dhir, Ravi Shankar, Pareekshit Reddy Gaddam, Anup Shirgaonkar

Organizations: JPMorganChase

Abstract

Recent work has shown that reinforcement learning from execution feedback can substantially improve text-to-SQL performance, often enabling smaller models to match or exceed much larger systems. However, most existing approaches treat SQL generation as a single-turn task, limiting the model's ability to recover from errors through iterative refinement. We present ReToolSQL, a two-stage training framework for text-to-SQL that combines (i) a supervised warm-start on rejection-sampled reasoning traces with (ii) agentic reinforcement fine-tuning (RFT) over multi-turn tool-use trajectories. The key insight is that the two stages act on complementary axes, the supervised fine-tuning (SFT) on verified privileged-teacher traces expands the set of solvable questions (raising pass@k coverage on the hardest cases), while RFT converts that expanded capability into higher single-pass accuracy by teaching the model when to verify, what evidence to retrieve, and how to repair faulty SQL from execution feedback. Applied to Gemma 4 instruction-tuned (31B), RFT alone achieves 73.66% execution accuracy (EX) on the BIRD-SQL development benchmark (74.12% EX with self-consistency). Initializing RFT from the SFT checkpoint (SFT $\to$ RFT) yields our strongest model at 74.32% EX single-pass and 74.77% EX with self-consistency. At the time of writing, this ranked first on the BIRD single-model development-set leaderboard. The approach uses composite rewards anchored on execution correctness, requires no human annotation beyond the benchmark itself, and operates within a single dense 31B model, showing that a properly designed SFT $\to$ RFT pipeline over tool-use trajectories is a practical path toward robust enterprise-grade text-to-SQL.

Figures & tables

Figure 1: High-level overview of ReToolSQL . A natural-language question and enriched schema are processed by the Gemma 4 31B policy, which interacts with database tools across multiple turns to produce verified SQL. In the training loop, $B_{1},B_{2},\ldots$ denote the prompt groups sampled per training step (each containing $G{=}16$ multi-turn trajectories for a single question), and $R_{1},R_{2},\ldots$ the trajectory-level composite rewards $R=\sum_{k}w_{k}R_{k}$ (Section 3.8 ) that yield the group-relative advantages driving the GRPO update. Groups with zero reward variance ( $\text{std}=0$ ) are discarded by dynamic sampling (Section 3.5 ), as they contribute no gradient signal.

Panel of Figure 2 — Figure 2: M-Schema encoding example (abbreviated). Each column carries its type, semantics, nullability, summary statistics, and representative values, enabling evidence-based schema linking.

Component	Type	Signal	$w_{k}$	Weighted range ( $w_{k}R_{k}$ )
$R_{\text{format}}$	binary	Correct output template compliance	0.2	$\{0,0.2\}$
$R_{\text{syntax}}$	binary	Predicted SQL parses and executes	0.5	$\{0,0.5\}$
$R_{\text{exec}}$	binary	BIRD EX: row-set equality with gold	2.0	$\{0,2.0\}$
$R_{\text{tables}}$	binary	Predicted table set $\supseteq$ gold table set	0.5	$\{0,0.5\}$
$R_{\text{columns}}$	continuous	Jaccard(pred columns, gold columns)	0.5	$[0,0.5]$
$R_{\text{nonempty}}$	binary	Executes and returns $\geq 1$ non-null cell	0.1	$\{0,0.1\}$

Table 1: Composite reward components and their weights

w_{k}

. The execution-result term

R_{\text{exec}}

dominates, while linking terms supply dense schema-grounding signal. The final column lists the already-weighted contribution

w_{k}R_{k}

of each term (not the raw signal

R_{k}

); summing the positive upper bounds gives a maximum attainable reward of

3.8

Configuration	SC	$\Delta$ vs. Base	EX (%)	pass@16 (%)
Gemma 4 31B Instruct (base)	✗	—	71.19	76.9
+ SFT	✗	+1.50	72.69	81.29
+ RFT (from base)	✗	+2.47	73.66	77.2
+ RFT + SC	✓	+2.93	74.12	—
+ SFT $\to$ RFT	✗	+3.13	74.32	81.94
+ SFT $\to$ RFT + SC	✓	+3.58	74.77	—

Table 2: Training-stage ablation on BIRD-SQL dev (1,534 questions) under a common evaluation protocol (parallel execution, 30s timeout, set-based comparison). All rows use the tool suite. SFT = supervised fine-tuning on rejection-sampled privileged-teacher and self-traces (Section 3.10 ); RFT = agentic reinforcement fine-tuning via GRPO from the base model (Section 3.4 ); SC = execution-based self-consistency (majority vote over 16 samples).

\Delta

is temp-0 (greedy) EX relative to the base model.

Rank	System	Params	SC	EX (%)
1	ReToolSQL (SFT $\to$ RFT) + SC ‡	31B	Many	74.77
2	ReToolSQL (SFT $\to$ RFT)	31B	—	74.32
3	Gemini-SQL2	UNK	Many	74.12
3	ReToolSQL (RFT) + SC ‡	31B	Many	74.12
4	SiriusAI-Text2SQL (Tencent)	27B	Many	73.70
5	ReToolSQL (RFT)	31B	—	73.66

Table 3: BIRD-SQL dev leaderboard (single-model track, 1,534 questions, Aug 26 2026). SC indicates self-consistency. Shaded rows show our experiments and contributions. ‡ pass@16 majority vote.

	Inference-Time Tools
Training	No Tools	+ Tools	Tool $\Delta$
No RL (Gemma 4 31B Instruct)	69.69%	71.71%	+2.02
Agentic RL ( ReToolSQL )	70.14%	73.66%	+3.52
RL $\Delta$	+0.45	+1.95
+ Self-Consistency (SC@16)	—	74.12%	+4.43 vs. base

Table 4: Component isolation: RL training

\times

tool access on BIRD-SQL dev (1,534 questions). The interaction effect (+1.5% beyond additive components) confirms that agentic RL specifically teaches the model to leverage tools effectively.

Model	Base EX	RL EX	$\Delta$ RL	SC EX
Gemma 4 E4B	65.12%	68.45%	+3.33%	70.73%
Gemma 4 (31B)	71.71%	73.66%	+1.95%	74.12%

Table 5: Scale comparison: Gemma 4 E4B Instruct vs. Gemma 4 31B Instruct. SC = self-consistency (pass@16).

Configuration	Simple	Moderate	Challenging
Gemma 4 31B Instruct (no tools)	69.30%	67.72%	74.89%
ReToolSQL (no tools)	70.23%	68.85%	72.29%
ReToolSQL (tools)	73.60%	72.91%	75.32%
ReToolSQL + SC@16	73.95%	73.81%	75.32%

Table 6: Accuracy by difficulty level for key configurations. All rows use the same difficulty annotation (860/443/231 split).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: A representative ReToolSQL trajectory. The policy drafts SQL, detects an empty result via execution, retrieves the correct literal with BM25 search, profiles the denominator column to discover nulls, and emits a guarded final query.

Figure 4: Entropy collapse in the Gemma 4 E4B no-KL run (

\beta=0

): execution accuracy, self-consistency (SC@16), and policy entropy versus training checkpoint (shared scale). Entropy collapses within

{\sim}40

steps, after which accuracy plateaus while pass@

k

headroom shrinks.

Figure 5: Gemma 4 E4B trained with the decaying beta schedule (

\beta=0.005\to 0.001\to 0

at steps 0/40/80): execution accuracy, self-consistency (SC@16), and policy entropy versus training checkpoint (shared scale). Retaining a small KL penalty early preserves entropy, avoiding the collapse of Figure 4 and sustaining learning.

Text-to-SQL

Text-to-SQL translates natural-language questions into executable database queries, linking user intent to schemas, values, and SQL logic.

See all 169 papers

Progress-SQL: Improving Reinforcement Learning for Text-to-SQL via Progressive Rewards
Jun 5, 2026Shihao Zhang, Xiaoman Wang, Yuan Liu +2RL for Language Models Reinforcement Learning
East China Normal University
FINER-SQL: Boosting Small Language Models for Text-to-SQL
May 5, 2026Thanh Dat Hoang, Thanh Trung Huynh, Matthias Weidlich +4Reward Modeling Small Language Models
Griffith University (Australia) · VinUniversity (Vietnam) · Humboldt-Universitat zu Berlin (Germany) +1
SERL-SQL: Selective Hindsight Distillation for Text-to-SQL Reinforcement Agentic Learning
Aug 1, 2026Tao Liu, Tao Feng, Xiangheng Li +9Reinforcement Learning Text-to-SQL
Zhejiang University, College of Artificial Intelligence · Zhengzhou University, School of Computer Science of Artificial Intelligence · Byte Dance

Jun 5, 2026J/K move · Enter open · S save

Progress-SQL: Improving Reinforcement Learning for Text-to-SQL via Progressive Rewards

Shihao Zhang, Xiaoman Wang, Yuan Liu, Yunshi Lan+1

East China Normal University

RL for Language Models Reinforcement Learning Reward Shaping Text-to-SQL Multi-Turn RL

Open paper AskPDF arXiv

Reinforcement learning has recently shown promise in improving large language models for Text-to-SQL generation, yet existing methods typically optimize one-shot rewards defined over a single SQL state. Such rewards provide limited guidance for iterative SQL correction and are insufficient to capture the improvement of multi-turn SQL refinement. In this paper, we propose Progress-SQL, a multi-turn reinforcement learning framework with progressive rewards for Text-to-SQL. Our approach introduces an Oracle-guided Diagnostic Tree (ODT), which abstracts SQL queries into clause-level structural profiles and produces diagnostic feedback for next-turn refinement. To provide dense and robust reward signals, we combine ODT-based structural alignment with lexical alignment and define a progressive reward that measures the improvement from the initial SQL to the final SQL. We further incorporate a progression latency reward that favors earlier correctness and an execution status reward that encourages recovery from the invalid SQL. Experiments on BIRD, Spider, and Spider robustness variants demonstrate that our method consistently improves Text-to-SQL performance across both primary and robustness evaluations.

Abstract

Text-to-SQL

Explore similar work

Progress-SQL: Improving Reinforcement Learning for Text-to-SQL via Progressive Rewards

FINER-SQL: Boosting Small Language Models for Text-to-SQL

SERL-SQL: Selective Hindsight Distillation for Text-to-SQL Reinforcement Agentic Learning