Organizations: Griffith University, Brisbane, Australia · Fudan University, Shanghai, China · Shanghai Innovation Institute, China · The University of Hong Kong, Hong Kong, China · Peking University, Beijing, China · InFly, China · Shanghai Academy of Artificial Intelligence for Science, China
Agentic Reinforcement Learning (RL) shows promise for complex tasks, but Text-to-SQL remains mostly restricted to single-turn paradigms. A primary bottleneck is the credit assignment problem. In traditional paradigms, rewards are determined solely by the final-turn feedback, which ignores the intermediate process and leads to ambiguous credit evaluation. To address this, we propose Agentic SQL, a framework featuring a universal two-tiered reward mechanism designed to provide effective trajectory-level evaluation and dense step-level signals. First, we introduce Aggregated Trajectory Reward (ATR) to resolve multi-turn credit assignment. Using an asymmetric transition matrix, ATR aggregates process-oriented scores to incentivize continuous improvement. Leveraging Lyapunov stability theory, we prove ATR acts as an energy dissipation operator, guaranteeing a cycle-free policy and monotonic convergence. Second, Column-Set Matching Reward (CSMR) provides immediate step-level rewards to mitigate sparsity. By executing queries at each turn, CSMR converts binary (0/1) feedback into dense [0, 1] signals based on partial correctness. Evaluations on BIRD show a 5% gain over binary-reward GRPO. Notably, our approach outperforms SOTA Arctic-Text2SQL-R1-7B on BIRD and Spider 2.0 using identical models, propelling Text-to-SQL toward a robust multi-turn agent paradigm.
Figures & tables
Figure 1: The framework of Agentic SQL. CSMR captures rich signals from error cases, whereas traditional binary rewards overlook this information.
Model
RL Data
Base Model
Bird (Greedy)
Spider-Dev (Greedy)
Spider2.0-Lite (Mj@8)
GPT-4o
—
—
64.4
80.4
15.6
Deepseek-V3
—
—
62.5
78.3
15.6
Qwen2.5-7B-Instruct Team (2024)
—
-
47.5
74.7
5.1
Qwen2.5-Coder-7B Team (2024)
—
-
58.2
77.4
2.2
SQL-R1 + Qwen2.5-Coder-7B Ma et al. (2025)
SynSQL-Complex-5K
Qwen2.5-Coder-7B
63.1
84.5
-
Reasoning-SQL-7B Ma et al. (2025)
BIRD
Qwen2.5-Coder-7B
64.0
78.7
-
Table 1: For Bird and Spider, we consistently use Greedy decoding, while for Spider 2.0, we use 8-time sampling followed by majority voting on the answers. *Gretel indicates Gretel-Synth-Filtered.
Figure 2: ATR reward analysis, showing the ATR on 0/1 reward (left) and the dense reward distribution by strategy type (right).
Figure 3: Turn count distribution for models using CSMR on the BIRD-Dev.
Figure 4: Agentic vs. Single-Turn training speed comparison.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Batch Size
128
Learning Rate
1e-6
Rollout Temperature
1.0
Rollout Top-p
0.95
Validation Temperature
0.6 / 1.0
Validation Top-p
0.95
Appendix
Table 2: Hyperparameters for RL Training
Figure 5: Turn count distribution for models using CSMR on the BIRD-Dev.
Model
Turn 1
Turn 2
Turn 3
Single-Turn GRPO
59.4
-
-
Agentic SQL
62.2
66.0
39.5
Agentic SQL + ATR
72.7
71.0
36.2
Appendix
Table 3: Model Performance on Turn-n Interactions.
Model
Bird-Dev
Single-Turn GRPO
α=0.6
59.2
α=0.7
60.1
α=0.8
59.4
Appendix
Table 4: Sensitivity Analysis of the Scaling Factor α in CSMR.
Recent Text-to-SQL systems increasingly rely on multi-turn interaction, execution feedback, and reinforcement learning. However, most existing methods use execution correctness only as a trajectory-level reward, which provides limited guidance for identifying the SQL decisions responsible for success or failure. We propose SERL-SQL, a selective execution-grounded reinforcement learning framework for multi-turn Text-to-SQL agents. SERL-SQL samples on-policy SQL interaction trajectories and uses a training-only teacher to re-score student actions with execution feedback. The resulting teacher--student likelihood gap is converted into bounded, masked weights that reweight GRPO advantages only on SQL and tool-action tokens. In this way, task rewards preserve the optimization direction, while execution hindsight provides localized credit assignment. Experiments on BIRD, Spider, and cross-domain benchmarks show that SERL-SQL achieves competitive performance, reaching 76.56% execution accuracy on BIRD-Dev and 89.92% on Spider-Test. Moreover, our reward-based selection strategy closely approaches the oracle Best-of-N upper bound and consistently outperforms consistency-based selection, showing that SERL-SQL produces high-quality candidates that can be reliably identified by lightweight execution-grounded rewards. Our code will be released at https://github.com/Ffunkytao/SERL-SQL.
Tao Liu, Tao Feng, Xiangheng Li +9
Zhejiang University, College of Artificial Intelligence · Zhengzhou University, School of Computer Science of Artificial Intelligence · Byte Dance
Tool-integrated Text-to-SQL parsing has emerged as a promising paradigm, framing SQL generation as a sequential decision-making process interleaved with tool execution. However, existing reinforcement learning approaches mainly rely on coarse-grained outcome supervision, resulting in a fundamental credit assignment problem: models receive the same reward for any trajectory that yields the correct answer, even when intermediate steps are redundant, inefficient, or erroneous. Consequently, models are encouraged to explore suboptimal reasoning spaces, limiting both efficiency and generalization. To address this problem, we propose FineStep, a novel framework for step-level credit assignment in tool-augmented Text-to-SQL. First, we introduce a reward design with independent process rewards to alleviate the signal sparsity of outcome supervision. Next, we present a step-level credit assignment mechanism to precisely quantify the value of each reasoning step. Finally, we develop a policy optimization method based on step-level advantages for efficient updates. Extensive experiments on BIRD benchmarks show that FineStep achieves state-of-the-art performance and reduces redundant tool interactions, with a 3.25% average EX gain over GRPO at the 4B scale.
Yaxun Dai, Baolin Sun, Junying Wang +6
Institute of Computer Science and Technology, Soochow University, China · Ant Digital Technologies, Ant Group · School of Management, University of Science and Technology of China
Recent work has shown that reinforcement learning from execution feedback can substantially improve text-to-SQL performance, often enabling smaller models to match or exceed much larger systems. However, most existing approaches treat SQL generation as a single-turn task, limiting the model's ability to recover from errors through iterative refinement. We present ReToolSQL, a two-stage training framework for text-to-SQL that combines (i) a supervised warm-start on rejection-sampled reasoning traces with (ii) agentic reinforcement fine-tuning (RFT) over multi-turn tool-use trajectories. The key insight is that the two stages act on complementary axes, the supervised fine-tuning (SFT) on verified privileged-teacher traces expands the set of solvable questions (raising pass@k coverage on the hardest cases), while RFT converts that expanded capability into higher single-pass accuracy by teaching the model when to verify, what evidence to retrieve, and how to repair faulty SQL from execution feedback. Applied to Gemma 4 instruction-tuned (31B), RFT alone achieves 73.66% execution accuracy (EX) on the BIRD-SQL development benchmark (74.12% EX with self-consistency). Initializing RFT from the SFT checkpoint (SFT→RFT) yields our strongest model at 74.32% EX single-pass and 74.77% EX with self-consistency. At the time of writing, this ranked first on the BIRD single-model development-set leaderboard. The approach uses composite rewards anchored on execution correctness, requires no human annotation beyond the benchmark itself, and operates within a single dense 31B model, showing that a properly designed SFT→RFT pipeline over tool-use trajectories is a practical path toward robust enterprise-grade text-to-SQL.