BIRD-RL: Scaling Agentic Reinforcement Learning over Stateful Data-Centric Environments

Ge Qu; Jinyang Li; Xiaohan Xu; Xiaolong Li; Bowen Qin; Nan Huo; Ziwei Tang; Chenhao Ma

cs.LGOct 21, 2026

BIRD-RL: Scaling Agentic Reinforcement Learning over Stateful Data-Centric Environments

Authors: Ge Qu, Jinyang Li, Xiaohan Xu, Xiaolong Li, Bowen Qin, Nan Huo, Ziwei Tang, Chenhao Ma,

Organizations: The University of Hong Kong · National University of Singapore · The Chinese University of Hong Kong, Shenzhen

Abstract

While agentic reinforcement learning (RL) has emerged as a promising paradigm for training small language models (SLMs), existing frameworks are predominantly tool-centric and struggle with data-centric environments. Specifically, applying tool-centric frameworks to stateful tasks like SQL debugging introduces severe spatial inefficiencies since proper environment isolation typically requires a separate database container per rollout. Naively reusing databases without tracking evolving states across multi-turn interactions causes trajectory contamination and cross-trajectory interference, which severely destabilizes policy updates. To overcome these bottlenecks, we introduce BIRD-RL, a novel four-stage agentic RL curriculum that alternately enhances foundational agent reasoning and trajectory-adaptive on-policy exploration to circumvent the instability inherently associated with direct end-to-end learning. Crucially, to support this curriculum, BIRD-RL is driven by a Trajectory-scoped Persistent Execution Infrastructure, which enables safe database replica reuse while preserving trajectory isolation. This design improves spatial RL efficiency by 12.8x, successfully enabling scalable agentic RL for complex data-centric tasks without needs of Kubernettes. Empirical evaluations on the BIRD-CRITIC-SQLite benchmark demonstrate that our 7B and 14B models achieve Success Rates of 44.40% and 48.00%, respectively, showing competitive performance against strong proprietary model-based agents such as Claude-Opus-4.6 Agent (48.20%). Furthermore, BIRD-RL successfully trains unified SLMs capable of both advanced SQL debugging and generation, highlighting its broader potential for multi-objective optimization across heterogeneous, real-world data-centric tasks.

Figures & tables

Figure 1: Overview of the Bird-RL framework. (Left) The four-stage agentic RL curriculum transitioning from static reasoning imitation to interactive stateful exploration. (Right) The Trajectory-Scoped Persistent Execution Infrastructure, enabling scalable and isolated database replica reuse via a customized agent loop and worker-partitioned pool.

Type	Model	Quer.	Mana.	Pers.	Overall
General-Purpose Models	GPT-5.4-Pro	44.01	38.67	39.72	42.00
	Claude-Opus-4.6	50.35	44.00	39.01	46.20
	Claude-Sonnet-4.5	46.83	36.00	34.75	41.80
	Gemini-3.1-Pro	53.52	44.00	41.84	48.80
	GLM-4.7	48.24	40.00	33.33	42.80
	Kimi-K2.5	46.48	37.33	35.46	42.00

Table 1: Success Rate (SR %) of different models on BIRD-CRITIC-SQLite , evaluated under greedy decoding and grouped by issue category. Bold numbers indicate the highest score in each column , and underlined numbers indicate the second highest. "Quer." refers to query-like issues, "Mana." refers to data-management issues, and "Pers." refers to personalized-function issues.

Model	Base	SQL-Act	Bird-Fixer	Bird-Talon	$\Delta$ (%)
Qwen-2.5-Coder-7B	26.20	29.80	33.60	44.40	+18.20
Qwen-2.5-Coder-14B	33.80	36.40	40.00	48.00	+14.20

Table 2: Detailed comparison of Bird-Talon with other strong baselines on BIRD-CRITIC-SQLite . The SQL-ACT agent employs SQLs as actions.

\Delta

denotes the absolute improvement of Bird-Talon compared with the base model.

Strategy	Training-time I/O	Emp. I/O ( $\downarrow$ )	Cost Ratio
Per-rollout Container	$O(NR\bar{S})$	2.94 TiB	$12.8\times$
Dynamic Reusable Pool	$O((B+q/k)R\bar{S})$	236 GiB	$1.0\times$

Table 3: Comparison of database lifecycle I/O during interactive rollout execution under trajectory-level isolation. “Emp. I/O” reports the measured online database lifecycle I/O from our implementation and excludes the one-time offline pool materialization cost.

Model	Gen.	Debug.
Think2SQL-7B	56.1	24.0
SQL-R1-7B	58.9	26.6
ReEX-SQL-7B	63.4	-
Reasoning-SQL-7B	64.0	-
Arctic-T2S-R1-7B	68.9	29.8
SQL-Trail-7B	60.1	-

Table 4: Comparison of Bird-Zeno with strong RL-based generation-specialized methods. "Gen." = "Generation" and "Debug." = "Debugging".

Setting	Reasoning Trace	Action Trace
w/o Stage 3	“… the problematic query joins budget and expense … To get the correct total budget per event, I need to aggregate budget amounts at the event level using MAX or MIN, since each event has one budget …”	SELECT ... SUM(expense.cost), MAX(budget.amount) AS total_budget ... GROUP BY budget.link_to ...
w/ Stage 3 (Ours)	“… there are multiple budget lines …, and each budget line has multiple expense records … The solution is to pre-aggregate expenses per budget line first , then join with budget …”	SELECT SUM(sub_exp.total_cost), SUM(budget.amount) ... FROM ( SELECT link_to_budget, SUM(cost) ... GROUP BY link_to_budget ) sub_exp ...

Table 5: Case study illustrating the effectiveness of Stage 3, with the full trajectory shown in Figure 4 .

Type	Method	Quer.	Mana.	Pers.	Overall
Type	BIRD-Talon-7B	46.13	45.33	40.43	44.40
Training Strategy	(a) Up to Reasoning RL (Stages 1-3)	45.77	36.00	33.33	40.60 ( $\downarrow$ 3.80)
	(b) Up to Tool-Use Imitation (Stages 1-2)	45.07	33.33	30.50	39.20 ( $\downarrow$ 5.20)
	(c) Reasoning Imitation Only (Stage 1)	41.55	28.00	26.24	35.20 ( $\downarrow$ 9.20)
Reward Design	(d) w/ Partial Correctness	45.77	41.33	37.59	42.80 ( $\downarrow$ 1.60)
	(e) w/ LLM-as-a-Judge	45.77	38.67	34.04	41.40 ( $\downarrow$ 3.00)

Table 6: Ablation study on the training curriculum and reward design of Bird-RL .

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Statistic	SIX-GYM-SQLite
Total Issues	5,000
# of query-like issues	3,094
# of management issues	1,087
# of personalization issues	819
user query length (mean/max)	162.55/917
issue SQL length (mean/max)	85.50/898

Appendix

Table 7: Data characteristics of SIX-GYM-SQLite. Length statistics are reported as mean/max token counts using the cl100k_base tokenizer.

Category	Count	User Query	Issue SQL	Solution SQL	Test Cases
Query	3,094	183.93/917	86.76/751	96.22/1,976	143.74/1,601
Management	1,087	120.20/535	73.51/802	83.84/802	446.16/1,085
Personalization	819	138.01/421	96.63/898	121.40/1,041	216.33/1,418
Overall	5,000	162.55/917	85.50/898	97.66/1,976	221.38/1,601

Appendix

Table 8: SIX-GYM-SQLite statistics grouped by task category. Length statistics are reported as mean/max token counts.

Panel of Figure 3 — Figure 3: Statistics of SIX-GYM-SQLite training data, covering SQL feature usage and database-domain distribution.

Model	GPU Hours
BIRD-Talon-7B	93.38
BIRD-Zeno-7B	80.36
BIRD-Talon-14B	152.75

Appendix

Table 9: GPU hours spent to train each backbone model.

Figure 4: Case study comparing debugging trajectories with and without Stage 3 adaptive reasoning reinforcement.

Figure 5: The success rate of Bird-Talon on unseen dialects shows its generalization ability.

Model	pass@1	pass@4	pass@8	pass@16
BIRD-Talon-14B	46.40	57.40	60.60	65.00

Appendix

Table 10: Test-time scaling results of Bird-Talon -14B on BIRD-CRITIC-SQLite .

Figure 6: The inference prompt used for SQL debugging task (system prompt).

Figure 7: The inference prompt used for SQL debugging task (user prompt).

Figure 8: The inference prompt used for SQL generation task (system prompt).

Figure 9: The inference prompt used for SQL generation task (user prompt).

Figure 10: The baseline prompt used for SQL debugging tasks.

Figure 11: The prompt used in Stage 1 reasoning imitation data construction to elicit reasoning thought from teacher model.

Figure 12: The prompt used in Stage 1 reasoning imitation data construction to verify whether teacher model could leverage the reasoning thought to regenerate the correct fixed SQL.

Figure 13: The prompt used to get LLM-as-judge reward (1/2).

Figure 14: The prompt used to get LLM-as-judge reward (2/2).

Figure 15: The prompt for the heuristic rubric reward: instructions used to extract a bug checklist from the task context.

Figure 16: The prompt for the heuristic rubric reward: instructions used to verify whether the reasoning of agent correctly identifies the checklist bugs.

Figure 17: The example of trajectory from Bird-Talon solving SQL debugging task.

Agentic RL

Agentic reinforcement learning trains AI agents to plan, use tools, and adapt through feedback from multi-step interactions.

See all 226 papers

SQL-ASTRA: Alleviating Sparse Feedback in Agentic SQL via Column-Set Matching and Trajectory Aggregation
Mar 17, 2026Long Li, Zhijian Zhou, Jiangxuan Long +5Text-to-SQL Step-Level Credit Assignment in RL
Griffith University, Brisbane, Australia · Fudan University, Shanghai, China · Shanghai Innovation Institute, China +4
HarnessSQL: Harness-Native Training for SQL Agents in Realistic Database Environments
Oct 8, 2026Haolin Yang, Jipeng Zhang, Zhiqiang Qian +4Text-to-SQL LLM Agent Training
Hong Kong University of Science and Technology · Microsoft Research · Tsinghua University +1
DualSQL: Text-to-SQL with Multi-Agent Reinforcement Learning
Sep 16, 2026Shijie Chen, Yu Gan, Yeounoh Chung +6Multi-Agent LLM Systems Text-to-SQL
The Ohio State University · Google LLC

Mar 17, 2026J/K move · Enter open · S save

SQL-ASTRA: Alleviating Sparse Feedback in Agentic SQL via Column-Set Matching and Trajectory Aggregation

Long Li, Zhijian Zhou, Jiangxuan Long, Peiyang Liu+4

Griffith University, Brisbane, Australia · Fudan University, Shanghai, China · Shanghai Innovation Institute, China+4

Text-to-SQL Step-Level Credit Assignment in RL RL for LLM Agents Sparse-Reward RL Agentic RL Dense Reward RL

Open paper AskPDF arXiv

Agentic Reinforcement Learning (RL) shows promise for complex tasks, but Text-to-SQL remains mostly restricted to single-turn paradigms. A primary bottleneck is the credit assignment problem. In traditional paradigms, rewards are determined solely by the final-turn feedback, which ignores the intermediate process and leads to ambiguous credit evaluation. To address this, we propose Agentic SQL, a framework featuring a universal two-tiered reward mechanism designed to provide effective trajectory-level evaluation and dense step-level signals. First, we introduce Aggregated Trajectory Reward (ATR) to resolve multi-turn credit assignment. Using an asymmetric transition matrix, ATR aggregates process-oriented scores to incentivize continuous improvement. Leveraging Lyapunov stability theory, we prove ATR acts as an energy dissipation operator, guaranteeing a cycle-free policy and monotonic convergence. Second, Column-Set Matching Reward (CSMR) provides immediate step-level rewards to mitigate sparsity. By executing queries at each turn, CSMR converts binary (0/1) feedback into dense [0, 1] signals based on partial correctness. Evaluations on BIRD show a 5% gain over binary-reward GRPO. Notably, our approach outperforms SOTA Arctic-Text2SQL-R1-7B on BIRD and Spider 2.0 using identical models, propelling Text-to-SQL toward a robust multi-turn agent paradigm.

Abstract

Agentic RL

Explore similar work

SQL-ASTRA: Alleviating Sparse Feedback in Agentic SQL via Column-Set Matching and Trajectory Aggregation

HarnessSQL: Harness-Native Training for SQL Agents in Realistic Database Environments

DualSQL: Text-to-SQL with Multi-Agent Reinforcement Learning