Rollout-Time Intervention: Unlocking Agentic Reinforcement Learning Beyond Policy Limits

Xi Chen; Ruixue Ding; Zhigaoyuan Wang; Fanrui Zhang; Qiang Zhang; Boli Chen; Shihang Wang; Tianfu Wang

cs.AIOct 9, 2026

Rollout-Time Intervention: Unlocking Agentic Reinforcement Learning Beyond Policy Limits

Authors: Xi Chen, Ruixue Ding, Zhigaoyuan Wang, Fanrui Zhang, Qiang Zhang, Boli Chen, Shihang Wang, Tianfu Wang,

Organizations: Alibaba Token Hub, Alibaba Group · HKUST (Guangzhou) · Amap, Alibaba Group

Abstract

Reinforcement learning (RL) has emerged as a key paradigm for improving large language model (LLM) agents, enabling stronger reasoning, planning, and tool-use in interactive environments. However, existing agent RL methods remain fundamentally constrained by an on-policy reasoning bottleneck: when tasks exceed the model's current capability boundary, effective trajectories become extremely sparse and training often collapses. We introduce Rollout-Time Intervention (RTI), a technique designed to elicit successful trajectories beyond the policy's intrinsic exploration capacity. RTI operates by leveraging reward-side evidence (i.e., evaluation rubrics) during rollout to convert erroneous trajectories into successful ones, providing effective learning signals for tasks that the policy rarely solves on its own. Although intervention trajectories are inherently off-policy, we find that they can effectively guide policy optimization when trajectories with excessive distributional shifts are filtered out. Extensive experiments across five benchmarks demonstrate that RTI consistently outperforms RL baselines, enabling the post-trained policy to approach the performance of substantially stronger contemporary LLMs and, more importantly, transcend its original capability boundary. The code is available at https://github.com/Alibaba-NLP/qqr.

Figures & tables

Figure 1: RTI enables successful trajectories where on-policy and teacher-guided RL fail.

Figure 2: Overview of the proposed RTI framework.

	Navigation			$\tau^{2}$ -Telecom			$\tau^{2}$ -Retail			$\tau^{2}$ -Airline			Claw-Eval-Hard20
Method	p@1	avg p@ $k$	avg p k	p@1	avg p@ $k$	avg p k	p@1	avg p@ $k$	avg p k	p@1	avg p@ $k$	avg p k	p@1	avg p@ $k$	avg p k
Qwen3-8B	22.67	28.50	14.17	36.84	57.02	21.49	45.61	71.93	35.53	31.00	50.50	16.00	18.75	24.38	15.94
Qwen3-235B-A22B	28.00	35.50	16.67	44.74	64.04	28.07	64.04	79.61	47.15	50.50	64.50	32.00	20.00	20.00	12.50
GPT-4.1	23.33	31.33	18.67	39.47	58.99	23.68	71.93	83.55	52.19	53.00	66.50	34.00	30.00	30.00	18.75
Claude-4-Sonnet	26.00	32.83	20.67	49.12	67.76	32.02	74.56	85.09	53.95	54.00	68.50	35.50	40.00	46.25	30.00
GRPO	35.33	40.00	29.00	36.84	55.48	21.71	67.54	80.92	50.22	46.50	57.50	26.50	22.50	24.17	20.83

Table 1: Main results on five benchmarks with the Qwen3-8B backbone.

	Navigation			$\tau^{2}$ -Telecom			$\tau^{2}$ -Retail			$\tau^{2}$ -Airline			Claw-Eval-Hard20
Method	p@1	avg p@ $k$	avg p k	p@1	avg p@ $k$	avg p k	p@1	avg p@ $k$	avg p k	p@1	avg p@ $k$	avg p k	p@1	avg p@ $k$	avg p k
Qwen3.5-9B	39.33	53.00	26.17	92.11	96.49	89.69	82.46	93.64	71.05	60.50	73.00	47.00	26.25	31.67	19.90
Qwen3.5-397B-A17B	44.00	53.67	38.00	95.61	98.25	93.64	84.21	94.74	73.90	63.00	77.00	51.00	30.00	40.00	26.25
GPT-5.1	42.00	54.00	30.17	95.61	98.03	93.64	77.19	92.11	62.28	66.50	79.00	53.00	40.00	47.50	40.00
Claude-4.6-Sonnet	58.00	64.17	53.83	97.37	98.90	96.05	91.23	96.93	85.53	80.00	88.50	71.50	51.25	59.27	44.17
GRPO	71.33	74.83	66.33	95.61	97.59	93.64	84.21	94.74	73.90	75.00	82.50	63.00	52.50	57.71	45.42

Table 2: Main results on five benchmarks with the Qwen3.5-9B backbone.

	Navigation								$\tau^{2}$ -Telecom
Method	pass@1	pass@2	pass@3	pass@4	pass 1	pass 2	pass 3	pass 4	pass@1	pass@2	pass@3	pass@4	pass 1	pass 2	pass 3	pass 4
GRPO	35.33	40.00	41.33	43.33	35.33	29.33	26.67	24.67	36.84	52.63	63.16	69.30	36.84	21.93	17.54	10.53
RTI w/o Post-Check	26.00	33.33	36.00	38.00	26.00	14.00	9.33	7.33	38.60	53.51	59.65	67.54	38.60	24.56	16.67	14.91
RTI w/o PPL Gating	50.00	60.00	64.00	65.33	50.00	37.33	29.33	24.00	54.39	68.42	75.44	79.82	54.39	27.19	21.05	18.42
RTI w/o Reweight	55.33	58.67	60.67	62.00	55.33	45.33	42.00	33.33	60.53	79.82	85.96	88.60	60.53	38.60	30.70	20.18
RTI	57.33	61.33	64.67	66.00	57.33	48.00	42.00	39.33	61.40	83.33	87.72	92.11	61.40	46.49	37.72	31.58

Table 3: Ablation study on Navigation and

\tau^{2}

-Telecom with the Qwen3-8B backbone.

Panel of Figure 5 — Figure 5: Further analysis of intervention PPL dynamics and resulting reasoning-pattern distributions.

Figure 6: Training curves on Claw-Eval-Hard20 with the Qwen3.5-9B backbone.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Statistic	$\tau^{2}$ -Telecom	$\tau^{2}$ -Retail	$\tau^{2}$ -Airline	Navigation	Claw-Eval-Hard20
Train	1148	563	271	600	160
Test	114	114	50	150	20

Appendix

Table 4: Statistics of the five datasets.

Task category	# Tasks	Pass@3 $=0/3$	Pass@3 $=1/3$
Office automation	12	12	0
Research and visual question answering	3	1	2
Document understanding and aggregation	5	4	1
Total	20	17	3

Appendix

Table 5: Composition and Qwen3.5-9B difficulty of Claw-Eval-Hard20.

Figure 7: Token-level heatmap of $\tilde{\rho}_{\mathrm{cal}}^{t}$ on an intervened trajectory

Statistic	Count	Ratio
Rollout groups with intervention triggered	150 / 1,000	15.0%
Rollout steps flagged as invalid by the Validator	8,739 / 15,440	56.60%
Corrections accepted by the Verifier	5,166 / 8,739	59.11%
Corrections rejected by the Verifier	3,573 / 8,739	40.89%
Intervened trajectories with $\geq 1$ accepted correction	1,169 / 1,200	97.42%
Avg. tool calls per trajectory (corrected vs. original)		0.31 vs. 0.55

Appendix

Table 6: Intermediate statistics of the MAS intervention pipeline on

\tau^{2}

-Telecom (Qwen3-8B backbone).

	Overall			Hard20-ID			Rest-OOD
Method	p@1	avg p@ $k$	avg p k	p@1	avg p@ $k$	avg p k	p@1	avg p@ $k$	avg p k
Claude-4.6-Sonnet	60.10	67.99	45.78	51.25	59.27	44.17	61.07	68.71	45.89
Base	51.51	61.79	41.23	26.25	31.67	19.90	54.34	65.13	43.62
GRPO	56.06	65.63	45.83	52.50	57.71	45.42	56.56	66.53	45.87
RTI	65.20	72.95	56.47	85.00	94.17	73.96	62.92	70.73	54.48

Appendix

Table 7: In-distribution (Hard20-ID) and out-of-distribution (Rest-OOD) generalization on the full unimodal (text-only) Claw-Eval suite (199 tasks) with the Qwen3.5-9B backbone after training on the 160 synthetic instances. Rest-OOD consists of the 179 tasks whose families never contribute to the training-data synthesis. All numbers are percentages.

RL for LLM Agents

Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.

See all 612 papers

Optimizing What Policies Learn From: Recoverability-aware Rollout Intervention Learning
Aug 5, 2026Zheyuan Zhang, Manqing Mao, Hong Wang +8RL for Language Models RL Post-Training
University of Notre Dame · Amazon, Inc
Learning Agentic Policy from Action Guidance
May 12, 2026Yuxiang Ji, Zengbin Wang, Yong Wang +6Agentic RL Reinforcement Learning
1Xiamen University · 2AMAP, Alibaba Group · Work done during internship at AMAP, Alibaba Group +1
Tool-Aware Optimization with Entropy Guidance for Efficient Agentic Reinforcement Learning
Jun 2, 2026Hongye Cao, Nuo Yan, Haoyuan Deng +5Agentic RL RL for Language Model Reasoning
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210093, China · Nanyang Technological University, Singapore · China Mobile NineVerse Artificial Intelligence Technology (Beijing) Co., Ltd., and Institute of Artificial Intelligence, NineVerse, Beijing 100032, China

Aug 5, 2026J/K move · Enter open · S save

Optimizing What Policies Learn From: Recoverability-aware Rollout Intervention Learning

Zheyuan Zhang, Manqing Mao, Hong Wang, Zhuoer Wang+7

University of Notre Dame · Amazon, Inc

RL for Language Models RL Post-Training Policy Learning

Open paper AskPDF arXiv

Critic-free group-based reinforcement learning has become a scalable approach for post-training large language models. However, most existing methods allocate the same number of rollouts to every task and trajectory state, even though some rollouts provide much more useful learning signals than others. Recent work has started to treat rollout generation as an adaptive decision, but two important limitations remain. First, intervention strategies are often based on fixed heuristics and therefore cannot adjust as the policy changes during training. Second, these methods usually decide only how many rollouts to generate, without explicitly controlling where and how to intervene. To address these limitations, we propose Recoverability-Aware Intervention Learning (RAIL), a training-time framework that learns how to generate rollouts based on the improvement produced by each intervention. RAIL models intervention selection as an online contextual-bandit problem and trains a recoverability controller using intervention traces collected through a shadow-to-live procedure. This allows the controller to keep learning while the underlying policy evolves. We evaluate RAIL in terms of effectiveness, adaptivity, expressiveness, and efficiency. Across multiple settings, RAIL consistently improves performance under limited rollout budgets. These results show that recoverability-aware intervention provides a principled way to generate more informative and less redundant rollouts, leading to stronger learning signals during post-training.

Abstract

RL for LLM Agents

Explore similar work

Optimizing What Policies Learn From: Recoverability-aware Rollout Intervention Learning

Learning Agentic Policy from Action Guidance

Tool-Aware Optimization with Entropy Guidance for Efficient Agentic Reinforcement Learning