SCOPE: Selective Critical-Step On-Policy Self-Distillation for Agentic Reinforcement Learning

Qirui Liu; Wei Cao; Yu Mi; Yichen Sun; Xuekun Zhang; Lei Liu; Yue Shen; Zhixuan Chu

cs.AIOct 28, 2026

SCOPE: Selective Critical-Step On-Policy Self-Distillation for Agentic Reinforcement Learning

Authors: Qirui Liu, Wei Cao, Yu Mi, Yichen Sun, Xuekun Zhang, Lei Liu, Yue Shen, Zhixuan Chu

Organizations: Zhejiang University · Ant Group

Abstract

Outcome-based reinforcement learning provides coarse supervision for long-horizon LLM agents: failed trajectories may contain correct steps, while successful trajectories can contain errors. Trajectory-level skills can also shift teacher preferences at unrelated steps, introducing off-target distillation signals. We introduce SCOPE, a framework for Selective Critical-Step On-Policy Self-Distillation. After a supervised warm-up, the policy analyzes its own trajectories, generates step-specific hindsight skills, and distills their guidance only at selected error steps. A bounded signed teacher-student coefficient stabilizes auxiliary updates while preserving positive reinforcement and negative correction. A success-relative pairwise length reward further encourages efficient task completion. Experiments on ALFWorld and WebShop demonstrate consistent improvements across multiple model backbones. With Qwen2.5-3B, SCOPE achieves 97.7% success on ALFWorld and 85.2% on WebShop, outperforming SEED by 5.9 and 6.3 percentage points, respectively. On unseen ALFWorld tasks, SCOPE reaches 89.6% success, improving over GRPO by 23.2 percentage points with 46.7% fewer interactions. Scaling to Qwen3-32B, SCOPE achieves 99.2% success on seen ALFWorld tasks and 94.0% on unseen tasks.

Figures & tables

Figure 1: Motivation and results of SCOPE. (a) Outcome–step mismatch. (b) Off-target skill signals. (c) Training dynamics (mean $\pm$ std. across seeds); Unbounded SCOPE uses raw-gap OPSD. (d) Qwen2.5-3B results at step 150.

Figure 2: Overview of SCOPE. Supervised initialization teaches the policy to analyze trajectories and generate corrective skills. During online training, the policy collects and analyzes its own rollouts, assigns skills to selected error steps, and combines selective self-distillation with efficiency-aware GRPO.

Figure 3: Anomaly during minibatch optimization. (a) Pre-optimization and training-stage gaps (symmetric-log axis). (b) Current student and cached teacher sampled-token log-probabilities. Dashed lines mark outer update 26.

Figure 4: Training dynamics on ALFWorld with Qwen2.5-3B. (A) Component ablations. (B) Signed-gap slope

\alpha

. (C) Tanh saturation bound

c

. (D) Pairwise length temperature

\tau_{\mathrm{len}}

. The full setting is

\alpha=2.5

c=1

, and

\tau_{\mathrm{len}}=0.1

, labeled All in (A) and by its parameter value in (B)–(D). Trajectory-level guidance uses one skill per trajectory and distills all steps. Lines connect observed checkpoints; the success axes start at 10% in (A) and 40% in (B)–(D).

Figure 5: Interaction efficiency during training. Mean environment interaction steps on ALFWorld across different training configurations.

Panel of Figure 6 — Figure 6: Parameter sensitivity of SCOPE. Left: effects of gap slope $\alpha$ , saturation bound $c$ , and pairwise temperature $\tau_{\mathrm{len}}$ on task success. Right: task-wise performance under different parameter settings, where Full denotes the default configuration ( $\alpha=2.5$ , $c=1$ , $\tau_{\mathrm{len}}=0.1$ ).

Figure 9: Training dynamics with Qwen3-32B. Single-run ALFWorld seen-evaluation curves comparing SCOPE and GRPO across six task families, overall success, and mean interaction steps. The dotted line marks step 150, corresponding to the seen results in Table 3 .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Maximum skill entries $K_{\max}$	5
Signed-gap slope $\alpha$	2.5
Saturation bound $c$	1
Distillation weight $\lambda$	0.01
Length-preference temperature $\tau_{\mathrm{len}}$	0.1
Length-bonus weight $\lambda_{\mathrm{len}}$	5

Appendix

Table 4: Key settings of full SCOPE. Parameter sweeps vary one of

\alpha

c

, or

\tau_{\mathrm{len}}

at a time.

Figure 10: Single-run validation curves on ALFWorld. Overall and task-family success rates, together with mean environment interaction steps, for SCOPE-3B and SCOPE-7B. The dotted vertical line indicates step 150, whose overall success rates match Table 1 .

Figure 11: Single-run validation curves on WebShop. Success rate, normalized task score, and mean environment interaction steps for SCOPE-3B and SCOPE-7B. The dotted vertical line indicates step 150; the success rates and normalized scores match Table 1 .

Agentic RL

Agentic reinforcement learning trains AI agents to plan, use tools, and adapt through feedback from multi-step interactions.

See all 232 papers

AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL
Aug 25, 2026Xiaolong Jin, Dingmin Wang, Vijay Lingam +1RL for LLM Agents LLM-Guided RL
AWS AI Labs · Purdue University
StepOPSD: Step-Aware Online Preference Distillation for Agent Reinforcement Learning
May 26, 2026Yanfei Zhang, Xu Lin, Chenglin WuCredit Assignment in RL Step-Level Credit Assignment in RL
Independent Researcher · Tencent · DeepWisdom
OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning
Jun 25, 2026Shuo Yang, Jinyang Wu, Zhengxi Lu +8Agentic RL LLM Agent Skill Learning
1Tsinghua University · 2Zhejiang University · 3The Chinese University of Hong Kong +2

Aug 25, 2026J/K move · Enter open · S save

AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL

Xiaolong Jin, Dingmin Wang, Vijay Lingam, Varun Kumar

AWS AI Labs · Purdue University

RL for LLM Agents LLM-Guided RL Multi-Turn RL Step-Level Credit Assignment in RL Self-Distillation

Open paper AskPDF arXiv

Training multi-turn LLM agents with reinforcement learning typically relies on trajectory-level rewards, which assign a uniform advantage to every step and cannot identify which decisions led to success or failure. Self-distillation methods can provide finer-grained supervision by augmenting RL with privileged information. However, existing approaches usually apply the same type of privileged information to every step in an indistinguishable manner, ignoring a key asymmetry: routine steps need little additional guidance, while critical error steps require corrective direction that environment feedback alone cannot provide. We propose AHEAD, a step-aware framework that matches different supervision sources to different step types. The teacher receives environment feedback on all steps as a grounded dense signal, and additionally receives LLM-generated corrective hints on error steps to supply the direction that environment feedback lacks. The method introduces minimal changes to the standard GRPO algorithm. Across ALFWorld, WebShop, and Search-based QA, and across three model scales, AHEAD raises task success (+13.3 points on ALFWorld and +11.0 on WebShop at 7B over GRPO), reaches a given success rate in fewer training steps, and solves tasks within tighter interaction budgets than outcome-only RL and prior self-distillation baselines.

Abstract

Agentic RL

Explore similar work

AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL

StepOPSD: Step-Aware Online Preference Distillation for Agent Reinforcement Learning

OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning