Outcome-based reinforcement learning provides coarse supervision for long-horizon LLM agents: failed trajectories may contain correct steps, while successful trajectories can contain errors. Trajectory-level skills can also shift teacher preferences at unrelated steps, introducing off-target distillation signals. We introduce SCOPE, a framework for Selective Critical-Step On-Policy Self-Distillation. After a supervised warm-up, the policy analyzes its own trajectories, generates step-specific hindsight skills, and distills their guidance only at selected error steps. A bounded signed teacher-student coefficient stabilizes auxiliary updates while preserving positive reinforcement and negative correction. A success-relative pairwise length reward further encourages efficient task completion. Experiments on ALFWorld and WebShop demonstrate consistent improvements across multiple model backbones. With Qwen2.5-3B, SCOPE achieves 97.7% success on ALFWorld and 85.2% on WebShop, outperforming SEED by 5.9 and 6.3 percentage points, respectively. On unseen ALFWorld tasks, SCOPE reaches 89.6% success, improving over GRPO by 23.2 percentage points with 46.7% fewer interactions. Scaling to Qwen3-32B, SCOPE achieves 99.2% success on seen ALFWorld tasks and 94.0% on unseen tasks.
Figures & tables
Figure 1: Motivation and results of SCOPE. (a) Outcome–step mismatch. (b) Off-target skill signals. (c) Training dynamics (mean ± std. across seeds); Unbounded SCOPE uses raw-gap OPSD. (d) Qwen2.5-3B results at step 150.
Figure 2: Overview of SCOPE. Supervised initialization teaches the policy to analyze trajectories and generate corrective skills. During online training, the policy collects and analyzes its own rollouts, assigns skills to selected error steps, and combines selective self-distillation with efficiency-aware GRPO.
Figure 3: Anomaly during minibatch optimization. (a) Pre-optimization and training-stage gaps (symmetric-log axis). (b) Current student and cached teacher sampled-token log-probabilities. Dashed lines mark outer update 26.
Figure 4: Training dynamics on ALFWorld with Qwen2.5-3B. (A) Component ablations. (B) Signed-gap slope α . (C) Tanh saturation bound c . (D) Pairwise length temperature τlen . The full setting is α=2.5 , c=1 , and τlen=0.1 , labeled All in (A) and by its parameter value in (B)–(D). Trajectory-level guidance uses one skill per trajectory and distills all steps. Lines connect observed checkpoints; the success axes start at 10% in (A) and 40% in (B)–(D).
Figure 5: Interaction efficiency during training. Mean environment interaction steps on ALFWorld across different training configurations.
Figure 6: Parameter sensitivity of SCOPE. Left: effects of gap slope α , saturation bound c , and pairwise temperature τlen on task success. Right: task-wise performance under different parameter settings, where Full denotes the default configuration ( α=2.5 , c=1 , τlen=0.1 ).
Figure 7
Figure 9: Training dynamics with Qwen3-32B. Single-run ALFWorld seen-evaluation curves comparing SCOPE and GRPO across six task families, overall success, and mean interaction steps. The dotted line marks step 150, corresponding to the seen results in Table 3 .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Maximum skill entries Kmax
5
Signed-gap slope α
2.5
Saturation bound c
1
Distillation weight λ
0.01
Length-preference temperature τlen
0.1
Length-bonus weight λlen
5
Appendix
Table 4: Key settings of full SCOPE. Parameter sweeps vary one of α , c , or τlen at a time.
Figure 10: Single-run validation curves on ALFWorld. Overall and task-family success rates, together with mean environment interaction steps, for SCOPE-3B and SCOPE-7B. The dotted vertical line indicates step 150, whose overall success rates match Table 1 .
Figure 11: Single-run validation curves on WebShop. Success rate, normalized task score, and mean environment interaction steps for SCOPE-3B and SCOPE-7B. The dotted vertical line indicates step 150; the success rates and normalized scores match Table 1 .
Training multi-turn LLM agents with reinforcement learning typically relies on trajectory-level rewards, which assign a uniform advantage to every step and cannot identify which decisions led to success or failure. Self-distillation methods can provide finer-grained supervision by augmenting RL with privileged information. However, existing approaches usually apply the same type of privileged information to every step in an indistinguishable manner, ignoring a key asymmetry: routine steps need little additional guidance, while critical error steps require corrective direction that environment feedback alone cannot provide. We propose AHEAD, a step-aware framework that matches different supervision sources to different step types. The teacher receives environment feedback on all steps as a grounded dense signal, and additionally receives LLM-generated corrective hints on error steps to supply the direction that environment feedback lacks. The method introduces minimal changes to the standard GRPO algorithm. Across ALFWorld, WebShop, and Search-based QA, and across three model scales, AHEAD raises task success (+13.3 points on ALFWorld and +11.0 on WebShop at 7B over GRPO), reaches a given success rate in fewer training steps, and solves tasks within tighter interaction budgets than outcome-only RL and prior self-distillation baselines.
Reinforcement learning for multi-turn agents suffers from a credit-assignment mismatch: rewards are sparse and trajectory-level, while success often hinges on a few local decisions. Existing online policy distillation (OPD) provides denser token-level supervision, but typically treats heterogeneous agent trajectories as monolithic strings rather than causal interaction units. We present StepOPSD, a post-rollout preference self-distillation framework that takes the agent step as the unit of credit redistribution. StepOPSD decomposes trajectories into action-centered step segments, rescoring them under hindsight-enriched teacher contexts and converting token-level log-probability gaps into sign-preserving advantage shaping with a normalized per-step credit budget before the GRPO update. Across ALFWorld and Search-QA with Qwen3-1.7B and Qwen2.5-3B-Instruct, StepOPSD attains best or second-best results on subsets most sensitive to local causal errors, including first-place performance on ALFWorld Heat (79.1%), PickTwo (95.0%), Search-QA TriviaQA (61.6%), and tied-best performance on HotpotQA (40.4%). The results further reveal a consistent two-knob law: smaller α_clip acts as a broadly stabilizing local trust region, whereas the optimal global mixing strength λ_mix remains task-dependent. These findings suggest that step-aware distillation is most useful when trajectory-level rewards are weakly aligned with the local action that determines downstream success.
Outcome-based reinforcement learning provides a stable optimization backbone for language agents, but its sparse trajectory-level rewards provide little guidance on which intermediate decisions should be reinforced or suppressed. On-policy self-distillation offers dense token-level supervision, yet existing skill-conditioned variants often rely on external skill memories or retrieved privileged context, which are costly to maintain and can be mismatched with the state distribution induced by the current policy in multi-turn interaction. We propose OPID (On-Policy Skill Distillation), a framework that extracts skill supervision directly from completed on-policy trajectories. OPID represents trajectory hindsight as hierarchical skills: episode-level skills capture global workflows or failure-avoidance rules, while step-level skills capture local decision knowledge at critical timesteps. A critical-first routing mechanism uses step-level skills when critical decisions are identified and falls back to episode-level skills as default guidance otherwise. The selected skill is injected into the interaction history, allowing the old policy to re-score the same sampled response under both original and skill-augmented contexts. The resulting log-probability shift yields a token-level self-distillation advantage, which is combined with the outcome advantage for policy optimization. OPID thus preserves RL as the primary training objective while introducing dense, distribution-matched hindsight supervision. Experiments on ALFWorld, WebShop and Search-based QA demonstrate that OPID generally improves agent performance, sample efficiency, and robustness over outcome-only RL and existing skill-distillation baselines. Our code is available at https://github.com/jinyangwu/OPID/tree/main.
Shuo Yang, Jinyang Wu, Zhengxi Lu +8
1Tsinghua University · 2Zhejiang University · 3The Chinese University of Hong Kong +2