Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents

Hao Wang; Guozhi Wang; Han Xiao; Yufeng Zhou; Yue Pan; Jichao Wang; Ke Xu; Yafei Wen

cs.LGApr 12, 2026

Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents

Authors: Hao Wang, Guozhi Wang, Han Xiao, Yufeng Zhou, Yue Pan, Jichao Wang, Ke Xu, Yafei Wen, +3 more

Organizations: Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences · vivo AI Lab · The Chinese University of Hong Kong · University of Science and Technology of China · University of Chinese Academy of Sciences

Abstract

Reinforcement learning (RL) has been widely used to train LLM agents for multi-turn interactive tasks, but its sample efficiency is severely limited by sparse rewards and long horizons. On-policy self-distillation (OPSD) alleviates this by providing dense token-level supervision from a privileged teacher that has access to ground-truth answers. However, such fixed privileged information cannot capture the diverse valid strategies in agent tasks, and naively combining OPSD with RL often leads to training collapse. To address these limitations, we introduce Skill-SD, a framework that turns the agent's own trajectories into dynamic training-only supervision. Completed trajectories are summarized into compact natural language skills that describe successful behaviors, mistakes, and workflows. These skills serve as dynamic privileged information conditioning only the teacher, while the student always acts under the plain task prompt and learns to internalize the guidance through distillation. To stabilize the training, we derive an importance-weighted reverse-KL loss to provide gradient-correct token-level distillation, and dynamically synchronize the teacher with the improving student. Experimental results on agentic benchmarks demonstrate that Skill-SD substantially outperforms the standard RL baseline, improving both vanilla GRPO (+14.0%/+10.9% on AppWorld/Sokoban) and vanilla OPD (+42.1%/+40.6%). Project page: https://k1xe.github.io/skill-sd/

Figures & tables

Figure 1: Skill-SD overview. (1) The student generates on-policy rollouts and receives task-level rewards. (2) Completed trajectories are asynchronously summarized into compact skills and stored in a UCB-indexed buffer. (3) The same token sequence is re-scored under the student prompt and the skill-augmented teacher prompt; GRPO provides task-level credit assignment while the importance-weighted SDL loss transfers token-level teacher knowledge. Token color intensity reflects prediction probability.

Figure 2: Token-level self-distillation dynamics on an AppWorld task. Token color intensity is proportional to prediction probability. At Epoch 0, student and teacher distributions differ substantially; after convergence, they have largely aligned. Right: the SDL loss decreases by 59.3% over training.

	AppWorld		Sokoban		Avg.
Method	Acc.	Comp.	Acc.	Comp.	Acc.	Comp.
Base Model	8.8 0.0	39.1 0.0	12.5 0.0	32.0 0.0	10.6 0.0	35.6 0.0
Vanilla OPD	22.8 ↑14.0	59.7 ↑20.6	21.9 ↑9.4	37.5 ↑5.5	22.4 ↑11.7	48.6 ↑13.0
Vanilla GRPO	50.9 ↑42.1	76.3 ↑37.2	51.6 ↑39.1	68.8 ↑36.8	51.2 ↑40.6	72.5 ↑36.9
Skill-Augmented GRPO	42.1 ↑33.3	76.1 ↑37.0	20.3 ↑7.8	37.5 ↑5.5	31.2 ↑20.6	56.8 ↑21.2
Skill-SD	64.9 ↑56.1	84.9 ↑45.8	62.5 ↑50.0	71.1 ↑39.1	63.7 ↑53.1	78.0 ↑42.4

Table 1: Main results on AppWorld and Sokoban (Qwen3-4B-Instruct-2507). Skill-SD uses on-policy student rollout with dynamic teacher synchronization. Subscripts denote absolute change from the base model.

Panel of Figure 3 — Figure 3: Training curves for all baselines on AppWorld (left) and Sokoban (right). On AppWorld, Skill-Augmented GRPO achieves the highest training accuracy while Skill-SD steadily narrows the gap. On Sokoban, Skill-SD overtakes Vanilla GRPO after step 60 and maintains steady gains.

		AppWorld		Sokoban		Avg.
Rollout	Teacher	Acc.	Comp.	Acc.	Comp.	Acc.	Comp.
On-policy	Frozen	49.1 ↓15.8	79.0 ↓5.9	50.0 ↓12.5	63.3 ↓7.8	49.6 ↓14.1	71.1 ↓6.9
On-policy	Dynamic	64.9 0.0	84.9 0.0	62.5 0.0	71.1 0.0	63.7 0.0	78.0 0.0
Off-policy ∗	Frozen	45.6 ↓19.3	78.8 ↓6.1	12.5 ↓50.0	31.3 ↓39.8	29.1 ↓34.6	55.0 ↓23.0
Off-policy ∗	Dynamic	42.1 ↓22.8	76.5 ↓8.4	10.9 ↓51.6	32.0 ↓39.1	26.5 ↓37.2	54.3 ↓23.7

Table 2: Ablation: effect of replacing Skill-SD’s design choices. Skill-SD uses on-policy student rollout with dynamic teacher synchronization (bolded). Subscripts denote absolute change from Skill-SD. ∗ Training collapsed during mid-training; values reflect the checkpoint before collapse.

Panel of Figure 4 — Figure 4: Training dynamics of the four teacher–student configurations on AppWorld (left) and Sokoban (right). Skill-SD (a, on-policy + dynamic) trains stably to high performance. On-policy + frozen (b) converges stably but to a lower plateau. Both off-policy variants (c, d) achieve strong early performance but collapse during mid-training. The pattern replicates across both environments.

Figure 5: Effect of SDL coefficient

\lambda

on AppWorld training (left) and validation (right) completion rate.

\lambda=0.001

achieves the best validation performance;

\lambda=0.01

over-regularizes and suppresses exploration, while

\lambda=0.0005

provides insufficient teacher guidance.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Quantity	Definition	Purpose
Main SDL weight $\rho_{t}^{\text{on}}$	$\pi_{\theta}^{\text{stu}}/\operatorname{sg}(\pi_{\theta_{\text{old}}}^{\text{stu}})$	Match student sampling in the main method
Off-policy SDL weight $\rho_{t}^{\text{off}}$	$\pi_{\theta}^{\text{stu}}/\operatorname{sg}(\pi_{\theta_{\text{old}}}^{\text{tea}})$	Match teacher sampling in the comparison branch
GRPO clipping ratio $r_{t}$	$\pi_{\theta}^{\text{stu}}/\operatorname{sg}(\pi_{\theta_{\text{old}}}^{\text{stu}})$	Center trust region at old student

Appendix

Table 3: Importance-weighted SDL and student-centered GRPO in Skill-SD.

Variant	Rollout Source	Teacher Refresh	Distillation
Off-policy + dynamic	Teacher	Dynamic self-evolution	Yes
Off-policy + frozen	Teacher	Frozen	Yes
On-policy + dynamic	Student	Dynamic self-evolution	Yes
On-policy + frozen	Student	Frozen	Yes
Skill-Augmented GRPO	Skill-conditioned student	N/A	No
Vanilla GRPO	Student	N/A	No

Appendix

Table 4: Variant definitions for the AppWorld ablation matrix.

Hyperparameter	Value
Group size $G$ (rollouts per task)	4
Train batch size	16
Optimization epochs per batch	1
Max prompt length	8192 tokens
Max response length	20480 tokens
Dynamic batch sizing	Enabled

Appendix

Table 5: Training hyperparameters for AppWorld and Sokoban experiments.

Explore similar work

Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents
Aug 10, 2026Tianjun Pan, Yuan Li, Hongda Wang +8LLM Agent Skill Learning Self-Distillation
Fudan University · Zhejiang University · Alibaba Group +1
SKILL-KD: Contrastive Skill Distillation for LLM Agents
Jul 30, 2026Qiming Shi, Yibo Dou, Jiawen Zhu +5LLM Agent Skill Learning LLM Agent Training
State Key Lab of CAD&CG, Zhejiang University · School of Software and Microelectronics, Peking University · School of Software Technology, Zhejiang University +1
Self-Distilled Agentic Reinforcement Learning
May 14, 2026Zhengxi Lu, Zhiyuan Yao, Zhuowen Han +8Agentic RL Self-Distillation
Zhejiang University · Meituan · Tsinghua University

Aug 10, 2026J/K move · Enter open · S save

Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents

Tianjun Pan, Yuan Li, Hongda Wang, Linbo Jin+7

Fudan University · Zhejiang University · Alibaba Group+1

LLM Agent Skill Learning Self-Distillation Knowledge Distillation RL for LLM Agents

Open paper AskPDF arXiv

External natural-language skills provide large language model (LLM) agents with reusable and editable guidance for solving complex tasks. Yet their effectiveness depends not only on skill quality, but also on whether the policy can translate the provided guidance into appropriate actions. However, methods specifically designed to improve this skill-utilization ability remain largely underexplored. In practice, skill-based agents are commonly trained with reinforcement learning objectives centered on task-level rewards, which offer limited supervision and struggle to capture subtle differences in how effectively the policy uses the provided skills. We propose BCSD (Bidirectional Context Self-Distillation), a framework that combines self-distillation with reinforcement learning to train LLM agents to use external skills more effectively. Unlike prior self-distillation methods that rely on a single privileged context, BCSD evaluates each trajectory from two complementary skill-context views. The augmented view introduces higher-level Meta-Skill guidance, while the reduced view prunes general guidance to highlight task-specific skills. Their complementary token-level signals are combined to rescale the RL advantage. Experiments on ALFWorld and WebShop demonstrate that BCSD achieves the strongest overall performance across model scales, enabling agents to utilize external skills more effectively. Ablation studies further verify the complementary contributions of the augmented and reduced context views. Code will be released to ensure full reproducibility.

Abstract

Explore similar work

Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents

SKILL-KD: Contrastive Skill Distillation for LLM Agents

Self-Distilled Agentic Reinforcement Learning