On-Policy Self-Distillation with Teacher Evolution

Shufan Shen; Zhongni Hou; Chao Bi; Yufei Zhang; Guojun Yin; Shuhui Wang

cs.LGOct 28, 2026

On-Policy Self-Distillation with Teacher Evolution

Authors: Shufan Shen, Zhongni Hou, Chao Bi, Yufei Zhang, Guojun Yin, Shuhui Wang

Organizations: State Key Lab. of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Meituan

Abstract

On-Policy Self-Distillation (OPSD), which aligns a student model's token-level distribution with that of its self-teacher prompted by privileged context, has emerged as a promising direction to mitigate reward sparsity in sequence-level post-training paradigms. However, OPSD often fails to achieve continuous performance gains during training. In this paper, we find that the model's limited self-teaching capability underlies this phenomenon by generating unreliable guidance on certain samples. To address this problem, we propose Teacher-Evolution Self-Distillation (TESD), which enables the self-teacher to generate rollouts and receive environmental feedback based on task correctness and context independence. First, using these rewards as training objectives, we perform teacher-oriented optimization to continuously enhance the model's self-teaching capability during distillation. Second, by estimating teacher guidance reliability from these rewards, we apply adaptive student-oriented optimization to down-weight samples with unreliable guidance. Extensive experiments across multiple datasets and model scales show that TESD consistently outperforms representative token-level and sequence-level methods, highlighting the necessity of improving supervision quality in OPSD.

Figures & tables

Figure 1: (a) In OPSD, the self-teacher provides guidance indiscriminately across all samples without improving its own capabilities, thereby degrading the quality of supervision delivered to the student. (b) TESD improves the self-teacher using the environmental rewards of its rollouts while delivering adaptive guidance by down-weighting samples with unreliable supervision. (c) TESD achieves more continuous performance improvement than current OPSD methods.

Figure 2: Top: We plot the task correctness and teacher confidence (measured by mean token probability) of each student rollout, with the color representing its edit distance to the privileged context. Bottom: The evolution of the model’s teaching ability during the training process, which is estimated by the mean spearman correlation between teacher confidence and actual rewards. Experiments are conducted on the Physics subset of SciKnowEval using Qwen3-8B.

Figure 3: Overall framework of TESD. Given an input query, the student first generates rollouts, which provide the privileged context for the self-teacher. Then, TESD optimizes the student and the self-teacher, respectively. For the self-teacher, it interacts with the environment and learns from the obtained rewards of correctness and independence. For the student, it is supervised by the logits of the self-teacher. The loss value is adaptively weighted by the teacher’s mean reward of the corresponding input query, which estimates the quality of teacher guidance.

Model	Method	Chemistry	Physics	Biology	Material	Tooluse	Mean
Qwen3-4B	SDPO	80.92	71.48	56.63	73.07	66.18	69.66
	GRPO	81.52	78.83	60.87	82.11	63.69	73.40
	SDPO+GRPO	79.58	75.37	54.62	81.85	62.87	70.86
	TESD (Ours)	80.30	77.50	63.50	83.31	67.00	74.32
Qwen3-8B	SDPO	77.83	67.73	62.63	79.99	62.87	70.21
	GRPO	80.71	80.47	63.13	82.11	68.29	74.94

Table 1: Performance comparisons on SciKnowEval and Tooluse benchmarks. All methods are trained to convergence, and scores are reported as the maximum mean@16 accuracy (%) achieved during training.

Method	Qwen3-1.7B	Qwen3-4B	Qwen3-8B	Mean
SDPO	45.35	56.97	62.38	54.90
GRPO	44.96	56.12	60.85	53.98
SDPO+GRPO	46.38	59.88	61.95	56.07
TESD (Ours)	45.98	60.07	63.31	56.45

Table 2: Performance comparisons on LCBv6 benchmark using different base models.

	Task	Holdout Tasks
Method	Physics	IFEval	MMLU-Pro	TruthfulQA	GSM8K	Math500	ARC-Challenge	HumanEval	HaluEval	Mean
Base	58.44	87.05	73.70	86.05	91.89	82.13	90.53	64.02	75.93	81.41
SDPO	65.08	86.33	68.70	87.64	87.19	74.60	89.33	65.85	75.68	79.42
GRPO	78.75	87.65	71.70	85.68	91.28	79.20	89.16	64.63	76.01	80.66
TESD	79.24	86.21	72.70	88.98	91.51	80.40	90.36	67.68	76.31	81.77

Table 3: Performance on avoiding catastrophic forgetting. We compare the base model (Qwen3-8B) and the final checkpoint of GRPO, SDPO and TESD on the training task (Physics) and holdout tasks.

Method	Physics	Biology	Material	Mean
Baseline	71.48	56.63	73.07	67.06
+Seq-level Loss	75.37	54.62	81.85	70.61
+Priv-Context	73.20	51.88	82.98	69.35
+Edit-Reward	76.48	63.75	81.65	73.96

Table 4: Ablations of the teacher-oriented optimization. Based on SDPO, we sequentially incorporating sequence-level loss (+Seq-level Loss), optimizing the self-teacher with privileged context (+Priv-Context), and adopting an edit-distance reward (+Edit-Reward).

Method	Physics	Biology	Material	Mean
Baseline	77.66	62.50	81.12	73.76
+Mean Reward	74.84	55.37	78.39	69.53
+Softmax	73.67	58.75	79.26	70.56
+Scaling	80.55	64.13	81.78	75.49

Table 5: Ablations of the adaptive weighting mechanism. We evaluate the impact of weighting the query losses based on the mean reward of teacher rollouts (+Mean Reward), normalizing these rewards via softmax (+Softmax), and scaling the normalized results (+Scaling).

Figure 4: We report the evolution of the standard deviation of accuracy across 16 evaluations (a), the model’s average response length (b), teaching ability (c), and the distillation loss (d) across the training process.

Figure 5: Token-level teacher credit heatmaps of SDPO and TESD with Qwen3-8B on the Physics dataset.

Figure 6: Distributions of the mean reward (top) and the adaptive weight (bottom) of the input batch.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: System prompt for ScienceQA.

Figure 8: User prompt for ScienceQA.

Figure 9: User prompt for ToolUse.

Figure 10: Teacher input template.

Hyperparameter	GRPO	SDPO	TESD
Rollouts / prompt (total)	16	16	16
student rollouts	—	16	8
teacher rollouts	—	0	8
PPO mini-batch size	8	32	32
Advantage estimator	GRPO	GRPO	GRPO
Policy loss	student PG	KL-Div	KL-Div + teacher PG

Appendix

Table 6: Hyperparameter settings of different methods.

Dataset	Train	Test
SciKnowEval-Biology	450	50
SciKnowEval-Physics	720	80
SciKnowEval-Chemistry	1890	210
SciKnowEval-Materials	841	94
ToolUse (ToolAlpaca)	4046	68
LCBv6 (LiveCodeBench)	131	131

Appendix

Table 7: Train/test sizes of the six tasks.

Figure 11: Case studies of different methods on the Physics subset of the SciKnowEval benchmark using Qwen3-8B.

Methods	$N=16$	$N=32$	$N=48$
SDPO	54.62	55.25	55.12
TESD	54.87	56.63	59.62

Appendix

Table 8: Performance comparison of SDPO and TESD across different numbers of rollouts

N

. The experiments are conducted on the Biology subset of the SciKnowEval benchmark using Qwen3-4B over

10

training epochs.

$\lambda_{\text{edit}}$	Biology	Material	Physics	Mean
0.1	51.88	78.99	76.48	69.12
0.3	53.75	79.45	76.25	69.82
0.5	54.00	79.92	75.23	69.72
0.7	50.88	82.78	73.05	68.90
1.0	54.87	81.65	75.37	70.63

Appendix

Table 9: Ablation studies on the hyperparameter

\lambda_{\text{edit}}

. Experiments are conducted on SciKnowEval benchmark using Qwen3-4B over 10 training epochs.

Figure 12: The time cost of different methods during the training process. Experiments are conducted on the LCBv6 benchmark using Qwen3-8B.

On-Policy Self-Distillation

On-policy self-distillation trains a model on its own generated trajectories using a teacher view to provide dense supervision.

See all 162 papers

Adaptive Supervised Anchoring for On-Policy Self-Distillation
Aug 8, 2026Meilin Yang, Zixuan Ding, Jianhao Nie +5Supervised Fine-Tuning On-Policy Self-Distillation
Renmin University of China, Beijing, China · Renmin University of China, Beijing, China.
DualOPSD: Closing the Feedback Loop in On-Policy Self-Distillation
Aug 26, 2026Yutong Chen, Guangfu Guo, Zhichao Xu +1Adaptive Knowledge Distillation On-Policy Self-Distillation
Department of Computer Science, Clemson University, 105 Sikes Hall, Clemson, SC 29634, USA · Department of Computer Science, University of Utah
Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement
Aug 31, 2026Yi Ding, Ruqi ZhangRL for Language Models Language Model Distillation
Department of Computer Science, Purdue University, USA

Aug 8, 2026J/K move · Enter open · S save

Adaptive Supervised Anchoring for On-Policy Self-Distillation

Meilin Yang, Zixuan Ding, Jianhao Nie, Weite Zhang+4

Renmin University of China, Beijing, China · Renmin University of China, Beijing, China.

Supervised Fine-Tuning On-Policy Self-Distillation Language Model Distillation LLM Reasoning

Open paper AskPDF arXiv

On-policy self-distillation (OPSD) adapts a language model by distilling guidance from a frozen teacher on trajectories sampled from the student. Its effectiveness, however, depends critically on the quality of those trajectories. We show that when student rollouts drift from target trajectories, conditioning the teacher on off-target prefixes substantially weakens its task-relevant supervision. Controlled prefix-corruption experiments expose this failure mode, which we term rollout-conditioned signal degradation. To address this problem, we propose a unified training framework that separates two complementary supervision pathways. The first retains rollout-conditioned distribution matching, providing guidance on states the student actually visits. The second applies supervised cross-entropy on canonical ground-truth contexts, avoiding the incompatibility of imposing target tokens on erroneous rollout prefixes. Token-level rollout-target alignment is used to adapt the strength of the canonical-context anchor, emphasizing it during cold start and relaxing it as rollout quality improves. Experiments across multiple model scales, two task families, and general-reasoning benchmarks show that the proposed approach improves task acquisition over OPSD while preserving general capabilities, resulting in a more favorable empirical plasticity-stability trade-off. These findings identify context quality as a central bottleneck in on-policy self-distillation and demonstrate the value of separating rollout-conditioned guidance from canonical supervision.

Abstract

On-Policy Self-Distillation

Explore similar work

Adaptive Supervised Anchoring for On-Policy Self-Distillation

DualOPSD: Closing the Feedback Loop in On-Policy Self-Distillation

Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement