Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning

Changhui Sun; Lanbo Liu; Hang Lei; Tong Ling; Jiahang Xie; Zhiyong Zheng; Yujia Wang; Hao Liu

cs.CLAug 17, 2026

Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning

Authors: Changhui Sun, Lanbo Liu, Hang Lei, Tong Ling, Jiahang Xie, Zhiyong Zheng, Yujia Wang, Hao Liu,

Organizations: State Key Laboratory for Novel Software Technology, Nanjing University, China

Abstract

On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories. This approach has achieved strong empirical gains and can often surpass conventional off-policy distillation with substantially less data. However, standard token-level OPD can provide only fragmented corrections along an erroneous student trajectory and cannot unfold a complete and correct repair path. Motivated by this limitation, we propose Step-Level On-Policy Distillation (SOPD), which combines the long-horizon correction of supervised fine-tuning (SFT) with the on-policy advantage of OPD to provide step-level supervision over complete student-generated trajectories. We show that, at different limits of step length, SOPD reduces to SFT or approximates OPD. Compared with SFT, the teacher responses in SOPD are conditioned on student trajectories and therefore align more closely with student-visited states; compared with OPD, SOPD provides longer-horizon corrections rather than fragmented token-level guidance. Across both reasoning and agent tasks, SOPD substantially outperforms conventional SFT and OPD. For example, on ALFWorld, SOPD improves the average success rate by 13.4 points over Vanilla OPD. We hope this work offers a new perspective for future research on distillation methods.

Figures & tables

Figure 1: Overview of Step-Level On-Policy Distillation (SOPD). The left and center panels compare On-Policy Distillation (OPD), On-Policy Expert Correction (OEC), and SOPD. The right panel illustrates the packed attention mask used for efficient training. We pack the input prompt, the complete student trajectory, and its multiple step-level teacher targets into a single sequence, enabling these step-level supervision instances to be processed in parallel within a single forward pass to ensure training efficiency. Each teacher target can attend only to the input prompt, its corresponding student prefix, and its own preceding causal teacher tokens, while future student steps and other teacher targets remain masked.

	Valid Seen		Valid Unseen		Hard
Method	SR $\uparrow$	Rounds $\downarrow$	SR $\uparrow$	Rounds $\downarrow$	SR $\uparrow$	Rounds $\downarrow$
Qwen2.5-7B-RL (Teacher)	85.71	10.61	76.87	13.06	6.61	27.31
Qwen2.5-3B Zero-Shot	7.86	28.73	2.24	29.63	0.83	29.88
Supervised fine-tuning (SFT)	32.14	22.85	25.37	24.16	4.96	29.12
Vanilla OPD	65.72	14.73	60.45	16.21	10.74	28.64
TCOD (B2F)	77.86 $\uparrow$ 12.14	12.57 $\downarrow$ 2.16	70.90 $\uparrow$ 10.45	14.56 $\downarrow$ 1.65	13.22 $\uparrow$ 2.48	28.16 $\downarrow$ 0.48

Table 1: ALFWorld results with a 3B student and 7B RL teacher. SR denotes success rate (%). TCOD supplies all rows except SOPD ( Wang et al., 2026 ) . Arrows show the absolute change from Vanilla OPD; bold marks the best student result in each column.

Figure 2: ALFWorld full-split evaluation over normalized Explorer progress. Seen and Unseen success improve over training while interaction length decreases overall. The curves summarize full-split evaluations at the saved checkpoints.

Figure 3: ALFWorld training diagnostics following TCOD’s multi-panel analysis. (a) Explorer-batch task success; (b) mean environment rounds; (c) mean tokens per generated teacher turn target; and (d) step-balanced SFT loss in the Trainer. The horizontal axis normalizes each signal to the endpoint of its corresponding component.

Method	AIME24	AIME25	HMMT25 (Feb.)	HMMT25 (Nov.)	$\mathrm{Avg}_{4}$
Teacher	58.0	54.6	32.5	38.9	46.0
$+$ continued RL (100 updates)	60.9	55.6	32.8	38.4	46.9
Student	21.5	21.9	10.0	8.0	15.4
OPD	61.9	57.0	32.5	39.6	47.7
SOPD	71.8	67.4	38.9	52.7	57.7

Table 2: Comparison of OPD and SOPD using the same-size teacher–student pair. Teacher denotes Qwen3-4B-Non-Thinking-RL-Math, and Student denotes the initial Qwen3-4B non-thinking model. The first three rows are reference results reported by G-OPD ( Yang et al., 2026 ) . Values are mean accuracy (%);

\mathrm{Avg}_{4}

averages the four benchmarks, and bold marks the best student method in each column.

Figure 4: SOPD step structure over mathematical training. (a) Mean natural steps per AIME24 evaluation response; (b) mean student tokens per natural step; and (c) mean supervised teacher tokens per retained step during training. All boundaries use the semantic-step rule and 1,024-token safety cap.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Training batch size	128
Micro-batch size	128
Rollouts per prompt	8
Maximum prompt length	2,048
Maximum response length	16,384
Temperature	1.0

Appendix

Table 3: GRPO hyperparameters used to train the mathematical domain teacher, following G-OPD ( Yang et al., 2026 ) .

Language Model Distillation

Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.

See all 489 papers

Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation
May 29, 2026Yanjiang Liu, Jie Lou, Xinyan Guan +7CoT Distillation Language Model Distillation
University of Chinese Academy of Sciences · 2Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences, Beijing, China · 3Xiaohongshu
Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement
Aug 31, 2026Yi Ding, Ruqi ZhangRL for Language Models Language Model Distillation
Department of Computer Science, Purdue University, USA
Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance
May 29, 2026Yuxuan Jiang, Francis FerraroCoT Distillation Language Model Distillation
University of Maryland, Baltimore County

May 29, 2026J/K move · Enter open · S save

Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation

Yanjiang Liu, Jie Lou, Xinyan Guan, Yuqiu Ji+6

University of Chinese Academy of Sciences · 2Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences, Beijing, China · 3Xiaohongshu

CoT Distillation Language Model Distillation On-Policy Distillation Teacher-Student Learning RL Fine-Tuning

Open paper AskPDF arXiv

On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher. However, we identify a critical bottleneck, Supervision Fidelity Decay (SFD): as student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative. Consequently, the teacher-dependent corrective signal in reverse-KL distillation weakens, causing student drift to compound across long reasoning chains. To mitigate SFD, we introduce Lookahead Group Reward (\ours{}). Building on the insight that next-step teacher confidence reflects the discriminative strength of future reverse-KL supervision, \ours{} evaluates the student's top-K candidate tokens by the teacher confidence they induce at the subsequent step and assigns a group-normalized reward. To maintain computational efficiency, we further design an entropy-triggered tree-attention mechanism. Across six math and code benchmarks, \ours{} improves mean@8 by 2.57 points over OPD for a 7B student, with gains increasing in longer-generation and reaching +4.92 points on AIME-26 at 39k tokens.

Abstract

Language Model Distillation

Explore similar work

Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation

Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance