Divergence-Routed On-Policy Distillation for Language Models

Tiantian Peng; Guannan Lv; Hongyang Li; Qiuhe Hong; Hang Cheng; Hongjian Dou; Xiangyu Wu; Yuyang Liu

cs.AIOct 14, 2026

Divergence-Routed On-Policy Distillation for Language Models

Authors: Tiantian Peng, Guannan Lv, Hongyang Li, Qiuhe Hong, Hang Cheng, Hongjian Dou, Xiangyu Wu, Yuyang Liu

Organizations: Peking University · Kuaishou Technology · Tsinghua University

Abstract

On-policy distillation (OPD) transfers knowledge from a teacher model to a student along student-generated trajectories, typically by minimizing the reverse Kullback-Leibler (RKL) divergence. However, RKL can provide weak corrective signals for teacher-supported tokens to which the student assigns very low probability. This raises two questions: which positions are most consequential for the behavioral gap between the student and teacher, and what additional supervision is needed at these positions? Through cross-sampling, we find that divergence-based routing more effectively pinpoints positions contributing to the teacher-student performance gap than entropy-based routing. Further analysis reveals teacher-mode under-coverage at high Jensen-Shannon (JS) divergence positions, where standard OPD increases the large teacher-student probability gaps. Motivated by these findings, we propose Divergence-Routed On-Policy Distillation (DrOPD). DrOPD retains RKL supervision at all positions and selectively adds forward Kullback-Leibler (FKL) divergence supervision at positions with high JS divergence. Experiments across six math benchmarks show that DrOPD improves over standard OPD and entropy-based routing. DrOPD also reduces severe under-coverage cases by 37.9 percent relative to OPD, supporting targeted FKL as an effective complement to OPD.

Figures & tables

Figure 1: Cross-sampling and teacher-mode under-coverage at high-JS positions. (a) AMC23 accuracy versus teacher-intervention rate; dashed lines indicate standalone Teacher and Base performance. (b) Mean positive probability gap across original JS deciles, computed over recorded tokens with positive gaps. (c) Relative change from Base to OPD in the fraction of positions with gaps at or above each threshold, evaluated on fixed original JS

>0.06

positions.

Figure 2: Overview of DrOPD. (a) We compute teacher–student JS divergence at each rollout position and select the top-20% with the highest disagreement. (b) High-JS positions can exhibit under-coverage of teacher-supported modes, for which RKL provides weak corrective signals. (c) DrOPD retains RKL supervision at all positions and adds FKL supervision at the high-JS positions.

Method	AIME24	AIME25	AMC23	Minerva	Olympiad	MATH500	Mean(6)
Baselines
1.7B-Base	0.42	1.25	14.38	6.34	8.18	23.88	9.07
4B-Base	10.00	5.42	30.00	6.48	24.78	40.38	19.51
4B-GRPO	24.17	17.50	60.63	30.19	48.53	83.10	44.02
8B-GRPO	25.83	21.67	68.13	35.29	51.52	86.00	48.07
Student: 1.7B-Base Teacher: 4B-GRPO

Table 1: Mathematical reasoning results (Avg@8, %). Both panels select and average over all six benchmarks. Results marked with

\dagger

are reproduced by us.

Figure 3: Additional forward-KL accelerates early alignment. Changes in (a) FKL and (b) JS divergence during training. (c) Full-softmax teacher mass on the top-16 intersection.

Figure 4: Under-coverage recovery and learning progress. (a) Fraction of fixed high-JS positions with probability gaps at or above each threshold. (b) Severe under-coverage (

\delta_{t}\geq 0.8

) on the same positions. (c) Mean Avg@8 across six benchmarks at successive training checkpoints.

Figure 5: Sensitivity to routing and candidate support. Avg@8 averaged over four benchmarks for (a) the routed fraction, (b) top-

k

per model, and (c) candidate-set construction. The default uses top-20% JS routing with a top-16 union. The dashed line denotes OPD, equivalent to

\rho=0

Method	Remaining positions	Selected positions	Mean(4)
OPD (Full RKL)	RKL	RKL	33.80
Full FKL	FKL	FKL	34.16
Sparse RKL	–	RKL	34.94
Sparse FKL	–	FKL	33.29
DrOPD	RKL	RKL + FKL	35.64

Table 2: Effect of loss direction and supervision allocation. . Selected positions are the top 20% ranked by JS; remaining positions are the other 80%. “–” denotes no loss at those positions.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Training framework / rollout engine	verl / vLLM
Training data	DAPO-Math-17K
Training epochs	1
Prompt batch / PPO minibatch size	64 / 64
Responses per training prompt	4
Learning rate / weight decay	$10^{-6}$ / 0.01

Appendix

Table 3: Training and evaluation settings. Candidate top-

k

specifies the support used for distillation, rather than a restriction on rollout sampling.

Model pair	Method	Mean step time (s)	Cumulative step time (h)	Mean response length (tokens)
1.7B–4B	OPD	141.43	10.96	3117
1.7B–4B	DrOPD	152.62	11.83	3733

Appendix

Table 4: Training cost for the 1.7B–4B pair over 279 steps.

JS			RKL			FKL
$\tau$	$\eta$ (%)	Avg@8	$\tau$	$\eta$ (%)	Avg@8	$\tau$	$\eta$ (%)	Avg@8
0.5	0.21	37.81	3	0.70	40.31	2.2	0.54	36.56
0.4	0.59	40.62	2.6	0.97	43.12	1.9	0.79	40.62
0.3	1.48	45.94	2.2	1.33	46.88	1.6	1.22	44.06
0.2	3.24	49.06	1.8	1.97	45.62	1	2.80	46.25
0.18	3.93	51.88	1.4	3.11	50.62	0.7	4.41	52.50

Appendix

Table 5: Divergence-based cross-sampling on AMC23.

\tau

is the routing threshold;

\eta

is the measured intervention rate. Avg@8 is reported in percent.

Student entropy			Teacher entropy			Random
$\tau$	$\eta$ (%)	Avg@8	$\tau$	$\eta$ (%)	Avg@8	$r$	$\eta$ (%)	Avg@8
1.8	0.82	34.69	1.6	3.83	25.00	0.04	3.67	16.56
1.6	1.30	37.19	1.4	4.51	27.81	0.06	5.36	16.88
1.4	2.03	40.00	1.2	5.42	29.69	0.08	7.30	17.50
1.2	3.12	42.50	1	6.28	37.50	0.1	9.03	20.00
1	4.79	42.19	0.9	6.78	31.25	0.12	10.95	20.31

Appendix

Table 6: Entropy-based and random cross-sampling on AMC23.

r

is the random teacher-intervention probability; the remaining columns follow Table 5 .

Student entropy	Teacher entropy	Intervention (%)	Avg@8 (%)
High	High	7.19	27.81
High	Low	3.82	47.50
Low	High	1.20	18.75
Low	Low	0.16	16.25

Appendix

Table 7: Cross-sampling restricted to individual entropy regions. Each row is a separate policy on the same 40 AMC23 problems. The column order explicitly distinguishes student and teacher entropy.

Gap threshold $\tau$	Base	OPD	DrOPD
0.2	73.01	51.40	56.86
0.4	39.97	30.82	31.16
0.5	26.35	22.05	20.68
0.6	16.03	14.78	12.48
0.7	8.72	8.88	6.57
0.8	3.83	4.48	2.79

Appendix

Table 8: Fraction (%) of fixed original high-JS positions above each recorded-token gap threshold. Every row uses the same 97,122 positions.

Figure 6: Student probability mass on the top-16 intersection. Faint lines show raw step values and solid lines show trailing 15-step means. This complements the teacher-mass statistic in Figure 3c using the corresponding student probabilities.

On-Policy Distillation

On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.

See all 507 papers

Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
Sep 27, 2026Wenze Lin, Jiyuan Long, Jiale Zhao +11Data-Free Knowledge Distillation Language Model Distillation
LeapLab, Tsinghua University · Qiuzhen College, Tsinghua University · Beihang University +3
OPD+: Rethinking the Advantage Design for On-Policy Distillation
May 31, 2026Hanyang Zhao, Haoxian Chen, Han Lin +3Language Model Distillation Policy Gradient Methods
Columbia University · Amazon · Meta +1
Distillation as Probability Transport: Routed On-Policy Distillation
Sep 8, 2026Tianle Xia, Lingxiang Hu, Yiding Sun +6Language Model Distillation On-Policy Distillation
Tencent

Sep 27, 2026J/K move · Enter open · S save

Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?

Wenze Lin, Jiyuan Long, Jiale Zhao, Shenzhi Wang+10

LeapLab, Tsinghua University · Qiuzhen College, Tsinghua University · Beihang University+3

Data-Free Knowledge Distillation Language Model Distillation Multi-Teacher Knowledge Distillation On-Policy Distillation Knowledge Distillation

Open paper AskPDF arXiv

Since the advent of knowledge distillation, KL divergence has been the standard loss in distillation. Recently, on-policy distillation (OPD) has emerged as an efficient post-training paradigm for LLMs. As a distillation method, OPD naturally inherits KL divergence as its standard loss. However, in this work, we find that KL divergence may not be necessary for OPD. We show that simply preserving the update direction is sufficient for effective OPD. As long as the update direction is toward the teacher, OPD works. More precisely, it is not the direction of every token, but the direction of a small subset of tokens where the teacher and student disagree strongly. We first show that simply assigning a reward of (+1) to tokens where the teacher probability is higher than the student probability and (-1) where it is lower, which merely encourages updates toward the teacher, reproduces almost the same training mode as OPD with reverse KL. We further show that only the direction of a small subset of tokens with large teacher-student disagreement is critical, and training works as long as their update direction is toward the teacher, even if other tokens are pulled away from the teacher. And as an application of these findings, we introduce Consensus Multi-Teacher On-Policy Distillation (C-MOPD) to improve Multi-Teacher On-Policy Distillation (MOPD). Unlike MOPD, which routes each sample to a single teacher and may cause capability conflicts across domains, C-MOPD lets every sample be supervised by all teachers. Experiments show that C-MOPD consistently outperforms MOPD on both math and code benchmarks. Our code is available at https://github.com/LeapLabTHU/KL-Free-OPD.

Abstract

On-Policy Distillation

Explore similar work

Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?

OPD+: Rethinking the Advantage Design for On-Policy Distillation

Distillation as Probability Transport: Routed On-Policy Distillation