Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?

Wenze Lin; Jiyuan Long; Jiale Zhao; Shenzhi Wang; Xitai Jiang; Ce Luo; Rui Lan; Qianli Ma

cs.LGSep 27, 2026

Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?

Authors: Wenze Lin, Jiyuan Long, Jiale Zhao, Shenzhi Wang, Xitai Jiang, Ce Luo, Rui Lan, Qianli Ma, +6 more

Organizations: LeapLab, Tsinghua University · Qiuzhen College, Tsinghua University · Beihang University · National University of Singapore · The Chinese University of Hong Kong · E Fund Management Co., Ltd.

Abstract

Since the advent of knowledge distillation, KL divergence has been the standard loss in distillation. Recently, on-policy distillation (OPD) has emerged as an efficient post-training paradigm for LLMs. As a distillation method, OPD naturally inherits KL divergence as its standard loss. However, in this work, we find that KL divergence may not be necessary for OPD. We show that simply preserving the update direction is sufficient for effective OPD. As long as the update direction is toward the teacher, OPD works. More precisely, it is not the direction of every token, but the direction of a small subset of tokens where the teacher and student disagree strongly. We first show that simply assigning a reward of (+1) to tokens where the teacher probability is higher than the student probability and (-1) where it is lower, which merely encourages updates toward the teacher, reproduces almost the same training mode as OPD with reverse KL. We further show that only the direction of a small subset of tokens with large teacher-student disagreement is critical, and training works as long as their update direction is toward the teacher, even if other tokens are pulled away from the teacher. And as an application of these findings, we introduce Consensus Multi-Teacher On-Policy Distillation (C-MOPD) to improve Multi-Teacher On-Policy Distillation (MOPD). Unlike MOPD, which routes each sample to a single teacher and may cause capability conflicts across domains, C-MOPD lets every sample be supervised by all teachers. Experiments show that C-MOPD consistently outperforms MOPD on both math and code benchmarks. Our code is available at https://github.com/LeapLabTHU/KL-Free-OPD.

Figures & tables

Student	Teacher	Dataset
Math
DeepSeek-Distill-Qwen-1.5B	JustRL-1.5B	DAPO-Math-17k
Qwen3-1.7B-Base	Qwen3-4B-Base-RL	DAPO-Math-17k
Llama-3.2-3B-Instruct	GT-Llama3.2-3B-MATH	DeepMath
Qwen3-4B-Non-Thinking	Qwen3-4B-Non-Thinking-RL-Math	DeepMath
Qwen3-30B-A3B-Non-Thinking	Qwen3-30B-A3B-Instruct-2507	DeepMath

Table 1: Student-teacher pairs and training datasets used in our experiments.

DeepSeek-Distill-Qwen-1.5B (Student) vs. JustRL-1.5B (Teacher)
Method	AIME24	AIME25	AMC	MATH500	Minerva	Olympiad	Avg
Student	23.7	17.0	59.7	80.9	23.7	41.4	41.0
Teacher	46.2	37.5	82.0	89.2	30.7	53.5	56.5
OPD	43.8	32.5	78.8	88.5	31.7	52.6	54.7
BinaryOPD	45.4	32.1	78.5	88.9	31.6	52.6	54.9
Qwen3-1.7B-Base (Student) vs. Qwen3-4B-Base-RL (Teacher)

Table 2: Math benchmark results. All results are averaged over 8 samples (Avg@8).

Qwen3-4B-Non-Thinking (Student) vs. Qwen3-4B-Non-Thinking-RL-Code (Teacher)
Method	LiveCodeBench v6	HumanEval	MBPP	Avg
Student	25.1	80.8	48.7	51.5
Teacher	36.0	90.6	56.2	60.9
OPD	27.9	85.4	53.0	55.4
BinaryOPD	27.8	85.7	52.1	55.2
DeepSeek-Distill-Qwen-1.5B (Student) vs. Nemotron-Research-Reasoning-1.5B (Teacher)

Table 3: Code benchmark results. All results are averaged over 8 samples (Avg@8).

Figure 1: Training dynamics of OPD and BinaryOPD on one math student-teacher pair and one code pair. Full training dynamics are provided in Appendix D .

Figure 2: Summary of the positive and negative reward experiments. The positive experiment shows that keeping only Group A or Groups A+B leads to successful training, while adding Group C causes failure. The negative experiment exhibits a symmetric pattern.

Figure 3: Training dynamics for the positive and negative reward experiments with

\epsilon=0.8

, using Qwen3-4B-Non-Thinking-RL-Math as the teacher and Qwen3-4B-Non-Thinking as the student.

Method	AIME24	AIME25	AMC	MATH500	Minerva	Olympiad	Avg
OPD	60.4	51.7	85.7	93.0	34.5	59.2	64.1
BinaryOPD	61.3	55.0	85.7	92.8	35.9	59.8	65.1
ALL+1(A)	60.8	52.0	85.9	92.9	35.0	59.5	64.5
ALL+1(A+B)	60.5	51.5	86.0	92.7	34.8	59.0	64.3
ALL+1(A+B+C)	24.0	15.5	56.0	79.5	26.0	41.0	40.3

Table 4: Math benchmark results for the positive reward experiment (avg@8, %). Note that ALL+1(A) keeps less than

1.5\%

of all tokens.

	Math						Code
Model	AIME24	AIME25	AMC	MATH500	Minerva	Olympiad	LCB v6	HumanEval	MBPP
Student	25.0	16.7	58.4	81.3	27.7	42.9	25.1	80.8	48.7
Math Teacher	61.2	56.3	88.0	93.2	36.7	60.5	–	–	–
Code Teacher	–	–	–	–	–	–	36.0	90.6	56.2
MOPD	60.0	52.5	86.4	92.9	34.3	58.8	27.9	85.9	53.5
C-MOPD ( $\epsilon=0.0$ )	59.8	52.8	86.2	92.8	34.6	58.9	28.2	85.7	53.8

Table 5: Math and code benchmark results for C-MOPD (avg@8, %).

Figure 4: Training dynamics of C-MOPD.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: State-space coverage grows slowly despite a rapidly expanding state library. For each current state, we search only among previously visited states from different prompts. Left: fraction of current-step states whose closest earlier neighbor exceeds cosine thresholds

0.9

0.95

, and

0.99

, as a function of library size. Expanding the library by

29\times

raises

\cos\geq 0.99

coverage from

10.4\%

18.4\%

. Right: mean similarity to the closest earlier state increases from

0.823

0.891

Figure 6: The geometry of visited states remains stable across training. Mean cross-prompt cosine similarity between states collected at training steps $i$ and $j$ . All off-diagonal entries lie in the narrow range $[0.138,0.149]$ , with no clear separation between early and late training steps.

Figure 7: Strong cross-prompt nearest-neighbor redundancy. Distribution of nearest-neighbor cosine similarity over all

61{,}440

states, where the neighbor is required to originate from a different prompt. The mean nearest-neighbor similarity is

0.893

;

29.5\%

of states have a neighbor with cosine similarity at least

0.95

, and

18.5\%

have one at least

0.99

	mean	median	p90	p95	$\%\geq 0.99$	$\%\geq 0.95$
Cross-prompt NN	0.893	0.901	0.994	0.995	18.5%	29.5%

Appendix

Table 6: Cross-prompt nearest-neighbor similarity over all

61{,}440

states. The last two columns report the fraction of states whose nearest different-prompt neighbor exceeds the corresponding threshold.

DeepSeek-Distill-Qwen-1.5B (Student) vs. JustRL-1.5B (Teacher)
Method	AIME24	AIME25	AMC	MATH500	Minerva	Olympiad	Avg
Student	23.7	17.0	59.7	80.9	23.7	41.4	41.0
Teacher	46.2	37.5	82.0	89.2	30.7	53.5	56.5
OPD	43.8	32.5	78.8	88.5	31.7	52.6	54.7
BinaryOPD	45.4	32.1	78.5	88.9	31.6	52.6	54.9
Verifier-Aligned	44.6	31.3	78.0	88.7	31.3	52.6	54.4

Appendix

Table 7: Math benchmark results. All results are averaged over 8 samples (Avg@8).

Qwen3-4B-Non-Thinking (Student) vs. Qwen3-4B-Non-Thinking-RL-Code (Teacher)
Method	LiveCodeBench v6	HumanEval	MBPP	Avg
Student	25.1	80.8	48.7	51.5
Teacher	36.0	90.6	56.2	60.9
OPD	27.9	85.4	53.0	55.4
BinaryOPD	27.8	85.7	52.1	55.2
Verifier-Aligned	28.1	84.6	52.2	55.0

Appendix

Table 8: Code benchmark results. All results are averaged over 8 samples (Avg@8).

Hyperparameter	Value
Learning Rate	1e-6
Train Batch Size	256
Max Response Length (Training)	8192
Max Response Length (Evaluation)	16384
Max Prompt Length	1024
Rollout Temperature	1.0

Appendix

Table 9: Hyperparameter settings.

Figure 8: Full training dynamics of all methods (OPD, BinaryOPD, Verifier-Aligned, Verifier-Conflicted) across all student-teacher pairs and datasets. From left to right: training accuracy, entropy, response length, and gradient norm.

Figure 9: Full training dynamics for the positive reward experiment (ALL+1) across different

\epsilon

values and student-teacher pairs.

Figure 10: Full training dynamics for the negative reward experiment (ALL-1) across different

\epsilon

values and student-teacher pairs.

Figure 11: Visualization of high-disagreement tokens. Blue tokens have $\log(p_{T}/p_{s})>0.8$ and red tokens have $\log(p_{T}/p_{s})<-0.8$ .

Explore similar work

Unbiased Top- $k$ Estimation for On-Policy Distillation
Sep 28, 2026Linjian Meng, Siyuan Gan, YuHan Li +5Efficient On-Policy Distillation Kullback-Leibler Divergence
Shanghai Artificial Intelligence Laboratory, Shanghai, China · State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China
KL for a KL: On-Policy Distillation with Control Variate Baseline
May 8, 2026Minjae Oh, Sangjun Song, Gyubin Choi +2Kullback-Leibler Divergence Post-Training
Graduate School of Data Science Seoul National University
Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives
Sep 29, 2026Qiwei Di, Xuheng Li, Kaixuan Ji +3Kullback-Leibler Divergence Feedback
Department of Computer Science, University of California, Los Angeles, CA 90095, USA

Sep 28, 2026, cs.CLJ/K move · Enter open · S save

Unbiased Top- $k$ Estimation for On-Policy Distillation

Linjian Meng, Siyuan Gan, YuHan Li, Xiran Wang+4

Shanghai Artificial Intelligence Laboratory, Shanghai, China · State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China

Efficient On-Policy Distillation Kullback-Leibler Divergence Top-K

Open paper AskPDF arXiv

On-policy distillation (OPD) is becoming an important component of large language model (LLM) post-training for transferring the reasoning capability of a strong teacher LLM to a weaker student LLM. OPD trains the student by minimizing the reverse KL divergence between the teacher and the student via rollouts generated by the student's policy. However, estimating the gradient of the reverse KL divergence in OPD remains a challenge. Using only the sampled token from the student-generated rollout is computationally cheap but provides limited distributional supervision, which will degrade accuracy. In addition, using the full vocabulary provides complete distributional supervision but is computationally expensive. Therefore, recent works propose Top-

k

OPD (TK-OPD) that use selected top-

k

tokens, which provides richer distributional supervision than sampled-token estimation at substantially lower computational cost than full-vocabulary estimation. Unfortunately, using only the selected top-

k

tokens induces bias, leading to accuracy degradation, as the probability mass outside the selected top-

k

tokens is discarded. To address the bias of TK-OPD, we propose Tail-Corrected Top-

k

On-Policy Distillation (TT-OPD). It preserves the advantages of TK-OPD, including rich distributional supervision and low computational cost, while providing an unbiased estimator of the gradient of the reverse KL divergence. The key insight of TT-OPD is to use not only the selected top-

k

tokens, but also the sampled token from the student-generated rollout, thereby recovering the discarded probability mass in expectation, avoiding the bias. Experimental results demonstrate that TT-OPD significantly outperforms other tested OPD variants.

Abstract

Explore similar work

Unbiased Top-kkk Estimation for On-Policy Distillation

KL for a KL: On-Policy Distillation with Control Variate Baseline

Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives

Unbiased Top- $k$ Estimation for On-Policy Distillation