Rethinking Relative Policy Differences Transfer of On-Policy Distillation

Rong Wu; Daocheng Fu; Yu Yang; Xuemeng Yang; Siqi Li; Shu Zou; Yao Zhu; Botian Shi

cs.LGOct 9, 2026

Rethinking Relative Policy Differences Transfer of On-Policy Distillation

Authors: Rong Wu, Daocheng Fu, Yu Yang, Xuemeng Yang, Siqi Li, Shu Zou, Yao Zhu, Botian Shi,

Organizations: Zhejiang University · Shanghai Artificial Intelligence Laboratory · Fudan University · The Australian National University ·

Abstract

On-policy distillation (OPD) trains a target policy on its own rollouts using token-level supervision from a stronger teacher, but typically relies on the teacher being substantially more capable than the target. Recent work relaxes this requirement by transferring the relative policy difference between an expert and a reference policy rather than the expert's absolute distribution. Yet it remains unclear which differences provide useful supervision and which targets can benefit from them. We develop a unified view in which the log-probability ratio between a proxy expert and a proxy reference defines an implicit reward for KL-regularized policy optimization. This formulation induces an on-policy update for the target, recovers conventional OPD as a special case, and enables any-to-any transfer across models that differ in scale, training stage or generation. Our systematic experiments yield three findings. First, a stronger expert is not necessarily a better source of supervision: differences induced by task-specific training, including reinforcement learning and supervised fine-tuning, transfer consistently, whereas those arising from model scale alone transfer weakly or not at all. Second, the expert and reference serve complementary roles: the expert provides the capability to be transferred, while the reference filters out shared behavior and isolates the relevant residual. Third, even a useful difference does not transfer to every target: success requires the induced update to move the target toward the expert and generalize beyond sampled rollouts. Raw policy distance does not predict transferability, whereas an appropriate reference can rescue otherwise unsuccessful transfer. These results characterize when relative policy differences provide effective supervision for OPD.

Figures & tables

Method	$\pi_{\phi}^{+}$	$\pi_{\phi}$	$\pi_{\mathrm{ref}}$	$\alpha/\beta$	Advantage
OPD	$T^{+}$	$S$	$S$	$1$	$\log\pi_{T^{+}}-\log\pi_{\theta}$
ExOPD	$T^{+}$	$S$	$S$	$\lambda$	$\lambda(\log\pi_{T^{+}}-\log\pi_{S})-(\log\pi_{\theta}-\log\pi_{S})$
Relative Policy Transfer	$T^{+}$	$T$	$S$	$\lambda$	$(\log\pi_{T^{+}}-\log\pi_{T})-\lambda(\log\pi_{\theta}-\log\pi_{S})$

Table 1: Existing OPD methods as special cases of any-to-any policy transfer (Eq. 7 ). Here,

T^{+}

T

, and

S

denote the teacher expert, teacher base, and student base, respectively.

Source	Proxy expert / reference	Proxy performance ( $G_{P}$ )	Target performance ( $G_{T}$ )	$K_{\mathrm{Induced}}$
Task-optimized differences
RL	Qwen3-1.7B-RL / Qwen3-1.7B	$8.92\!\rightarrow\!25.15$ ( $+16.23$ )	$18.43\!\rightarrow\!28.21$ ( $+9.78$ )	0.0773
RL	Qwen3-4B-RL / Qwen3-4B	$17.82\!\rightarrow\!44.71$ ( $+26.89$ )	$18.43\!\rightarrow\!45.27$ ( $+26.84$ )	0.1066
RL	Skywork-7B / Qwen2.5-7B	$1.79\!\rightarrow\!43.93$ ( $+42.14$ )	$18.43\!\rightarrow\!50.66$ ( $+32.23$ )	0.2292
RL †	Qwen3-4B-RL / Qwen3-1.7B	$8.92\!\rightarrow\!44.71$ ( $+35.79$ )	$18.43\!\rightarrow\!44.28$ ( $+25.85$ )	0.3531
SFT	Qwen3-4B-SFT / Qwen3-4B	$17.82\!\rightarrow\!36.03$ ( $+18.21$ )	$18.43\!\rightarrow\!36.62$ ( $+18.19$ )	0.1725

Table 2: Transfer results for policy differences from different sources, with Qwen3-8B fixed as the target. † Alternative reference: the proxy reference is not the expert’s training origin.

Panel of Figure 1 — Figure 1: Transferability across sources of policy difference.

Proxy expert	Proxy reference	Proxy performance ( $G_{P}$ )	Target performance ( $G_{T}$ )	$K_{\mathrm{Base}}$	$R_{\mathrm{perf}}$	$\mathcal{O}_{E,R}$
Varying the expert; reference = Qwen3-4B-Base
Qwen3-8B	Qwen3-4B-Base	$17.82\!\rightarrow\!18.43$ ( $+0.61$ )	$18.43\!\rightarrow\!17.40$ ( $-1.03$ )	0.0533	$-1.680$	–
Qwen3-14B	Qwen3-4B-Base	$18.43\!\rightarrow\!19.54$ ( $+1.72$ )	$18.43\!\rightarrow\!18.45$ ( $+0.02$ )	0.0550	$0.016$	–
Qwen3-4B-SFT	Qwen3-4B-Base	$17.82\!\rightarrow\!36.03$ ( $+18.21$ )	$18.43\!\rightarrow\!36.62$ ( $+18.19$ )	0.0708	0.999	–
Qwen3-4B-Early-RL	Qwen3-4B-Base	$17.82\!\rightarrow\!39.91$ ( $+22.09$ )	$18.43\!\rightarrow\!40.85$ ( $+22.42$ )	0.0465	1.015	–
Qwen3-4B-RL	Qwen3-4B-Base	$17.82\!\rightarrow\!44.71$ ( $+26.89$ )	$18.43\!\rightarrow\!45.27$ ( $+26.84$ )	0.0484	0.998	0.839

Table 3: Effects of the proxy expert and reference on transfer . The shaded row is equivalent to standard OPD. Full policy-distance statistics are in Appendix B .

Figure 3: Effects of the proxy expert and reference on transfer. Qwen3-8B is fixed as the target.

Proxy Expert	Proxy Reference	Target	$K_{\text{Base}}$	$G_{T}$	$R_{\mathrm{perf}}$	Avg
Qwen3-1.7B-RL	Qwen3-1.7B	Qwen3-1.7B	0.000	16.11	- 0.993	25.03
Qwen3-1.7B-RL	Qwen3-1.7B	Qwen3-4B	0.106	13.65	- 0.841	31.47
Qwen3-1.7B-RL	Qwen3-1.7B	Qwen3-8B	0.111	9.78	- 0.603	28.21
Qwen3-1.7B-RL	Qwen3-1.7B	Qwen2.5-7B	0.462	-1.64	-0.101	0 0.15
Qwen3-1.7B-RL	Qwen3-1.7B	Qwen2.5-7B-Ins	0.656	-1.69	-0.104	0 5.17
Qwen3-1.7B-RL	Qwen3-1.7B	Qwen2.5-7B-Math	0.628	-0.08	-0.005	0 0.69

Table 4: Transfer results across proxy pairs and targets. The shaded rows denote Qwen2.5-generation targets.

Figure 4: The outcome is not a function of policy distance. Performance realization ratio (left) and transferred math accuracy (right) against the proxy–target base KL.

Panel of Figure 5 — Figure 5: Benefiting targets move toward the proxy expert (fixed Qwen3-1.7B proxy pair).

Panel of Figure 6 — Figure 6: Failure is not undertraining, and matching the reference generation recovers the benefit.

			Math		Code
Proxy Expert	Proxy Reference	Target	Avg	$G_{T}$	Avg	$G_{T}$
Same-generation transfer (Qwen3 $\rightarrow$ Qwen3)
Qwen3-4B-RL	Qwen3-4B	Qwen3-8B	45.27	$+26.8$	56.44	$+2.3$
Qwen3-4B-RL	Qwen3-1.7B	Qwen3-8B	44.28	$+25.9$	56.03	$+1.9$
Cross-generation transfer (Qwen3 proxy $\rightarrow$ Qwen2.5 target)
Qwen3-4B-RL	Qwen3-4B	Qwen2.5-7B	0 0.05	$-1.7$	22.83	$-6.4$

Table 5: Representative successful and failed settings performance on both math and code.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Table 6: Base model and proxy expert performance.

Table 7: Configurations, validation results, and distributional and efficiency metrics for policy difference experiments.

Table 8: Configurations, validation results, and distributional and efficiency metrics for varying proxy reference while fixed proxy expert.

Table 9: Configurations, validation results, and distributional and efficiency metrics for varying proxy expert while fixed proxy reference.

Table 10: Configurations, validation results, and distributional and efficiency metrics for varying target while fixed useful differences.

On-Policy Distillation

On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.

See all 517 papers

Oct 16, 2026

SimOPD: Relational Knowledge Transfer for On-Policy Distillation

A key challenge in on-policy distillation (OPD) is extracting effective supervision from teacher knowledge. Supervision from token probabilities may underutilize internal knowledge, while direct hidden-state matching forces the student to shift to the teacher's feature space and complicates cross-architecture transfer. Inspired by contrastive learning's use of relationships between instances, we propose SimOPD to transfer teacher internal knowledge through response similarities. Specifically, SimOPD uses paired memory banks of student-generated responses to extract supervision from the teacher's internal representations as similarity distributions. The student then matches these distributions within its own representation space, retaining representational freedom and removing the need for cross-model projection. Extensive experiments show that SimOPD improves the overall average over the strongest baselines by up to 2.6 and 3.1 percentage points in same- and cross-architecture settings.

Kai Tang, Xing Zheng, Haobo Wang +6

Zhejiang University · Ant Group

Language Model Distillation On-Policy Distillation Relational Knowledge Distillation Cross-Architecture Knowledge Distillation

Ask PDF

Sep 29, 2026

Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives

On-policy distillation (OPD) learns from teacher feedback on student-generated responses and has shown promise in reducing forgetting relative to supervised fine-tuning (SFT). However, its benefits and fragility remain incompletely understood. We study sequential distillation from multiple teachers, where the student minimizes its average divergence from the teachers. Forward Kullback--Leibler (KL) divergence yields a weighted arithmetic mixture, while reverse KL yields a normalized weighted geometric aggregate. We develop algorithms that learn these targets under off-policy and on-policy feedback, respectively, establishing logarithmic regret bounds in the tabular setting and extending the analysis to function approximation. By analyzing these aggregation targets, we identify mechanisms that help explain both the benefits and fragility of OPD. Relative to forward KL, reverse KL can better retain a confident expert's preferences under uninformative feedback, but is more sensitive to teachers that assign very low probabilities to correct responses. Its token-level conditionals also reveal a dependence on continuation distributions that can favor incorrect prefixes over long horizons.

Qiwei Di, Xuheng Li, Kaixuan Ji +3

Department of Computer Science, University of California, Los Angeles, CA 90095, USA

Multi-Teacher Knowledge Distillation On-Policy Distillation Policy Learning Knowledge Distillation Kullback-Leibler Divergence

Ask PDF

Sep 26, 2026

Scaling Properties of Same-Family On-Policy Distillation

Reinforcement learning (RL) can induce substantial reasoning capabilities in large language models (LLMs), but how much of this capability transfers across model scales, and how quickly, remains unclear. We study the scaling properties of on-policy distillation (OPD) across weak-to-strong, same-base, and strong-to-weak teacher--student setups. We find that early OPD training dynamics uniformly exhibit a regular useful-transfer regime, in which held-out accuracy (the gold score,

G

) rises approximately linearly in

d=\sqrt{\mathrm{KL}(π_θ\Vert π_{\mathrm{ref}})}

, the square root of token-level reverse KL divergence from the student initialization. In every observed weak-to-strong pair, the student's peak gold score exceeds its teacher's own, so a compact RL expert can transfer capability to a much larger student via OPD. To estimate OPD outcomes, we fit power laws for how

G_{\mathrm{peak}}

and the slope of the useful-transfer regime scale with student and teacher parameter counts and with teacher gold score. These laws show that peak gold score improves with teacher scale only up to roughly the student's scale, and that at a matched gold score smaller teachers transfer better, so a teacher's score alone does not define its supervision value. We also study the scaling effects of two OPD variants, bootstrapping weak-to-strong OPD, and the degree of on-policy supervision.

Yuntai Bao, Qinfeng Li, Guoqing Jiang +5

Zhejiang University · Kuaishou Technology

On-Policy Distillation Language Model Distillation Knowledge Distillation Weak-to-Strong Generalization

Ask PDF

Abstract

On-Policy Distillation

Explore similar work

SimOPD: Relational Knowledge Transfer for On-Policy Distillation

Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives

Scaling Properties of Same-Family On-Policy Distillation