Scaling Properties of Same-Family On-Policy Distillation

Yuntai Bao; Qinfeng Li; Guoqing Jiang; Liwei Chen; Zhiheng Qin; Xuanping Li; Wenqi Zhang; Xuhong Zhang

cs.LGSep 26, 2026

Scaling Properties of Same-Family On-Policy Distillation

Authors: Yuntai Bao, Qinfeng Li, Guoqing Jiang, Liwei Chen, Zhiheng Qin, Xuanping Li, Wenqi Zhang, Xuhong Zhang

Organizations: Zhejiang University · Kuaishou Technology

Abstract

Reinforcement learning (RL) can induce substantial reasoning capabilities in large language models (LLMs), but how much of this capability transfers across model scales, and how quickly, remains unclear. We study the scaling properties of on-policy distillation (OPD) across weak-to-strong, same-base, and strong-to-weak teacher--student setups. We find that early OPD training dynamics uniformly exhibit a regular useful-transfer regime, in which held-out accuracy (the gold score, $G$ ) rises approximately linearly in $d=\sqrt{\mathrm{KL}(π_θ\Vert π_{\mathrm{ref}})}$ , the square root of token-level reverse KL divergence from the student initialization. In every observed weak-to-strong pair, the student's peak gold score exceeds its teacher's own, so a compact RL expert can transfer capability to a much larger student via OPD. To estimate OPD outcomes, we fit power laws for how $G_{\mathrm{peak}}$ and the slope of the useful-transfer regime scale with student and teacher parameter counts and with teacher gold score. These laws show that peak gold score improves with teacher scale only up to roughly the student's scale, and that at a matched gold score smaller teachers transfer better, so a teacher's score alone does not define its supervision value. We also study the scaling effects of two OPD variants, bootstrapping weak-to-strong OPD, and the degree of on-policy supervision.

Figures & tables

Figure 1: OPD transfers expertise from an RL-trained teacher to a student policy. Left: the three teacher–student setups, with post-RL experts as teachers and SFT bases as students. Center: gold score rises linearly with KL-indexed training progress $d$ during the useful-transfer phase, after which dynamics turn noisy. Right: the fitted joint power law predicts peak gold error from teacher and student scale.

Figure 2: Gold score for the 0.5B, 3B, and 14B students as teacher scale varies. Dashed lines show linear fits to the first 30 observations. Full results are in Figures 12 and 13 .

Figure 3: Fitted initial slope

m

across teacher scale, including only teachers no larger than each fixed student.

Figure 4: Peak gold score across model scale. Left: the complete 25-cell Vanilla grid. Middle: fixed-student Vanilla trends using teachers no larger than the student, with color-matched dashed levels marking each student’s direct-RL endpoint. Right: Vanilla peaks grouped by teacher, together with the direct-RL endpoint and the SFT baseline, on the same y axis as the middle panel. Parameter axes are logarithmic and all curves are one-run descriptions.

OPD method	$A$	$\alpha$	$\beta$	$\zeta$
Vanilla-OPD	0.97	0.30	$-0.27$	0.95
Delta-OPD	1.02	0.34	$-0.33$	1.01

Table 1: Fitted joint power-law coefficients ( Equation 6 ).

OPD method	Peak law	Leave-one-scale-out ( $\downarrow$ )	Extrapolation S / T ( $\downarrow$ )
Vanilla-OPD	Scales only ( Equation 7 )	3.43	.64 / .75
	Teacher score only	2.55	2.54 / .70
	Joint ( Equation 6 )	1.66	.55 / .68
Delta-OPD	Scales only ( Equation 7 )	2.47	.21 / .29
	Teacher score only	2.07	.93 / .46
	Joint ( Equation 6 )	0.82	.20 / .32

Table 4: Validation of the peak laws, in accuracy points. Leave-one-scale-out RMSE refits each law with all cells sharing one student or teacher scale withheld and predicts the withheld cells (mean over the two split types); extrapolation RMSE withholds the largest student (S) or teacher (T) scale. Protocols and further diagnostics are in appendix H . Best results are highlighted in bold.

Teacher	Teacher score	Observed peak	Scales only	Score only	Joint
1.5B endpoint (step 580)	63.6	77.5	77.0	75.6	77.1
3B intermediate (step 58)	66.0	73.8	79.5	76.3	74.1
3B endpoint (step 580)	73.6	80.3	79.5	78.7	79.6

Table 5: Out-of-sample validation of the Vanilla-OPD peak laws with an intermediate 3B teacher checkpoint, in gold-score points. Each row is a 7B-student cell; predictions use the full-fit coefficients of the scale-only ( Equation 7 ), score-only ( Equation 19 ), and joint ( Equation 6 ) laws. The intermediate-teacher cell is held out from every fit; only the joint law orders it correctly relative to the 1.5B endpoint teacher.

Figure 5: Trajectories of the intermediate-teacher validation. Gold score against

d

for the 7B student distilled from the 1.5B RL endpoint, the score-matched intermediate 3B checkpoint at step 58, and the 3B RL endpoint. Stars mark peaks and dashed levels each teacher’s own gold score.

Figure 6: Vanilla-OPD

Figure 9: Peak gold score against student scale for Delta-OPD, Vanilla-OPD, and direct RL, grouped by teacher.

Figure 10: Peak gold score under three degrees of on-policy supervision (0.5B expert as teacher).

Figure 11: Full 0.5B

\to

14B bootstrapped chains against direct baselines.

Symbol	Meaning
$x,{\mathcal{D}}$	Prompt and training-prompt distribution.
$y,y_{<t},y_{t},\|y\|$	Complete rollout, prefix before token $t$ , token $t$ , and rollout length.
$\pi_{\theta},\pi_{\mathrm{ref}}$	Student policy parameterized by $\theta$ and its fixed SFT reference.
$\pi_{\mathrm{T}},\pi_{\mathrm{T}}^{\mathrm{base}}$	RL-trained teacher and its pre-RL reference checkpoint.
$r_{t},r_{t}^{\Delta}$	Vanilla-OPD teacher-induced token reward and Delta-OPD teacher-shift reward.
$A_{t}^{\mathrm{V}},A_{t}^{\Delta}$	Advantages of Vanilla-OPD and Delta-OPD, equal to the token rewards under the zero-discount update.

Table 6: Summary of notations.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Qwen2.5-0.5B	Qwen2.5-1.5B	Qwen2.5-3B	Qwen2.5-7B	Qwen2.5-14B
Learning rate	2.00e-5	1.52e-5	1.28e-5	1.03e-5	8.70e-6
Batch size	200
Epochs	2
Samples per epoch	300K
Learning rate schedule	Linear decay to 1/10 maximum LR
Warmup ratio	3%

Appendix

Table 7: Hyperparameters for Qwen2.5 SFT models.

Hyperparameter	Value
Learning rate	1e-6
Learning rate schedule	Constant
Global batch size	256
Global mini-batch size	64
GRPO group size	8
Epochs	10

Appendix

Table 8: Hyperparameters for Qwen2.5 RL teacher models.

Hyperparameter	Value
Learning rate	1e-6
Learning rate schedule	Constant
Batch size	256
Epochs	$\leq$ 10
Max rollout length	2048
Optimizer	AdamW

Appendix

Table 9: Hyperparameters for OPD on Qwen2.5 models.

Hyperparameter	Value
Temperature	1.0
Top-p	1.0
Top-k	-1
Max rollout length	2048

Appendix

Table 10: Hyperparameters for evaluation.

Figure 12: Qwen2.5 mathematics trajectory grid with token-mean-KL ticks on square-root-spaced axes. Diamonds mark transfer endpoints (hollow when right-censored), and stars mark global gold-score maxima. Dotted purple curves show the teacher-induced proxy reward on right-hand axes for the pairs where it was logged. Rows share the student scale and columns share the teacher scale; panels within a row share axis ranges.

Figure 13: Delta-OPD trajectory grid over weak-to-strong and same-base cells with token-mean-KL ticks on square-root-spaced axes. Diamonds mark transfer endpoints (hollow when right-censored), and stars mark global gold-score maxima. Dotted purple curves show the token-mean Delta-OPD reward on right-hand axes, with the same ticks as Figure 12 . Rows share the student scale and columns share the teacher scale; panels within a row share axis ranges.

Pair	Relation	$m_{\mathrm{V}}$	$m_{\Delta}$	$R^{2}_{\Delta}$	Gain V	Gain $\Delta$	Endpoint
0.5B $\leftarrow$ 0.5B	Same-base	.573	.557	.985	23.3	22.6	Cens.
0.5B $\leftarrow$ 1.5B	S2W	.539	.570	.984	23.3	23.6	Cens.
1.5B $\leftarrow$ 0.5B	W2S	.564	.554	.984	14.2	16.8	Cens.
1.5B $\leftarrow$ 1.5B	Same-base	.721	.744	.981	24.6	24.6	Cens.
1.5B $\leftarrow$ 3B	S2W	.629	.723	.973	24.8	24.6	Cens.
3B $\leftarrow$ 0.5B	W2S	.378	.444	.963	8.2	12.4	Obs.

Appendix

Table 11: Per-pair Delta-OPD results. Peak gains are percentage points relative to each run’s own initialization; endpoint status is observed (Obs.) or right-censored (Cens.).

Figure 14: Direct-RL gold score against token-mean reverse KL. Each panel emphasizes the first four chronological observations, annotates the free-intercept local fit, and extrapolates it across the remaining trajectory. The black square marks the SFT initialization.

Figure 15: Student-scale SFT baseline. Points show equally weighted Vanilla step-zero means with within-scale standard deviations. The solid curve is the five-scale remaining-error fit in Equation 18 ; the dashed curve is refit through 7B and its open square is the 14B outer-scale prediction.

Method	Peak-error law	S/T RMSE	AICc
Vanilla	Multiplicative	1.81 / 1.60	$-155.6$
	Weighted additive	1.97 / 1.76	$-140.8$
	Equal additive	4.02 / 2.67	$-100.0$
Delta	Multiplicative	.91 / .85	$-108.3$
	Weighted additive	1.66 / 1.26	$-88.7$
	Equal additive	3.99 / 3.04	$-53.9$

Appendix

Table 12: Peak-error functional-form comparison. S/T are leave-one-scale-out RMSE over student and teacher scales, in percentage points.

Figure 16: Peak-performance scaling under the multiplicative and weighted-additive laws. Rows separate Vanilla-OPD and Delta-OPD. The left column fixes student scale and retains teachers no larger than the student; the right column fixes teacher scale. Points are observed peaks, solid curves are multiplicative-law fits, and dashed curves are weighted-additive fits over each series’ support.

Method	Target	Teacher law	Student exp.	Teacher exponents	Hold-out RMSE	AICc	$R^{2}$
Vanilla-OPD	Peak	Size	0.278	$\beta=0.147$	3.43	$-101.2$	0.915
		Perf.	0.269	$\zeta=0.425$	2.55	$-129.7$	0.967
		Joint	0.304	$\beta=-0.267,\ \zeta=0.952$	1.66	$-190.2$	0.996
	Rate	Size	0.240	$\delta=0.217$	19.1	$-13.0$	0.243
		Perf.	0.269	$\xi=0.697$	17.0	$-21.3$	0.427
		Joint	0.189	$\delta=-0.609,\ \xi=1.896$	16.3	$-28.2$	0.587

Appendix

Table 13: Teacher-size, teacher-performance, and joint laws for the peak and rate targets, fitted on the grid cells plus the auxiliary bootstrap-teacher cells (five for Vanilla-OPD, three for Delta-OPD). Leave-one-scale-out RMSE is the mean of the student-scale and teacher-scale hold-out splits, in accuracy points for the peak target and in points per unit

d

for the rate target. The teacher exponent acts on

\widetilde{N}_{T}^{\mathrm{eff}}

for the size law and on

1-G_{T}^{\mathrm{eff}}

for the performance law; the joint law fits both, with 95% bootstrap intervals in Table 16 .

Figure 17: Teacher size versus teacher performance, fitted on the grid plus the auxiliary bootstrap-teacher cells; rows are Vanilla-OPD (top) and Delta-OPD (bottom). Left two columns: leave-one-scale-out RMSE of the size, performance, and joint laws for the peak and rate targets. Third column: bootstrap joint-law teacher exponents for both targets, concentrated away from zero on anticorrelated ridges. Right: capped teacher size against capped-teacher error; red diamonds mark OPD-product teachers, which sit off the RL-endpoint size trend and break the collinearity.

Holdout	Method	Train/Test	Joint	Weighted additive	Joint teacher
Largest student	Vanilla	20/5	$.64\ [.33,2.27]$	$1.50\ [.67,3.60]$	$.55\ [.28,1.80]$
	Delta	12/5	$.21\ [.12,.89]$	$.92\ [.46,2.35]$	$.20\ [.11,.96]$
Largest teacher	Vanilla	20/5	$.75\ [.32,1.36]$	$1.06\ [.67,1.64]$	$.68\ [.33,1.25]$
	Delta	16/1	$.29\ [.04,.72]$	$.37\ [.02,1.49]$	$.32\ [.07,.68]$

Appendix

Table 14: Outer-scale peak extrapolation. Entries are accuracy-point RMSE with 95% training-cell-bootstrap intervals. Train counts refer to the size-only laws; the joint teacher law adds the chain cells below the held-out scale (four Vanilla-OPD and two Delta-OPD for the student holdout, five and three for the teacher holdout).

Figure 18: Peak-law extrapolation after fitting only smaller scales. Rows hold out the largest student or teacher scale; columns separate Vanilla-OPD and Delta-OPD. Stars are held-out peaks, solid curves are joint-law predictions, and dashed curves are weighted-additive predictions.

Method	Peak S/T	Rate S/T	Endpoint model
Vanilla	.55 / .68	.203 / .124	two-scale $C=.37$
Delta	.20 / .32	.177 / .104	two-scale $C=.34$

Appendix

Table 15: Scale-law validation under the joint teacher laws of Equation 6 : peak uses outer-scale holdout errors (S/T) in points; rate uses leave-one-scale-out RMSE (S/T) in accuracy per unit

d

Figure 19: Transfer-extent diagnostics. Observed departures or censored lower bounds against the accelerated-failure-time fits of Equation 20 for Vanilla-OPD (top) and Delta-OPD (bottom).

Method	Target	Coefficients
Vanilla	Peak error	$A=.97\,[.93,1.04],\ \alpha=.304\,[.286,.317],$
		$\beta=-.267\,[-.311,-.229],\ \zeta=.952\,[.898,1.035]$
Delta	Peak error	$A=1.02\,[.93,1.11],\ \alpha=.336\,[.327,.351],$
		$\beta=-.325\,[-.372,-.276],\ \zeta=1.009\,[.904,1.114]$
SFT	Baseline	$A_{\mathrm{SFT}}=.682\,[.680,.683],\ \alpha_{\mathrm{SFT}}=.326\,[.324,.328]$
Vanilla	Rate	$B=.116\,[.050,.263],\ \gamma=.189\,[.011,.306],$

Appendix

Table 16: Power-law coefficients with 95% cell-bootstrap intervals. Peak and rate report the joint teacher laws of Equation 6 ; intervals quantify cross-pair variation rather than seed uncertainty. Extent rows report the median-budget laws of Equation 20 ,

\operatorname{median}(d_{\mathrm{transfer}})=C\widetilde{N}_{S}^{-u}\widetilde{N}_{T}^{\,v}

, with the teacher scale capped or uncapped as marked.

One-sided band	Two deviations	Three deviations	Four deviations
90%	12	9	9
95%	9	9	9
99%	8	8	8

Appendix

Table 17: Endpoint-rule sensitivity: number of Vanilla-OPD trajectories with an observed departure from the initial line.

Figure 20: Gold-score dynamics of the five Vanilla-OPD and three Delta-OPD bootstrap chains. Stars mark the observed peak.

Figure 21: Peak gold score under three degrees of on-policy supervision, per student, over weak-to-strong and same-base cells. Dash-dotted levels mark SFT initializations.

Figure 22: Cold-start OPD against pure OPD on the 14 shared cells. Each trajectory uses the reverse KL from its own initialization, so the two curves in a panel have different reference policies.

On-Policy Distillation

On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.

See all 511 papers

Jul 28, 2026

Weak-to-Strong On-Policy Distillation

On-policy distillation (OPD), which aligns a student with the teacher's token-level distribution on the student's own rollouts, is an effective paradigm for transferring capabilities across LLMs. Prevailing approaches assume a teacher at least as capable as the student: they either distill a larger model into a smaller one, which fails at the frontier where no larger teacher exists, or consolidate multiple domain experts trained from a shared base, which requires costly training at the student's scale. We introduce Weak-to-Strong On-Policy Distillation (W2S-OPD), a simple yet effective OPD framework that improves the strong student by distilling from multiple weak models. W2S-OPD constructs a proxy teacher in logit space from a contrast pair of a positive and a negative model, both smaller than the student and cheap to obtain. Their logit difference isolates the capability direction, which is added to the student's own base model, yielding a proxy teacher that couples this direction while staying distributionally adjacent to the student. The student then distills it by minimizing the per-token reverse KL on its own rollouts. We instantiate the contrast pair as i) a post-RL expert against its pre-RL initialization, isolating the skill RL instills, ii) a larger against a smaller base model, isolating the capability from scale, and iii) a small base model with correct versus wrong hints, isolating the instance-level direction toward the solution. Across four math and three code benchmarks, W2S-OPD outperforms OPD, enables the student to surpass the domain teacher, and keeps improving the student even when every supervision source is weaker. Analysis shows different contrasts yield distinct signals: the post-RL and hint contrasts emphasize reasoning frameworks, while the scale contrast emphasizes the solving procedure. Our code will be available at https://github.com/Yu-Fangxu/W2S-OPD.

Fangxu Yu, Weijia Xu, Michael Xu +2

University of Maryland, College Park · Microsoft Research · MBZUAI

Language Model Distillation On-Policy Distillation Knowledge Distillation

Ask PDF

Apr 14, 2026

Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

On-policy distillation (OPD) has become a core technique in the post-training of large language models, yet its training dynamics remain poorly understood. This paper provides a systematic investigation of OPD dynamics and mechanisms. We first identify that two conditions govern whether OPD succeeds or fails: (i) the student and teacher should share compatible thinking patterns; and (ii) even with consistent thinking patterns and higher scores, the teacher must offer genuinely new capabilities beyond what the student has seen during training. We validate these findings through weak-to-strong reverse distillation, showing that same-family 1.5B and 7B teachers are distributionally indistinguishable from the student's perspective. Probing into the token-level mechanism, we show that successful OPD is characterized by progressive alignment on high-probability tokens at student-visited states, a small shared token set that concentrates most of the probability mass (97%-99%). We further propose two practical strategies to recover failing OPD: off-policy cold start and teacher-aligned prompt selection. Finally, we show that OPD's apparent free lunch of dense token-level reward comes at a cost, raising the question of whether OPD can scale to long-horizon distillation.

Yaxuan Li, Yuxin Zuo, Bingxiang He +8

Tsinghua University · ShanghaiTech University · University of Illinois Urbana-Champaign +1

Language Model Distillation On-Policy Distillation Knowledge Distillation Teacher-Student Learning

Ask PDF

Jun 21, 2026

On the Position Bias of On-Policy Distillation

On-Policy Distillation (OPD) improves the learning efficiency of standard reinforcement learning through dense, token-level supervision from teachers. In the standard KL objective of OPD, token-level losses are uniformly averaged, implying equal weights for all tokens. However, we discover that not all tokens are created equal: as student rollouts grow longer, they deviate further from the teacher's distribution, leading to degraded supervision quality at later positions. As a result, OPD using only the first 30% of tokens can perform comparably to using all tokens, whereas OPD using only the last 30% of tokens barely learns anything. In this work, we provide a principled understanding of this issue through the lens of constrained optimization. Based on these insights, we derive Importance-Weighted On-Policy Distillation (IW-OPD), in which the weight assigned to each token depends on the accumulated discrepancy between the student's and teacher's distributions, naturally upweighting earlier tokens and downweighting later ones with larger deviations. We show that IW-OPD converges significantly faster than OPD, with better learning efficiency, and achieves better final performance than standard OPD in both same-size and cross-scale settings, improving performance up to 6.9 points on AIME-2025.

Yan Xie, Sijie Zhu, Tiansheng Wen +2

Xidian University · Georgia Institute of Technology · Amazon AGI SF Lab

Importance Weighting Language Model Distillation On-Policy Distillation RL Post-Training Teacher-Student Learning Policy Learning

Ask PDF

Abstract

On-Policy Distillation

Explore similar work

Weak-to-Strong On-Policy Distillation

Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

On the Position Bias of On-Policy Distillation