Learning Dynamics of On-Policy Distillation

Guan Zhe Hong; Sumeet Ramesh Motwani; Liang Chen; Yangchen Pan; Junchi Yu; Jialin Yu; Philip Torr; Yi Ren

cs.LGNov 5, 2026

Learning Dynamics of On-Policy Distillation

Authors: Guan Zhe Hong, Sumeet Ramesh Motwani, Liang Chen, Yangchen Pan, Junchi Yu, Jialin Yu, Philip Torr, Yi Ren

Organizations: University of Oxford · EPFL · City University of Hong Kong

Abstract

On-policy distillation (OPD) is emerging as an important paradigm for language-model post-training, combining dense teacher supervision with training on student-generated trajectories. This creates a feedback loop where each update changes the student and thereby the training distribution that generates subsequent updates. Yet, how this feedback shapes learning and generalization remains poorly understood. We conduct a mechanistic analysis of OPD's learning dynamics, distinguishing the two feedback loop stages: conversion of the distillation signal into parameter updates, and broadcast through which these updates change predictions across prefixes and reshape the rollout distribution. This framework helps us understand both limited student responsiveness despite substantial teacher-student disagreement, and degeneration when updates that improve local teacher agreement also increase the probability of unsuccessful continuations, exposing the student to unreliable supervision in subsequent rollouts. Finally, we show that offline distillation can prepare a student for OPD without significantly improving immediate held-out accuracy. While OPD alone may degrade generation, it can yield substantial accuracy gains after sufficient offline preparation. Together, our results connect teacher supervision, student geometry, and the evolving rollout distribution in determining the learning dynamics of OPD and its effects on downstream performance.

Figures & tables

Figure 1: OPD’s learning dynamics, visualizing the feedback loop at the top (detailed in Section 3.1 ), and illustrative successful and failing processes at the bottom in reasoning settings we analyzed empirically. In principle, OPD propagates current teacher requests

\Gamma_{t}(s)|_{s\sim\mu_{t}}

to motion at any future prefix

x

through shared parameters, captured by the Neural Tangent Kernel

K(x,s)

. Such policy motion shapes how the autoregressive rollout (prefix visitation) and supervision distribution changes (

\dot{\mu}_{t},\dot{\Gamma}_{t}

). In practice, a healthy dynamics needs not only locally “tangent-accessible” and coherent requests ( Section 3.2 ), but also the student to be sufficiently “prepared”, so that it can sustain meaningful rollouts with learning value as it absorbs corrections requested by the teacher, instead of primarily performing local mimicry and falling into chaotic rollouts hard for the teacher to rescue, let alone improve on ( Sections 3.3 and 3.4 ).

Figure 2: Initial/end AIME accuracy; diamonds mark teachers. Red: sibling-taught; blue: larger-teacher taught.

Figure 3: Teacher request fields differ systematically. In (a), each curve shows how much more (or less) often a semantic or token-position class appears among a teacher’s top 50%, 20%, …, 1% of requests than among all annotated positions, in percentage points (so 0 = no preference). The RL-sibling teachers’ increasingly emphasize requests on sentence openings and “control-oriented” moves, while the larger teachers’ stay in sentence interiors and “technical-oriented” content. In (b) we show finer semantic categories of the request fields. Each dumbbell pairs the teachers’ enrichment among their

1\%

largest-normed of initial requests.

1

denotes corpus-level prevalence. See Section D.2 for finer details on the classes and how we conduct the interpretability analysis.

Panel of Figure 4 — Figure 4: Collective conversion and acquired responsiveness distinguish the pairings. (a) Healthy/stalled ratios; segments show the range across the control and technical strata. (b) Median leading-three eigenvalue gain $G_{3}$ and eigen-subspace overlap $O_{3}$ . (c) Step-50 paired trained-minus-other-field projection gain. Definitions and full results are in Sections D.4 and D.5 .

Panel of Figure 5 — Figure 5: Routing into unreliable prefixes, then fitting them. Weak Qwen: (a,b) cumulative routing, drift, and total changes in local loss and entropy, each on its own scale. The vertical guide at step 20 marks the approximate onset-to-fitting transition; the effects overlap. (c,d) Student surprisal and teacher top-1 probability; darker means greater uncertainty. Columns are 512-token windows. Details are in Section E.1 .

Figure 6: Initial-policy continuation success rate on trained prefixes. Negative values indicate harder prefixes for reaching the right answer.

Figure 7: Conditional checking refinement. Filled circles: initial over-use; open triangles: initial under-use.

Figure 8: Offline-prepared OPD improves sampled coverage and reliability. Left: pooled AIME 2024/2025 pass@

k

for

k\leq 32

(top) and the fraction of problems with empirical solve rate

\geq r>0

(bottom), using 32 responses per problem from batch a. Right: preparation dose versus strict MATH-500 pass@1, using the step-100 OPD checkpoint at 1.5B and the best evaluated checkpoint within 100 steps at 7B; error bars are 95% problem-bootstrap intervals. See Sections F.1 and F.2 for protocols and paired comparisons.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Composition of the strongest requests as the percentile rises. Shares of sentence openings (first three tokens), control positions and technical positions among each teacher’s top 50%, 20%, …, 1% of requests. Dotted: all annotated positions; bands: 95% bootstrap intervals over the annotated rollouts.

	1.5B student		7B student
Position category	Sibling	Larger	Sibling	Larger
Reframe	1.63	0.70	2.61	0.65
Check-conclude	1.52	0.91	1.86	0.62
Doubt flag	1.15	1.84	2.95	0.68
Check-initiate	1.37	1.20	2.64	1.32
Planning: opening	3.59	0.30	4.13	1.06

Appendix

Table 1: Enrichment values for the main semantic figure. Each value compares the frequency of a category among the strongest requests with its frequency among all annotated positions. Figure 10 shows the bootstrap intervals for each teacher.

Figure 10: Where the strongest requests occur. The full set of semantic categories, sentence positions, and token kinds. A ratio of one means that a category has the same frequency among the strongest requests as in the annotated rollouts. Planning sentences are split into their opening and stated approach. Intervals are clipped at the plot edges; left-pointing markers denote estimates below the plotted range, including zero. The main figure selects twelve rows.

Run	Class	$P_{0}$	$A_{0}/10^{3}$	$C_{0}$	$\Xi_{0}/10^{3}$	$\\|u_{0}\\|$
7B healthy	Control	0.192	119.8	0.081	9.70	43.1
7B healthy	Technical	0.151	106.3	0.055	5.85	29.7
7B stalled	Control	0.521	148.1	0.025	3.70	43.9
7B stalled	Technical	0.599	128.8	0.020	2.58	39.3
1.5B healthy	Control	6.024	39.6	0.041	1.62	98.9
1.5B healthy	Technical	2.569	54.5	0.064	3.49	94.7

Appendix

Table 2: Initial conversion estimates by semantic class. Each row uses

n=80

positions selected for one teacher and class.

P

and

A

use measurements at individual positions.

C

uses the finite-sample correction above.

\Xi

and

\|u\|

are approximate values computed from these estimates. Comparisons are within scale and parameterization.

Run	Class	Step	$P_{t}/P_{0}$	$A_{t}/A_{0}$	$C_{t}/C_{0}$	$\Xi_{t}/\Xi_{0}$	$A_{t}[B_{0}]/A_{0}[B_{0}]$
7B healthy	Control	200	0.005	0.983	0.012	0.012	1.414
7B healthy	Technical	200	0.007	2.159	0.036	0.079	1.232
7B stalled	Control	150	0.740	0.827	0.520	0.430	0.810
7B stalled	Technical	150	0.740	0.884	0.500	0.442	0.889
1.5B healthy	Control	200	0.009	1.216	0.341	0.415	2.829
1.5B healthy	Technical	200	0.015	3.628	0.031	0.113	5.399

Appendix

Table 3: Remaining requests and responsiveness at the endpoint. The positions are the same as at initialization. The pressure and live conversion columns use the current requests

B_{t}

; the last column uses the initial requests

B_{0}

. All diagnostic columns are ratios to their initial values. Endpoints are step 200, except for stalled 7B at step 150. Coherence estimates are rounded, especially at healthy endpoints.

Run	Step	$G_{3}$	$1-O_{3}$	$\Delta w_{1:3}$ (pp)	Median $a_{t}/a_{0}$	$A_{t}[B_{0}]/A_{0}[B_{0}]$
7B healthy	50	1.19	0.034	-0.23	1.04	1.23
7B healthy	200	1.25	0.041	-0.36	1.07	1.34
7B stalled	50	0.79	0.018	+1.12	0.93	0.90
7B stalled	150	0.74	0.032	+1.51	0.91	0.85
1.5B healthy	50	1.00	0.232	+5.49	1.19	3.95
1.5B healthy	200	2.31	0.369	+5.07	2.08	3.78

Appendix

Table 4: Changes in local geometry. Except in the last column, ratios and differences are computed at each position before taking the median. The last column compares accessibility averaged across positions with weights proportional to initial request energy. The median and weighted average describe different aspects of the same sample. Here pp denotes percentage points.

Panel of Figure 11 — Figure 11: Changes in the kernel and its response to the initial requests. Left: $A_{t}[B_{0}]/A_{0}[B_{0}]$ on the combined sample, with equal numbers of positions from the two classes. Right: median leading strength and subspace change across positions. Red/blue denote healthy/stalled runs; circles/squares denote 7B/1.5B.

Panel of Figure 12 — Figure 12: Endpoint eigenvalues relative to initialization. At each sorted rank, we compute the endpoint-to-initial ratio at each position and report the median over 160 positions per run. Red/blue denote healthy/stalled runs. Endpoints are step 200, except for stalled 7B at step 150.

Run	Trained gain (pp)	Other gain (pp)	Paired difference (pp)	95% interval
7B healthy	-0.23	+0.07	-0.16	$[-0.33,+0.27]$
7B stalled	+1.12	+1.25	+0.00	$[-0.53,+0.60]$
1.5B healthy	+5.49	+2.42	+1.79	$[+0.94,+3.13]$
1.5B stalled	+1.06	+1.00	-0.16	$[-0.29,+0.05]$

Appendix

Table 5: Comparison of the two teachers’ requests at step 50. Values are medians over the same 160 positions. The paired difference is the median of the per-position differences. For each of 3,000 bootstrap samples, we resample the eleven initial rollouts and retain all selected positions in each sampled rollout. The two fields remain paired and the trained checkpoints remain fixed.

	Local loss				Conditional entropy
Learner	Initial	Final	$\sum\widehat{R}$	$\sum\widehat{D}$	Initial	Final	$\sum\widehat{R}$	$\sum\widehat{D}$
DeepSeek	0.263	0.015	-0.033	-0.214	0.664	0.707	-0.049	0.093
Unprepared	0.579	0.073	0.474	-0.980	0.650	3.988	3.666	-0.327
Prepared	0.707	0.362	0.108	-0.453	2.171	0.842	-1.183	-0.146

Appendix

Table 6: Cumulative routing and drift contributions to loss and entropy. The comparisons run from initialization through batch 161, including evaluations at substitute checkpoints. Values are rounded independently. At full precision, the routing and drift contributions sum to the change from the initial to the final mean.

Learner	Reference class	Positions	$c_{W}$ at reference	$c_{W}$ at 200	$\Delta$ log-odds
DeepSeek	Above teacher	33,010	0.0789	0.0032	-2.073
DeepSeek	Below teacher	11,336	0.1915	0.0149	+1.381
Prepared Qwen	Above teacher	25,794	0.0276	0.0580	-0.606
Prepared Qwen	Below teacher	18,211	0.5627	0.3813	+0.721

Appendix

Table 7: Checking probability on fixed prefixes. We report discrepancy from the teacher,

c_{W}

, and the change in log-odds from the reference checkpoint to checkpoint 200. The reference is checkpoint 0 for DeepSeek and checkpoint 20 for prepared Qwen. Each class contains the same prefixes at both checkpoints. The main-text routing and drift plots instead compare successive banks.

Figure 13: Local agreement and teacher uncertainty. Left: local loss averaged over tokens in the full recorded batches, through the period with fixed prompt-language composition. Right: teacher entropy on the 32-rollout samples from each batch. Colors match Figures 5 and 14 . The teacher is generally less uncertain on DeepSeek and prepared-Qwen prefixes than on unprepared-Qwen prefixes.

	log-odds of	correctness			change in two-branch value
Evaluator	answering	answer	continue	natural	$\Delta V$	probability term	value term
initial student	$+2.4$	0.30	0.22	0.27	—	—	—
update 20	$-3.9$	0.30	0.29	0.29	$+0.01$	$-0.03$	$+0.05$
update 30	$-4.9$	0.33	0.32	0.31	$+0.04$	$-0.03$	$+0.07$
update 40	$-6.1$	0.29	0.27	0.26	$-0.01$	$-0.04$	$+0.03$
update 50	$-6.7$	0.31	0.17	0.21	$-0.09$	$-0.08$	$-0.02$

Appendix

Table 8: Answer probability and continuation success on 86 fixed prefixes. Log-odds are averaged over prefixes. The answer and continue columns use forced tokens; natural uses unconstrained continuations. The final columns give the change in the two-branch value and the contributions from changes in branch probability and branch values. We compute these at each prefix before averaging. At update 100, the bootstrap intervals are

[-0.32,-0.16]

for

\Delta V

[-0.20,-0.10]

for the probability term, and

[-0.14,-0.04]

for the value term. Values are rounded.

Figure 14: Loss and checking during the prepared run. Left: cumulative routing and drift contributions to local loss. Middle: WAIT-family tokens per thousand response tokens, measured in the first 512 tokens. Right: cumulative routing (dashed) and drift (solid) of mean log-WAIT probability at candidate sentence starts in the same window. The last panel decomposes log probability, which is distinct from emitted frequency. Open markers denote substituted evaluator checkpoints.

Figure 15: Changes on fixed prefixes and changes in teacher requests. Left: mean WAIT probability at the same sentence starts in the first 512 tokens of batch-41 rollouts, evaluated by each checkpoint. Right: mean request at sampled WAIT occurrences before an answer in full training batches, with the reported 95% intervals. Positive requests favor the sampled WAIT token; negative requests suppress it. The two panels use different prefix sets.

Panel of Figure 16 — Figure 16: Uncertainty at different response depths during training. Student surprisal and teacher top-1 probability are measured on the same student-generated prefixes, in 512-token windows. Hatched cells have no reported value. The plots describe the observed training trajectory.

	Student surprisal (nats)		Teacher top-1 probability
Batch	First 512	1536–2047	First 512	1536–2047
1	.85	—	.79	—
5	.69	—	.81	—
10	.58	.12	.83	.94
15	.58	—	.83	—
20	.62	3.24	.81	.51

Appendix

Table 9: Two response windows from the heatmaps. The first column gives the rollout batch. Batch

\ell

is generated after

\ell-1

updates. Student values are sampled-token surprisal. A dash indicates no reported value.

Cut depth	Step 50	Step 100	Step 200
256	$+2.1$	$-1.7$	$-5.9$
512	$-1.3$	$-2.7$	$-6.1$
1,024	$-8.1$	$-7.6$	$-8.5$

Appendix

Table 10: Change in initial-policy completion success,

100[F_{t}^{0}(h)-J_{Y}(0)]

, in percentage points. The initial student completes prefixes from each trained checkpoint; its own-rollout baseline is

48.4\%

. These are the values plotted in Figure 6 , reported as rounded aggregate point estimates from one training run.

Student	Offline-control steps	Steps in the OPD arm	Sequence presentations per arm
Qwen 1.5B	200 offline	200 OPD	51,200
Qwen 7B	100 offline	100 OPD	25,600
Llama 8B	200 offline	200 OPD	51,200

Appendix

Table 11: Comparison budgets from the same prepared checkpoint within each model pair, after 512 OMI2 preparation steps. Sequence presentations include repetitions.

Student	Arm	pass@1	pass@32	Coverage	$\geq 1/2$
Qwen 1.5B	Base	$0.0115$	$0.1333$	8	0
	Cold start	$0.0099$	$0.1333$	8	0
	+ offline	$0.0078$	$0.0833$	5	0
	+ OPD	$0.0281$	$0.1833$	11	0
Qwen 7B	Base	$0.0953$	$0.3167$	19	4
	Cold start	$0.1302$	$0.4167$	25	7

Appendix

Table 12: Pooled AIME endpoints and observed reliability at 32 samples per problem (batch a). Coverage, at-least-half, and all-32 columns count problems out of 60. Both Llama comparison arms start from the same prepared checkpoint and use the budgets in Table 11 .

Student	Year	$\Delta$ pass@1 [95% interval]	$\Delta$ pass@32 [95% interval]
Qwen 1.5B	2024	$+0.032\ [+0.005,+0.065]$	$+0.200\ [+0.033,+0.367]$
	2025	$+0.008\ [-0.004,+0.027]$	$+0.000\ [-0.100,+0.100]$
Qwen 7B	2024	$+0.086\ [+0.047,+0.130]$	$+0.133\ [+0.033,+0.267]$
	2025	$+0.071\ [+0.033,+0.114]$	$+0.200\ [+0.033,+0.367]$
Llama 8B	2024	$+0.040\ [+0.005,+0.082]$	$+0.100\ [+0.000,+0.233]$
	2025	$+0.029\ [-0.006,+0.069]$	$+0.133\ [+0.000,+0.300]$

Appendix

Table 13: OPD minus offline control at 32 samples per problem (batch a), with 95% paired problem-bootstrap percentile intervals computed separately within each year. Values are in probability units.

Student	Passes	Cold start	After OPD
Qwen 1.5B	1	$0.459$ [0.411, 0.506]	$0.459$ [0.409, 0.509]
	2	$0.457$ [0.409, 0.504]	$0.530$ [0.476, 0.585]
	4	$0.471$ [0.420, 0.522]	$0.562$ [0.510, 0.614]
Qwen 7B	1	$0.683$ [0.638, 0.725]	$0.823$ [0.782, 0.862]
	2	$0.693$ [0.649, 0.736]	$0.828$ [0.791, 0.864]
	4	$0.711$ [0.669, 0.753]	$0.853$ [0.817, 0.887]

Appendix

Table 14: The dose panels of Figure 8 : strict MATH-500 pass@1 (first 250 problems, 4 samples per problem) after 1, 2 and 4 preparation passes. The OPD column uses step 100 at 1.5B and the best evaluated checkpoint within 100 steps at 7B. Brackets give 95% problem-bootstrap intervals at each checkpoint.

On-Policy Distillation

On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.

See all 538 papers

Proposing Better Rollouts for On-Policy Distillation
Oct 27, 2026Sirui Chen, Yunzhe Qi, Haobo Xu +4Language Model Distillation On-Policy Distillation
University of Illinois Urbana-Champaign · Stanford University
Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective
Oct 2, 2026Han Cui, Jianhao Yan, Yun Luo +3On-Policy Distillation Language Model Distillation
Westlake University
Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
Mar 26, 2026Yuqian Fu, Haohuan Huang, Kaiwen Jiang +4Language Model Distillation On-Policy Distillation
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, UCAS · Fudan University +1

Oct 27, 2026J/K move · Enter open · S save

Proposing Better Rollouts for On-Policy Distillation

Sirui Chen, Yunzhe Qi, Haobo Xu, Jiaru Zou+3

University of Illinois Urbana-Champaign · Stanford University

Language Model Distillation On-Policy Distillation

Open paper AskPDF arXiv

On-policy distillation (OPD) improves language-model post-training by letting the student learn from its own sampled trajectories under teacher supervision, aligning the training process with inference-time behavior. However, the rollout policy in OPD is usually treated as fixed: each round samples directly from the current student. We show that this choice can create a compounding teacher--student mismatch, because it requires the teacher to provide effective supervision on whichever trajectories the student samples, including ones less reachable by the teacher itself. On such trajectories, teacher supervision can become noisy or even misleading, leading to undesirable student patterns such as repetition. Moreover, since the updated student in turn defines the rollout policy of subsequent rounds, these patterns persist and potentially amplify over training. This failure mode suggests that improving OPD requires treating the rollout policy itself as an optimization target, rather than only modifying the distillation objective. We propose Mixture Proposal On-Policy Distillation (MPOPD), which derives a student-anchored, teacher-guided rollout distribution from a local trust-region objective based on relative teacher support. The resulting target is a geometric mixture of the student and teacher policies, and we introduce a proposal policy that approximates this mixture, enabling rollout generation in a single forward pass. Experiments on 16 math, knowledge, and medical reasoning benchmarks show consistent gains over OPD and recent variants. On math reasoning, our method improves average accuracy by 6.91 over OPD on Qwen3-4B-Base while reducing average generation length by 41%, suggesting that rollout policy design provides an effective way to improve OPD.

Abstract

On-Policy Distillation

Explore similar work

Proposing Better Rollouts for On-Policy Distillation

Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective

Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes