On-policy distillation (OPD) is emerging as an important paradigm for language-model post-training, combining dense teacher supervision with training on student-generated trajectories. This creates a feedback loop where each update changes the student and thereby the training distribution that generates subsequent updates. Yet, how this feedback shapes learning and generalization remains poorly understood. We conduct a mechanistic analysis of OPD's learning dynamics, distinguishing the two feedback loop stages: conversion of the distillation signal into parameter updates, and broadcast through which these updates change predictions across prefixes and reshape the rollout distribution. This framework helps us understand both limited student responsiveness despite substantial teacher-student disagreement, and degeneration when updates that improve local teacher agreement also increase the probability of unsuccessful continuations, exposing the student to unreliable supervision in subsequent rollouts. Finally, we show that offline distillation can prepare a student for OPD without significantly improving immediate held-out accuracy. While OPD alone may degrade generation, it can yield substantial accuracy gains after sufficient offline preparation. Together, our results connect teacher supervision, student geometry, and the evolving rollout distribution in determining the learning dynamics of OPD and its effects on downstream performance.
Figures & tables
Figure 1: OPD’s learning dynamics, visualizing the feedback loop at the top (detailed in Section 3.1 ), and illustrative successful and failing processes at the bottom in reasoning settings we analyzed empirically. In principle, OPD propagates current teacher requests Γt(s)∣s∼μt to motion at any future prefix x through shared parameters, captured by the Neural Tangent Kernel K(x,s) . Such policy motion shapes how the autoregressive rollout (prefix visitation) and supervision distribution changes ( μ˙t,Γ˙t ). In practice, a healthy dynamics needs not only locally “tangent-accessible” and coherent requests ( Section 3.2 ), but also the student to be sufficiently “prepared”, so that it can sustain meaningful rollouts with learning value as it absorbs corrections requested by the teacher, instead of primarily performing local mimicry and falling into chaotic rollouts hard for the teacher to rescue, let alone improve on ( Sections 3.3 and 3.4 ).
Figure 3: Teacher request fields differ systematically. In (a), each curve shows how much more (or less) often a semantic or token-position class appears among a teacher’s top 50%, 20%, …, 1% of requests than among all annotated positions, in percentage points (so 0 = no preference). The RL-sibling teachers’ increasingly emphasize requests on sentence openings and “control-oriented” moves, while the larger teachers’ stay in sentence interiors and “technical-oriented” content. In (b) we show finer semantic categories of the request fields. Each dumbbell pairs the teachers’ enrichment among their 1% largest-normed of initial requests. 1 denotes corpus-level prevalence. See Section D.2 for finer details on the classes and how we conduct the interpretability analysis.
Figure 4: Collective conversion and acquired responsiveness distinguish the pairings. (a) Healthy/stalled ratios; segments show the range across the control and technical strata. (b) Median leading-three eigenvalue gain G3 and eigen-subspace overlap O3 . (c) Step-50 paired trained-minus-other-field projection gain. Definitions and full results are in Sections D.4 and D.5 .
Figure 5: Routing into unreliable prefixes, then fitting them. Weak Qwen: (a,b) cumulative routing, drift, and total changes in local loss and entropy, each on its own scale. The vertical guide at step 20 marks the approximate onset-to-fitting transition; the effects overlap. (c,d) Student surprisal and teacher top-1 probability; darker means greater uncertainty. Columns are 512-token windows. Details are in Section E.1 .
Figure 6: Initial-policy continuation success rate on trained prefixes. Negative values indicate harder prefixes for reaching the right answer.
Figure 8: Offline-prepared OPD improves sampled coverage and reliability. Left: pooled AIME 2024/2025 pass@ k for k≤32 (top) and the fraction of problems with empirical solve rate ≥r>0 (bottom), using 32 responses per problem from batch a. Right: preparation dose versus strict MATH-500 pass@1, using the step-100 OPD checkpoint at 1.5B and the best evaluated checkpoint within 100 steps at 7B; error bars are 95% problem-bootstrap intervals. See Sections F.1 and F.2 for protocols and paired comparisons.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Composition of the strongest requests as the percentile rises. Shares of sentence openings (first three tokens), control positions and technical positions among each teacher’s top 50%, 20%, …, 1% of requests. Dotted: all annotated positions; bands: 95% bootstrap intervals over the annotated rollouts.
1.5B student
7B student
Position category
Sibling
Larger
Sibling
Larger
Reframe
1.63
0.70
2.61
0.65
Check-conclude
1.52
0.91
1.86
0.62
Doubt flag
1.15
1.84
2.95
0.68
Check-initiate
1.37
1.20
2.64
1.32
Planning: opening
3.59
0.30
4.13
1.06
Appendix
Table 1: Enrichment values for the main semantic figure. Each value compares the frequency of a category among the strongest requests with its frequency among all annotated positions. Figure 10 shows the bootstrap intervals for each teacher.
Figure 10: Where the strongest requests occur. The full set of semantic categories, sentence positions, and token kinds. A ratio of one means that a category has the same frequency among the strongest requests as in the annotated rollouts. Planning sentences are split into their opening and stated approach. Intervals are clipped at the plot edges; left-pointing markers denote estimates below the plotted range, including zero. The main figure selects twelve rows.
Run
Class
P0
A0/103
C0
Ξ0/103
∥u0∥
7B healthy
Control
0.192
119.8
0.081
9.70
43.1
7B healthy
Technical
0.151
106.3
0.055
5.85
29.7
7B stalled
Control
0.521
148.1
0.025
3.70
43.9
7B stalled
Technical
0.599
128.8
0.020
2.58
39.3
1.5B healthy
Control
6.024
39.6
0.041
1.62
98.9
1.5B healthy
Technical
2.569
54.5
0.064
3.49
94.7
Appendix
Table 2: Initial conversion estimates by semantic class. Each row uses n=80 positions selected for one teacher and class. P and A use measurements at individual positions. C uses the finite-sample correction above. Ξ and ∥u∥ are approximate values computed from these estimates. Comparisons are within scale and parameterization.
Run
Class
Step
Pt/P0
At/A0
Ct/C0
Ξt/Ξ0
At[B0]/A0[B0]
7B healthy
Control
200
0.005
0.983
0.012
0.012
1.414
7B healthy
Technical
200
0.007
2.159
0.036
0.079
1.232
7B stalled
Control
150
0.740
0.827
0.520
0.430
0.810
7B stalled
Technical
150
0.740
0.884
0.500
0.442
0.889
1.5B healthy
Control
200
0.009
1.216
0.341
0.415
2.829
1.5B healthy
Technical
200
0.015
3.628
0.031
0.113
5.399
Appendix
Table 3: Remaining requests and responsiveness at the endpoint. The positions are the same as at initialization. The pressure and live conversion columns use the current requests Bt ; the last column uses the initial requests B0 . All diagnostic columns are ratios to their initial values. Endpoints are step 200, except for stalled 7B at step 150. Coherence estimates are rounded, especially at healthy endpoints.
Run
Step
G3
1−O3
Δw1:3 (pp)
Median at/a0
At[B0]/A0[B0]
7B healthy
50
1.19
0.034
-0.23
1.04
1.23
7B healthy
200
1.25
0.041
-0.36
1.07
1.34
7B stalled
50
0.79
0.018
+1.12
0.93
0.90
7B stalled
150
0.74
0.032
+1.51
0.91
0.85
1.5B healthy
50
1.00
0.232
+5.49
1.19
3.95
1.5B healthy
200
2.31
0.369
+5.07
2.08
3.78
Appendix
Table 4: Changes in local geometry. Except in the last column, ratios and differences are computed at each position before taking the median. The last column compares accessibility averaged across positions with weights proportional to initial request energy. The median and weighted average describe different aspects of the same sample. Here pp denotes percentage points.
Figure 11: Changes in the kernel and its response to the initial requests. Left: At[B0]/A0[B0] on the combined sample, with equal numbers of positions from the two classes. Right: median leading strength and subspace change across positions. Red/blue denote healthy/stalled runs; circles/squares denote 7B/1.5B.
Figure 12: Endpoint eigenvalues relative to initialization. At each sorted rank, we compute the endpoint-to-initial ratio at each position and report the median over 160 positions per run. Red/blue denote healthy/stalled runs. Endpoints are step 200, except for stalled 7B at step 150.
Run
Trained gain (pp)
Other gain (pp)
Paired difference (pp)
95% interval
7B healthy
-0.23
+0.07
-0.16
[−0.33,+0.27]
7B stalled
+1.12
+1.25
+0.00
[−0.53,+0.60]
1.5B healthy
+5.49
+2.42
+1.79
[+0.94,+3.13]
1.5B stalled
+1.06
+1.00
-0.16
[−0.29,+0.05]
Appendix
Table 5: Comparison of the two teachers’ requests at step 50. Values are medians over the same 160 positions. The paired difference is the median of the per-position differences. For each of 3,000 bootstrap samples, we resample the eleven initial rollouts and retain all selected positions in each sampled rollout. The two fields remain paired and the trained checkpoints remain fixed.
Local loss
Conditional entropy
Learner
Initial
Final
∑R
∑D
Initial
Final
∑R
∑D
DeepSeek
0.263
0.015
-0.033
-0.214
0.664
0.707
-0.049
0.093
Unprepared
0.579
0.073
0.474
-0.980
0.650
3.988
3.666
-0.327
Prepared
0.707
0.362
0.108
-0.453
2.171
0.842
-1.183
-0.146
Appendix
Table 6: Cumulative routing and drift contributions to loss and entropy. The comparisons run from initialization through batch 161, including evaluations at substitute checkpoints. Values are rounded independently. At full precision, the routing and drift contributions sum to the change from the initial to the final mean.
Learner
Reference class
Positions
cW at reference
cW at 200
Δ log-odds
DeepSeek
Above teacher
33,010
0.0789
0.0032
-2.073
DeepSeek
Below teacher
11,336
0.1915
0.0149
+1.381
Prepared Qwen
Above teacher
25,794
0.0276
0.0580
-0.606
Prepared Qwen
Below teacher
18,211
0.5627
0.3813
+0.721
Appendix
Table 7: Checking probability on fixed prefixes. We report discrepancy from the teacher, cW , and the change in log-odds from the reference checkpoint to checkpoint 200. The reference is checkpoint 0 for DeepSeek and checkpoint 20 for prepared Qwen. Each class contains the same prefixes at both checkpoints. The main-text routing and drift plots instead compare successive banks.
Figure 13: Local agreement and teacher uncertainty. Left: local loss averaged over tokens in the full recorded batches, through the period with fixed prompt-language composition. Right: teacher entropy on the 32-rollout samples from each batch. Colors match Figures 5 and 14 . The teacher is generally less uncertain on DeepSeek and prepared-Qwen prefixes than on unprepared-Qwen prefixes.
log-odds of
correctness
change in two-branch value
Evaluator
answering
answer
continue
natural
ΔV
probability term
value term
initial student
+2.4
0.30
0.22
0.27
—
—
—
update 20
−3.9
0.30
0.29
0.29
+0.01
−0.03
+0.05
update 30
−4.9
0.33
0.32
0.31
+0.04
−0.03
+0.07
update 40
−6.1
0.29
0.27
0.26
−0.01
−0.04
+0.03
update 50
−6.7
0.31
0.17
0.21
−0.09
−0.08
−0.02
Appendix
Table 8: Answer probability and continuation success on 86 fixed prefixes. Log-odds are averaged over prefixes. The answer and continue columns use forced tokens; natural uses unconstrained continuations. The final columns give the change in the two-branch value and the contributions from changes in branch probability and branch values. We compute these at each prefix before averaging. At update 100, the bootstrap intervals are [−0.32,−0.16] for ΔV , [−0.20,−0.10] for the probability term, and [−0.14,−0.04] for the value term. Values are rounded.
Figure 14: Loss and checking during the prepared run. Left: cumulative routing and drift contributions to local loss. Middle: WAIT-family tokens per thousand response tokens, measured in the first 512 tokens. Right: cumulative routing (dashed) and drift (solid) of mean log-WAIT probability at candidate sentence starts in the same window. The last panel decomposes log probability, which is distinct from emitted frequency. Open markers denote substituted evaluator checkpoints.
Figure 15: Changes on fixed prefixes and changes in teacher requests. Left: mean WAIT probability at the same sentence starts in the first 512 tokens of batch-41 rollouts, evaluated by each checkpoint. Right: mean request at sampled WAIT occurrences before an answer in full training batches, with the reported 95% intervals. Positive requests favor the sampled WAIT token; negative requests suppress it. The two panels use different prefix sets.
Figure 16: Uncertainty at different response depths during training. Student surprisal and teacher top-1 probability are measured on the same student-generated prefixes, in 512-token windows. Hatched cells have no reported value. The plots describe the observed training trajectory.
Student surprisal (nats)
Teacher top-1 probability
Batch
First 512
1536–2047
First 512
1536–2047
1
.85
—
.79
—
5
.69
—
.81
—
10
.58
.12
.83
.94
15
.58
—
.83
—
20
.62
3.24
.81
.51
Appendix
Table 9: Two response windows from the heatmaps. The first column gives the rollout batch. Batch ℓ is generated after ℓ−1 updates. Student values are sampled-token surprisal. A dash indicates no reported value.
Cut depth
Step 50
Step 100
Step 200
256
+2.1
−1.7
−5.9
512
−1.3
−2.7
−6.1
1,024
−8.1
−7.6
−8.5
Appendix
Table 10: Change in initial-policy completion success, 100[Ft0(h)−JY(0)] , in percentage points. The initial student completes prefixes from each trained checkpoint; its own-rollout baseline is 48.4% . These are the values plotted in Figure 6 , reported as rounded aggregate point estimates from one training run.
Student
Offline-control steps
Steps in the OPD arm
Sequence presentations per arm
Qwen 1.5B
200 offline
200 OPD
51,200
Qwen 7B
100 offline
100 OPD
25,600
Llama 8B
200 offline
200 OPD
51,200
Appendix
Table 11: Comparison budgets from the same prepared checkpoint within each model pair, after 512 OMI2 preparation steps. Sequence presentations include repetitions.
Student
Arm
pass@1
pass@32
Coverage
≥1/2
All 32
Qwen 1.5B
Base
0.0115
0.1333
8
0
0
Cold start
0.0099
0.1333
8
0
0
+ offline
0.0078
0.0833
5
0
0
+ OPD
0.0281
0.1833
11
0
0
Qwen 7B
Base
0.0953
0.3167
19
4
0
Cold start
0.1302
0.4167
25
7
0
Appendix
Table 12: Pooled AIME endpoints and observed reliability at 32 samples per problem (batch a). Coverage, at-least-half, and all-32 columns count problems out of 60. Both Llama comparison arms start from the same prepared checkpoint and use the budgets in Table 11 .
Student
Year
Δ pass@1 [95% interval]
Δ pass@32 [95% interval]
Qwen 1.5B
2024
+0.032[+0.005,+0.065]
+0.200[+0.033,+0.367]
2025
+0.008[−0.004,+0.027]
+0.000[−0.100,+0.100]
Qwen 7B
2024
+0.086[+0.047,+0.130]
+0.133[+0.033,+0.267]
2025
+0.071[+0.033,+0.114]
+0.200[+0.033,+0.367]
Llama 8B
2024
+0.040[+0.005,+0.082]
+0.100[+0.000,+0.233]
2025
+0.029[−0.006,+0.069]
+0.133[+0.000,+0.300]
Appendix
Table 13: OPD minus offline control at 32 samples per problem (batch a), with 95% paired problem-bootstrap percentile intervals computed separately within each year. Values are in probability units.
Student
Passes
Cold start
After OPD
Qwen 1.5B
1
0.459 [0.411, 0.506]
0.459 [0.409, 0.509]
2
0.457 [0.409, 0.504]
0.530 [0.476, 0.585]
4
0.471 [0.420, 0.522]
0.562 [0.510, 0.614]
Qwen 7B
1
0.683 [0.638, 0.725]
0.823 [0.782, 0.862]
2
0.693 [0.649, 0.736]
0.828 [0.791, 0.864]
4
0.711 [0.669, 0.753]
0.853 [0.817, 0.887]
Appendix
Table 14: The dose panels of Figure 8 : strict MATH-500 pass@1 (first 250 problems, 4 samples per problem) after 1, 2 and 4 preparation passes. The OPD column uses step 100 at 1.5B and the best evaluated checkpoint within 100 steps at 7B. Brackets give 95% problem-bootstrap intervals at each checkpoint.
On-policy distillation (OPD) improves language-model post-training by letting the student learn from its own sampled trajectories under teacher supervision, aligning the training process with inference-time behavior. However, the rollout policy in OPD is usually treated as fixed: each round samples directly from the current student. We show that this choice can create a compounding teacher--student mismatch, because it requires the teacher to provide effective supervision on whichever trajectories the student samples, including ones less reachable by the teacher itself. On such trajectories, teacher supervision can become noisy or even misleading, leading to undesirable student patterns such as repetition. Moreover, since the updated student in turn defines the rollout policy of subsequent rounds, these patterns persist and potentially amplify over training. This failure mode suggests that improving OPD requires treating the rollout policy itself as an optimization target, rather than only modifying the distillation objective. We propose Mixture Proposal On-Policy Distillation (MPOPD), which derives a student-anchored, teacher-guided rollout distribution from a local trust-region objective based on relative teacher support. The resulting target is a geometric mixture of the student and teacher policies, and we introduce a proposal policy that approximates this mixture, enabling rollout generation in a single forward pass. Experiments on 16 math, knowledge, and medical reasoning benchmarks show consistent gains over OPD and recent variants. On math reasoning, our method improves average accuracy by 6.91 over OPD on Qwen3-4B-Base while reducing average generation length by 41%, suggesting that rollout policy design provides an effective way to improve OPD.
Sirui Chen, Yunzhe Qi, Haobo Xu +4
University of Illinois Urbana-Champaign · Stanford University
On-policy distillation (OPD) has become an important approach to language model post-training. However, despite its performance gains, OPD can also collapse into excessively long and repetitive generation, and the mechanism underlying these divergent outcomes remains poorly understood. We explain these outcomes through a reinforcement learning perspective: the teacher implicitly rewards student behaviors, even those it rarely exhibits itself. From this perspective, our experiments show that OPD improves performance without expanding the student's capabilities. When the implicit reward model is reliable, OPD makes correct responses easier to sample. In contrast, when the preference misaligns with quality, reward hacking happens: the implicit reward model amplifies overlong, repetitive student rollouts, even though it rarely generates such text itself. Guided by this diagnosis, we find that masking unhealthy responses during training and using SFT initialization can each effectively mitigate the collapse. Together, these findings show that OPD amplifies student behaviors favored by the teacher's implicit feedback, shifting the focus from how well the teacher generates to how reliably it evaluates student rollouts. Our code is available at https://github.com/HancCui/opd_hacking.
On-policy distillation (OPD) is increasingly used in LLM post-training because it can leverage a teacher model to provide dense supervision on student rollouts. The standard implementation, however, usually reduces distribution matching to a sampled-token log-ratio, which can make the learning signal fragile on long rollouts whose prefixes drift away from the teacher's typical support. We revisit this formulation from both theoretical and implementation perspectives. Theoretically, token-level OPD is biased relative to sequence-level reverse-KL minimization, but admits a substantially tighter worst-case variance bound; a controlled synthetic study further shows that stronger future-reward coupling increases gradient variance and destabilizes training. Empirically, we identify three failure modes of sampled-token OPD: imbalanced token-level supervision, unreliable teacher guidance on student-generated prefixes, and tokenizer or special-token mismatch. These findings motivate teacher top-K local support matching, a truncated reverse-KL objective that compares teacher and student distributions over a teacher-supported token set at each prefix, together with top-p rollout sampling and special-token masking. Across single-task reasoning and multi-task benchmarks spanning agentic and reasoning settings, this objective improves optimization stability and yields a +19.8% performance gain over standard sampled-token OPD baselines, providing a practical recipe for more stable on-policy distillation.
Yuqian Fu, Haohuan Huang, Kaiwen Jiang +4
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, UCAS · Fudan University +1