On-policy distillation (OPD) is increasingly used for language model post-training, yet how teacher supervision shapes learning and forgetting remains unclear. We present OPDArena, a framework comparing supervised fine-tuning (SFT), reinforcement learning with verifiable rewards (RLVR), and OPD across five pretrained models and diverse teacher sources. It jointly evaluates mathematical learning, general capability retention, and predictive distribution changes. Teacher choice strongly shapes retention: students of RLVR-trained teachers improve on the target task while retaining prior capabilities, whereas students of SFT-trained teachers incur greater capability loss. Across post-training methods, out-of-domain (OOD) distribution change shows a more consistent positive association with forgetting than in-domain change. To understand how teacher choice affects these changes, we examine the additional information teachers provide relative to the student's pretrained model. Controlled teacher comparisons show that the task-specific preferences teachers acquire through post-training make student updates more concentrated in-domain.
Figures & tables
Figure 1: Training methods and joint learning–retention outcomes. (a) Training paths and feedback sources from a shared Base model. (b) In-domain accuracy gain versus OOD accuracy gain from Base; colors identify methods and shapes identify models. Ellipses summarize variation across model–teacher configurations. RLVR and Self-OPD (RLVR) combine task learning with low forgetting.
Method (Teacher)
Qwen2.5-1.5B
Qwen2.5-7B
In-domain
OOD
In-domain
OOD
Base
13.93
35.20
26.08
49.82
SFT
11.50 (−2.42)
34.25 (−0.95)
35.87 (+9.79)
45.32 (−4.50)
RLVR
19.08 (+5.15)
36.23 (+1.03)
32.56 (+6.48)
50.06 (+0.23)
OPD (Q2.5-7B)
15.85 (+1.93)
34.54 (−0.66)
–
–
OPD (Q2.5-14B)
16.18 (+2.26)
34.27 (−0.93)
25.03 (−1.05)
48.47 (−1.35)
Table 1: Main results for four Base models: accuracy (%) with changes from Base in parentheses (percentage points). Negative OOD changes indicate forgetting. RLVR and Self-OPD (RLVR) combine in-domain gains with strong capability retention.
Figure 2: Distribution change and forgetting. Horizontal axes show in-domain (top) and OOD (bottom) JS from Base; larger values indicate greater distribution change. The vertical axis shows OOD accuracy loss from Base; higher values indicate greater forgetting. Each point pairs JS and OOD accuracy at the same checkpoint. JS is in nats, averaged equally over five in-domain or seven OOD tasks; r is Pearson correlation. Qwen3-1.7B is shown in Appendix Figure 14 .
Figure 3: Prediction changes from Base across post-training methods. The first two columns show in-domain and OOD JS in 10−2 nats; the last shows relative OOD influence ρ . Boxes mark the lowest ρ for each model.
Teacher
Teacher acc.
Student acc.
Math
OOD
Math
OOD
ρ
Student: Qwen2.5-1.5B Base
13.93
35.20
Qwen3-8B Base
31.32
51.37
16.33
33.37
1.998
Qwen2.5-1.5B RLVR
19.08
36.23
20.00
36.60
0.131
Student: Qwen2.5-7B Base
26.08
49.82
Qwen3-8B Base
31.32
51.37
24.17
47.13
1.648
Table 2: Teacher and student accuracies (%) and student ρ at step 780. Each block compares a larger Base teacher with the student’s own RLVR teacher; shaded headings identify the student and its Base accuracies. Datasets are weighted equally within each domain.
Table 3: Student distribution changes and capabilities under different information sources. All A–E conditions use student step 120. JS uses shared prefixes, is relative to Base, and is reported in 10−2 nats. Parentheses show accuracy changes from Base in percentage points. Datasets are weighted equally within each domain; ρ is undefined at Base.
Figure 4: Accuracy changes from Base under weight scaling. Stars mark sweet-spot coefficients that improve both accuracies over the original endpoint, as defined in Appendix F .
Figure 5: Qwen3-4B parameter distances for SFT, RLVR, and their Self-OPD students (left), and student–teacher JS for the same students (right). Each distance is normalized by its corresponding initial Base–teacher gap.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Input
AMC23
AIME24
AIME25
HMMT25
Math500
Mean
Qwen2.5-1.5B Base
Raw
20.00
1.25
0.83
0.42
47.13
13.93
Qwen2.5-1.5B Base
Template
1.88
0.42
0.00
0.00
4.58
1.37
Qwen2.5-7B Base
Raw
46.88
9.17
5.00
2.08
67.28
26.08
Qwen2.5-7B Base
Template
31.25
6.67
3.33
0.42
57.03
19.74
Qwen2.5-14B Base
Raw
51.25
10.42
7.08
2.50
73.23
28.90
Qwen3-1.7B Base
Raw
32.50
4.17
4.58
0.00
59.58
20.17
Appendix
Table 4: In-domain acc@8 (%) of the released Base models. The Mean column weights the five datasets equally. Raw and Template denote inputs without and with the corresponding Qwen chat template.
Model
Input
AMC23
AIME24
AIME25
HMMT25
Math500
Mean
Qwen2.5-1.5B Base
Raw
2359.4
5101.0
3827.6
3450.3
1074.3
3162.5
Qwen2.5-1.5B Base
Template
25725.3
25467.4
25473.3
22321.9
13228.8
22443.4
Qwen2.5-7B Base
Raw
1160.0
3131.1
3121.3
2364.9
787.3
2112.9
Qwen2.5-7B Base
Template
1541.7
1985.4
2171.4
2038.7
696.8
1686.8
Qwen2.5-14B Base
Raw
912.9
1214.1
1483.7
1317.1
614.7
1108.5
Qwen3-1.7B Base
Raw
1807.0
5101.9
4117.2
4692.0
1073.2
3358.3
Appendix
Table 5: Mean in-domain output length in tokens for the same Base-model evaluations. The Mean column weights the five datasets equally. Math500 uses a 16,384-token response limit; the other datasets use 31,744 tokens.
Model
HellaSwag
ARC-C
MMLU
IFEval
TriviaQA
BBH
NQ-Open
Macro
Qwen2.5-1.5B Base
49.22
39.45
61.40
20.70
22.00
42.59
11.05
35.20
Qwen2.5-7B Base
61.33
44.53
74.21
31.79
44.40
72.96
19.53
49.82
Qwen2.5-14B Base
62.11
50.78
78.07
31.79
55.20
79.63
–
–
Qwen3-1.7B Base
46.09
37.11
65.26
24.03
20.60
50.37
9.17
36.09
Qwen3-4B Base
53.13
41.02
75.09
31.98
34.00
66.67
16.29
45.45
Qwen3-8B Base
59.77
44.53
76.49
37.52
47.20
72.59
21.50
51.37
Appendix
Table 6: OOD accuracy (%) of the released Base models with Raw input. The macro is an unweighted mean over the seven datasets. HellaSwag, ARC-Challenge, and MMLU use candidate likelihood; the other tasks score one generated response. Response budgets are listed in Table 8 .
Setting
SFT
RLVR
OPD variants / OPSD
Train records
49,993
49,993
49,993
Epochs
1
1
1
Global batch
64
64
64
Learning rate
10−5
10−6
10−6
Optimizer
AdamW
AdamW
AdamW
Adam (β1,β2)
(0.9,0.999)
(0.9,0.999)
(0.9,0.999)
Appendix
Table 7: Current OpenR1-50K training settings. OPD, Self-OPD, and OPSD share the listed optimization settings; the teacher input or token signal differs.
Domain
Dataset
Evaluation
Examples
Response budget (tokens)
Primary score
In-domain
AMC23
eight generated responses
40
31,744
acc@8
AIME24
eight generated responses
30
31,744
acc@8
AIME25
eight generated responses
30
31,744
acc@8
HMMT25
eight generated responses
30
31,744
acc@8
Math500
eight generated responses
500
16,384
acc@8
OOD
HellaSwag
candidate likelihood
256
–
normalized accuracy
Appendix
Table 8: Target evaluation protocol: fixed validation suite and response budgets in tokens. OOD category subsamples use seed 42 and are reused at every checkpoint. A dash denotes likelihood scoring without response generation. The IFEval budget applies to the response prefix used for scoring.
Figure 6: In-domain learning and OOD retention throughout training. Curves show task-group mean accuracy; dashed horizontal lines mark Base. RLVR and Self-OPD (RLVR) combine in-domain learning with sustained capability retention.
Figure 7: Task accuracies during training for Qwen2.5-1.5B Base.
Figure 8: Task accuracies during training for Qwen2.5-7B Base.
Figure 9: Task accuracies during training for Qwen3-1.7B Base.
Figure 10: Task accuracies during training for Qwen3-4B Base.
Figure 11: Task accuracies during training for Qwen3-8B Base.
Student
Teacher
Parameter distance
Prediction distance
dθ
d⊥
dout,in
dout,OOD
Qwen2.5-1.5B
SFT
0.825
0.355
0.493
0.361
RLVR
0.894
0.744
0.010
0.354
Qwen2.5-7B
SFT
0.976
0.471
0.601
0.173
RLVR
0.943
0.758
0.010
0.243
Qwen3-1.7B
SFT
1.003
0.516
0.705
0.187
Appendix
Table 9: Self-OPD distances to the corresponding SFT or RLVR teacher at step 780 for the four models supplementing Figure 5 . Parameter and prediction distances are normalized by the initial Base–teacher gap; d⊥ measures departure from the Base–teacher line.
Figure 12: Accuracy changes from Base for source training (dashed) and the corresponding Self-OPD student (solid). Students use a fixed endpoint teacher.
Prefix source
Base
SFT@60
SFT@240
RLVR@60
RLVR@240
Teacher
Qwen2.5-7B
SFT
1.057
0.871
0.580
1.563
1.567
0.379
RLVR
0.230
0.330
0.436
0.106
0.096
0.094
Qwen3-1.7B
SFT
0.967
0.839
0.665
1.855
1.822
0.458
RLVR
0.300
0.332
0.444
0.145
0.136
0.135
Appendix
Table 10: Mean negative log probability over the first 128 tokens of teacher responses, in nats/token. SFT/RLVR columns denote students of the corresponding teacher source; @ denotes student training step. Each teacher contributes all 256 responses; the final column is scored by the teacher that generated them.
Prefix source
Tokens 1–128
Tokens 33–128
Qwen2.5-7B
SFT
0.678
0.514
RLVR
0.136
0.122
Qwen3-1.7B
SFT
0.509
0.353
RLVR
0.165
0.132
Appendix
Table 11: Scoring gaps after correcting for teacher self-probability, in nats/token. Larger values indicate a higher predictive cost under Base relative to the generating teacher.
In-domain accuracy
OOD accuracy
Model
Ordinary
Zeroed
Δ
Ordinary
Zeroed
Δ
Qwen2.5-1.5B
10.33
14.23
+3.91
34.93
34.58
-0.35
Qwen2.5-7B
28.49
34.57
+6.08
47.02
46.50
-0.52
Qwen3-1.7B
17.16
20.54
+3.38
35.59
35.71
+0.12
Qwen3-4B
34.71
38.82
+4.11
43.45
43.35
-0.10
Qwen3-8B
44.79
44.12
-0.66
47.10
47.63
+0.53
Appendix
Table 12: Step-780 accuracy (%) with ordinary feedback and first-position feedback zeroed. Δ is zeroed minus ordinary accuracy, in percentage points. Evaluation leaves all output positions unchanged.
JS to Base
JS to SFT teacher
Model
Feedback
In-domain
OOD
In-domain
OOD
Qwen2.5-1.5B
Ordinary
7.200
2.136
3.658
1.007
First-position zeroed
4.172
1.750
4.917
0.860
Qwen2.5-7B
Ordinary
7.137
1.963
4.625
0.462
First-position zeroed
5.355
1.620
4.779
0.753
Qwen3-1.7B
Ordinary
6.165
2.149
4.555
0.521
Appendix
Table 13: Step-780 JS in 10−2 nats for the same ordinary and first-position-zeroed students. The references are the shared Base and the corresponding frozen SFT teacher.
Figure 13: Complete accuracy trajectories for ordinary Self-OPD (SFT) and its first-position feedback intervention across five models (columns), with in-domain accuracy above OOD accuracy. Both conditions use the same Base and frozen SFT teacher. Dotted horizontal lines mark Base accuracy.
Teacher supervision
Distribution change
Accuracy (%)
Din
DOOD
ρ
Math ( Δ )
OOD ( Δ )
Qwen3-4B → Qwen3-1.7B
Student Base
–
–
–
20.17
36.09
A: Larger Base teacher
0.2692
0.1772
0.658
19.38
(−0.78)
35.96
(−0.13)
D: Larger RLVR teacher
4.1255
0.3643
0.088
24.21
(+4.04)
36.19
(+0.10)
E: RLVR increment
4.1056
0.3302
0.080
24.08
(+3.92)
36.05
(−0.04)
Appendix
Table 14: Additional A/D/E comparisons for Qwen3-1.7B at student step 120. JS is relative to student Base in 10−2 nats. Accuracies are percentages, with changes from Base in parentheses. Domains use equal dataset weights.
Figure 14: Distribution change and forgetting for Qwen3-1.7B. Left: in-domain JS; right: OOD JS. Both panels show OOD accuracy loss at the corresponding checkpoint, with r denoting Pearson correlation.
Teacher
Raw
Chat template
Raw + answer boundary
Qwen2.5-7B
24.12
0.0100
0.0028
Qwen3-1.7B
15.36
0.0385
0.0034
Appendix
Table 15: Teacher end-of-sequence probability (%) before the first student response token, with the problems and reference solutions fixed.
Student
Metric
Base
Raw
Chat template
Raw + boundary
Qwen2.5-1.5B
Math
13.93
5.63
6.86
12.64
OOD
35.20
35.21
35.06
35.09
Qwen2.5-7B
Math
26.08
8.61
22.44
25.46
OOD
49.82
48.91
49.00
49.35
Qwen3-1.7B
Math
20.16
9.62
19.34
19.68
OOD
36.09
35.09
36.22
35.63
Appendix
Table 16: OPSD accuracy (%) at step 780 under three teacher input formats. Students always receive Raw problems. Base is the untrained student; each metric averages its datasets equally.
Student
Teacher input
Din
DOOD
ρ
Qwen2.5-1.5B
Raw
2.965
0.440
0.148
Chat template
4.978
0.472
0.095
Raw + answer boundary
4.691
0.599
0.128
Qwen2.5-7B
Raw
2.308
0.189
0.082
Chat template
4.789
0.192
0.040
Raw + answer boundary
4.585
0.188
0.041
Appendix
Table 17: OPSD distribution changes at step 780 for the same runs as Table 16 . JS is measured relative to Base in 10−2 nats; ρ=DOOD/Din .
Figure 15: OPSD mathematical and OOD accuracy throughout training under the three teacher input formats, one student per row. Dotted horizontal lines mark the common Base evaluation. Each curve uses measured checkpoints at 20-step intervals.
Update direction
In-domain JS ×105
OOD JS ×105
ρ
Qwen3-1.7B
A: Base teacher
1.079
0.659
0.611
D: RLVR teacher
1.072
0.079
0.074
E: RLVR increment
0.938
0.053
0.056
RLVR parameter change (step 60)
1.228
0.071
0.058
RLVR parameter change (step 780)
1.098
0.069
0.063
Appendix
Table 18: Independent-prefix measurements after calibrating in-domain JS to 10−5 . Incorporating the teacher’s RLVR changes reduces relative OOD influence and brings it closer to the paired direct RLVR parameter directions.
On-policy distillation (OPD) has become a core technique in the post-training of large language models, yet its training dynamics remain poorly understood. This paper provides a systematic investigation of OPD dynamics and mechanisms. We first identify that two conditions govern whether OPD succeeds or fails: (i) the student and teacher should share compatible thinking patterns; and (ii) even with consistent thinking patterns and higher scores, the teacher must offer genuinely new capabilities beyond what the student has seen during training. We validate these findings through weak-to-strong reverse distillation, showing that same-family 1.5B and 7B teachers are distributionally indistinguishable from the student's perspective. Probing into the token-level mechanism, we show that successful OPD is characterized by progressive alignment on high-probability tokens at student-visited states, a small shared token set that concentrates most of the probability mass (97%-99%). We further propose two practical strategies to recover failing OPD: off-policy cold start and teacher-aligned prompt selection. Finally, we show that OPD's apparent free lunch of dense token-level reward comes at a cost, raising the question of whether OPD can scale to long-horizon distillation.
Yaxuan Li, Yuxin Zuo, Bingxiang He +8
Tsinghua University · ShanghaiTech University · University of Illinois Urbana-Champaign +1
On-policy distillation (OPD) and on-policy self-distillation (OPSD) have emerged as promising post-training methods for large language models, offering dense token-level supervision on trajectories sampled from the model's own policy. However, existing results on their effectiveness remain mixed: while OP(S)D has shown promise in system prompt and knowledge internalization, recent studies also report instability and degradation. In this work, we present a comprehensive empirical study of when OPD and OPSD work, when they fail, and why. We find that OPD on mathematical reasoning is highly sensitive to teacher choice and loss formulation, whereas OPSD fails in our tested settings due to test-time absence of instance-specific privileged information (PI). In contrast, OPSD is effective when PI represents a shared latent rule, such as a system prompt or alignment preference. We identify three failure mechanisms: (1) distribution mismatch between teacher and student caused by conditioning on student-generated prefixes, (2) optimization instability from biased TopK reverse-KL gradients, and (3) an OPSD-specific limitation where the student learns a PI-free policy that aggregates PI-conditioned teachers, which is insufficient when PI is instance-specific. We further show that stop-gradient TopK objectives, RLVR-adapted teachers, and SFT-stabilized students mitigate these failures.
Siqi Zhu, Xuyan Ye, Hongyu Lu +2
1UIUC · 2Renmin University of China · †Work done during an internship at UIUC. +1
On-policy distillation (OPD) has become an important approach to language model post-training. However, despite its performance gains, OPD can also collapse into excessively long and repetitive generation, and the mechanism underlying these divergent outcomes remains poorly understood. We explain these outcomes through a reinforcement learning perspective: the teacher implicitly rewards student behaviors, even those it rarely exhibits itself. From this perspective, our experiments show that OPD improves performance without expanding the student's capabilities. When the implicit reward model is reliable, OPD makes correct responses easier to sample. In contrast, when the preference misaligns with quality, reward hacking happens: the implicit reward model amplifies overlong, repetitive student rollouts, even though it rarely generates such text itself. Guided by this diagnosis, we find that masking unhealthy responses during training and using SFT initialization can each effectively mitigate the collapse. Together, these findings show that OPD amplifies student behaviors favored by the teacher's implicit feedback, shifting the focus from how well the teacher generates to how reliably it evaluates student rollouts. Our code is available at https://github.com/HancCui/opd_hacking.