Organizations: Key Laboratory of Big Data & Artificial Intelligence in Transportation, Beijing Jiaotong University · School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China · Weixin AI, Tencent, China
On-policy distillation (OPD) mitigates the student's exposure bias by optimizing on student-generated trajectories. However, student-generated prefixes may contain reasoning errors or follow patterns unfamiliar to the teacher, undermining the reliability of its token-level supervision. This degradation becomes more pronounced as prefixes grow longer. To address this teacher-side distribution mismatch, we propose Student-Prefix Teacher Adaptation (SPTA), which adapts the teacher to the states on which it is queried during distillation. Specifically, before OPD starts, we collect student rollout prefixes with random lengths and continually train the teacher to continue from the prefixes using reinforcement learning with verifiable rewards (RLVR). By treating each prefix as context and rewarding correct final answers, the teacher learns to recover from student-induced states rather than imitate the student's reasoning. Then, the adapted teacher is used for standard OPD, with no changes to the distillation objective. Experiments on mathematical reasoning benchmarks show that our SPTA improves average student performance over standard OPD and OPD with task-level RLVR-adapted teachers. These results highlight the importance of adapting teachers not only to solve problems independently, but also to provide reliable guidance on the states their students actually visit.
Figures & tables
Figure 1: Standard OPD and Student-Prefix Teacher Adaptation ( SPTA ). (a) In standard OPD, student rollouts may enter regions unfamiliar to the teacher, reducing supervision reliability. (b) SPTA first adapts the teacher through prefix-conditioned RLVR, using student prefixes as context and final-answer correctness as reward, then freezes it for standard OPD. Shaded regions indicate prefixes familiar to the teacher before and after adaptation.
Figure 2: Student-prefix adaptation improves teacher recovery. Mean accuracy (avg@ 16 across five benchmarks) versus the fraction of a student rollout retained as context. Each panel compares the original teacher with its SPTA -Iter1 version on identical sampled student prefixes; 0% denotes clean-prompt generation. Recovery improves at every evaluated nonzero ratio, even though Qwen3-4B’s clean-prompt accuracy decreases.
Table 1: Student accuracy after distillation (%, avg@ 16 ). Avg. is the unweighted benchmark mean. All distillation runs use a single adaptation/distillation round, and within each pairing they share initialization and settings. Bold marks the best result in each column, including ties; the undistilled S0 row means the original student performance. ∗ Prune-OPD scores are quoted from Yang et al. (2026) and were not reproduced by us, so they are listed for context rather than as a controlled comparison.
Clean
Recovery
Student
Teacher
r=0
r=25%
r=50%
r=75%
r=90%
Avg.
(1) Qwen3-4B
Original
60.99
32.76
28.91
24.82
22.36
31.52
SPTA
54.39
50.46
49.88
48.56
47.68
35.30
Δ
−6.60
+17.70
+20.97
+23.74
+25.32
+3.78
Student πS (reference)
12.49
12.49
12.49
12.49
12.49
–
Table 2: Recovery gains are distinct from clean-prompt gains. Teacher accuracies are means over five benchmarks (avg@ 16 ) at each student-prefix ratio r ; benchmark-level results are in Appendix B . Δ is SPTA minus Original in percentage points. The last column gives the corresponding distilled student’s average from Table 1 , not a teacher metric. The italic reference row in each block is the prefix-source student’s own accuracy, which is constant in r (see text) and serves as the break-even level below which the teacher carries no information the student lacks.
Method
AIME24
AIME25
AMC23
HMMT24
HMMT25
Avg.
OPD
33.54
26.04
65.51
16.04
16.46
31.52
SPTA -OPD (Iter1)
37.71
33.54
71.91
16.04
17.29
35.30
SPTA -OPD-Iter2
39.79
33.96
74.32
17.29
17.50
36.57
Δ (Iter2 − Iter1)
+2.08
+0.42
+2.41
+1.25
+0.21
+1.27
Table 3: Effect of a second SPTA round in configuration (1) (Qwen3-4B → Qwen3-4B-Base), accuracy (%, avg@ 16 ). Iter1 is the SPTA -OPD row of Table 1 ; Iter2 refreshes the prefix source from the Iter1 student and warm-starts both models from the preceding round, so it uses an additional round of compute and is not matched to the rows above it.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Teacher adaptation
Student OPD
Training objective
GRPO
PG-OPD ( k1 )
Inputs per training batch
256
64
Continuations per input
8
4
Sequences per rollout batch
2048
256
PPO mini-batch size (inputs)
64
64
Maximum input length
7168
1024
Appendix
Table 4: Shared training settings across all teacher–student pairings. Input limits apply to the full conditioned input, including the student prefix in Stage I.
Teacher
Prefix
AIME24
AIME25
AMC23
HMMT24
HMMT25
Avg.
(1) Qwen3-4B
Original
0%
72.08
64.17
88.93
35.42
44.37
60.99
Original
25%
38.33
27.08
64.01
17.71
16.67
32.76
Original
50%
31.67
23.54
63.10
15.00
11.25
28.91
Original
75%
26.25
21.67
57.23
11.87
7.08
24.82
Original
90%
25.00
16.88
53.24
10.21
6.46
22.36
Appendix
Table 5: Full teacher recovery results. Accuracy (%, avg@ 16 ) at each student-prefix ratio. All adapted teachers are from Iter1. These are teacher continuation results, not distilled-student scores.
On-policy distillation (OPD) improves the reasoning capabilities of small language models through token-level teacher supervision on student-generated trajectories. Yet can teachers that excel at solving problems independently also guide student reasoning effectively? Prior work shows that when student prefixes follow reasoning paths that differ from the teacher's own or contain errors, teachers can be less accurate when continuing from these prefixes than when solving problems independently. To this end, we propose Prep-OPD, which uses reinforcement learning (RL) before distillation to train the teacher to adapt to the student's existing reasoning state and correct course when errors arise. Training optimizes teacher continuations from fixed student prefixes using final-answer correctness as the reward. The prepared teacher then trains the student through trajectory guidance and token-level supervision. We evaluate Prep-OPD on eight mathematical reasoning benchmarks, using Qwen3-4B-Instruct-2507 as the teacher and Qwen3-0.6B and Qwen3-1.7B as students. With the 4B teacher and 1.7B student, Prep-OPD improves average accuracy over standard OPD and the strongest baseline, Relay-OPD, by 8.28 and 2.30 percentage points, respectively. Controlled experiments further show that teacher RL conditioned on student-generated prefixes yields higher student accuracy than problem-start teacher RL with and without handoff on Qwen3-1.7B. Reusing the same prepared teacher also improves Qwen3-0.6B.
Xiaoyu Ma, Haoyue Liu, Zhichao Wang +2
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · Shenzhen Future Network of Intelligence Institute (FNiI-Shenzhen) · Guangdong Provincial Key Laboratory of Future Networks of Intelligence, CUHK(SZ)
On-policy distillation (OPD), which samples trajectories from the student model and supervises them with a teacher at the token level, avoids relying solely on verifiable terminal rewards and can yield better generalization than off-policy distillation. However, OPD requires expensive on-the-fly sampling of the student policy during training, which substantially increases training cost, especially for long responses. Our initial analysis shows that, during OPD, training signals are often concentrated in the prefix of each output, and that even a short teacher-generated prefix can significantly help the student produce the correct answer. Motivated by these observations, we propose a simple yet effective modification of OPD: we apply the distillation objective only to prefixes of student-generated outputs and terminate each sampling early during distillation. Experiments on a suite of AI-for-Math and out-of-domain benchmarks show that on-policy prefix distillation matches the performance of full OPD while reducing training FLOP by 2x-47x.
On-policy distillation (OPD) trains a student on its own responses using token-level feedback from a teacher. As responses grow longer, the student's prefix can depart from the teacher's own reasoning, weakening its feedback on later tokens. Updates based on this feedback can reinforce the same deviations in later responses and destabilize training. Prior approaches such as length control and loss reweighting limit learning from problematic responses but leave their prefixes unchanged. Fixed cutoffs can also discard later reasoning. We propose Adaptive Teacher Intervention for On-Policy Distillation (ATI-OPD) to improve subsequent teacher guidance by revising the student's reasoning prefix. ATI-OPD replaces the earliest high-disagreement segment with a short teacher segment, then lets the student resume its own reasoning from the revised text. To handle renewed disagreement, it excludes persistently divergent suffixes from the distillation loss while retaining useful student continuation. Training diagnostics show less teacher-student disagreement, fewer gradient spikes, and fewer extremely long responses. Across five mathematical reasoning benchmarks and two teacher-student pairs, ATI-OPD improves average student-only accuracy over standard OPD.
Jiannan Guan, Youxin Jiang, Yutai Hou +9
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology · Huawei Technologies Co., Ltd.