Organizations: Key Laboratory of Big Data & Artificial Intelligence in Transportation, Beijing Jiaotong University · School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China · Weixin AI, Tencent, China
On-policy distillation (OPD) mitigates the student's exposure bias by optimizing on student-generated trajectories. However, student-generated prefixes may contain reasoning errors or follow patterns unfamiliar to the teacher, undermining the reliability of its token-level supervision. This degradation becomes more pronounced as prefixes grow longer. To address this teacher-side distribution mismatch, we propose Student-Prefix Teacher Adaptation (SPTA), which adapts the teacher to the states on which it is queried during distillation. Specifically, before OPD starts, we collect student rollout prefixes with random lengths and continually train the teacher to continue from the prefixes using reinforcement learning with verifiable rewards (RLVR). By treating each prefix as context and rewarding correct final answers, the teacher learns to recover from student-induced states rather than imitate the student's reasoning. Then, the adapted teacher is used for standard OPD, with no changes to the distillation objective. Experiments on mathematical reasoning benchmarks show that our SPTA improves average student performance over standard OPD and OPD with task-level RLVR-adapted teachers. These results highlight the importance of adapting teachers not only to solve problems independently, but also to provide reliable guidance on the states their students actually visit.
Figures & tables
Figure 1: Standard OPD and Student-Prefix Teacher Adaptation ( SPTA ). (a) In standard OPD, student rollouts may enter regions unfamiliar to the teacher, reducing supervision reliability. (b) SPTA first adapts the teacher through prefix-conditioned RLVR, using student prefixes as context and final-answer correctness as reward, then freezes it for standard OPD. Shaded regions indicate prefixes familiar to the teacher before and after adaptation.
Figure 2: Student-prefix adaptation improves teacher recovery. Mean accuracy (avg@ 16 across five benchmarks) versus the fraction of a student rollout retained as context. Each panel compares the original teacher with its SPTA -Iter1 version on identical sampled student prefixes; 0% denotes clean-prompt generation. Recovery improves at every evaluated nonzero ratio, even though Qwen3-4B’s clean-prompt accuracy decreases.
Table 1: Student accuracy after distillation (%, avg@ 16 ). Avg. is the unweighted benchmark mean. All distillation runs use a single adaptation/distillation round, and within each pairing they share initialization and settings. Bold marks the best result in each column, including ties; the undistilled S0 row means the original student performance. ∗ Prune-OPD scores are quoted from Yang et al. (2026) and were not reproduced by us, so they are listed for context rather than as a controlled comparison.
Clean
Recovery
Student
Teacher
r=0
r=25%
r=50%
r=75%
r=90%
Avg.
(1) Qwen3-4B
Original
60.99
32.76
28.91
24.82
22.36
31.52
SPTA
54.39
50.46
49.88
48.56
47.68
35.30
Δ
−6.60
+17.70
+20.97
+23.74
+25.32
+3.78
Student πS (reference)
12.49
12.49
12.49
12.49
12.49
–
Table 2: Recovery gains are distinct from clean-prompt gains. Teacher accuracies are means over five benchmarks (avg@ 16 ) at each student-prefix ratio r ; benchmark-level results are in Appendix B . Δ is SPTA minus Original in percentage points. The last column gives the corresponding distilled student’s average from Table 1 , not a teacher metric. The italic reference row in each block is the prefix-source student’s own accuracy, which is constant in r (see text) and serves as the break-even level below which the teacher carries no information the student lacks.
Method
AIME24
AIME25
AMC23
HMMT24
HMMT25
Avg.
OPD
33.54
26.04
65.51
16.04
16.46
31.52
SPTA -OPD (Iter1)
37.71
33.54
71.91
16.04
17.29
35.30
SPTA -OPD-Iter2
39.79
33.96
74.32
17.29
17.50
36.57
Δ (Iter2 − Iter1)
+2.08
+0.42
+2.41
+1.25
+0.21
+1.27
Table 3: Effect of a second SPTA round in configuration (1) (Qwen3-4B → Qwen3-4B-Base), accuracy (%, avg@ 16 ). Iter1 is the SPTA -OPD row of Table 1 ; Iter2 refreshes the prefix source from the Iter1 student and warm-starts both models from the preceding round, so it uses an additional round of compute and is not matched to the rows above it.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Teacher adaptation
Student OPD
Training objective
GRPO
PG-OPD ( k1 )
Inputs per training batch
256
64
Continuations per input
8
4
Sequences per rollout batch
2048
256
PPO mini-batch size (inputs)
64
64
Maximum input length
7168
1024
Appendix
Table 4: Shared training settings across all teacher–student pairings. Input limits apply to the full conditioned input, including the student prefix in Stage I.
Teacher
Prefix
AIME24
AIME25
AMC23
HMMT24
HMMT25
Avg.
(1) Qwen3-4B
Original
0%
72.08
64.17
88.93
35.42
44.37
60.99
Original
25%
38.33
27.08
64.01
17.71
16.67
32.76
Original
50%
31.67
23.54
63.10
15.00
11.25
28.91
Original
75%
26.25
21.67
57.23
11.87
7.08
24.82
Original
90%
25.00
16.88
53.24
10.21
6.46
22.36
Appendix
Table 5: Full teacher recovery results. Accuracy (%, avg@ 16 ) at each student-prefix ratio. All adapted teachers are from Iter1. These are teacher continuation results, not distilled-student scores.
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · Shenzhen Future Network of Intelligence Institute (FNiI-Shenzhen) · Guangdong Provincial Key Laboratory of Future Networks of Intelligence, CUHK(SZ)