On-policy distillation (OPD) improves language-model reasoning through dense teacher feedback on student-generated responses. We first observe an amplitude disparity in this feedback: on math reasoning responses, style-related positions exhibit 3.5 times the mean absolute teacher--student log-probability difference of mathematical positions. Scaling OPD updates by these discrepancies thus directs much of the update strength toward matching the teacher's expression. We instead take the student's own pretrained base as a reference: the student--base difference records the token preferences acquired during post-training and provides supervision that does not depend on the student's distance from the teacher. Building on this reference, we propose Student-Delta On-Policy Distillation (SD-OPD), which uses centered student--base differences as update coefficients, keeps each coefficient only when its sign agrees with centered teacher--student feedback, and recomputes the student signal every training round. Across 5 teacher--student configurations with Qwen3 students of 1.7B to 8B parameters, SD-OPD improves six-task mathematics average pass@1 (avg@16) over OPD by 7.76 to 10.37 percentage points. Code is available at https://github.com/aim-uofa/sd-opd.
Figures & tables
Figure 1: SD-OPD overview. Each round scores student-generated responses on shared prefixes. The student–base difference supplies the coefficient, and teacher–student feedback selects its admissible direction. Here, AT=AOPD , Adirect=ASD-OPD , and ρt=wt . The diagram shows full-vocabulary centering; training uses the top-1,024 approximation in Appendix A.2 . Advantages remain fixed within each round and are recomputed after refreshing the student. Teacher T and base SB remain frozen.
Figure 2: Feedback on the same student responses: raw (a) teacher–student rt and (b) student–base dt , and both after the same-sign mask mt=1[rtdt>0] , (c) mtrt and (d) mtdt . Panels (c) and (d) retain identical positions and directions and differ only in magnitude source. All panels share the scale, categories, and averaging procedure in Equation 2 ; filtered-out positions contribute zero and remain in the denominator.
Figure 3: Student–base preferences and teacher selection. (a, b) Fifty representative strings per sign of dt , with larger, darker text indicating greater averaged amplitude. A piecewise-linear size scale separates ten high-amplitude examples from smaller context strings. (c) Six representative strings: parts of dt with dt>0 (right of zero) and dt<0 (left), split into the part retained by 1[rtdt>0] and the part removed, with the retained share of absolute amplitude. Appendix C.4 describes aggregation and string selection.
Method
AIME24
AIME25
AIME26
HMMT25
AMC23
MATH500
Average
Qwen3-4B-Instruct-2507 → Qwen3-1.7B
Initial student
13.12
8.54
7.71
6.67
44.69
69.40
25.02
OPD
32.08
26.25
24.38
16.88
73.75
83.90
42.87
ExOPD
40.42
28.12
28.75
20.00
79.06
86.11
47.08
SD-OPD
45.42
33.33
40.62
21.04
86.72
88.72
52.64
Δ vs. OPD
+13.33
+7.08
+16.25
+4.17
+12.97
+4.83
+9.77
Table 1: Pass@1 (%) across 5 teacher–student settings (teacher → student), averaged over 16 sampled responses per problem. Average is the six-task mean; averages and gains use unrounded scores. Gray rows at the bottom of each block give teacher reference scores; boldface marks the highest student scores within each setting, and green rows report SD-OPD gains over OPD in percentage points.
Variant
AIME24
AIME25
AIME26
HMMT25
AMC23
MATH500
Average
SD-OPD
45.42
33.33
40.62
21.04
86.72
88.72
52.64
Frozen
23.75
17.08
13.12
11.25
63.12
78.46
34.47
Ungated
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Table 2: Teacher guidance and source refresh ablations for the 1.7B student, using matched training and evaluation settings. Frozen substitutes the initial student in the student–base difference. Ungated removes the teacher sign gate and teacher scoring.
Figure 4: Early training dynamics with Qwen3-30B-A3B-Instruct-2507 teaching Qwen3-1.7B. Each step is one rollout round with four optimizer updates. Length is the rollout mean; entropy averages the four logged update means. Curves are unsmoothed. Appendix B.1 shows all 272 training steps.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Rollout rounds
272
Prompts per round
64
Responses per prompt
4
Response batch / microbatch
64 / 1
Updates per round / total
4 / 1,088
Maximum prompt tokens
1,024
Appendix
Table A1: Shared training hyperparameters for OPD and SD-OPD . The same settings are used for the Frozen and Ungated ablations.
Figure B1: Full training trajectories in the 30B-to-1.7B setting: 272 rollout steps for each method. The dotted line marks step 50, the endpoint of Figure 4 . Each step contains four optimizer updates. Length is the rollout mean; entropy averages the four update means. All recorded points are shown without smoothing.
Ungated
SD-OPD
Round
Length
Entropy
Length
Entropy
1
1752.5
0.2536
1663.9
0.2628
20
1793.8
0.3392
5669.3
0.1966
40
280.6
2.0328
5880.6
0.1636
60
1519.9
2.1405
5777.3
0.1469
100
30.1
0.5400
4906.2
0.1170
Appendix
Table B1: Training dynamics at selected rounds. Length is the mean number of response tokens generated at the start of the round. Entropy is the arithmetic mean of the four logged per-update response-token entropy means. Each method generates its own responses.
Style-to-math ratio
Removed (%)
Before
After
Setting
OPD
S–B
OPD
S–B
Style
Math
Original
3.530
4.097
3.717
3.064
69.1
58.7
30B-A3B teacher
3.262
4.097
3.739
3.520
63.3
57.2
GSM8K data †
5.646
5.246
3.811
2.318
81.0
57.1
Appendix
Table D1: Category-level feedback patterns for Qwen3-1.7B across teachers and datasets. OPD uses teacher–student log-probability differences; student–base uses the student’s differences from its pretrained base. Ratios compare mean absolute amplitudes at style-related and mathematical positions, aggregated as in Equation 2 . Before/After refer to sign gating, which retains positions where the two signals agree; S–B denotes student–base, and Removed gives the share of student–base amplitude removed by gating. † For GSM8K, averages use responses and questions containing the category: style-related tokens occur in 8,105 responses across 1,021 questions; mathematical tokens occur in all responses.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
On-policy distillation (OPD) improves the reasoning capabilities of small language models through token-level teacher supervision on student-generated trajectories. Yet can teachers that excel at solving problems independently also guide student reasoning effectively? Prior work shows that when student prefixes follow reasoning paths that differ from the teacher's own or contain errors, teachers can be less accurate when continuing from these prefixes than when solving problems independently. To this end, we propose Prep-OPD, which uses reinforcement learning (RL) before distillation to train the teacher to adapt to the student's existing reasoning state and correct course when errors arise. Training optimizes teacher continuations from fixed student prefixes using final-answer correctness as the reward. The prepared teacher then trains the student through trajectory guidance and token-level supervision. We evaluate Prep-OPD on eight mathematical reasoning benchmarks, using Qwen3-4B-Instruct-2507 as the teacher and Qwen3-0.6B and Qwen3-1.7B as students. With the 4B teacher and 1.7B student, Prep-OPD improves average accuracy over standard OPD and the strongest baseline, Relay-OPD, by 8.28 and 2.30 percentage points, respectively. Controlled experiments further show that teacher RL conditioned on student-generated prefixes yields higher student accuracy than problem-start teacher RL with and without handoff on Qwen3-1.7B. Reusing the same prepared teacher also improves Qwen3-0.6B.
Xiaoyu Ma, Haoyue Liu, Zhichao Wang +2
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · Shenzhen Future Network of Intelligence Institute (FNiI-Shenzhen) · Guangdong Provincial Key Laboratory of Future Networks of Intelligence, CUHK(SZ)
On-policy distillation (OPD) trains a student on its own generated responses using dense, token-level supervision from a stronger teacher. Vanilla OPD treats all teacher signals equally, assuming that the teacher's supervision is equally important for every token. However, teacher signals at different tokens may have very different effects on the student's performance: some correct important reasoning errors, while others have little effect on the final answer. Motivated by this observation, we introduce Dr. OPD (OPD Done Right), which defines the optimal weighted OPD to maximize the student's performance. We formulate Dr. OPD as a bilevel optimization problem in which the student learns from weighted teacher supervision, while the weights are selected to maximize the expected reward of the resulting student. To solve Dr. OPD, we develop an efficient iterative solver that updates the token weights and student policy alternatively. At each round, it updates weights in closed form and then takes one gradient step on the resulting weighted OPD objective. Under regularity conditions, we show that this weighted update achieves a higher expected reward than a vanilla OPD update. Empirically, across strong-to-weak and same-size distillation on math and code, Dr. OPD consistently outperforms all evaluated baselines. In particular, in the strong-to-weak distillation setting, Dr. OPD improves average math performance by 9.7 points over vanilla OPD, and enables the smaller student to surpass its larger teacher.
On-policy distillation (OPD) trains a student on its own responses using token-level feedback from a teacher. As responses grow longer, the student's prefix can depart from the teacher's own reasoning, weakening its feedback on later tokens. Updates based on this feedback can reinforce the same deviations in later responses and destabilize training. Prior approaches such as length control and loss reweighting limit learning from problematic responses but leave their prefixes unchanged. Fixed cutoffs can also discard later reasoning. We propose Adaptive Teacher Intervention for On-Policy Distillation (ATI-OPD) to improve subsequent teacher guidance by revising the student's reasoning prefix. ATI-OPD replaces the earliest high-disagreement segment with a short teacher segment, then lets the student resume its own reasoning from the revised text. To handle renewed disagreement, it excludes persistently divergent suffixes from the distillation loss while retaining useful student continuation. Training diagnostics show less teacher-student disagreement, fewer gradient spikes, and fewer extremely long responses. Across five mathematical reasoning benchmarks and two teacher-student pairs, ATI-OPD improves average student-only accuracy over standard OPD.
Jiannan Guan, Youxin Jiang, Yutai Hou +9
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology · Huawei Technologies Co., Ltd.