On-policy distillation (OPD) improves language-model reasoning through dense teacher feedback on student-generated responses. We first observe an amplitude disparity in this feedback: on math reasoning responses, style-related positions exhibit 3.5 times the mean absolute teacher--student log-probability difference of mathematical positions. Scaling OPD updates by these discrepancies thus directs much of the update strength toward matching the teacher's expression. We instead take the student's own pretrained base as a reference: the student--base difference records the token preferences acquired during post-training and provides supervision that does not depend on the student's distance from the teacher. Building on this reference, we propose Student-Delta On-Policy Distillation (SD-OPD), which uses centered student--base differences as update coefficients, keeps each coefficient only when its sign agrees with centered teacher--student feedback, and recomputes the student signal every training round. Across 5 teacher--student configurations with Qwen3 students of 1.7B to 8B parameters, SD-OPD improves six-task mathematics average pass@1 (avg@16) over OPD by 7.76 to 10.37 percentage points. Code is available at https://github.com/aim-uofa/sd-opd.
Figures & tables
Figure 1: SD-OPD overview. Each round scores student-generated responses on shared prefixes. The student–base difference supplies the coefficient, and teacher–student feedback selects its admissible direction. Here, AT=AOPD , Adirect=ASD-OPD , and ρt=wt . The diagram shows full-vocabulary centering; training uses the top-1,024 approximation in Appendix A.2 . Advantages remain fixed within each round and are recomputed after refreshing the student. Teacher T and base SB remain frozen.
Figure 2: Feedback on the same student responses: raw (a) teacher–student rt and (b) student–base dt , and both after the same-sign mask mt=1[rtdt>0] , (c) mtrt and (d) mtdt . Panels (c) and (d) retain identical positions and directions and differ only in magnitude source. All panels share the scale, categories, and averaging procedure in Equation 2 ; filtered-out positions contribute zero and remain in the denominator.
Figure 3: Student–base preferences and teacher selection. (a, b) Fifty representative strings per sign of dt , with larger, darker text indicating greater averaged amplitude. A piecewise-linear size scale separates ten high-amplitude examples from smaller context strings. (c) Six representative strings: parts of dt with dt>0 (right of zero) and dt<0 (left), split into the part retained by 1[rtdt>0] and the part removed, with the retained share of absolute amplitude. Appendix C.4 describes aggregation and string selection.
Method
AIME24
AIME25
AIME26
HMMT25
AMC23
MATH500
Average
Qwen3-4B-Instruct-2507 → Qwen3-1.7B
Initial student
13.12
8.54
7.71
6.67
44.69
69.40
25.02
OPD
32.08
26.25
24.38
16.88
73.75
83.90
42.87
ExOPD
40.42
28.12
28.75
20.00
79.06
86.11
47.08
SD-OPD
45.42
33.33
40.62
21.04
86.72
88.72
52.64
Δ vs. OPD
+13.33
+7.08
+16.25
+4.17
+12.97
+4.83
+9.77
Table 1: Pass@1 (%) across 5 teacher–student settings (teacher → student), averaged over 16 sampled responses per problem. Average is the six-task mean; averages and gains use unrounded scores. Gray rows at the bottom of each block give teacher reference scores; boldface marks the highest student scores within each setting, and green rows report SD-OPD gains over OPD in percentage points.
Variant
AIME24
AIME25
AIME26
HMMT25
AMC23
MATH500
Average
SD-OPD
45.42
33.33
40.62
21.04
86.72
88.72
52.64
Frozen
23.75
17.08
13.12
11.25
63.12
78.46
34.47
Ungated
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Table 2: Teacher guidance and source refresh ablations for the 1.7B student, using matched training and evaluation settings. Frozen substitutes the initial student in the student–base difference. Ungated removes the teacher sign gate and teacher scoring.
Figure 4: Early training dynamics with Qwen3-30B-A3B-Instruct-2507 teaching Qwen3-1.7B. Each step is one rollout round with four optimizer updates. Length is the rollout mean; entropy averages the four logged update means. Curves are unsmoothed. Appendix B.1 shows all 272 training steps.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Rollout rounds
272
Prompts per round
64
Responses per prompt
4
Response batch / microbatch
64 / 1
Updates per round / total
4 / 1,088
Maximum prompt tokens
1,024
Appendix
Table A1: Shared training hyperparameters for OPD and SD-OPD . The same settings are used for the Frozen and Ungated ablations.
Figure B1: Full training trajectories in the 30B-to-1.7B setting: 272 rollout steps for each method. The dotted line marks step 50, the endpoint of Figure 4 . Each step contains four optimizer updates. Length is the rollout mean; entropy averages the four update means. All recorded points are shown without smoothing.
Ungated
SD-OPD
Round
Length
Entropy
Length
Entropy
1
1752.5
0.2536
1663.9
0.2628
20
1793.8
0.3392
5669.3
0.1966
40
280.6
2.0328
5880.6
0.1636
60
1519.9
2.1405
5777.3
0.1469
100
30.1
0.5400
4906.2
0.1170
Appendix
Table B1: Training dynamics at selected rounds. Length is the mean number of response tokens generated at the start of the round. Entropy is the arithmetic mean of the four logged per-update response-token entropy means. Each method generates its own responses.
Style-to-math ratio
Removed (%)
Before
After
Setting
OPD
S–B
OPD
S–B
Style
Math
Original
3.530
4.097
3.717
3.064
69.1
58.7
30B-A3B teacher
3.262
4.097
3.739
3.520
63.3
57.2
GSM8K data †
5.646
5.246
3.811
2.318
81.0
57.1
Appendix
Table D1: Category-level feedback patterns for Qwen3-1.7B across teachers and datasets. OPD uses teacher–student log-probability differences; student–base uses the student’s differences from its pretrained base. Ratios compare mean absolute amplitudes at style-related and mathematical positions, aggregated as in Equation 2 . Before/After refer to sign gating, which retains positions where the two signals agree; S–B denotes student–base, and Removed gives the share of student–base amplitude removed by gating. † For GSM8K, averages use responses and questions containing the category: style-related tokens occur in 8,105 responses across 1,021 questions; mathematical tokens occur in all responses.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · Shenzhen Future Network of Intelligence Institute (FNiI-Shenzhen) · Guangdong Provincial Key Laboratory of Future Networks of Intelligence, CUHK(SZ)