On-policy distillation (OPD) trains a student language model with dense feedback from a stronger teacher on student-generated trajectories. Yet standard OPD weights token-level distillation terms uniformly, implicitly treating local teacher preference as a proxy for correction utility. A decision's task value, however, depends on how the student completes the subsequent reasoning. This mismatch can cause imitation to suppress viable student strategies or reinforce paths the student cannot reliably execute. Verified trajectory outcomes provide complementary evidence about continuation quality, but do not directly identify the utility of individual decisions. We introduce Reward-Aligned Reweighting for On-Policy Distillation (R2-OPD), which uses outcome agreement and the magnitude of teacher--student disagreement to continuously reallocate teacher supervision. It gives reward-aligned corrections greater relative influence while retaining dense feedback, moving beyond uniform imitation and hard filtering. Our analysis formalizes the mismatch between local teacher preference and student continuation value and establishes sufficient conditions for reallocation to improve first-order task progress over uniform OPD. Across seven mathematical reasoning benchmarks, R2-OPD achieves the highest average accuracy among the compared training methods in both cross-size and same-size distillation. It outperforms standard OPD on all seven benchmarks, with average gains of 3.5 and 2.4 percentage points for 1.7B and 4B students, respectively. An extension to code generation yields an average gain of 1.6 percentage points over standard OPD. These results highlight outcome-guided supervision allocation as an effective way to translate dense teacher feedback into stronger student performance across model scales and task domains.
Figures & tables
Method
HumanEval+
LiveCodeBench
Avg.
Student
64.0
21.1
42.6
Teacher
78.0
46.5
62.3
Vanilla OPD
75.4
43.7
59.6
OPDVR
76.2
44.2
60.2
R 2 -OPD
77.3
45.1
61.2
Table 2: Code-generation avg@4 (%) under cross-size distillation from Qwen3-4B-GRPO to Qwen3-1.7B-Base. Bold and underlined scores indicate the best and second-best results among student-training methods.
Variant
AIME24
MATH500
Minerva
Δavg
R 2 -OPD (full)
30.1
77.8
29.4
–
Reversed alignment
25.4
73.5
26.9
−3.8
Outcome-permuted
27.9
75.9
28.5
−1.7
Sign-only gate
27.3
76.6
28.7
−1.6
Magnitude-only gate
26.1
75.8
28.2
−2.4
No mass-norm
28.1
76.7
29.8
−0.9
Table 3: Accuracy (%) for ablation studies and GRPO composition. Δavg denotes the change in mean accuracy across the three benchmarks relative to full R 2 -OPD.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Optimization
Optimizer
AdamW
Learning rate ( η )
1×10−6
Schedule / warmup
Constant / none
AdamW (β1,β2)
(0.9,0.99)
AdamW ϵ
10−8
Appendix
Table 4: Training settings and mathematical evaluation.
Variant
AIME24
AIME25
AMC23
AMC24
MATH500
Minerva
Olympiad
Δavg
R 2 -OPD (full)
30.1
24.5
65.0
54.4
77.8
29.4
45.9
–
Success-only
31.3
22.3
63.7
50.0
76.3
29.1
45.8
-1.2
Failure-only
28.1
20.0
63.7
52.2
75.1
28.7
43.7
-2.2
Appendix
Table 5: Accuracy (%) on seven mathematical reasoning benchmarks for outcome-branch controls. Δavg reports the change in the seven-benchmark average relative to full R 2 -OPD.
University of Chinese Academy of Sciences · 2Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences, Beijing, China · 3Xiaohongshu