On-policy distillation (OPD) has emerged as an effective framework for post-training language models by pairing student-generated trajectories with dense token-level supervision from a teacher. However, OPD implicitly assumes that teacher-derived rewards are an appropriate proxy for reasoning progress, and therefore treats all teacher feedback equally during policy optimization. While in practice, this assumption does not always hold. We observe that teacher-derived rewards often conflict with genuine reasoning progress, as reasoning steps with clear reasoning advancement may still receive lower distillation rewards, simply due to deviation from teacher's outputs. To address this mismatch, we propose Reasoning-Progress-Aware Reward Filtering for On-Policy Distillation (R2-OPD), which constructs two within-trajectory rankings of reasoning spans, one from teacher-derived rewards and the other from independently estimated progress reward. Distillation rewards are selectively suppressed whenever the two rankings disagree, reducing supervision that conflicts with reasoning progress while preserving effective teacher guidance. Our approach shows consistent improvement over standard OPD especially regarding reasoning performances.
Figures & tables
Figure 1: Distribution of evaluation outcomes for DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-1.7B on DAPO (500 samples, max 7,168 response tokens).
Figure 2: representative mismatch between reasoning progress and the OPD divergence signal. A reasoning regressive segment ( J9 ) receives a smaller KL divergence under Equation ( 9 ) than segments that derive or recover the correct solution, respectively( J2 and J10 ).
Figure 3: Overview of the proposed Reasoning-Progress-Aware Reward Filtering OPD. Adjacent reasoning spans with consistent progress signals are first merged. For each merged segment, we sum its process rewards and average its token-level KL divergences. Rank conflicts between the resulting segment-level signals are then used to identify unreliable spans, which are masked during the OPD policy update.
Method
AIME 24
AIME 25
Olympiad
Avg.
avg@4
pass@4
avg@4
pass@4
avg@4
pass@4
avg@4
pass@4
Student
22.50
43.33
23.33
36.67
43.19
58.31
29.67
46.10
Teacher
41.67
56.67
30.83
43.44
53.28
68.43
41.92
56.18
OPD ( Agarwal et al. 2024 )
28.33
50.00
22.50
30.00
46.86
62.10
32.55
47.37
E-OPD ( Jin et al. 2026 )
18.33
36.67
12.50
23.33
49.91
64.56
26.91
41.52
TIP-OPD ( Xu et al. 2026b )
17.50
33.33
11.67
20.00
48.24
62.83
25.80
38.72
Table 1: OPD performance on DeepSeek-R1-Distill-Qwen-1.5B with JustRL as the teacher model.
Dataset
Base
OPD
R 2 -OPD
avg@4
pass@4
avg@4
pass@4
avg@4
pass@4
AIME 24
24.17
30.00
22.50
36.67
25.00
40.00
AIME 25
18.75
20.00
27.50
33.33
25.83
36.67
Olympiad
50.10
63.87
53.82
67.09
54.31
67.91
Avg.
31.01
37.96
34.61
45.70
35.04
48.19
Table 2: Transfer performance of R 2 -OPD on Qwen3-1.7B with e3-1.7B as the teacher model.
Figure 4: Sensitivity of R 2 -OPD to the masking ratio q . A moderate ratio of q=30 provides the best overall performance, whereas overly aggressive masking substantially degrades accuracy on AIME 2024 and AIME 2025.
Figure 5: PR–KL rank agreement during training with and without sign-consistent segment merging. Light lines show per-step values, and dark lines show 15-step moving averages. The two vertical axes use the same numerical span.
Dataset
No Merge
R 2 -OPD
Δ
AIME 24
17.5
32.50
15.0
AIME 25
11.67
25.83
14.16
Olympiad
47.8
46.86
-0.96
Table 3: Effect of sign-consistent segment merging on avg@4 accuracy. Δ denotes the improvement of the full R 2 -OPD over its no-merging variant.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Optimization and sequence settings
Training epochs
1
Optimizer
AdamW
Learning rate
5×10−6
Global batch size
64
Maximum prompt length
1,024 tokens
Appendix
Table 4: Optimization, process-reward estimation, and filtering settings used in all R 2 -OPD experiments.
Condition
Action
The string-level pre-check does not find gi in yi , or segmentation yields fewer than two segments.
Skip process-reward rollouts and retain all token-level OPD supervision.
The response has fewer than max(3,nmin) merged segments.
Do not apply segment masking.
No strict adjacent pair is available after process-reward ranking.
Do not apply segment masking.
All candidate segments have zero inconsistency score.
Do not apply segment masking.
At least one candidate has a positive inconsistency score.
Mask up to the response-specific budget; retain every other token.
Appendix
Table 5: Eligibility checks and conservative fallback actions used by R 2 -OPD.
Category
Matched markers
Reconsideration
wait , hold on , let me reconsider , hmm
Correction
actually
Verification
let me check
Alternative reasoning
alternatively
Appendix
Table 6: Complete lexicon of discourse markers used to propose reasoning-segment boundaries.
On-policy distillation is a powerful way to transfer reasoning ability from a strong teacher to a smaller student: the student samples trajectories from its own policy, and the teacher provides dense token-level supervision on the states the student actually visits. However, this supervision is not always reliable: a teacher can assign high likelihood to plausible but incorrect solutions, or low likelihood to correct student solutions that follow different reasoning paths. Unconditionally distilling the teacher can therefore reinforce bad modes or erase useful student behavior. To address these limitations, we introduce RG-OPD: Reward-Gated On-Policy Distillation that uses verifier feedback to decide when teacher logits should be trusted. RG-OPD bridges sparse verifier rewards and dense teacher logits, preserving token-level supervision while filtering misleading teacher signals. Across reasoning and coding benchmarks, RG-OPD produces stronger distilled students, outperforming both vanilla reverse-KL distillation and the recent TSD-KD baseline. At 1K generation length, RG-OPD improves over reverse-KL by 2.9 points and over TSD-KD by 4.9 points; in the long-generation setting, it improves over the untuned student by 8.2 points. Our code is available at https://github.com/UoC-tail/RG-OPD.
Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi +3
On-policy distillation is an alternative post-training method in reinforcement learning that alleviates the constraints imposed by reward models by providing token-level supervision from a teacher model. Although on-policy distillation has been studied and applied across various settings, its fundamental design remains underexplored. In this paper, we introduce a new distillation reward, termed the delta signal, instead of directly imitating the teacher's output distribution. The delta signal is defined as the difference between the teacher model and its base model prior to instruction tuning for reasoning capability. It therefore captures the changes induced by reasoning tuning and provides a more direct signal for transferring reasoning capabilities. Using extensive empirical evidence, we show that the delta signal substantially improves on-policy distillation and refer to the new distillation method as On-Policy Delta Distillation (OPD2). Experiments across mathematics, science, and code-reasoning benchmarks demonstrate that OPD2 consistently outperforms conventional on-policy distillation, enabling reasoning LLMs to achieve strong performance with only a short post-training period. Code will be available at https://github.com/naver-ai/opd2
On-policy distillation (OPD) trains a student language model with dense feedback from a stronger teacher on student-generated trajectories. Yet standard OPD weights token-level distillation terms uniformly, implicitly treating local teacher preference as a proxy for correction utility. A decision's task value, however, depends on how the student completes the subsequent reasoning. This mismatch can cause imitation to suppress viable student strategies or reinforce paths the student cannot reliably execute. Verified trajectory outcomes provide complementary evidence about continuation quality, but do not directly identify the utility of individual decisions. We introduce Reward-Aligned Reweighting for On-Policy Distillation (R2-OPD), which uses outcome agreement and the magnitude of teacher--student disagreement to continuously reallocate teacher supervision. It gives reward-aligned corrections greater relative influence while retaining dense feedback, moving beyond uniform imitation and hard filtering. Our analysis formalizes the mismatch between local teacher preference and student continuation value and establishes sufficient conditions for reallocation to improve first-order task progress over uniform OPD. Across seven mathematical reasoning benchmarks, R2-OPD achieves the highest average accuracy among the compared training methods in both cross-size and same-size distillation. It outperforms standard OPD on all seven benchmarks, with average gains of 3.5 and 2.4 percentage points for 1.7B and 4B students, respectively. An extension to code generation yields an average gain of 1.6 percentage points over standard OPD. These results highlight outcome-guided supervision allocation as an effective way to translate dense teacher feedback into stronger student performance across model scales and task domains.
Haofeng Xu, Junwei Su, Lansong Diao +2
The University of Hong Kong · Alibaba Group · University of Science and Technology of China