Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress

Chen Yang; Haiyuan Wan; Rengrong Xiong; Yize Chen; Danny H. K. Tsang

cs.AIAug 19, 2026

Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress

Authors: Chen Yang, Haiyuan Wan, Rengrong Xiong, Yize Chen, Danny H. K. Tsang

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Tsinghua University · Zhejiang University · University of Alberta

Abstract

On-policy distillation (OPD) has emerged as an effective framework for post-training language models by pairing student-generated trajectories with dense token-level supervision from a teacher. However, OPD implicitly assumes that teacher-derived rewards are an appropriate proxy for reasoning progress, and therefore treats all teacher feedback equally during policy optimization. While in practice, this assumption does not always hold. We observe that teacher-derived rewards often conflict with genuine reasoning progress, as reasoning steps with clear reasoning advancement may still receive lower distillation rewards, simply due to deviation from teacher's outputs. To address this mismatch, we propose Reasoning-Progress-Aware Reward Filtering for On-Policy Distillation (R2-OPD), which constructs two within-trajectory rankings of reasoning spans, one from teacher-derived rewards and the other from independently estimated progress reward. Distillation rewards are selectively suppressed whenever the two rankings disagree, reducing supervision that conflicts with reasoning progress while preserving effective teacher guidance. Our approach shows consistent improvement over standard OPD especially regarding reasoning performances.

Figures & tables

Figure 1: Distribution of evaluation outcomes for DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-1.7B on DAPO (500 samples, max 7,168 response tokens).

Figure 2: representative mismatch between reasoning progress and the OPD divergence signal. A reasoning regressive segment ( $J_{9}$ ) receives a smaller KL divergence under Equation ( 9 ) than segments that derive or recover the correct solution, respectively( $J_{2}$ and $J_{10}$ ).

Figure 3: Overview of the proposed Reasoning-Progress-Aware Reward Filtering OPD. Adjacent reasoning spans with consistent progress signals are first merged. For each merged segment, we sum its process rewards and average its token-level KL divergences. Rank conflicts between the resulting segment-level signals are then used to identify unreliable spans, which are masked during the OPD policy update.

Method	AIME 24		AIME 25		Olympiad		Avg.
Method	avg@4	pass@4	avg@4	pass@4	avg@4	pass@4	avg@4	pass@4
Student	22.50	43.33	23.33	36.67	43.19	58.31	29.67	46.10
Teacher	41.67	56.67	30.83	43.44	53.28	68.43	41.92	56.18
OPD ( Agarwal et al. 2024 )	28.33	50.00	22.50	30.00	46.86	62.10	32.55	47.37
E-OPD ( Jin et al. 2026 )	18.33	36.67	12.50	23.33	49.91	64.56	26.91	41.52
TIP-OPD ( Xu et al. 2026b )	17.50	33.33	11.67	20.00	48.24	62.83	25.80	38.72

Table 1: OPD performance on DeepSeek-R1-Distill-Qwen-1.5B with JustRL as the teacher model.

Dataset	Base		OPD		R 2 -OPD
Dataset	avg@4	pass@4	avg@4	pass@4	avg@4	pass@4
AIME 24	24.17	30.00	22.50	36.67	25.00	40.00
AIME 25	18.75	20.00	27.50	33.33	25.83	36.67
Olympiad	50.10	63.87	53.82	67.09	54.31	67.91
Avg.	31.01	37.96	34.61	45.70	35.04	48.19

Table 2: Transfer performance of R 2 -OPD on Qwen3-1.7B with e3-1.7B as the teacher model.

Figure 4: Sensitivity of R 2 -OPD to the masking ratio $q$ . A moderate ratio of $q=30$ provides the best overall performance, whereas overly aggressive masking substantially degrades accuracy on AIME 2024 and AIME 2025.

Figure 5: PR–KL rank agreement during training with and without sign-consistent segment merging. Light lines show per-step values, and dark lines show 15-step moving averages. The two vertical axes use the same numerical span.

Dataset	No Merge	R 2 -OPD	$\Delta$
AIME 24	17.5	32.50	15.0
AIME 25	11.67	25.83	14.16
Olympiad	47.8	46.86	-0.96

Table 3: Effect of sign-consistent segment merging on avg@4 accuracy.

\Delta

denotes the improvement of the full R 2 -OPD over its no-merging variant.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Optimization and sequence settings
Training epochs	1
Optimizer	AdamW
Learning rate	$5\times 10^{-6}$
Global batch size	64
Maximum prompt length	1,024 tokens

Appendix

Table 4: Optimization, process-reward estimation, and filtering settings used in all R 2 -OPD experiments.

Condition	Action
The string-level pre-check does not find $g_{i}$ in $y_{i}$ , or segmentation yields fewer than two segments.	Skip process-reward rollouts and retain all token-level OPD supervision.
The response has fewer than $\max(3,n_{\min})$ merged segments.	Do not apply segment masking.
No strict adjacent pair is available after process-reward ranking.	Do not apply segment masking.
All candidate segments have zero inconsistency score.	Do not apply segment masking.
At least one candidate has a positive inconsistency score.	Mask up to the response-specific budget; retain every other token.

Appendix

Table 5: Eligibility checks and conservative fallback actions used by R 2 -OPD.

Category	Matched markers
Reconsideration	wait , hold on , let me reconsider , hmm
Correction	actually
Verification	let me check
Alternative reasoning	alternatively

Appendix

Table 6: Complete lexicon of discourse markers used to propose reasoning-segment boundaries.

Explore similar work

Reward-Gated On-Policy Distillation
Jul 4, 2026Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi +3Language Model Distillation On-Policy Distillation
University of Cologne · AWS AI · UT Austin +1
On-Policy Delta Distillation
Jul 16, 2026Byeongho Heo, Jaehui Hwang, Sangdoo Yun +1RL for Language Model Reasoning Language Model Distillation
NAVER AI Lab
Reward-Aligned Reweighting for On-Policy Distillation
Sep 28, 2026Haofeng Xu, Junwei Su, Lansong Diao +2Language Model Distillation On-Policy Distillation
The University of Hong Kong · Alibaba Group · University of Science and Technology of China

Jul 4, 2026J/K move · Enter open · S save

Reward-Gated On-Policy Distillation

Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi, Chanakya Ekbote+2

University of Cologne · AWS AI · UT Austin+1

Language Model Distillation On-Policy Distillation LLM Reasoning

Open paper AskPDF arXiv

On-policy distillation is a powerful way to transfer reasoning ability from a strong teacher to a smaller student: the student samples trajectories from its own policy, and the teacher provides dense token-level supervision on the states the student actually visits. However, this supervision is not always reliable: a teacher can assign high likelihood to plausible but incorrect solutions, or low likelihood to correct student solutions that follow different reasoning paths. Unconditionally distilling the teacher can therefore reinforce bad modes or erase useful student behavior. To address these limitations, we introduce RG-OPD: Reward-Gated On-Policy Distillation that uses verifier feedback to decide when teacher logits should be trusted. RG-OPD bridges sparse verifier rewards and dense teacher logits, preserving token-level supervision while filtering misleading teacher signals. Across reasoning and coding benchmarks, RG-OPD produces stronger distilled students, outperforming both vanilla reverse-KL distillation and the recent TSD-KD baseline. At 1K generation length, RG-OPD improves over reverse-KL by 2.9 points and over TSD-KD by 4.9 points; in the long-generation setting, it improves over the untuned student by 8.2 points. Our code is available at https://github.com/UoC-tail/RG-OPD.

Abstract

Explore similar work

Reward-Gated On-Policy Distillation

On-Policy Delta Distillation

Reward-Aligned Reweighting for On-Policy Distillation