SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting

Binbin Zheng; Xing Ma; Yiheng Liang; Jingqing Ruan; Xiaoliang Fu; Kepeng Lin; Benchang Zhu; Ke Zeng

cs.LGApr 12, 2026

SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting

Authors: Binbin Zheng, Xing Ma, Yiheng Liang, Jingqing Ruan, Xiaoliang Fu, Kepeng Lin, Benchang Zhu, Ke Zeng,

Organizations: University of Science and Technology of China · Meituan LongCat Interaction Team · Nanjing University · Fudan University ·

Abstract

On-policy reinforcement learning has become the dominant paradigm for reasoning alignment in large language models, yet its sparse, outcome-level rewards make token-level credit assignment notoriously difficult. On-Policy Distillation (OPD) alleviates this by introducing dense, token-level KL supervision from a teacher model, but typically applies this supervision uniformly across all rollouts, ignoring fundamental differences in signal quality. We propose Signal-Calibrated On-Policy Distillation Enhancement (SCOPE), a dual-path adaptive training framework that routes on-policy rollouts by correctness into two complementary supervision paths. For incorrect trajectories, SCOPE performs teacher-perplexity-weighted KL distillation to prioritize instances where the teacher demonstrates genuine corrective capability, while down-weighting unreliable guidance. For correct trajectories, it applies student-perplexity-weighted MLE to concentrate reinforcement on low-confidence samples at the capability boundary rather than over-reinforcing already mastered ones. Both paths employ a group-level normalization to adaptively calibrate weight distributions, accounting for the intrinsic difficulty variance across prompts. Extensive experiments on six reasoning benchmarks show that SCOPE achieves an average relative improvement of 11.42% in Avg@32 and 7.30% in Pass@32 over competitive baselines, demonstrating its consistent effectiveness.

Figures & tables

Figure 1: (a) Performance changes on the AIME24 benchmark before and after training. Both PSR and OPD training enhance pass@1 at the expense of pass@32, highlighting a clear trade-off between accuracy and reasoning diversity. (b) Recovery rate of the teacher model across varying truncation levels, conditioned on truncated student error trajectories as prefixes and stratified by perplexity.

Figure 2: (a) Standard OPD applies uniform supervision to all samples. (b) Our SCOPE framework refines the learning process by first dividing trajectories into correct $\Omega_{C}$ and incorrect $\Omega_{W}$ sets, applying dual-path perplexity-based weighting, and finally optimizing the weighted branches via a unified objective.

Model	AIME24		AIME25		AMC23		MATH500		Minerva		Olympiad		Average
	A@32	P@32	A@32	P@32	A@32	P@32	A@32	P@32	A@32	P@32	A@32	P@32	A@32	P@32
Teacher: SkyWork-OR1-7B $\longrightarrow$ Student: DeepSeek-R1-Distill-Qwen-1.5B
R1-Distill-Qwen-1.5B	29.4	76.5	23.9	46.9	72.7	94.7	84.6	97.3	32.3	55.9	44.2	67.5	47.9	73.1
w/ GRPO	35.5	68.3	24.5	45.1	75.1	95.0	87.0	96.7	35.1	53.5	40.5	67.7	49.6	71.1
w/ KD	26.6	71.4	22.2	45.6	69.1	96.3	84.1	97.4	30.7	54.1	39.7	66.9	45.4	72.0
w/ OPD	40.2	75.0	28.9	48.5	75.9	95.0	89.0	97.7	34.9	53.0	44.9	69.3	52.3	73.1

Table 1: Main results on mathematical reasoning benchmarks under different teacher–student configurations. We report Avg@32 (A@32) and Pass@32 (P@32) for each benchmark. Bold denotes the best performance and underlined the second-best.

Panel of Figure 3 — Figure 3: Training dynamics comparing GRPO, OPD, and SCOPE (Ours): (a) entropy loss across training steps, and Avg@32 (%) performance on (b) AIME24 and (c) AIME25.

Panel of Figure 4 — Figure 4: Pass@ $k$ (%) performance comparison of GRPO, OPD, and SCOPE (Ours) on the AIME24, AIME25, and AMC23 benchmarks using the DeepSeek-R1-Distill-Qwen-1.5B model.

Figure 5: Ablation study on the dual-path adaptive weighting mechanism (DPAW) of SCOPE. We report the performance of Avg@32 (%) and Pass@32 (%) on AIME24 (a) and AIME25 (b).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	GRPO	OPD	SCOPE
Optimization Parameters
Learning Rate	$5\times 10^{-5}$	$5\times 10^{-5}$	$5\times 10^{-5}$
LR Scheduler	Constant	Constant	Constant
Weight Decay	0.01	0.01	0.01
Train Batch Size (Prompts)	256	256	256
Mini Batch Size (Prompts)	256	256	256

Appendix

Table 3: Training config for GRPO, OPD, and SCOPE

Parameter	Value
Max Tokens	32768
Temperature	0.6
Top-p	0.95
Top-k	20
Repetition Penalty (Qwen3-1.7B)	1.08
Repetition Penalty (Distill-1.5B)	1.0

Appendix

Table 4: Evaluation parameters for all models.

Benchmark	Evaluation Focus & Characteristics
AIME 2024	Evaluates precise arithmetic and algebraic reasoning under strict answer-format constraints, where solutions are typically given as integer-valued responses.
AIME 2025	Serves as a held-out benchmark for assessing whether models can handle unseen and challenging mathematical reasoning patterns.
AMC 2023	Includes foundational competition-style problems in algebra, geometry, and combinatorics, serving as a measure of core mathematical problem-solving ability.
MATH-500	Uses a representative subset of the MATH benchmark spanning multiple mathematical areas, requiring symbolic manipulation, formula interpretation, and robust answer parsing.
Minerva	Consists of STEM-oriented mathematical problems, emphasizing technical notation, quantitative reasoning, and multi-step solution construction.
OlympiadBench	Collects difficult olympiad-style problems, targeting rigorous logical reasoning, advanced mathematical generalization, and complex problem-solving skills.

Appendix

Table 5: Overview of evaluation benchmarks and their key characteristics, covering mathematical reasoning and code generation tasks.

Bucket	PPL Mean	NLL Mean
Q1 (Lowest PPL)	1.361	0.305
Q2	1.565	0.448
Q3	1.710	0.536
Q4 (Highest PPL)	2.383	0.688
Overall	1.755	0.494

Appendix

Table 6: Teacher PPL and NLL statistics for each perplexity bucket over incorrect student trajectories.

Trunc. Ratio	Q1	Q2	Q3	Q4	Q1–Q4 Gap
0.2	64.9	59.7	54.0	45.4	+19.4
0.4	55.8	53.2	50.1	40.2	+15.6
0.6	44.8	43.1	41.0	34.5	+10.3
0.8	35.8	35.3	32.6	28.6	+7.2

Appendix

Table 7: Teacher error recovery rate (%) under different truncation ratios and PPL buckets. Each cell reports the mean accuracy over

n=4

completions per sample. The Q1–Q4 gap (rightmost column) quantifies the within-truncation-level spread attributable to PPL stratification.

Figure 6: Impact of the temperature hyperparameter

\tau

on model performance across AIME24, AIME25, and AMC23. The results indicate that the default configuration of

\tau=1.0

consistently yields the best performance across all benchmarks compared to

\tau=0.5

and

\tau=2.0

Component	GRPO	OPD	SCOPE (Ours)
Generation	264.5	164.5	247.7
Old Logprob	34.0	5.2	31.4
Reward Computation	4.8	0.7	4.6
Actor Update	151.8	22.9	154.1
Teacher Scoring	—	31.2	200.0
Total (step)	459.0	227.5	641.9

Appendix

Table 8: Per-step wall-clock time breakdown (seconds) for each training method. Values are means over the stable training region. The number of rollouts is set to 8 for GRPO and SCOPE, and 1 for OPD.

Problem	Let $a_{1},a_{2},\ldots,a_{100}$ be integers such that $\frac{\sum a_{i}^{2}}{\sum a_{i}}=100.$ Find the maximum possible value of $a_{1}$ .
Ground Truth	$550$
Student Answer	$100$
Error Type	Numerical collapse
Key Excerpt	... $10100m^{2}+202md+2d^{2}-2000000m-20000d+$ 9999...9 [thousands of repeated digits] $\times\,6999\times$ 9999...9 [continued overflow] That seems like I’m not doing this... The problem is just to be solved. ... $\verb\|\boxed{100}\|$

Appendix

Table 9: Case 1: Numerical collapse.

Problem	Determine whether the perimeter of the Mandelbrot set is finite or infinite.
Ground Truth	infinite
Student Answer	finite
Error Type	Repetitive reasoning loop; fails to commit to correct conclusion
Key Excerpt	Okay, so I have this question... I remember that the Mandelbrot set is this intricate fractal... [paragraph repeated verbatim 4 times] But the question isn’t about whether a point is in the set, it’s about the perimeter. I know that fractals often have infinite perimeters, but I’m not entirely sure about the Mandelbrot set... [loop continues without resolution]

Appendix

Table 10: Case 2: Infinite reasoning loop.

Problem	Is a real-valued function $F$ on $[a,b]$ with bounded variation and the intermediate value property guaranteed to be continuous?
Ground Truth	Yes
Student Answer	No
Error Type	Incorrect application of Cantor function as counterexample
Key Excerpt	...the Cantor function is constant on the middle-thirds Cantor set, so it’s not continuous, but it’s still of bounded variation. [...] Therefore, it’s clear that such functions can be discontinuous. So, the answer is no, it’s not guaranteed to be continuous.
Note	The Cantor function does not satisfy the intermediate value property, making this a flawed counterexample. The student fails to verify the premise.

Appendix

Table 11: Case 3: Incorrect application of a theorem.

Problem	Find the integer value of $K=\left\lfloor\sum_{r=1}^{80}\int_{0}^{1}x^{\sqrt{r}-1}\,dx\right\rfloor$ .
Ground Truth	$16$
Student Answer	$17$
Error Type	Incorrect integral bounds in Euler–Maclaurin approximation
Key Excerpt	Wait, that seems contradictory because 16 is greater than 16.888. That can’t be. Wait, maybe I misapplied the bounds. [...] But in the initial manual addition, I had the sum as about 16.888, but in reality, it’s about 17.28. Therefore, the answer is 17.
Note	The student correctly computes $\sum\approx 16.888$ at one point, then contradicts this with an erroneous re-approximation, ultimately reporting $\lfloor 17.28\rfloor=17$ instead of $\lfloor 16.888\rfloor=16$ .

Appendix

Table 12: Case 4: Off-by-one arithmetic error in integral approximation.

On-Policy Distillation

On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.

See all 507 papers

Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress
Aug 19, 2026Chen Yang, Haiyuan Wan, Rengrong Xiong +2Language Model Distillation On-Policy Distillation
The Hong Kong University of Science and Technology (Guangzhou) · Tsinghua University · Zhejiang University +1
On-Policy Delta Distillation
Jul 16, 2026Byeongho Heo, Jaehui Hwang, Sangdoo Yun +1RL for Language Model Reasoning Language Model Distillation
NAVER AI Lab
On-policy Distillation with Verifiable Reward
Aug 25, 2026Wenze Lin, Jiale Zhao, Xitai Jiang +5RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
LeapLab, Tsinghua University · Qiuzhen College, Tsinghua University · Beihang University +2

Aug 19, 2026J/K move · Enter open · S save

Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress

Chen Yang, Haiyuan Wan, Rengrong Xiong, Yize Chen+1

The Hong Kong University of Science and Technology (Guangzhou) · Tsinghua University · Zhejiang University+1

Language Model Distillation On-Policy Distillation RL for Language Model Reasoning

Open paper AskPDF arXiv

On-policy distillation (OPD) has emerged as an effective framework for post-training language models by pairing student-generated trajectories with dense token-level supervision from a teacher. However, OPD implicitly assumes that teacher-derived rewards are an appropriate proxy for reasoning progress, and therefore treats all teacher feedback equally during policy optimization. While in practice, this assumption does not always hold. We observe that teacher-derived rewards often conflict with genuine reasoning progress, as reasoning steps with clear reasoning advancement may still receive lower distillation rewards, simply due to deviation from teacher's outputs. To address this mismatch, we propose Reasoning-Progress-Aware Reward Filtering for On-Policy Distillation (R2-OPD), which constructs two within-trajectory rankings of reasoning spans, one from teacher-derived rewards and the other from independently estimated progress reward. Distillation rewards are selectively suppressed whenever the two rankings disagree, reducing supervision that conflicts with reasoning progress while preserving effective teacher guidance. Our approach shows consistent improvement over standard OPD especially regarding reasoning performances.

Abstract

On-Policy Distillation

Explore similar work

Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress

On-Policy Delta Distillation

On-policy Distillation with Verifiable Reward