SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting
Authors: Binbin Zheng, Xing Ma, Yiheng Liang, Jingqing Ruan, Xiaoliang Fu, Kepeng Lin, Benchang Zhu, Ke Zeng, Xunliang Cai
Organizations: University of Science and Technology of China · Meituan LongCat Interaction Team · Nanjing University · Fudan University · Huazhong University of Science and Technology
On-policy reinforcement learning has become the dominant paradigm for reasoning alignment in large language models, yet its sparse, outcome-level rewards make token-level credit assignment notoriously difficult. On-Policy Distillation (OPD) alleviates this by introducing dense, token-level KL supervision from a teacher model, but typically applies this supervision uniformly across all rollouts, ignoring fundamental differences in signal quality. We propose Signal-Calibrated On-Policy Distillation Enhancement (SCOPE), a dual-path adaptive training framework that routes on-policy rollouts by correctness into two complementary supervision paths. For incorrect trajectories, SCOPE performs teacher-perplexity-weighted KL distillation to prioritize instances where the teacher demonstrates genuine corrective capability, while down-weighting unreliable guidance. For correct trajectories, it applies student-perplexity-weighted MLE to concentrate reinforcement on low-confidence samples at the capability boundary rather than over-reinforcing already mastered ones. Both paths employ a group-level normalization to adaptively calibrate weight distributions, accounting for the intrinsic difficulty variance across prompts. Extensive experiments on six reasoning benchmarks show that SCOPE achieves an average relative improvement of 11.42% in Avg@32 and 7.30% in Pass@32 over competitive baselines, demonstrating its consistent effectiveness.
Figures & tables
Figure 1: (a) Performance changes on the AIME24 benchmark before and after training. Both PSR and OPD training enhance pass@1 at the expense of pass@32, highlighting a clear trade-off between accuracy and reasoning diversity. (b) Recovery rate of the teacher model across varying truncation levels, conditioned on truncated student error trajectories as prefixes and stratified by perplexity.
Figure 2: (a) Standard OPD applies uniform supervision to all samples. (b) Our SCOPE framework refines the learning process by first dividing trajectories into correct ΩC and incorrect ΩW sets, applying dual-path perplexity-based weighting, and finally optimizing the weighted branches via a unified objective.
Table 1: Main results on mathematical reasoning benchmarks under different teacher–student configurations. We report Avg@32 (A@32) and Pass@32 (P@32) for each benchmark. Bold denotes the best performance and underlined the second-best.
Figure 3: Training dynamics comparing GRPO, OPD, and SCOPE (Ours): (a) entropy loss across training steps, and Avg@32 (%) performance on (b) AIME24 and (c) AIME25.
Figure 4: Pass@ k (%) performance comparison of GRPO, OPD, and SCOPE (Ours) on the AIME24, AIME25, and AMC23 benchmarks using the DeepSeek-R1-Distill-Qwen-1.5B model.
Figure 5: Ablation study on the dual-path adaptive weighting mechanism (DPAW) of SCOPE. We report the performance of Avg@32 (%) and Pass@32 (%) on AIME24 (a) and AIME25 (b).
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
GRPO
OPD
SCOPE
Optimization Parameters
Learning Rate
5×10−5
5×10−5
5×10−5
LR Scheduler
Constant
Constant
Constant
Weight Decay
0.01
0.01
0.01
Train Batch Size (Prompts)
256
256
256
Mini Batch Size (Prompts)
256
256
256
Appendix
Table 3: Training config for GRPO, OPD, and SCOPE
Parameter
Value
Max Tokens
32768
Temperature
0.6
Top-p
0.95
Top-k
20
Repetition Penalty (Qwen3-1.7B)
1.08
Repetition Penalty (Distill-1.5B)
1.0
Appendix
Table 4: Evaluation parameters for all models.
Benchmark
Evaluation Focus & Characteristics
AIME 2024
Evaluates precise arithmetic and algebraic reasoning under strict answer-format constraints, where solutions are typically given as integer-valued responses.
AIME 2025
Serves as a held-out benchmark for assessing whether models can handle unseen and challenging mathematical reasoning patterns.
AMC 2023
Includes foundational competition-style problems in algebra, geometry, and combinatorics, serving as a measure of core mathematical problem-solving ability.
MATH-500
Uses a representative subset of the MATH benchmark spanning multiple mathematical areas, requiring symbolic manipulation, formula interpretation, and robust answer parsing.
Minerva
Consists of STEM-oriented mathematical problems, emphasizing technical notation, quantitative reasoning, and multi-step solution construction.
Table 5: Overview of evaluation benchmarks and their key characteristics, covering mathematical reasoning and code generation tasks.
Bucket
PPL Mean
NLL Mean
Q1 (Lowest PPL)
1.361
0.305
Q2
1.565
0.448
Q3
1.710
0.536
Q4 (Highest PPL)
2.383
0.688
Overall
1.755
0.494
Appendix
Table 6: Teacher PPL and NLL statistics for each perplexity bucket over incorrect student trajectories.
Trunc. Ratio
Q1
Q2
Q3
Q4
Q1–Q4 Gap
0.2
64.9
59.7
54.0
45.4
+19.4
0.4
55.8
53.2
50.1
40.2
+15.6
0.6
44.8
43.1
41.0
34.5
+10.3
0.8
35.8
35.3
32.6
28.6
+7.2
Appendix
Table 7: Teacher error recovery rate (%) under different truncation ratios and PPL buckets. Each cell reports the mean accuracy over n=4 completions per sample. The Q1–Q4 gap (rightmost column) quantifies the within-truncation-level spread attributable to PPL stratification.
Figure 6: Impact of the temperature hyperparameter τ on model performance across AIME24, AIME25, and AMC23. The results indicate that the default configuration of τ=1.0 consistently yields the best performance across all benchmarks compared to τ=0.5 and τ=2.0 .
Component
GRPO
OPD
SCOPE (Ours)
Generation
264.5
164.5
247.7
Old Logprob
34.0
5.2
31.4
Reward Computation
4.8
0.7
4.6
Actor Update
151.8
22.9
154.1
Teacher Scoring
—
31.2
200.0
Total (step)
459.0
227.5
641.9
Appendix
Table 8: Per-step wall-clock time breakdown (seconds) for each training method. Values are means over the stable training region. The number of rollouts is set to 8 for GRPO and SCOPE, and 1 for OPD.
Problem
Let a1,a2,…,a100 be integers such that ∑ai∑ai2=100. Find the maximum possible value of a1 .
Ground Truth
550
Student Answer
100
Error Type
Numerical collapse
Key Excerpt
... 10100m2+202md+2d2−2000000m−20000d+ 9999...9 [thousands of repeated digits] ×6999× 9999...9 [continued overflow] That seems like I’m not doing this... The problem is just to be solved. ... \boxed{100}
Appendix
Table 9: Case 1: Numerical collapse.
Problem
Determine whether the perimeter of the Mandelbrot set is finite or infinite.
Ground Truth
infinite
Student Answer
finite
Error Type
Repetitive reasoning loop; fails to commit to correct conclusion
Key Excerpt
Okay, so I have this question... I remember that the Mandelbrot set is this intricate fractal... [paragraph repeated verbatim 4 times] But the question isn’t about whether a point is in the set, it’s about the perimeter. I know that fractals often have infinite perimeters, but I’m not entirely sure about the Mandelbrot set... [loop continues without resolution]
Appendix
Table 10: Case 2: Infinite reasoning loop.
Problem
Is a real-valued function F on [a,b] with bounded variation and the intermediate value property guaranteed to be continuous?
Ground Truth
Yes
Student Answer
No
Error Type
Incorrect application of Cantor function as counterexample
Key Excerpt
...the Cantor function is constant on the middle-thirds Cantor set, so it’s not continuous, but it’s still of bounded variation. [...] Therefore, it’s clear that such functions can be discontinuous. So, the answer is no, it’s not guaranteed to be continuous.
Note
The Cantor function does not satisfy the intermediate value property, making this a flawed counterexample. The student fails to verify the premise.
Appendix
Table 11: Case 3: Incorrect application of a theorem.
Problem
Find the integer value of K=⌊∑r=180∫01xr−1dx⌋ .
Ground Truth
16
Student Answer
17
Error Type
Incorrect integral bounds in Euler–Maclaurin approximation
Key Excerpt
Wait, that seems contradictory because 16 is greater than 16.888. That can’t be. Wait, maybe I misapplied the bounds. [...] But in the initial manual addition, I had the sum as about 16.888, but in reality, it’s about 17.28. Therefore, the answer is 17.
Note
The student correctly computes ∑≈16.888 at one point, then contradicts this with an erroneous re-approximation, ultimately reporting ⌊17.28⌋=17 instead of ⌊16.888⌋=16 .
Appendix
Table 12: Case 4: Off-by-one arithmetic error in integral approximation.
On-policy distillation (OPD) has emerged as an effective framework for post-training language models by pairing student-generated trajectories with dense token-level supervision from a teacher. However, OPD implicitly assumes that teacher-derived rewards are an appropriate proxy for reasoning progress, and therefore treats all teacher feedback equally during policy optimization. While in practice, this assumption does not always hold. We observe that teacher-derived rewards often conflict with genuine reasoning progress, as reasoning steps with clear reasoning advancement may still receive lower distillation rewards, simply due to deviation from teacher's outputs. To address this mismatch, we propose Reasoning-Progress-Aware Reward Filtering for On-Policy Distillation (R2-OPD), which constructs two within-trajectory rankings of reasoning spans, one from teacher-derived rewards and the other from independently estimated progress reward. Distillation rewards are selectively suppressed whenever the two rankings disagree, reducing supervision that conflicts with reasoning progress while preserving effective teacher guidance. Our approach shows consistent improvement over standard OPD especially regarding reasoning performances.
Chen Yang, Haiyuan Wan, Rengrong Xiong +2
The Hong Kong University of Science and Technology (Guangzhou) · Tsinghua University · Zhejiang University +1
On-policy distillation is an alternative post-training method in reinforcement learning that alleviates the constraints imposed by reward models by providing token-level supervision from a teacher model. Although on-policy distillation has been studied and applied across various settings, its fundamental design remains underexplored. In this paper, we introduce a new distillation reward, termed the delta signal, instead of directly imitating the teacher's output distribution. The delta signal is defined as the difference between the teacher model and its base model prior to instruction tuning for reasoning capability. It therefore captures the changes induced by reasoning tuning and provides a more direct signal for transferring reasoning capabilities. Using extensive empirical evidence, we show that the delta signal substantially improves on-policy distillation and refer to the new distillation method as On-Policy Delta Distillation (OPD2). Experiments across mathematics, science, and code-reasoning benchmarks demonstrate that OPD2 consistently outperforms conventional on-policy distillation, enabling reasoning LLMs to achieve strong performance with only a short post-training period. Code will be available at https://github.com/naver-ai/opd2
Reinforcement Learning with Verifiable Rewards (RLVR) and on-policy distillation (OPD) have become two widely adopted paradigms for post-training large language models. However, RLVR suffers from sparse task-level feedback, while OPD provides dense token-level guidance but ignores trajectory correctness, limiting its performance to that of the teacher. Combining them is a promising direction: OPD supplies dense supervisory signals, while RLVR provides task-level correctness. Nevertheless, existing integrations often rely on weighted combination or heuristic switching, introducing extra hyperparameters and trade-offs. We propose On-policy Distillation with Verifiable Reward (OPDVR), a simple yet effective method that seamlessly combines OPD and RLVR without adding any hyperparameters. We first reformulate the implicit reward of sampled-token OPD based on trajectory correctness, then apply a ReLU gating mechanism to ensure that correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards---thereby aligning the distillation signal with task success while preserving the teacher's distributional guidance. Furthermore, our modification transforms sampled-token OPD into a proper RLVR method, making it readily combinable with any policy gradient algorithm, such as GRPO. Experiments on six reasoning benchmarks show that OPDVR consistently outperforms standard OPD. Our code is available at https://github.com/LeapLabTHU/OPDVR.
Wenze Lin, Jiale Zhao, Xitai Jiang +5
LeapLab, Tsinghua University · Qiuzhen College, Tsinghua University · Beihang University +2