cs.LGMar 11, 2026

Scaling Reasoning Efficiently via Relaxed On-Policy Distillation

Authors: Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, Pashmina Cameron

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

On-policy distillation is pivotal for transferring reasoning capabilities to capacity-constrained models, yet remains prone to instability and negative transfer. We show that on-policy distillation can be interpreted, both theoretically and empirically, as a form of policy optimization, where the teacher-student log-likelihood ratio acts as a token reward. From this insight, we introduce REOPOLD (Relaxed On-Policy Distillation) a framework that stabilizes optimization by relaxing the strict imitation constraints of standard on-policy distillation. Specifically, REOPOLD temperately and selectively leverages rewards from the teacher through mixture-based reward clipping, entropy-based token-level dynamic sampling, and a unified exploration-to-refinement training strategy. Empirically, REOPOLD surpasses its baselines with superior sample efficiency during training and enhanced test-time scaling at inference, across mathematical, visual, and agentic tool-use reasoning tasks. Specifically, REOPOLD outperforms recent RL approaches achieving 6.7~12x greater sample efficiency and enables a 7B student to match a 32B teacher in visual reasoning with a ~3.32x inference speedup.

Figures & tables

Figure 2: Illustration of Reopold . While standard on-policy distillation (a) often introduces instability and inefficiency by forcing the student to mimic the teacher excessively, our proposed Reopold (b) fosters a more stable and effective learning environment. By establishing a formal connection between distillation and RL via a stop-gradient operation (Section 3.1 ), Reopold utilizes teacher signals temperately (Section 4.1 ) and selectively (Section 4.2 ). As depicted, this approach filters out potentially harmful signals, preventing the student from deviating excessively from its original distribution.
Figure 3: Comparison of training dynamics between vanilla RKL and RKL with stop-gradient. (a) Training loss dynamics exhibit similar trends, aligning with the theoretical equivalence in Remark 3.1 . (b) The gradient norm is markedly lower and more stable when stop-gradient is applied, which (c) leads to superior validation performance. This confirms that treating the log-likelihood ratio as a fixed reward signal is beneficial for optimization stability.
Figure 4: Log-scale histogram of token-level log-likelihood ratio rewards. The red dashed ellipse indicates degenerate near-zero rewards, while the purple dashed region highlights the heavy tail of negative rewards. These distributions pose challenges for RL-based on-policy distillation by causing gradient vanishing and training instability, respectively.
Figure 5: Correlation between token entropy and log-likelihood ratio rewards. Experimental results on math reasoning and visual reasoning benchmarks demonstrate that rewards in the bottom 60th entropy percentile are heavily concentrated around zero. This suggests that while teacher and student policies may diverge overall, they remain highly consistent on low-entropy, deterministic tokens, with significant deviations occurring primarily in high-entropy regimes.
Figure 6: Analysis of exploration-to-refinement multi-stage training. (Left) The proposed multi-stage approach prevents entropy collapse and preserves diverse modes during the initial exploration phase by masking strongly negative rewards, whereas the baseline suffers from rapid entropy collapse. (Right) The multi-stage strategy (solid lines) achieves superior final performance to the baseline (dashed lines) in both quality (i.e., Avg@32 ) and diversity (i.e., Pass@32 ).
ModelAIME-24AIME-25AMC-23MATH-500Minerva MathOlympiad BenchAVG.
SkyWork-OR1-Math-7B ( He et al., 2025a ) →\rightarrow DeepSeek-R1-Distill-Qwen-1.5B ( Guo et al., 2025 )
SkyWork-OR1-Math-7B69.852.394.195.849.373.572.5
R1-Distill-Qwen-1.5B28.622.762.682.926.443.644.4
+ GRPO31.823.762.085.433.849.847.8
+ SFT33.524.676.186.636.455.651.5
+ RKL37.130.680.288.034.656.054.4
Table 1: Performance comparison of Reopold and distillation baselines (SFT and RKL) for on mathematical reasoning benchmarks across different teacher models. Accuracy (%) is reported for all benchmarks. The best result in each column is shown in bold , and the second-best is underlined . Δ%\Delta\% indicates relative improvement of Reopold compared to RKL. The RL-based approach is colored in gray .
Figure 7: Validation accuracy on AIME with a 7B student policy. With a 7B student, vanilla RKL becomes unstable, whereas Reopold enables stable and consistent improvement.
ModelGeo3KMathVerseMathVisionMathVistaWeMathHallusionAVG.
Qwen2.5-VL-32B-Instruct + NoisyRollout56.7458.8639.8278.3075.5172.4563.61
Qwen2.5-VL-32B-Instruct + NoisyRollout ( Liu et al., 2025b ) →\rightarrow Qwen2.5-VL-3B-Instruct
Qwen2.5-VL-3B-Instruct26.4635.5822.8359.4053.4161.5143.20
+ PAPO ( Wang et al., 2025b )32.9540.6524.1665.1063.6261.6248.02
+ RKL48.0933.2524.7862.5064.4860.8348.99
+ Reopold50.5846.4026.3961.5064.6063.6252.18
Table 2: Performance comparison of vision-language models on a visual reasoning and perception benchmarks. Accuracy (%) is reported for all benchmarks. The best result in each column is shown in bold , and the second-best is underlined . † denotes the results from Liu et al. (2025b) as original model is not available. Δ%\Delta\% indicates relative improvement of Reopold compared to RKL. The RL-based approaches are colored in gray .
Table 9
Figure 10
Figure 11
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 11: Ratio of Clipped ρ(θ)\rho(\theta) .
Figure 12: Sensitivity analysis of λ\lambda .
 Math (Section 5.1 )Visual (Section 5.2 )Agentic (Section 5.4 )
Model Size1.5B7B3B7B3B
Total step KK300200606040
Learning rate η\eta1×10−51\times 10^{-5}3×10−63\times 10^{-6}5×10−65\times 10^{-6}5×10−65\times 10^{-6}1×10−61\times 10^{-6}
Clipping coefficient λ\lambda0.30.30.30.30.3
Entropy percentile β\beta0.20.20.20.20.4
Switch step TswitchT_{\text{switch}}10050202010
Appendix
Table 7: Hyperparameter values used in Reopold experiments in Section 5 .
Figure 13: Extended results of Figure 1 (b). We visualize the accuracy (Pass@ KK and Maj@ KK ) against inference latency as the sample budget KK increases (up to 64 for Geometry3K; 16 for MathVerse). Reopold (solid lines) consistently yields a better trade-off than the RKL baseline (faded lines). Notably, the 7B student matches or beats the 32B teacher’s accuracy with significantly lower latency, confirming the efficiency of our distillation.
Figure 14: Breakdown of training wall-clock time per step.
 GKDGKD †ReopoldReopold †
Geo3KOOM51.8553.5853.41
MathVerseOOM48.1451.9750.96
MathVisionOOM29.9031.1230.94
MathVistaOOM71.2573.6073.64
WeMathOOM70.8771.8471.80
HallusionOOM70.2170.8771.52
Appendix
Table 8: Comparison to full vocabulary on-policy distillation. We train the model for 300 training steps. † indicates distillation with Top-5 approximation.
Figure 15: Qualitative comparison on Hallusion Bench. While the baseline trained with RKL suffers from visual perception degradation (hallucinating a flat trend despite the visual evidence), Reopold maintains robust visual grounding, accurately identifying the peak in the chart.
Figure 16: Qualitative comparison on MathVerse. The baseline (Vanilla RKL) gets trapped in circular logic (Step 5) and hallucinates the final calculation (Step 8). Our model initially derives an incorrect value ( 96∘96^{\circ} ) but explicitly triggers a self-correction process (”However, we need to re-evaluate…”) to reach the correct solution ( 84∘84^{\circ} ).
More in this topic

On-Policy Distillation

On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.

See all 507 papers

Explore similar work

CardsList
Jul 4, 2026

Reward-Gated On-Policy Distillation

On-policy distillation is a powerful way to transfer reasoning ability from a strong teacher to a smaller student: the student samples trajectories from its own policy, and the teacher provides dense token-level supervision on the states the student actually visits. However, this supervision is not always reliable: a teacher can assign high likelihood to plausible but incorrect solutions, or low likelihood to correct student solutions that follow different reasoning paths. Unconditionally distilling the teacher can therefore reinforce bad modes or erase useful student behavior. To address these limitations, we introduce RG-OPD: Reward-Gated On-Policy Distillation that uses verifier feedback to decide when teacher logits should be trusted. RG-OPD bridges sparse verifier rewards and dense teacher logits, preserving token-level supervision while filtering misleading teacher signals. Across reasoning and coding benchmarks, RG-OPD produces stronger distilled students, outperforming both vanilla reverse-KL distillation and the recent TSD-KD baseline. At 1K generation length, RG-OPD improves over reverse-KL by 2.9 points and over TSD-KD by 4.9 points; in the long-generation setting, it improves over the untuned student by 8.2 points. Our code is available at https://github.com/UoC-tail/RG-OPD.

Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi +3

University of Cologne · AWS AI · UT Austin +1

Language Model DistillationOn-Policy DistillationLLM Reasoning
AskPDF
May 10, 2026

On-Policy Distillation with Best-of-N Teacher Rollout Selection

On-policy distillation (OPD), which supervises a student on its own sampled trajectories, has emerged as a data-efficient post-training method for improving reasoning while avoiding the reward dependence of reinforcement learning and the catastrophic forgetting often observed in standard supervised fine-tuning. However, standard OPD typically computes teacher supervision under noisy student-generated contexts and often relies on a single stochastic teacher rollout per prompt. As a result, the supervision signal can be high-variance: the sampled teacher trajectory can be incorrect, uninformative, or poorly matched to the student's current reasoning behavior. To address this limitation, we propose BRTS, a Best-of-N Rollout Teacher Selection framework for on-policy distillation. BRTS augments standard student-context OPD with a teacher-context supervision branch constructed from the curated teacher trajectory. Rather than distilling from the first sampled teacher rollout, BRTS samples a small pool of teacher trajectories and selects the auxiliary trajectory using a simple priority rule: correctness first, student alignment second. When multiple correct teacher trajectories are available, BRTS chooses the one most aligned with the student's current behavior; when unconditioned teacher samples fail on harder prompts, it invokes a ground-truth-conditioned recovery step to elicit a natural derivation. The selected trajectory is then used to provide reliable teacher-context supervision inside the OPD loop, augmented with an auxiliary loss on the teacher trajectory. Experiments on AIME 2024, AIME 2025, and AMC 2023 show that BRTS improves over standard OPD on challenging reasoning benchmarks, with the largest gains on harder datasets. Our code is available at https://github.com/BWGZK-keke/BRTS.

Ke Zhang, Yunjie Tian, Dongdi Zhao +4

Johns Hopkins University · TikTok · University of California, San Diego +1

CoT DistillationBest-of-N SamplingMathematical ReasoningLanguage Model DistillationOn-Policy DistillationTeacher-Student Learning
AskPDF
Jul 16, 2026

On-Policy Delta Distillation

On-policy distillation is an alternative post-training method in reinforcement learning that alleviates the constraints imposed by reward models by providing token-level supervision from a teacher model. Although on-policy distillation has been studied and applied across various settings, its fundamental design remains underexplored. In this paper, we introduce a new distillation reward, termed the delta signal, instead of directly imitating the teacher's output distribution. The delta signal is defined as the difference between the teacher model and its base model prior to instruction tuning for reasoning capability. It therefore captures the changes induced by reasoning tuning and provides a more direct signal for transferring reasoning capabilities. Using extensive empirical evidence, we show that the delta signal substantially improves on-policy distillation and refer to the new distillation method as On-Policy Delta Distillation (OPD2^2). Experiments across mathematics, science, and code-reasoning benchmarks demonstrate that OPD2^2 consistently outperforms conventional on-policy distillation, enabling reasoning LLMs to achieve strong performance with only a short post-training period. Code will be available at https://github.com/naver-ai/opd2

Byeongho Heo, Jaehui Hwang, Sangdoo Yun +1

NAVER AI Lab

RL for Language Model ReasoningLanguage Model DistillationOn-Policy Distillation
AskPDF