Scaling Reasoning Efficiently via Relaxed On-Policy Distillation

Jongwoo Ko; Sara Abdali; Young Jin Kim; Tianyi Chen; Pashmina Cameron

cs.LGMar 11, 2026

Scaling Reasoning Efficiently via Relaxed On-Policy Distillation

Authors: Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, Pashmina Cameron

Abstract

On-policy distillation is pivotal for transferring reasoning capabilities to capacity-constrained models, yet remains prone to instability and negative transfer. We show that on-policy distillation can be interpreted, both theoretically and empirically, as a form of policy optimization, where the teacher-student log-likelihood ratio acts as a token reward. From this insight, we introduce REOPOLD (Relaxed On-Policy Distillation) a framework that stabilizes optimization by relaxing the strict imitation constraints of standard on-policy distillation. Specifically, REOPOLD temperately and selectively leverages rewards from the teacher through mixture-based reward clipping, entropy-based token-level dynamic sampling, and a unified exploration-to-refinement training strategy. Empirically, REOPOLD surpasses its baselines with superior sample efficiency during training and enhanced test-time scaling at inference, across mathematical, visual, and agentic tool-use reasoning tasks. Specifically, REOPOLD outperforms recent RL approaches achieving 6.7~12x greater sample efficiency and enables a 7B student to match a 32B teacher in visual reasoning with a ~3.32x inference speedup.

Figures & tables

Figure 2: Illustration of Reopold . While standard on-policy distillation (a) often introduces instability and inefficiency by forcing the student to mimic the teacher excessively, our proposed Reopold (b) fosters a more stable and effective learning environment. By establishing a formal connection between distillation and RL via a stop-gradient operation (Section 3.1 ), Reopold utilizes teacher signals temperately (Section 4.1 ) and selectively (Section 4.2 ). As depicted, this approach filters out potentially harmful signals, preventing the student from deviating excessively from its original distribution.

Figure 3: Comparison of training dynamics between vanilla RKL and RKL with stop-gradient. (a) Training loss dynamics exhibit similar trends, aligning with the theoretical equivalence in Remark 3.1 . (b) The gradient norm is markedly lower and more stable when stop-gradient is applied, which (c) leads to superior validation performance. This confirms that treating the log-likelihood ratio as a fixed reward signal is beneficial for optimization stability.

Figure 4: Log-scale histogram of token-level log-likelihood ratio rewards. The red dashed ellipse indicates degenerate near-zero rewards, while the purple dashed region highlights the heavy tail of negative rewards. These distributions pose challenges for RL-based on-policy distillation by causing gradient vanishing and training instability, respectively.

Figure 5: Correlation between token entropy and log-likelihood ratio rewards. Experimental results on math reasoning and visual reasoning benchmarks demonstrate that rewards in the bottom 60th entropy percentile are heavily concentrated around zero. This suggests that while teacher and student policies may diverge overall, they remain highly consistent on low-entropy, deterministic tokens, with significant deviations occurring primarily in high-entropy regimes.

Figure 6: Analysis of exploration-to-refinement multi-stage training. (Left) The proposed multi-stage approach prevents entropy collapse and preserves diverse modes during the initial exploration phase by masking strongly negative rewards, whereas the baseline suffers from rapid entropy collapse. (Right) The multi-stage strategy (solid lines) achieves superior final performance to the baseline (dashed lines) in both quality (i.e., Avg@32 ) and diversity (i.e., Pass@32 ).

Model	AIME-24	AIME-25	AMC-23	MATH-500	Minerva Math	Olympiad Bench	AVG.
SkyWork-OR1-Math-7B ( He et al., 2025a ) $\rightarrow$ DeepSeek-R1-Distill-Qwen-1.5B ( Guo et al., 2025 )
SkyWork-OR1-Math-7B	69.8	52.3	94.1	95.8	49.3	73.5	72.5
R1-Distill-Qwen-1.5B	28.6	22.7	62.6	82.9	26.4	43.6	44.4
+ GRPO	31.8	23.7	62.0	85.4	33.8	49.8	47.8
+ SFT	33.5	24.6	76.1	86.6	36.4	55.6	51.5
+ RKL	37.1	30.6	80.2	88.0	34.6	56.0	54.4

Table 1: Performance comparison of Reopold and distillation baselines (SFT and RKL) for on mathematical reasoning benchmarks across different teacher models. Accuracy (%) is reported for all benchmarks. The best result in each column is shown in bold , and the second-best is underlined .

\Delta\%

indicates relative improvement of Reopold compared to RKL. The RL-based approach is colored in gray .

Figure 7: Validation accuracy on AIME with a 7B student policy. With a 7B student, vanilla RKL becomes unstable, whereas Reopold enables stable and consistent improvement.

Model	Geo3K	MathVerse	MathVision	MathVista	WeMath	Hallusion	AVG.
Qwen2.5-VL-32B-Instruct + NoisyRollout	56.74	58.86	39.82	78.30	75.51	72.45	63.61
Qwen2.5-VL-32B-Instruct + NoisyRollout ( Liu et al., 2025b ) $\rightarrow$ Qwen2.5-VL-3B-Instruct
Qwen2.5-VL-3B-Instruct	26.46	35.58	22.83	59.40	53.41	61.51	43.20
+ PAPO ( Wang et al., 2025b )	32.95	40.65	24.16	65.10	63.62	61.62	48.02
+ RKL	48.09	33.25	24.78	62.50	64.48	60.83	48.99
+ Reopold	50.58	46.40	26.39	61.50	64.60	63.62	52.18

Table 2: Performance comparison of vision-language models on a visual reasoning and perception benchmarks. Accuracy (%) is reported for all benchmarks. The best result in each column is shown in bold , and the second-best is underlined . † denotes the results from Liu et al. (2025b) as original model is not available.

\Delta\%

indicates relative improvement of Reopold compared to RKL. The RL-based approaches are colored in gray .

Table 9

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 11: Ratio of Clipped

\rho(\theta)

Figure 12: Sensitivity analysis of

\lambda

	Math (Section 5.1 )		Visual (Section 5.2 )		Agentic (Section 5.4 )
Model Size	1.5B	7B	3B	7B	3B
Total step $K$	300	200	60	60	40
Learning rate $\eta$	$1\times 10^{-5}$	$3\times 10^{-6}$	$5\times 10^{-6}$	$5\times 10^{-6}$	$1\times 10^{-6}$
Clipping coefficient $\lambda$	0.3	0.3	0.3	0.3	0.3
Entropy percentile $\beta$	0.2	0.2	0.2	0.2	0.4
Switch step $T_{\text{switch}}$	100	50	20	20	10

Appendix

Table 7: Hyperparameter values used in Reopold experiments in Section 5 .

Figure 13: Extended results of Figure 1 (b). We visualize the accuracy (Pass@

K

and Maj@

K

) against inference latency as the sample budget

K

increases (up to 64 for Geometry3K; 16 for MathVerse). Reopold (solid lines) consistently yields a better trade-off than the RKL baseline (faded lines). Notably, the 7B student matches or beats the 32B teacher’s accuracy with significantly lower latency, confirming the efficiency of our distillation.

Figure 14: Breakdown of training wall-clock time per step.

	GKD	GKD †	Reopold	Reopold †
Geo3K	OOM	51.85	53.58	53.41
MathVerse	OOM	48.14	51.97	50.96
MathVision	OOM	29.90	31.12	30.94
MathVista	OOM	71.25	73.60	73.64
WeMath	OOM	70.87	71.84	71.80
Hallusion	OOM	70.21	70.87	71.52

Appendix

Table 8: Comparison to full vocabulary on-policy distillation. We train the model for 300 training steps. † indicates distillation with Top-5 approximation.

Figure 15: Qualitative comparison on Hallusion Bench. While the baseline trained with RKL suffers from visual perception degradation (hallucinating a flat trend despite the visual evidence), Reopold maintains robust visual grounding, accurately identifying the peak in the chart.

Figure 16: Qualitative comparison on MathVerse. The baseline (Vanilla RKL) gets trapped in circular logic (Step 5) and hallucinates the final calculation (Step 8). Our model initially derives an incorrect value (

96^{\circ}

) but explicitly triggers a self-correction process (”However, we need to re-evaluate…”) to reach the correct solution (

84^{\circ}

On-Policy Distillation

On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.

See all 507 papers

Reward-Gated On-Policy Distillation
Jul 4, 2026Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi +3Language Model Distillation On-Policy Distillation
University of Cologne · AWS AI · UT Austin +1
On-Policy Distillation with Best-of-N Teacher Rollout Selection
May 10, 2026Ke Zhang, Yunjie Tian, Dongdi Zhao +4CoT Distillation Best-of-N Sampling
Johns Hopkins University · TikTok · University of California, San Diego +1
On-Policy Delta Distillation
Jul 16, 2026Byeongho Heo, Jaehui Hwang, Sangdoo Yun +1RL for Language Model Reasoning Language Model Distillation
NAVER AI Lab

Jul 4, 2026J/K move · Enter open · S save

Reward-Gated On-Policy Distillation

Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi, Chanakya Ekbote+2

University of Cologne · AWS AI · UT Austin+1

Language Model Distillation On-Policy Distillation LLM Reasoning

Open paper AskPDF arXiv

On-policy distillation is a powerful way to transfer reasoning ability from a strong teacher to a smaller student: the student samples trajectories from its own policy, and the teacher provides dense token-level supervision on the states the student actually visits. However, this supervision is not always reliable: a teacher can assign high likelihood to plausible but incorrect solutions, or low likelihood to correct student solutions that follow different reasoning paths. Unconditionally distilling the teacher can therefore reinforce bad modes or erase useful student behavior. To address these limitations, we introduce RG-OPD: Reward-Gated On-Policy Distillation that uses verifier feedback to decide when teacher logits should be trusted. RG-OPD bridges sparse verifier rewards and dense teacher logits, preserving token-level supervision while filtering misleading teacher signals. Across reasoning and coding benchmarks, RG-OPD produces stronger distilled students, outperforming both vanilla reverse-KL distillation and the recent TSD-KD baseline. At 1K generation length, RG-OPD improves over reverse-KL by 2.9 points and over TSD-KD by 4.9 points; in the long-generation setting, it improves over the untuned student by 8.2 points. Our code is available at https://github.com/UoC-tail/RG-OPD.

Abstract

On-Policy Distillation

Explore similar work

Reward-Gated On-Policy Distillation

On-Policy Distillation with Best-of-N Teacher Rollout Selection

On-Policy Delta Distillation