Self-Distilled RLVR

Chenxu Yang; Chuanyu Qin; Qingyi Si; Minghui Chen; Naibin Gu; Dingyu Yao; Zheng Lin; Weiping Wang

cs.LGApr 3, 2026

Self-Distilled RLVR

Authors: Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang,

Organizations: Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China

Abstract

On-policy distillation (OPD) has become a popular training paradigm in the LLM community. This paradigm selects a larger model as the teacher to provide dense, fine-grained signals for each sampled trajectory, in contrast to reinforcement learning with verifiable rewards (RLVR), which only obtains sparse signals from verifiable outcomes in the environment. Recently, the community has explored on-policy self-distillation (OPSD), where the same model serves as both teacher and student, with the teacher receiving additional privileged information such as reference answers to enable self-evolution. This paper demonstrates that learning signals solely derived from the privileged teacher result in severe information leakage and unstable long-term training. Accordingly, we identify the optimal niche for self-distillation and propose RLSD (RLVR with Self-Distillation). Specifically, we leverage self-distillation to obtain token-level policy differences for determining fine-grained update magnitudes, while continuing to use RLVR to derive reliable update directions from environmental feedback (e.g., response correctness). This enables RLSD to simultaneously harness the strengths of both RLVR and OPSD, achieving a higher convergence ceiling and superior training stability.

Figures & tables

Figure 1: Performance on the trained Qwen3-VL-8B-Instruct model. In (a), OPSD reaches its peak performance early and degrades, whereas RLSD inherits the training stability of GRPO while achieving a higher convergence ceiling. In (b), GRPO and RLSD report results at 200 training steps, while GRPO (2 $\times$ steps) reports results at 400 steps; RLSD at 200 steps already surpasses GRPO trained for twice as many steps, demonstrating faster convergence.

Method	Trajectory	Efficiency	Leakage Risk	Signal	Direction Anchoring
SFT	✗ Off-policy	✓ High	✓ N/A	✓ Rich	✗ Teacher
RLVR (GRPO)	✓ On-policy	✓ High	✓ N/A	✗ Weak	✓ Environment
OPD	✓ On-policy	✗ Low	✓ N/A	✓ Rich	✗ Teacher
OPSD	✓ On-policy	✓ High	✗ Severe	✓ Rich	✗ Teacher
RLSD (Ours)	✓ On-policy	✓ High	✓ N/A	✓ Rich	✓ Environment

Table 1: Comparison of training paradigms for post-training LLMs.

Panel of Figure 2 — Figure 2: A representative example illustrating the privileged information leakage exhibited by the OPSD-trained model, where the model appeals to an invisible reference solution during inference.

Figure 3: Leakage, KL divergence, and validation performance of OPSD and its ablated variants.

Figure 4: An overview of our RLSD method.

Method	MMMU	MathVista	MathVision	ZeroBench	Wemath	Avg.
Base LLM	62.44	73.80	47.37	19.76	54.10	51.49
GRPO	65.11	76.20	48.82	22.60	56.57	53.86
OPSD	63.82	75.10	47.53	21.06	54.95	52.49
SDPO	65.11	74.00	47.27	25.15	52.19	52.74
GRPO+OPSD	63.22	75.90	48.52	22.16	54.76	52.91
RLSD (Ours)	67.22	78.10	52.73	24.85	58.00	56.18

Table 2: Multimodal reasoning results on the Qwen3-VL-8B-Instruct model. Bold indicates the best result among all models. Avg. is computed over the five selected benchmarks.

Figure 5: Training dynamics on the multimodel reasoning tasks.

Figure 6: Token-level credit heatmaps under RLSD for two multimodal reasoning examples. Top: in a correct trajectory, larger credit is concentrated on the decisive counting and subtraction steps. Bottom: in an incorrect trajectory, the strongest blame is assigned to the misread bar-model relation and the derived wrong answer. Green denotes larger token-level credit magnitude and red denotes smaller token-level credit magnitude.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Property	OPSD (Frozen Teacher)	OPSD (Online Teacher)	RLSD
(a) Objective stability	$\checkmark$	$\times$	$\checkmark$
(b) Sustained improvement	$\times$	$\checkmark$	$\checkmark$
(c) Leakage-free training	$\times$	$\times$	$\checkmark$

Appendix

Table 3: Comparison of OPSD parameter management strategies and RLSD across the three desiderata identified in the impossibility trilemma.

On-Policy Self-Distillation

On-policy self-distillation trains a model on its own generated trajectories using a teacher view to provide dense supervision.

See all 154 papers

Teach Me After You Learn It: Lookahead Self-Distillation for RLVR
Oct 13, 2026Zepeng Zhai, Meilin ChenRL for Language Model Reasoning Self-Distillation
Xiaohongshu Inc.
SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation
Sep 29, 2026Zhenrui Yue, Huimin Zeng, Yueqi Wang +8RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
University of Illinois · UC San Diego · Rochester Institute of Technology +2
RISE: Recursive Improvement via Self-Extrapolating Policy Distillation
Sep 4, 2026Yang Li, Semih Yavuz, Shafiq JotyLanguage Model Distillation On-Policy Distillation
Salesforce AI Research

Oct 13, 2026J/K move · Enter open · S save

Teach Me After You Learn It: Lookahead Self-Distillation for RLVR

Zepeng Zhai, Meilin Chen

Xiaohongshu Inc.

RL for Language Model Reasoning Self-Distillation On-Policy Distillation Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

On-policy self-distillation (OPSD) has emerged as a popular training paradigm for LLMs, providing dense, token-level supervision by aligning a model's own distribution with the distribution induced by privileged information. However, existing OPSD methods derive supervision solely from privileged information of the same instance, leaving inter-instance supervision largely unexplored. We propose Lookahead Self-Distillation (LEAD), which introduces inter-instance distillation by allowing the teacher to first learn from other on-policy rollouts before teaching the student. LEAD constructs lookahead teachers through virtual RL updates on disjoint subsets of on-policy rollouts, and uses each teacher to provide cross-group token-level supervision for the other subset. To reconcile the complementary yet potentially conflicting signals from OPSD and RLVR, we introduce harmonious modulation, which adaptively modulates the GRPO advantage based on the agreement between the two signals. Extensive experiments on scientific reasoning and interactive-agent benchmarks show that LEAD achieves the best in-domain average performance across model scales while maintaining out-of-domain generalization comparable to GRPO.

Abstract

On-Policy Self-Distillation

Explore similar work

Teach Me After You Learn It: Lookahead Self-Distillation for RLVR

SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation

RISE: Recursive Improvement via Self-Extrapolating Policy Distillation