Teach Me After You Learn It: Lookahead Self-Distillation for RLVR

Zepeng Zhai; Meilin Chen

cs.LGOct 13, 2026

Teach Me After You Learn It: Lookahead Self-Distillation for RLVR

Authors: Zepeng Zhai, Meilin Chen

Organizations: Xiaohongshu Inc.

Abstract

On-policy self-distillation (OPSD) has emerged as a popular training paradigm for LLMs, providing dense, token-level supervision by aligning a model's own distribution with the distribution induced by privileged information. However, existing OPSD methods derive supervision solely from privileged information of the same instance, leaving inter-instance supervision largely unexplored. We propose Lookahead Self-Distillation (LEAD), which introduces inter-instance distillation by allowing the teacher to first learn from other on-policy rollouts before teaching the student. LEAD constructs lookahead teachers through virtual RL updates on disjoint subsets of on-policy rollouts, and uses each teacher to provide cross-group token-level supervision for the other subset. To reconcile the complementary yet potentially conflicting signals from OPSD and RLVR, we introduce harmonious modulation, which adaptively modulates the GRPO advantage based on the agreement between the two signals. Extensive experiments on scientific reasoning and interactive-agent benchmarks show that LEAD achieves the best in-domain average performance across model scales while maintaining out-of-domain generalization comparable to GRPO.

Figures & tables

Figure 1: Overview of LEAD. LEAD consists of three steps: Lookahead, Cross Scoring, and Harmonious Modulation. Refer to Section 3.1 for the details.

	Scientific Reasoning					Interactive Agent
Method	Bio.	Chem.	Mate.	Phy.	Avg.	AlfWorld	WebShop	Search	Avg.
Qwen3-4B
Base Model	31.86	43.67	61.74	79.67	54.24	30.68	2.73	27.03	20.15
GRPO	52.50	78.05	80.51	94.77	76.46	71.40	75.80	41.60	62.93
SDPO	52.12	77.48	73.72	86.99	72.58	57.00	79.20	39.56	58.59
RLSD	52.88	78.51	80.38	94.67	76.61	68.80	78.10	40.50	62.47

Table 1: In-domain results across the scientific-reasoning and interactive-agent suites. Each result is evaluated on the test split of the same domain used for training. Maximum mean@16 (%) and success rate (%) is reported on the scientific-reasoning and interactive-agent suites, respectively. Base is the untrained model and is listed for reference and excluded from the ranking. Bold indicates the best result among the four training methods and underline the second best. On the scientific-reasoning suite, both the training and the test splits are curated as described in Appendix B , and all methods use the same curated splits.

	Scientific Reasoning					Interactive Agent
Method	Bio.	Chem.	Mate.	Phy.	Avg.	AlfWorld	WebShop	Search	Avg.
Qwen3-4B
Base Model	61.69	57.76	51.73	45.76	54.24	14.88	28.85	16.70	20.14
GRPO	60.52	58.89	50.06	48.37	54.46	14.28	30.67	20.45	21.80
SDPO	33.98	33.53	25.92	38.23	32.92	12.93	25.05	21.35	19.78
RLSD	59.88	58.63	48.83	47.20	53.64	14.43	28.41	18.02	20.29

Table 2: Out-of-domain results across the scientific-reasoning and interactive-agent suites. Each result is the averaged performance of the model trained on the corresponding domain while evaluated on the remaining domains of the same suite. Bold indicates the best result among the four training methods and underline the second best.

Lookahead
	In-domain					Out-of-domain
Variant	Bio.	Chem.	Mate.	Phy.	Avg.	Bio.	Chem.	Mate.	Phy.	Avg.
LEAD	53.87	81.19	85.82	93.65	78.63	58.98	57.31	52.72	48.07	54.27
w/o lookahead	51.50	79.34	80.51	93.06	76.10	60.04	56.23	51.60	48.49	54.09
Cross-Scoring
w/ self-induced	52.38	80.92	81.45	92.83	76.90	59.25	56.05	53.06	46.43	53.70

Table 3: Ablation study on the scientific-reasoning suite (Qwen3-4B). In-domain is the maximum mean@16 (%) reached during training on that domain; out-of-domain is the accuracy of the same model on the remaining three domains, averaged over the last three evaluation checkpoints. Bold indicates the best result in each column and underline the second best.

Method	Generation (s/step)	Update (s/step)	Total step (s/step)
GRPO	16.30	9.06	25.36
SDPO	16.03	13.52	29.55
RLSD	15.88	9.92	25.80
LEAD (ours)	15.78	11.57	27.35

Table 4: Wall-clock time per training step (seconds), scientific-reasoning suite with Qwen3-4B , averaged over the four domains. Generation is the rollout stage; Update covers the teacher forward passes, the log-probability recomputation and the policy update.

Figure 2: Reward curve during training on Chemistry and Materials Science datasets of the scientific-reasoning suite.

Figure 3: Evaluation dynamics on the scientific-reasoning suite. Top row: in-domain mean@16 on the training domain. Bottom row: out-of-domain mean@16, averaged over the three domains that were not used for training.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

	Training split					Test split
Domain	Original	Unchanged	Relabelled	Rewritten	Removed	Original	Unchanged	Relabelled	Rewritten	Removed
Biology	$450$	$447$	$0$	$0$	$3$	$50$	$50$	$0$	$0$	$0$
Chemistry	$1{,}890$	$1{,}856$	$1$	$4$	$29$	$210$	$208$	$1$	$0$	$1$
Material	$841$	$801$	$1$	$0$	$39$	$94$	$93$	$0$	$0$	$1$
Physics	$720$	$524$	$26$	$64$	$106$	$80$	$50$	$4$	$7$	$19$
Total	$3{,}901$	$3{,}628$	$28$	$68$	$177$	$434$	$401$	$5$	$7$	$21$

Appendix

Table 5: Curation results for the SciKnowEval L3 reasoning subset. Unchanged denotes items whose original question, options, and ground-truth label are retained. Relabelled denotes items for which the ground-truth letter is corrected while the question and options remain unchanged. Rewritten denotes items for which both models agree that none of the provided options is correct and derive the same answer; the labelled option is then replaced with the derived answer. Removed denotes items for which the two models reach inconsistent judgments.

	GRPO	SDPO	RLSD	LEAD (ours)
Learning rate	$1$ e $\text{-}6$	$1$ e $\text{-}5$	$1$ e $\text{-}6$	$1$ e $\text{-}6$
Entropy coefficient	$0$	$0$	$0.001$	$0$
KL loss coefficient	$0.001$	off	$0.01$	$0.001$
Optimizer	AdamW
Clip ratio (low, high)	$0.2$ , $0.28$ ; dual-clip $c_{\text{clip}}=3.0$
Sampling parameters (training)	temperature $1.0$ with $16$ samples at top- $p$ $0.95$

Appendix

Table 6: Method-specific hyperparameters. Training hyperparameters for GRPO, SDPO, RLSD, and LEAD.

Domain / split	Original item	Original label	Curation
Physics / train	Question: A current-carrying helix has 518 turns with a radius of 7 cm and a current of 5 A: Calculate the magnetic field at a point that is located on the axis of the helix and is 74 cm away from its center. Additionally, please select the option from the given choices that you believe is closest to the correct answer! (Please note that the following content involves constants might be used in the calculations : \mu_0 = 1.26 * 10 ˆ (-6) V $\cdot$ s/(A $\cdot$ m) B: 1.94 x 10 ˆ (-5) Tesla C: 8.65 x 10 ˆ (-5) Tesla D: 4.53 x 10 ˆ (-5) Tesla	A	relabel : ground truth A $\rightarrow$ B
Physics / test	Question: Determine the magnetic field strength at a distance of 1 cm from a straight wire carrying a current of 2 A: The wire has a length of 2 cm, and the current flows in the direction perpendicular to the wire. Additionally, please select the option from the given choices that you believe is closest to the correct answer! (Please note that the following content involves constants might be used in the calculations : \mu_0 = 1.26 * 10 ˆ (-6) V $\cdot$ s/(A $\cdot$ m) B: 66.43 x 10 ˆ (-6) Tesla C: 80.87 x 10 ˆ (-6) Tesla D: 40.0 x 10 ˆ (-6) Tesla	C	relabel : ground truth C $\rightarrow$ D
Physics / train	Question: What is the density of a plasma with a temperature of 11 electron volts and an electron number density of 2.4e+19 m ˆ -3? Use the Saha equation to determine the ionization state of the plasma assuming it consists of only hydrogen atoms. Please note that the following content involves constants might be used in the calculations : Vacuum tolerance $\varepsilon$ = 8.85e-12 ; Boltzmann constant k = 1.38e-23 ; electronic mass m_e = 9.11e-31 kg ; e = 1.6e-19 C ; proton mass m_p = 1.673e-27 kg ; neutral hydrogen atom mass m_h = 1.673e-27 kg. Additionally, please select the option from the given choices that you believe is closest to the correct answer! A: 4.02e-27 kg/m ˆ 3 B: 1.28e-27 kg/m ˆ 3 C: 8.26e-27 kg/m ˆ 3 D: 8.2e-27 kg/m ˆ 3	A	rewrite : option A rewritten to 4.02e-08 kg/m ˆ 3 and ground truth stays A

Appendix

Table 7: Curated cases of Physics domain.

Domain / split	Original item	Original label	Curation
Chemistry / train	What is the molar weight (g/mol) of the molecule with the canonical SMILES representation ’C1CC(NC1)C(=O)O’? A: 118.150 B: 152.140 C: 114.150 D: 115.150	A	relabel : A $\rightarrow$ D
Chemistry / train	What is the molar weight (g/mol) of the molecule with the canonical SMILES representation ’C1CC(NC1)C(=O)O’? A: 122.170 B: 145.200 C: 114.150 D: 182.020	A	rewrite : option A rewritten to 115.130 and ground truth stays A
Material / train	Given the following data on dilute solute elements and their diffusion barriers in different host elements, compare these values and answer the question. The diffusion barrier of the dilute solute elements (in eV) is calculated using DFT methods. Choose the combination with the highest diffusion rate. Given data: Solute Element 1: Fe, Host Element 1: Pd, Diffusion Barrier 1: -0.14447 eV Solute Element 2: Fe, Host Element 2: Pt, Diffusion Barrier 2: -0.00302 eV Solute Element 3: Fe, Host Element 3: Sc, Diffusion Barrier 3: -0.28547 eV Solute Element 4: Fe, Host Element 4: Ta, Diffusion Barrier 4: -0.19922 eV A: Fe in Pd B: Fe in Pt C: Fe in Sc D: Fe in Ta	B	relabel : B $\rightarrow$ C

Appendix

Table 8: Curated cases of Chemistry and Materials Science domains.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,087 papers

Self-Distilled RLVR
Apr 3, 2026Chenxu Yang, Chuanyu Qin, Qingyi Si +7On-Policy Self-Distillation Reinforcement Learning with Verifiable Rewards
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
DemoPSD: Disagreement-Modulated Policy Self-Distillation
Jul 2, 2026Yunhe Li, Hao Shi, Wenhao Liu +5On-Policy Self-Distillation RL for Language Model Reasoning
1City University of Hong Kong · 2Tsinghua University · 3Shenzhen University of Advanced Technology +1
Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR
May 11, 2026Jeonghye Kim, Jiwon Jeon, Dongsheng Li +1RL for Language Model Reasoning Language Model Distillation
Microsoft Research · KAIST

Apr 3, 2026J/K move · Enter open · S save

Self-Distilled RLVR

Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen+6

Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China

On-Policy Self-Distillation Reinforcement Learning with Verifiable Rewards On-Policy Distillation

Open paper AskPDF arXiv

On-policy distillation (OPD) has become a popular training paradigm in the LLM community. This paradigm selects a larger model as the teacher to provide dense, fine-grained signals for each sampled trajectory, in contrast to reinforcement learning with verifiable rewards (RLVR), which only obtains sparse signals from verifiable outcomes in the environment. Recently, the community has explored on-policy self-distillation (OPSD), where the same model serves as both teacher and student, with the teacher receiving additional privileged information such as reference answers to enable self-evolution. This paper demonstrates that learning signals solely derived from the privileged teacher result in severe information leakage and unstable long-term training. Accordingly, we identify the optimal niche for self-distillation and propose RLSD (RLVR with Self-Distillation). Specifically, we leverage self-distillation to obtain token-level policy differences for determining fine-grained update magnitudes, while continuing to use RLVR to derive reliable update directions from environmental feedback (e.g., response correctness). This enables RLSD to simultaneously harness the strengths of both RLVR and OPSD, achieving a higher convergence ceiling and superior training stability.

Abstract

RL for Language Model Reasoning

Explore similar work

Self-Distilled RLVR

DemoPSD: Disagreement-Modulated Policy Self-Distillation

Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR