Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing

Gengsheng Li; Tianyu Yang; Junfeng Fang; Mingyang Song; Mao Zheng; Haiyun Guo; Dan Zhang; Jinqiao Wang

cs.LGApr 2, 2026

Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing

Authors: Gengsheng Li, Tianyu Yang, Junfeng Fang, Mingyang Song, Mao Zheng, Haiyun Guo, Dan Zhang, Jinqiao Wang, +1 more

Organizations: Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · National University of Singapore · Tencent · Wuhan AI Research

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training large language models. While Group Relative Policy Optimization (GRPO) is widely adopted, its coarse credit assignment uniformly penalizes failed rollouts, lacking the token-level focus needed to efficiently address specific deviations. Self-Distillation Policy Optimization (SDPO) addresses this by providing denser, more targeted logit-level supervision that facilitates rapid early improvement, yet it frequently collapses during prolonged training. We trace this late-stage instability to two intrinsic flaws: self-distillation on already-correct samples introduces optimization ambiguity, and the self-teacher's signal reliability progressively degrades. To resolve these issues, we propose Sample-Routed Policy Optimization (SRPO), a unified on-policy framework that routes correct samples to GRPO's reward-aligned reinforcement and failed samples to SDPO's targeted logit-level correction. SRPO further incorporates an entropy-aware dynamic weighting mechanism to suppress high-entropy, unreliable distillation targets while emphasizing confident ones. Evaluated across five benchmarks and two model scales, SRPO achieves both the rapid early improvement of SDPO and the long-horizon stability of GRPO. It consistently surpasses the peak performance of both baselines, raising the five-benchmark average on Qwen3-8B by 3.4% over GRPO and 6.3% over SDPO, while simultaneously yielding moderate response lengths and lowering per-step compute cost by up to 17.2%.

Figures & tables

Figure 1: Training dynamics and diagnostic analysis on Chemistry with Qwen3-8B. (a) SDPO improves faster than GRPO in early training, but is later overtaken and collapses, whereas SRPO achieves both rapid initial improvement and stable long-horizon optimization. (b) Restricting SDPO updates to incorrect samples retains most of its overall benefit, whereas applying SDPO only to correct samples degrades performance and destabilizes training, supporting the necessity of sample routing. (c) The self-teacher’s token-level entropy rises during training, indicating that the distillation signal becomes increasingly dominated by uncertain predictions. Curves show a 5-step rolling mean and shaded bands denote $\pm 1$ std.

Figure 2: Overview of SRPO. Given a prompt $x$ , the policy $\pi_{\theta}$ generates a group of on-policy rollouts. A correctness check routes each rollout to one of two branches: correct samples are sent to the GRPO branch (top), where group-relative advantages provide a reward-aligned policy update; incorrect samples with available teacher information are sent to the SDPO branch (bottom), where a feedback-conditioned self-teacher produces logit-level distillation targets via $\mathrm{KL}(P\;\|\;\mathrm{stopgrad}(Q))$ for dense corrective supervision.

	Chemistry			Physics			Biology			Materials			Tool Use			Average
	1h	5h	10h	1h	5h	10h	1h	5h	10h	1h	5h	10h	1h	5h	10h	1h	5h	10h
Qwen3-8B	41.1			58.7			30.5			59.3			57.9			49.5
+ GRPO	62.1	75.9	78.9	61.0	72.3	73.6	46.9	68.1	70.6	74.7	77.6	77.8	64.3	68.5	69.0	61.8	72.5	74.0
+ SDPO	71.6	80.6	80.6	67.6	74.0	74.0	52.1	58.5	58.5	68.1	76.6	76.6	64.8	65.7	65.7	64.8	71.1	71.1
+ SRPO	69.2	81.8	83.0	69.5	77.1	78.4	55.8	68.3	72.8	74.9	79.2	81.5	65.2	71.2	71.2	66.9	75.5	77.4
Qwen3-4B	43.6			59.8			30.8			61.2			58.8			50.8

Table 1: Main results on five benchmarks at three training budgets. Each entry reports the highest achieved avg@16 accuracy (%) within the corresponding wall-clock budget. The last three columns report the mean over the five benchmarks. Within each model scale, the best result in each column is in bold and the second-best is underlined.

Figure 3: Training curves on three representative benchmarks for Qwen3-8B. We plot avg@16 against wall-clock training time on (a) Chemistry, (b) Biology, and (c) Tool Use. These curves complement Table 1 , which reports the highest achieved result within each training budget. All curves show a 5-step rolling mean and shaded bands denote

\pm 1

std.

Ablation Target	Variant	1h	5h	10h
Mixing Strategy	SRPO w/o dynamic weighting	66.5	74.8	75.6
	Advantage Mix	67.2 +0.7	72.3 -2.5	72.3 -3.3
Dynamic Weighting	SRPO	66.9	75.5	77.4
	SRPO w/o dynamic weighting	66.5 -0.4	74.8 -0.7	75.6 -1.8

Table 2: Ablation results on Qwen3-8B, reported as avg@16 accuracy (%) across five benchmarks. The first block isolates the mixing strategy, and the second isolates the additional effect of dynamic weighting on top of sample routing. Colored deltas are measured relative to the reference row within each block.

Figure 4: Response length and per-step compute time for Qwen3-8B. (a) Response length on Chemistry: GRPO remains consistently long, SDPO drops rapidly, and SRPO stays moderate. Curves show a 5-step rolling mean and shaded bands denote

\pm 1

std. (b) Average seconds per training step, averaged across five benchmarks and measured over the 1h, 5h, and 10h windows. SRPO incurs a modest overhead relative to GRPO in the early stage of training, but becomes faster than both GRPO and SDPO over longer training horizons.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Parameters	GRPO	SDPO	SRPO
General
Model	Qwen3-{4B, 8B}	Qwen3-{4B, 8B}	Qwen3-{4B, 8B}
Thinking	False	False	False
Data
Max. prompt length	2048	2048	2048
Max. response length	8192	8192	8192

Appendix

Table 3: Hyperparameters for GRPO, SDPO, and SRPO. For GRPO and SDPO, we use the configurations from Hübotter et al. (2026) . For SRPO, the GRPO-branch and SDPO-branch loss parameters are inherited from the respective baselines; only the learning rate and the dynamic-weighting temperature

\beta

differ. Entries marked “–” indicate parameters not applicable to that method.

Benchmark	Source	Train	Test	Total
Chemistry	SciKnowEval	1,890	210	2,100
Physics	SciKnowEval	720	80	800
Biology	SciKnowEval	450	50	500
Materials	SciKnowEval	841	94	935
Tool Use	ToolAlpaca	4,046	68	4,114

Appendix

Table 4: Dataset statistics for all five benchmarks. The four Science Q&A benchmarks are drawn from the reasoning subset (Level 3) of SciKnowEval ( Feng et al., 2024 ) ; Tool Use is drawn from ToolAlpaca ( Tang et al., 2023 ) . All splits are identical to those used by SDPO ( Hübotter et al., 2026 ) .

Benchmark	Question (excerpt)	Answer
Chemistry	What is the correct logarithmic solubility value of the molecule “Cc1cc(=O)[nH]c(=S)[nH]1” in aqueous solutions? A: $-3.01$ B: $-2.436$ C: $-4.576$ D: $1.1$	B
Physics	A charged particle produces an electric field with a magnitude of $2.0\;\mathrm{N/C}$ at a point that is $50\;\mathrm{cm}$ away from the particle. What is the magnitude of the particle’s charge? A: 50 pC B: 56 pC C: 60 pC D: 64 pC	B
Biology	What is the folding stability score of the protein sequence “GSSTTRYRFLDEEEARRAAKEWARRGYQVHVTQNGTYWEVEVR”? A: $-0.01$ B: $1.69$ C: $2.49$ D: $0.45$	B
Materials	Given the following crystal structure parameters for the material RbLa 9 (IrO 6 ) 4 (Material ID: mp-560657), calculate the volume of the unit cell (in Å 3 ). Lattice: $a{=}7.82$ , $b{=}7.82$ , $c{=}17.88$ Å; $\alpha{=}\beta{=}\gamma{=}90^{\circ}$ . A: 1025.67 B: 1094.31 C: 1200.45 D: 1150.78	B
Tool Use	(Given the Axolotl API specification) Question: “I’m looking for an axolotl that is wild in color and medium in size. Can you help me find some pictures?”	searchAxolotlImages({"color": "wild", "gender": "", "size": "medium", "page": 1})

Appendix

Table 5: One representative example from each benchmark. Science Q&A examples show the question stem and four answer options; the Tool Use example shows the user query and the expected structured tool call (API specification and answer omitted for brevity; see Appendix B.3 for the full template).

Rollout	Correct? ( $c_{i}$ )	Teacher avail.? ( $m_{i}$ )	Route	Explanation
$y_{0}$ (incorrect)	0	1	SDPO	Uses $y_{2}$ ’s response as teacher info
$y_{2}$ (correct)	1	1	GRPO	Correct $\Rightarrow$ GRPO; $y_{5}$ available but unused
$y_{5}$ (correct)	1	1	GRPO	Correct $\Rightarrow$ GRPO; $y_{2}$ available but unused
$y_{7}$ (incorrect)	0	1	SDPO	Uses $y_{2}$ ’s response as teacher info

Appendix

Table 6: Routing decisions for a prompt with two correct rollouts (

y_{2}

y_{5}

) and six incorrect ones. All incorrect rollouts have teacher information available (

m_{i}=1

) because at least one correct sibling exists.

Correct? ( $c_{i}$ )	Teacher avail.? ( $m_{i}$ )	Teacher prompt content	Route
✓	✓	Question + sibling solution	GRPO
✓	✗	Question only (no sibling)	GRPO
✗	✓	Question + sibling solution	SDPO
✗	✗	Question only (no sibling)	GRPO (fallback)

Appendix

Table 7: Complete routing decision matrix. Only incorrect rollouts with available teacher information are routed to the SDPO branch; all other cases default to GRPO.

Figure 5: Routing statistics during SRPO training of Qwen3-8B in Chemistry. (a) Fraction of samples routed to the GRPO branch. (b) Fraction of samples routed to the SDPO branch. (c) Fraction of samples in each batch for which teacher information can be constructed. As training progresses, the policy improves and generates more correct rollouts, causing the SDPO fraction to decrease steadily while the GRPO fraction increases correspondingly. All curves show a 5-step rolling mean and shaded bands denote

\pm 1

std.

Explore similar work

SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation
Sep 29, 2026Zhenrui Yue, Huimin Zeng, Yueqi Wang +8RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
University of Illinois · UC San Diego · Rochester Institute of Technology +2
I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization
Aug 13, 2026Yubo Zhang, Xinhong Ma, Zezhong Tan +1On-Policy Self-Distillation Group Relative Policy Optimization
Qwen Large Model Application Team, Alibaba
Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance
Aug 1, 2026Zhuowen Han, Jinwei Xiao, Zhengxi Lu +9Self-Training for Language Models RL for Language Model Reasoning
TJUNLP Lab, School of Computer Science and Technology, Tianjin University · Meituan Longcat Team

Sep 29, 2026J/K move · Enter open · S save

SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation

Zhenrui Yue, Huimin Zeng, Yueqi Wang, Yaokun Liu+7

University of Illinois · UC San Diego · Rochester Institute of Technology+2

RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards On-Policy Distillation Knowledge Distillation

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for improving large language models (LLMs) on various tasks, yet its sparse outcome rewards lack token-level credit assignment for intermediate steps. To address this, on-policy self-distillation (OPSD) leverages a self-teacher with privileged context to provide additional dense learning signals. However, because the self-teacher is often overconfident and imposes excessive penalties on long reasoning trajectories, OPSD frequently struggles in practice. To mitigate this, we propose self-instructing policy optimization (SIPO) with a contrastive self-teacher to provide dense credit. At each iteration, SIPO samples multiple rollouts per prompt from the current policy, scores them with environment rewards, and constructs two teacher contexts for each rollout by pairing the reference answer with mistakes made within the group. The model then re-evaluates its own responses under both contexts, using the difference between the two teacher log-probabilities as token-level feedback, so that biases shared by both contexts are expected to largely cancel. The resulting objective yields a token-level advantage for every rollout: the reward still sets the main direction of each update while the self-teacher redistributes credit across tokens. Even in groups where every rollout fails and group-relative advantages vanish, SIPO still provides a learning signal. By preserving direct optimization of the task reward while providing dense, token-level feedback, this approach bridges reinforcement learning and on-policy self-distillation. Extensive experiments across multiple reasoning and code-generation benchmarks demonstrate that SIPO outperforms both RLVR and OPSD baselines without an external teacher or additional generation.

Abstract

Explore similar work

SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation

I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization

Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance