Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing
Authors: Gengsheng Li, Tianyu Yang, Junfeng Fang, Mingyang Song, Mao Zheng, Haiyun Guo, Dan Zhang, Jinqiao Wang, +1 more
Organizations: Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · National University of Singapore · Tencent · Wuhan AI Research
Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training large language models. While Group Relative Policy Optimization (GRPO) is widely adopted, its coarse credit assignment uniformly penalizes failed rollouts, lacking the token-level focus needed to efficiently address specific deviations. Self-Distillation Policy Optimization (SDPO) addresses this by providing denser, more targeted logit-level supervision that facilitates rapid early improvement, yet it frequently collapses during prolonged training. We trace this late-stage instability to two intrinsic flaws: self-distillation on already-correct samples introduces optimization ambiguity, and the self-teacher's signal reliability progressively degrades. To resolve these issues, we propose Sample-Routed Policy Optimization (SRPO), a unified on-policy framework that routes correct samples to GRPO's reward-aligned reinforcement and failed samples to SDPO's targeted logit-level correction. SRPO further incorporates an entropy-aware dynamic weighting mechanism to suppress high-entropy, unreliable distillation targets while emphasizing confident ones. Evaluated across five benchmarks and two model scales, SRPO achieves both the rapid early improvement of SDPO and the long-horizon stability of GRPO. It consistently surpasses the peak performance of both baselines, raising the five-benchmark average on Qwen3-8B by 3.4% over GRPO and 6.3% over SDPO, while simultaneously yielding moderate response lengths and lowering per-step compute cost by up to 17.2%.
Figures & tables
Figure 1: Training dynamics and diagnostic analysis on Chemistry with Qwen3-8B. (a) SDPO improves faster than GRPO in early training, but is later overtaken and collapses, whereas SRPO achieves both rapid initial improvement and stable long-horizon optimization. (b) Restricting SDPO updates to incorrect samples retains most of its overall benefit, whereas applying SDPO only to correct samples degrades performance and destabilizes training, supporting the necessity of sample routing. (c) The self-teacher’s token-level entropy rises during training, indicating that the distillation signal becomes increasingly dominated by uncertain predictions. Curves show a 5-step rolling mean and shaded bands denote ±1 std.
Figure 2: Overview of SRPO. Given a prompt x , the policy πθ generates a group of on-policy rollouts. A correctness check routes each rollout to one of two branches: correct samples are sent to the GRPO branch (top), where group-relative advantages provide a reward-aligned policy update; incorrect samples with available teacher information are sent to the SDPO branch (bottom), where a feedback-conditioned self-teacher produces logit-level distillation targets via KL(P∥stopgrad(Q)) for dense corrective supervision.
Chemistry
Physics
Biology
Materials
Tool Use
Average
1h
5h
10h
1h
5h
10h
1h
5h
10h
1h
5h
10h
1h
5h
10h
1h
5h
10h
Qwen3-8B
41.1
58.7
30.5
59.3
57.9
49.5
+ GRPO
62.1
75.9
78.9
61.0
72.3
73.6
46.9
68.1
70.6
74.7
77.6
77.8
64.3
68.5
69.0
61.8
72.5
74.0
+ SDPO
71.6
80.6
80.6
67.6
74.0
74.0
52.1
58.5
58.5
68.1
76.6
76.6
64.8
65.7
65.7
64.8
71.1
71.1
+ SRPO
69.2
81.8
83.0
69.5
77.1
78.4
55.8
68.3
72.8
74.9
79.2
81.5
65.2
71.2
71.2
66.9
75.5
77.4
Qwen3-4B
43.6
59.8
30.8
61.2
58.8
50.8
Table 1: Main results on five benchmarks at three training budgets. Each entry reports the highest achieved avg@16 accuracy (%) within the corresponding wall-clock budget. The last three columns report the mean over the five benchmarks. Within each model scale, the best result in each column is in bold and the second-best is underlined.
Figure 3: Training curves on three representative benchmarks for Qwen3-8B. We plot avg@16 against wall-clock training time on (a) Chemistry, (b) Biology, and (c) Tool Use. These curves complement Table 1 , which reports the highest achieved result within each training budget. All curves show a 5-step rolling mean and shaded bands denote ±1 std.
Ablation Target
Variant
1h
5h
10h
Mixing Strategy
SRPO w/o dynamic weighting
66.5
74.8
75.6
Advantage Mix
67.2 +0.7
72.3 -2.5
72.3 -3.3
Dynamic Weighting
SRPO
66.9
75.5
77.4
SRPO w/o dynamic weighting
66.5 -0.4
74.8 -0.7
75.6 -1.8
Table 2: Ablation results on Qwen3-8B, reported as avg@16 accuracy (%) across five benchmarks. The first block isolates the mixing strategy, and the second isolates the additional effect of dynamic weighting on top of sample routing. Colored deltas are measured relative to the reference row within each block.
Figure 4: Response length and per-step compute time for Qwen3-8B. (a) Response length on Chemistry: GRPO remains consistently long, SDPO drops rapidly, and SRPO stays moderate. Curves show a 5-step rolling mean and shaded bands denote ±1 std. (b) Average seconds per training step, averaged across five benchmarks and measured over the 1h, 5h, and 10h windows. SRPO incurs a modest overhead relative to GRPO in the early stage of training, but becomes faster than both GRPO and SDPO over longer training horizons.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Parameters
GRPO
SDPO
SRPO
General
Model
Qwen3-{4B, 8B}
Qwen3-{4B, 8B}
Qwen3-{4B, 8B}
Thinking
False
False
False
Data
Max. prompt length
2048
2048
2048
Max. response length
8192
8192
8192
Appendix
Table 3: Hyperparameters for GRPO, SDPO, and SRPO. For GRPO and SDPO, we use the configurations from Hübotter et al. (2026) . For SRPO, the GRPO-branch and SDPO-branch loss parameters are inherited from the respective baselines; only the learning rate and the dynamic-weighting temperature β differ. Entries marked “–” indicate parameters not applicable to that method.
Benchmark
Source
Train
Test
Total
Chemistry
SciKnowEval
1,890
210
2,100
Physics
SciKnowEval
720
80
800
Biology
SciKnowEval
450
50
500
Materials
SciKnowEval
841
94
935
Tool Use
ToolAlpaca
4,046
68
4,114
Appendix
Table 4: Dataset statistics for all five benchmarks. The four Science Q&A benchmarks are drawn from the reasoning subset (Level 3) of SciKnowEval ( Feng et al., 2024 ) ; Tool Use is drawn from ToolAlpaca ( Tang et al., 2023 ) . All splits are identical to those used by SDPO ( Hübotter et al., 2026 ) .
Benchmark
Question (excerpt)
Answer
Chemistry
What is the correct logarithmic solubility value of the molecule “Cc1cc(=O)[nH]c(=S)[nH]1” in aqueous solutions? A: −3.01 B: −2.436 C: −4.576 D: 1.1
B
Physics
A charged particle produces an electric field with a magnitude of 2.0N/C at a point that is 50cm away from the particle. What is the magnitude of the particle’s charge? A: 50 pC B: 56 pC C: 60 pC D: 64 pC
B
Biology
What is the folding stability score of the protein sequence “GSSTTRYRFLDEEEARRAAKEWARRGYQVHVTQNGTYWEVEVR”? A: −0.01 B: 1.69 C: 2.49 D: 0.45
B
Materials
Given the following crystal structure parameters for the material RbLa 9 (IrO 6 ) 4 (Material ID: mp-560657), calculate the volume of the unit cell (in Å 3 ). Lattice: a=7.82 , b=7.82 , c=17.88 Å; α=β=γ=90∘ . A: 1025.67 B: 1094.31 C: 1200.45 D: 1150.78
B
Tool Use
(Given the Axolotl API specification) Question: “I’m looking for an axolotl that is wild in color and medium in size. Can you help me find some pictures?”
Table 5: One representative example from each benchmark. Science Q&A examples show the question stem and four answer options; the Tool Use example shows the user query and the expected structured tool call (API specification and answer omitted for brevity; see Appendix B.3 for the full template).
Rollout
Correct? ( ci )
Teacher avail.? ( mi )
Route
Explanation
y0 (incorrect)
0
1
SDPO
Uses y2 ’s response as teacher info
y2 (correct)
1
1
GRPO
Correct ⇒ GRPO; y5 available but unused
y5 (correct)
1
1
GRPO
Correct ⇒ GRPO; y2 available but unused
y7 (incorrect)
0
1
SDPO
Uses y2 ’s response as teacher info
Appendix
Table 6: Routing decisions for a prompt with two correct rollouts ( y2 , y5 ) and six incorrect ones. All incorrect rollouts have teacher information available ( mi=1 ) because at least one correct sibling exists.
Correct? ( ci )
Teacher avail.? ( mi )
Teacher prompt content
Route
✓
✓
Question + sibling solution
GRPO
✓
✗
Question only (no sibling)
GRPO
✗
✓
Question + sibling solution
SDPO
✗
✗
Question only (no sibling)
GRPO (fallback)
Appendix
Table 7: Complete routing decision matrix. Only incorrect rollouts with available teacher information are routed to the SDPO branch; all other cases default to GRPO.
Figure 5: Routing statistics during SRPO training of Qwen3-8B in Chemistry. (a) Fraction of samples routed to the GRPO branch. (b) Fraction of samples routed to the SDPO branch. (c) Fraction of samples in each batch for which teacher information can be constructed. As training progresses, the policy improves and generates more correct rollouts, causing the SDPO fraction to decrease steadily while the GRPO fraction increases correspondingly. All curves show a 5-step rolling mean and shaded bands denote ±1 std.
Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for improving large language models (LLMs) on various tasks, yet its sparse outcome rewards lack token-level credit assignment for intermediate steps. To address this, on-policy self-distillation (OPSD) leverages a self-teacher with privileged context to provide additional dense learning signals. However, because the self-teacher is often overconfident and imposes excessive penalties on long reasoning trajectories, OPSD frequently struggles in practice. To mitigate this, we propose self-instructing policy optimization (SIPO) with a contrastive self-teacher to provide dense credit. At each iteration, SIPO samples multiple rollouts per prompt from the current policy, scores them with environment rewards, and constructs two teacher contexts for each rollout by pairing the reference answer with mistakes made within the group. The model then re-evaluates its own responses under both contexts, using the difference between the two teacher log-probabilities as token-level feedback, so that biases shared by both contexts are expected to largely cancel. The resulting objective yields a token-level advantage for every rollout: the reward still sets the main direction of each update while the self-teacher redistributes credit across tokens. Even in groups where every rollout fails and group-relative advantages vanish, SIPO still provides a learning signal. By preserving direct optimization of the task reward while providing dense, token-level feedback, this approach bridges reinforcement learning and on-policy self-distillation. Extensive experiments across multiple reasoning and code-generation benchmarks demonstrate that SIPO outperforms both RLVR and OPSD baselines without an external teacher or additional generation.
Zhenrui Yue, Huimin Zeng, Yueqi Wang +8
University of Illinois · UC San Diego · Rochester Institute of Technology +2
Group Relative Policy Optimization (GRPO) learns from reward differences within a rollout group, but receives no useful relative signal when every sampled response is incorrect. Privileged self-distillation can fill this gap with dense token supervision, yet applying it throughout training creates a different failure mode: the teacher is a biased, low-variance surrogate for the reward objective, so persistent imitation can oppose reward-improving updates after the policy becomes capable of producing successful trajectories. We introduce I-SDPO (Instance-Level Adaptive Self-Distillation Policy Optimization), which treats teacher reliance as capability-dependent. I-SDPO makes one routing decision per input instance and shares it across that instance's rollout group: all-incorrect groups use a privileged self-distillation objective, whereas any-success groups remain intact for GRPO. This design uses imitation only where group-relative rewards are uninformative. A local analysis characterizes when teacher and reward directions align and shows that a non-vanishing biased distillation weight induces an optimization bias floor. The routing rule automatically reduces the expected distillation rate as success probability rises, withdrawing teacher influence without a hand-designed schedule. On SciKnowEval, I-SDPO obtains the best result in all four scientific domains and improves average mean@16 accuracy from 56.67% with GRPO to 70.31%, with a maximum domain gain of 18.24 points.
Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training large language models (LLMs). While Group Relative Policy Optimization (GRPO) is widely adopted, it suffers from sparse reward signals and loses gradients entirely when all responses within a group receive identical rewards. On-policy distillation (OPD) offers a natural remedy by providing dense, token-level supervision from a teacher model. However, naively combining GRPO with OPD leads to degraded performance, due to three underlying causes: not all samples benefit from distillation; fitting too quickly to the teacher undermines the exploratory capacity of RL; and OPD's advantages are asymmetric, suppressing most tokens. To address these challenges, we propose RSTG (Recovering Learning Signals via Adaptive Teacher Guidance), which applies distillation selectively and precisely where it matters most. At the sample level, OPD is restricted to negative zero-variance prompts with each sample weighted by the teacher's confidence score. At the token level, distillation targets only tokens with high student entropy or large teacher-student divergence. We further augment training with SFT on correct trajectories generated by the teacher model, injecting positive gradient signals where RL yields none. Experiments demonstrate that RSTG substantially outperforms naive GRPO+OPD by +4.02% on math and +3.05% on code.
Zhuowen Han, Jinwei Xiao, Zhengxi Lu +9
TJUNLP Lab, School of Computer Science and Technology, Tianjin University · Meituan Longcat Team