RELAY: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learning

Kunwang Zheng; Xinyu Sun; Yu Cao; Shuocheng Li; Lei Cheng; Ying Liu; Qilong Zhang; Jinlong Chen

cs.LGOct 13, 2026

RELAY: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learning

Authors: Kunwang Zheng, Xinyu Sun, Yu Cao, Shuocheng Li, Lei Cheng, Ying Liu, Qilong Zhang, Jinlong Chen,

Organizations: University of Science and Technology of China · Fudan University · Alibaba Group

Abstract

Rubric-based reinforcement learning grades a response against many written criteria but collapses them into one reward, which discards the detail the rubric carries and gives no GRPO reward gradient when a group's responses tie. A frozen teacher that reads the rubric as privileged context provides token-level supervision, but we observe an early plateau in the gains over GRPO. RELAY uses advantages and rewards to select positive-advantage and low-scoring zero-advantage rollouts, and monitors the distillation loss to decide when to retire the teacher. Stage 1 combines selective distillation from a frozen teacher with GRPO; a plateau detector then removes the teacher, and Stage 2 continues with GRPO alone. Calibrated on one model-domain pair and reused unchanged in three others, the plateau detector fires between steps 122 and 171. RELAY scores highest in-domain among the compared methods in all four settings, ahead of GRPO on RaR-Medicine by 4.62 points with Qwen3-1.7B and 6.02 with Qwen3-4B. On Qwen3-1.7B/Medicine, removing the teacher from a shared checkpoint outperforms keeping, weakening or refreshing it. Distilling selected rollouts covering about half the response tokens also outperforms distilling all rollouts at a matched average coefficient.

Figures & tables

Figure 2: Overview of Relay . Stage 1 adds distillation to GRPO on selected rollouts. A plateau detector then removes the teacher, and Stage 2 continues with GRPO alone.

Rollout	GRPO reward gradient	Distilled
$A_{i}>\epsilon_{A}$	towards the rollout	yes
$A_{i}<-\epsilon_{A}$	away from it	no
$\|A_{i}\|\leq\epsilon_{A}$ , $r_{i}<\kappa_{t}$	approximately zero	yes
$\|A_{i}\|\leq\epsilon_{A}$ , $r_{i}\geq\kappa_{t}$	approximately zero	no

Table 1: Rollout classes and their treatment.

	Medicine			Science
Method	RaR-Med.	HealthBench	LLMEval-Med	RaR-Sci.	ResearchQA	Aggregate
Qwen3-1.7B
Base model	34.87	21.91	54.11	47.79	43.54	40.44
+ OPSD	40.20 $\uparrow$ 5.33	18.79 $\downarrow$ 3.12	55.71 $\uparrow$ 1.60	49.25 $\uparrow$ 1.46	45.31 $\uparrow$ 1.77	41.85 $\uparrow$ 1.41
+ GRPO	42.15 $\uparrow$ 7.28	23.87 $\uparrow$ 1.96	56.28 $\uparrow$ 2.17	52.79 $\uparrow$ 5.00	51.11 $\uparrow$ 7.57	45.24 $\uparrow$ 4.80
+ CriPO	42.37 $\uparrow$ 7.50	27.97 $\uparrow$ 6.06	54.99 $\uparrow$ 0.88	55.79 $\uparrow$ 8.00	60.58 $\uparrow$ 17.04	48.34 $\uparrow$ 7.90

Table 2: GPT-5.1 scores averaged over three evaluations (standard deviations: Table 7 ). Models are trained separately by domain; Aggregate averages the five benchmarks equally. Best/second best per model size: bold / underlined . Arrows show point changes from the same-size base model (

\uparrow

\downarrow

: increase/decrease).

Figure 3: Per-token divergence during Stage 1 of Relay across the four settings. Vertical lines mark the plateau-rule trigger. The divergence scale differs across settings, but the rule tests relative change and transfers without re-tuning.

Branch at step 200	210–260	270–300	310–350	360–390
Teacher removed	2.31	4.14	4.71	5.06
Kept at $\alpha{=}0.003$	2.49	3.45	3.73	4.11
Teacher refreshed	1.77	3.22	3.55	3.76
Kept at $\alpha{=}0.01$	1.70	2.49	2.67	2.88

Table 3: Training-judge validation gain from the shared step-200 branch point, averaged over the indicated steps. Bold marks the column maximum.

Teacher leaves at	141–160	181–200	221–240	281–300
Step 140	0.74	0.58	0.46	0.32
Step 171 (the rule)	0.73	0.72	0.57	0.42
Step 200	0.73	0.72	0.60	0.38
Never	0.73	0.72	0.70	0.60
GRPO (no teacher)	0.25	0.22	0.21	0.18

Table 4: Training-rollout token entropy by hand-off time (Qwen3-1.7B/Medicine), averaged over the steps shown. The four distillation runs share one Stage-1 trajectory and peak at 0.76 near step 130; each turns down after its own hand-off.

Distilled	Tok.	Best $\Delta$	Len.	Tr.
Pos. + low zero	.49	7.9	1,482	0.04
Pos. + all zero	.54	7.6	1,643	0.13
Positive only	.42	7.5	1,309	0.02
Pos. + negative	.91	7.3	1,752	0.35
All rollouts	.92	6.8	2,262	2.39
All, $\alpha{=}.005$	.98	7.1	1,978	1.17

Table 5: Rollout selection in Stage 1 (Qwen3-1.7B/Medicine). Tok.: distilled share of response tokens; low zero: rollouts with

|A_{i}|\leq\epsilon_{A}

and

r_{i}<\kappa_{t}

. Best

\Delta

: peak training-judge gain over Stage 1. Len., Tr.: rollout length and truncation (%) at steps 151–200. The all-rollout variant with

\alpha=0.005

matches Relay ’s average coefficient.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Rollout budget	400 steps
Prompts $\times$ responses per step	64 $\times$ 8
Policy updates per step	2 (32 prompts each)
Sampling (train)	$T{=}1$ , top- $p{=}1$ , top- $k{=}{-}1$
Prompt / response limit	1,024 / 5,120 tokens
Optimizer	Adam, LR $10^{-6}$ constant, no warmup

Appendix

Table 6: Training configuration ( Relay -specific rows at the bottom).

	Medicine			Science
Method	RaR-Med.	HealthBench	LLMEval-Med	RaR-Sci.	ResearchQA	Aggregate
Qwen3-1.7B
Base model	$34.87\pm 0.14$	$21.91\pm 1.00$	$54.11\pm 0.78$	$47.79\pm 0.10$	$43.54\pm 0.19$	40.44
+ OPSD	$40.20\pm 0.23$	$18.79\pm 0.19$	$55.71\pm 0.27$	$49.25\pm 0.54$	$45.31\pm 0.06$	41.85
+ GRPO	$42.15\pm 0.33$	$23.87\pm 0.11$	$56.28\pm 0.22$	$52.79\pm 0.06$	$51.11\pm 0.28$	45.24
+ CriPO	$42.37\pm 0.31$	$27.97\pm 0.27$	$54.99\pm 0.05$	$55.79\pm 0.22$	$60.58\pm 0.19$	48.34

Appendix

Table 7: Repeated GPT-5.1 evaluation of the checkpoints in Table 2 . Benchmark scores are means

\pm

standard deviations over three evaluations with fresh rollouts; Aggregate reports only the mean. Medicine and science columns use separately trained models.

Dataset	Use	Size
RaR-Medicine train	training	17,711
RaR-Medicine val.	validation	2,240
RaR-Medicine test	in-domain test	2,242
HealthBench subset	additional benchmark	500
LLMEval-Med	additional benchmark	571 (667 turns)
RaR-Science train	training	18,256

Appendix

Table 8: Datasets.

Figure 4: In-domain test-score gains in the first GPT-5.1 evaluation, with 95% question-bootstrap confidence intervals. Each domain uses a separately trained model.

	$\omega$	Criterion	Relay	CriPO	GRPO
1	$+5$	substitutes $S_{\mathrm{samp}}$	✓	✓	✓
2	$+5$	rearranges the equation	✓	✓	✓
3	$+4$	computes $0.394/0.296$	✓	✓	✓
4	$+4$	handles the units	✓	✓	✓
5	$+4$	explains the intercept	✓	—	—
6	$+3$	step-by-step derivation	✓	✓	✓

Appendix

Table 9: Rubric verdicts for the case of Appendix D (RaR-Science, Qwen3-4B). A check mark means the judge found the criterion present, which for the penalty row means the fault was committed. Scores follow Eq. ( 1 ), using the total positive rubric weight as the denominator and clipping the result to

[0,1]

Figure 6: CriPO token entropy at

\beta=0.05

(published) and

\beta=0.01

(used).

Setting	Min	Mean	Max
1.7B / Med.	0.402	0.496	0.594
1.7B / Sci.	0.403	0.499	0.588
4B / Med.	0.413	0.505	0.601
4B / Sci.	0.370	0.476	0.565

Appendix

Table 11: Fraction of response tokens selected for distillation, over all Stage-1 steps of each reported run.

Figure 7: Stage 1 response length (left) and per-token divergence (right) on Qwen3-1.7B/Medicine. Four selection rules from Table 5 use

\alpha=0.01

: Relay , positive only, positive plus negative, and all rollouts; GRPO is shown for reference.

Figure 8: Validation score (starting points aligned) and token entropy for GRPO and Relay across all four settings (10-step average). Dotted lines mark Relay ’s hand-off.

Schedule	Best $\Delta$	Step	Length
GRPO	9.8	370	1,374
Always on	12.3	390	2,155
Hand-off @140	13.3	350	2,008
Hand-off @200	14.1	370	2,658
Relay (rule @171)	14.8	400	2,557

Appendix

Table 12: Hand-off schedules (Qwen3-1.7B/Medicine). Best

\Delta

: peak training-judge gain from step 0 (3-step smoothed), reached at the step shown. Length: mean training-rollout response length near the peak. Always on keeps

\alpha{=}0.01

throughout.

Adaptive Knowledge Distillation

See all 13 papers

OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillation
Oct 2, 2026Xinpeng Wang, Wei Shi, Yu-Chia Chen +3RL for Language Models Rubric-Based RL
New York University · Work done at Meta · Meta +1
Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance
Aug 1, 2026Zhuowen Han, Jinwei Xiao, Zhengxi Lu +9Self-Training for Language Models RL for Language Model Reasoning
TJUNLP Lab, School of Computer Science and Technology, Tianjin University · Meituan Longcat Team
Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers
Jun 10, 2026MohammadHossein Rezaei, Anas Mahmoud, Zihao Wang +6Rubric-Based RL Language Model Distillation
Scale AI

Oct 2, 2026J/K move · Enter open · S save

OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillation

Xinpeng Wang, Wei Shi, Yu-Chia Chen, Maria Zontak+2

New York University · Work done at Meta · Meta+1

RL for Language Models Rubric-Based RL Rubric-Based Evaluation On-Policy Distillation

Open paper AskPDF arXiv

Many useful language-model tasks cannot be evaluated by exact outcome verification. Rubric-based reinforcement learning (RL) addresses this issue by scoring open-ended responses against explicit criteria. However, because the reward is assigned after the complete response, the training signal does not directly identify which individual decisions contributed to the final score. We propose a two-stage training framework that uses rubrics first as privileged teacher context for dense token-level supervision, then as rewards for further RL. In the first stage, rubric-privileged on-policy distillation (RP-OPD), a student without access to the rubric matches a rubric-aware teacher's next-token distributions at student-generated prefixes. In the second stage, RL directly optimizes the rubric reward and improves beyond the observed distillation plateau. We evaluate the framework on health and science tasks using open-weight models. Across HealthBench, ResearchQA, and RubricHub Science, we compare post-training methods and vary the amount of SFT or RP-OPD training before RL, finding that our two-stage framework achieves the highest scores among the methods evaluated. RP-OPD + RL shows limited signs of reward hacking on RubricHub Science, whereas the SFT + RL baseline increasingly receives high rewards for claims of rubric compliance without providing the required content. These findings support using rubrics to guide on-policy distillation before applying rubric-based RL.

Abstract

Adaptive Knowledge Distillation

Explore similar work

OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillation

Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance

Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers