Many useful language-model tasks cannot be evaluated by exact outcome verification. Rubric-based reinforcement learning (RL) addresses this issue by scoring open-ended responses against explicit criteria. However, because the reward is assigned after the complete response, the training signal does not directly identify which individual decisions contributed to the final score. We propose a two-stage training framework that uses rubrics first as privileged teacher context for dense token-level supervision, then as rewards for further RL. In the first stage, rubric-privileged on-policy distillation (RP-OPD), a student without access to the rubric matches a rubric-aware teacher's next-token distributions at student-generated prefixes. In the second stage, RL directly optimizes the rubric reward and improves beyond the observed distillation plateau. We evaluate the framework on health and science tasks using open-weight models. Across HealthBench, ResearchQA, and RubricHub Science, we compare post-training methods and vary the amount of SFT or RP-OPD training before RL, finding that our two-stage framework achieves the highest scores among the methods evaluated. RP-OPD + RL shows limited signs of reward hacking on RubricHub Science, whereas the SFT + RL baseline increasingly receives high rewards for claims of rubric compliance without providing the required content. These findings support using rubrics to guide on-policy distillation before applying rubric-based RL.
Figures & tables
Figure 1: Overview of RP-OPD + Rubric-RL . Stage 1 ( RP-OPD ): The student receives dense supervision from a rubric-conditioned teacher by minimizing the KL divergence between their token distributions at each position in student-generated responses. Stage 2 ( Rubric-RL ): An LLM judge scores the student’s responses against the rubric. These scores serve as rewards for RL, starting from the policy trained in Stage 1.
Figure 2: Rubric scores measured on held-out evaluation sets during training for SFT, OPD, and rubric-based RL on HealthBench (a), ResearchQA (b), and RubricHub Science (c). SFT and OPD are compared with and without teacher access to the rubric. Dashed gold lines show the performance of the rubric-conditioned teacher that SFT + rubric and OPD + rubric learn from. Teacher access to rubrics improves SFT and OPD, but OPD alone falls short of direct rubric-based RL.
Figure 3: RL trajectories on 500 HealthBench prompts overlapping with training, from RP-OPD (top) and SFT (bottom) across three models. Dashed curves show stage-one training, grey curves show direct RL, and colors identify starting checkpoints. White circles mark RL starts. RP-OPD warm starts achieve higher downstream scores and are less sensitive to prolonged warm-start training than SFT.
Figure 4: RL trajectories on 500 held-out HealthBench prompts, from RP-OPD (top) and SFT (bottom) across three models. Longer SFT warm starts lead to lower downstream scores for Qwen2.5-7B and Llama-3.1-8B, while RP-OPD is less sensitive to warm-start duration.
Figure 5: HealthBench rubric scores (a,b) and token entropy (c,d) during RP-OPD , SFT, and subsequent RL for Qwen2.5-3B. Dashed curves show RP-OPD or SFT training, grey curves show direct RL, and colored curves show RL from different warm starts. Legend numbers give the number of RP-OPD or SFT updates before RL begins. Before RL, SFT and RP-OPD reach similar rubric scores, but SFT has substantially lower token entropy. Subsequent RL achieves larger gains from RP-OPD warm starts.
Method
HealthBench
ResearchQA
RubricHub Science
Qwen2.5-3B
Rubric-based RL, no SFT warm-up
0.567
0.667
0.620
Rubric-conditioned SFT
0.527
0.540
0.606
RP-OPD (ours, stage 1 only)
0.495
0.594
0.642
Rubric-conditioned SFT + RL
0.612
0.743
0.682
RP-OPD + Rubric-RL (ours)
0.632
0.776
0.735
Qwen2.5-7B
Rubric-based RL, no SFT warm-up
0.583
0.672
0.651
Table 1: Performance on held-out evaluation sets using shortcut-aware GPT-4o-mini grading. Bold marks the highest score in each column within each block. RP-OPD + Rubric-RL achieves the highest scores in every evaluated model–dataset combination.
Figure 6: Rubric scores and response lengths for our models and public models across three benchmarks. Error bars denote 95% confidence intervals. Our 7B model matches GPT-5 scores on ResearchQA at a similar response length. On RubricHub Science, our 3B and 7B models score higher than GPT-5 with longer responses.
Figure 8: Reward-hacking comparison between the SFT + RL baseline and RP-OPD + RL on RubricHub Science. Panels (a,b) compare original and regraded scores. Panel (c) tracks rubric mentions and claims of correctness. Such claims become frequent in the SFT + RL baseline, whose scores fall sharply after regrading. RP-OPD + RL shows limited signs of these behaviors, and its scores remain nearly unchanged.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Setting
HealthBench
ResearchQA
RubricHub Science
SFT
Learning rate
10−5
10−5
10−5
Global batch size
256
256
256
Sequence length cap
2,048
4,096
3,072
RP-OPD
Learning rate
10−5
10−5
3×10−5
Global batch size
512
512
512
Response length cap
1,024
2,048
2,048
Appendix
Table 2: SFT and RP-OPD settings for Qwen2.5-3B with a Qwen2.5-32B teacher. Values are default settings.
Setting
Value
RL algorithm
DAPO-style GRPO
Clipping
clip_cov , ratio 0.05
Advantage normalization
batch_dapo
Overlong-response control
Penalty buffer
Learning rate
1×10−6
Actor KL coefficient
0.001
Appendix
Table 3: Default downstream RL settings for Qwen2.5-3B.
Figure 9: Stage-one distillation with 3B, 14B, and 32B teachers and a fixed Qwen2.5-3B student, scored by Qwen3-32B.
Figure 10: Effect of SFT response selection on subsequent RL performance on HealthBench. (a) Scores during RL. (b) Scores after 300 RL steps versus the scores of the teacher responses used for SFT. Results show means and ranges over three runs. After 300 RL steps, selecting the highest-scoring or lowest-scoring teacher responses leads to similar performance, while RP-OPD achieves the highest mean score.
Figure 11: Effect of teacher sampling settings on SFT and subsequent RL. Left: token entropy during SFT. Right: training reward during the first 200 RL steps. SFT with higher-temperature, higher-top- p teacher responses has higher entropy but similar early RL reward. RP-OPD is shown for reference.
Figure 12: Forward and reverse KL during distillation on RubricHub Science. Left: rubric scores on held-out responses. Right: token entropy computed over the 20 most probable tokens, with their probabilities renormalized. Forward KL achieves higher scores at most checkpoints and higher entropy at every measured checkpoint in this comparison.
Scoring stage
Mean score
Original rubric score
0.8297
LLM regrading before evidence checks
0.4224
After evidence checks
0.4040
Appendix
Table 4: Scoring the same 4,096 SFT + RL600 responses. Regrading applies to 3,082 flagged responses. Other scores are unchanged.
Figure 13: Qwen3-32B and two GPT-4o-mini grading procedures applied to the same 500 Qwen2.5-7B responses per checkpoint.
Figure 14: Answer outlines during RL with the revised teacher prompt. Panels (a,b) show Qwen3-32B scores before and after regrading the same responses. Panel (c) shows the fraction flagged by the rule for detecting answer outlines. SFT + RL increasingly receives credit for instructions to provide an answer, while RP-OPD + RL scores change little after regrading.
On-policy distillation (OPD) is a powerful paradigm for model alignment, yet its reliance on teacher logits restricts its application to white-box scenarios. We contend that structured semantic rubrics can serve as a scalable alternative to teacher logits, enabling OPD using only teacher-generated responses. To prove it, we introduce ROPD, a simple yet foundational framework for rubric-based OPD. Specifically, ROPD induces prompt-specific rubrics from teacher-student contrasts, and then utilizes these rubrics to score the student rollouts for on-policy optimization. Empirically, ROPD outperforms the advanced logit-based OPD methods across most scenarios, and achieving up to a 10x gain in sample efficiency. These results position rubric-based OPD as a flexible, black-box-compatible alternative to the prevailing logit-based OPD, offering a simple yet strong baseline for scalable distillation across proprietary and open-source LLMs. Code is available at https://github.com/Peregrine123/ROPD_official.
Junfeng Fang, Zhepei Hong, Mao Zheng +7
National University of Singapore · University of Science and Technology of China · Tencent
On-policy self-distillation (OPSD), where a single model acts as both student and teacher with different contexts, has shown promise in verifiable domains like math, where hard privileged information (PI) in the form of ground-truth answers structurally constrains valid continuations. We extend OPSD to open-ended generation using soft PI in the form of rubrics that guide preferences but admit many valid responses. Rubrics have served as scalar rewards for reinforcement learning (RL); we show that they provide substantially richer signal as dense PI for distillation, and contrary to intuition, soft rubric PI provides a larger and more effective training signal on student roll-outs than hard reference completion PI in this regime. A reference completion is one point in a set of valid responses, so distilling towards it over-constrains the student, while rubrics specify the preference structure shared across the set of valid responses. We show the effectiveness of using rubrics as PI for open-ended generation across Qwen and Llama model families and show that it outperforms rubric-as-reward (RaR) RL using HealthBench, a benchmark that grades open-ended health responses against physician-created rubrics, providing dense token-level supervision for open-ended tasks; RuPI beats RaR RL by up to +0.10 absolute score and, under matched recipe and KL direction, beats reference-PI by +0.034 to +0.079 absolute score across three models. We further show that these findings generalize to training on the RubricHub Science corpus and evaluating on ResearchQA: soft rubric PI outperforms both reference-PI distillation and RaR RL (66.6% vs. 64.2% and 57.6%).
Rubrics have emerged as an alternative to RLVR in open-ended domains where a single ground-truth final answer is not available. Existing rubric-based training methods rely on an LLM verifier that scores each rollout against rubrics. This introduces substantial training-time overhead, exposes optimization to verifier-specific biases, and reduces rubric feedback to a sparse end-of-trajectory signal. We propose Rubric-Guided Self-Distillation (RGSD), a verifier-free training method in which the base policy, conditioned on the rubric, serves as the teacher for the unconditioned student. RGSD distills the rubric-conditioned teacher distribution into the student token-by-token, replacing sparse trajectory-level rewards with dense per-token learning signals and removing the LLM judge from the training loop entirely. Across Qwen-2.5 (3B, 7B) and Qwen3-Thinking (4B, 8B) models on medical and science domains, RGSD achieves rubric satisfaction comparable to judge-based GRPO while using one on-policy rollout per prompt and no training-time verifier calls. Ablations show that raw rubrics provide a stronger teacher enrichment signal than self-generated reference responses, while a stronger GRPO judge can outperform RGSD in some settings, positioning RGSD as a complementary verifier-free alternative when verifier cost or reliability is the bottleneck.
MohammadHossein Rezaei, Anas Mahmoud, Zihao Wang +6