Rubric-based reinforcement learning (RL) extends RL with verifiable rewards to open-ended tasks by scoring each response against query-specific rubrics. Under group-relative optimization, however, a rubric that every sampled response fails (unreachable) or passes (saturated) gives all responses the same outcome and no learning signal; in Qwen3-4B medical training, 60.88% of query-rubric observations are inactive. We find that the two states lack different supervision: stating an unreachable rubric in the prompt raises its satisfaction from 1.13% to 91.25%, whereas responses that all pass a saturated rubric still differ in quality. We therefore propose RuACT (Rubrics as Adaptive Critics and Teachers), which assigns each inactive rubric the role that supplies what it lacks. Rubric as Teacher uses unreachable rubrics as privileged information for reward-gated on-policy self-distillation, and Rubric as Critic decomposes saturated rubrics into child rubrics that separate current responses while sharing the parent's weight. RuACT outperforms GRPO, rubric reweighting, and rubric-guided exploration on all four medical and scientific benchmarks with Qwen3-4B and achieves the best average with Qwen3-1.7B. On initially unreachable rubrics, unguided recovery rises from 35.02% to 42.86%, while child refinement improves reward-quality alignment on independent rollouts from 57.74% to 60.77%. Code is available at https://github.com/ZJU-REAL/RuACT.
Figures & tables
Figure 1: Inactive rubrics and RuACT. (a) Illustration of unreachable, informative, and saturated rubrics and their distribution through training. (b) How RuACT recovers inactive rubrics: reward-verified guidance helps elicit missing behaviors, while finer-grained criticism exposes details for further improvement.
Figure 2: Probes of inactive rubrics on frozen Qwen3-4B . (a) Fraction of initially unreachable rubrics satisfied by at least one of K independent ordinary samples; shading denotes 95% confidence intervals. (b) Fraction of initially unreachable and saturated pairs whose estimated quality variation Vq exceeds v .
Figure 3: Rubrics as adaptive critics and teachers. Rollout judgments identify unreachable ( p=0 ), informative ( 0<p<1 ), and saturated ( p=1 ) rubrics. As teachers, unreachable rubrics provide privileged guidance, with reward-gated forward-KL supervision on prefixes of the best ordinary rollout. As critics, saturated rubrics are expanded into discriminative children sharing the parent weight budget. Original and updated rubric rewards drive GRPO, while teacher supervision contributes an auxiliary objective.
Medicine
Science
Model
Group
Method
RaR-Med.
HealthBench
RaR-Sci.
ResearchQA
Avg.
Qwen3-1.7B
Vanilla
Base
38.57 ↑ 0.00
23.18 ↑ 0.00
52.56 ↑ 0.00
50.51 ↑ 0.00
41.21 ↑ 0.00
Baselines
GRPO
43.26 ↑ 4.69
26.45 ↑ 3.27
55.67 ↑ 3.11
54.79 ↑ 4.28
45.04 ↑ 3.84
Reweight
42.60 ↑ 4.03
25.61 ↑ 2.43
56.77 ↑ 4.21
56.11 ↑ 5.60
45.27 ↑ 4.07
RuscaRL
43.39 ↑ 4.82
23.95 ↑ 0.77
56.41 ↑ 3.85
59.37 ↑ 8.86
45.78 ↑ 4.58
OPSD ∗
26.30
38.50
39.60
38.40
35.70
Table 1: Main results across medical and scientific reasoning benchmarks. Scores are reported in % on a normalized 0–100 scale; Avg. is the unweighted mean across the four benchmarks. Arrows indicate absolute score changes relative to the corresponding vanilla checkpoint. Bold and underlined values denote the best and second-best results within each model scale, respectively. OPSD ∗ results are reported from CriPO ( Xia et al., 2026 ) using GPT-4o-mini as the judge.
Figure 4: Recovery of initially unreachable rubrics (Qwen3-4B, RaR-Medicine). A rubric is recovered if any of eight unguided rollouts satisfies it. (a) During training. (b) Grouped by whether rubric guidance elicits the behavior from the base model.
Scoring rule
Quality alignment (%) ↑
Reward coverage (%) ↑
parent rubric-only
57.74
65.92
w/ Decomposed
60.77
88.85
w/ Shuffled Decomposed
53.62
90.44
Table 2: Quality alignment of refined rewards. Reward-quality alignment and coverage.
Figure 5: Reward discrimination on independent rollouts. Within-group reward STD with and without child-rubric refinement for (a) RuACT-Critic and (b) RuACT on Qwen3-4B RaR-Medicine. Rollouts are sampled independently of those used for child selection.
Method
RaR-Med.
HealthBench
Avg.
Base
50.38 ↑ 0.00
35.45 ↑ 0.00
42.92 ↑ 0.00
GRPO
53.55 ↑ 3.17
37.62 ↑ 2.17
45.59 ↑ 2.67
DAPO-style
56.29 ↑ 5.91
39.27 ↑ 3.82
47.78 ↑ 4.87
GRPO+OPSD
55.59 ↑ 5.21
38.63 ↑ 3.18
47.11 ↑ 4.20
OPSD ∗
42.00
42.10
42.05
RuACT
56.34 ↑ 5.96
40.12 ↑ 4.67
48.23 ↑ 5.32
Table 3: Comparison with alternative optimization strategies on Qwen3-4B. Scores are reported in %; arrows indicate gains over Base. DAPO-style adapts dynamic sampling to rubric-based GRPO ( Yu et al., 2026 ) .
Figure 6: Policy-level gains across inactive rubrics. The x-axis measures unreachable-rate reduction over 21,500 test query–rubric pairs. The y-axis measures child-rubric satisfaction gains on 1,954 fixed Base-saturated parents with shared children. Both axes show percentage-point improvements relative to GRPO.
MT-Bench
FollowBench
Model
Method
R1
R2
HSR
SSR
Qwen3-1.7B
Base
7.13
5.90
43.52
56.44
RuACT
7.66
6.03
45.37
57.69
Qwen3-4B
Base
8.31
7.60
58.50
67.28
RuACT
8.64
7.88
59.89
67.75
Table 4: General capability preservation on OOD benchmarks. MT-Bench scores (0–10) and FollowBench success rates (%) are evaluated using Qwen3-Max as the judge.
Variant
RaR-Med.
HealthBench
Avg.
RuACT
56.34
40.12
48.23
Rubric as Teacher
RuACT-Teacher
55.15
38.57
46.86
w/o Gate
55.18
37.45
46.32
Rubric as Critic
RuACT-Critic
55.50
38.77
47.14
Table 5: Ablation study of RuACT on Qwen3-4B. Scores are reported in % on RaR-Medicine and HealthBench, with Avg. denoting their unweighted mean.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Train
Evaluation
RaR-Medicine
17,926
2,242
RaR-Science
18,302
2,292
HealthBench
–
500
ResearchQA
–
500
Appendix
Table 6: Dataset sizes used in our experiments. Counts refer to examples in the prepared splits. HealthBench and ResearchQA are evaluation subsets.
Hyperparameter
Value
Training steps
200
Queries per batch
64
Responses per query, G
8
PPO mini-batch size
32
PPO epochs per batch
1
Learning rate
10−6
Appendix
Table 7: Policy-training settings. Sequence limits are for the standard RaR-Medicine configuration.
Hyperparameter
Setting
Teacher
Target rubrics per query
3
Guided responses per selected query
1
Effective β (1.7B / 4B)
0.10 / 0.05
Coefficient warmup steps
50
Critic
Appendix
Table 8: RuACT hyperparameters. Critic settings describe the Qwen3-4B RaR-Medicine configuration. Selection and retention counts are maxima.
Figure 7: Rubric inactivity persists and co-occurs throughout GRPO training. (a) Query-group composition by unreachable (U) and saturated (S) rubric occurrence across five 40-step phases. (b) Mean number of U or S rubrics per affected group, conditional on at least one occurrence. Statistics use the original rubrics and eight rollouts per query group from Qwen3-4B training on RaR-Medicine.
Figure 8: Training reward on Qwen3-4B RaR-Medicine. Reward magnitudes are not directly comparable across different scoring rules.
Figure 9: Policy entropy during training. Comparison over 200 training steps.
Figure 10: Response length during training. Comparison over 200 training steps.
Medicine
Science
Model
Group
Approach
RaR-Med.
HealthBench
RaR-Sci.
ResearchQA
Avg.
Qwen3-1.7B
Vanilla
Base
41.34
20.88
49.92
48.13
40.07
Baselines
GRPO
46.08
24.80
53.06
52.71
44.16
Reweight
45.60
24.50
53.44
53.47
44.25
RuscaRL
46.51
21.29
53.13
58.44
44.84
Ours
RuACT-Teacher
46.44
23.69
54.78
55.71
45.15
Appendix
Table 9: Robustness to evaluation judges. Results on medical and scientific reasoning benchmarks re-evaluated using Qwen3.7-Max. Avg. denotes the unweighted mean across four benchmarks. Bold and underlined values indicate the best and second-best results within each model scale.
ID
Rubric-relevant response excerpts
P
c1
c2
c3
R1
“ External Oblique : Contracts to pull the ribcage downward and inward (indirectly aiding expiration). … Internal Oblique : Also helps in pulling the ribcage inward. … Transverse Abdominis : Contracts to increase abdominal pressure, which helps push the diaphragm upward.”
1
0
1
1
R2
“ Transverse Abdominis : Contracts to compress the abdominal cavity, helping push the diaphragm upward. … Oblique Muscles (Internal and External Obliques): Contract to stabilize the abdominal wall and assist in forced expiration.”
1
0
0
0
R3
“ External and Internal Obliques : Contract to compress the abdominal cavity, further helping to expel air from the lungs. … Transverse Abdominis : Contracts to increase intra-abdominal pressure, assisting in expiration.”
1
1
1
1
R4
“ External Oblique : Contracts to pull the ribcage inward and compress the abdominal cavity. … Internal Oblique : Similarly contributes to abdominal compression. … Transversus Abdominis : Contracts to increase abdominal pressure, helping push the diaphragm up.”
1
0
1
1
R5
“ External Oblique and Internal Oblique : Contract to compress the abdominal cavity, helping push the diaphragm upward. … Transversus Abdominis : Contracts to increase abdominal pressure, assisting in expelling air.”
1
0
0
0
R6
“ External and Internal Oblique Muscles : Contract to compress the abdominal cavity, further increasing intra-abdominal pressure and aiding in expiration. … Transversus Abdominis : Contracts to tighten the abdominal wall, assisting in expelling air from the lungs.”
1
1
1
1
Appendix
Table 10: Critic-side case study on RaR-Medicine. A saturated parent is satisfied by all eight responses, whereas its finer child rubrics yield non-uniform judgments on the same rollout group.
Rubric-based reinforcement learning, in which an LLM judge scores responses against prompt-specific criteria, has become a standard way to post-train language models on tasks without verifiable answers. The rubric, however, is only a proxy for response quality, and a policy trained against it long enough learns to exploit the difference. We train models with GRPO on medical and science rubrics and grade their responses on out-of-distribution (OOD) benchmarks throughout training with two judges: the training judge and a stronger OOD judge. The two scores first rise together and then diverge. The policy is learning to satisfy the training judge without writing better responses, which is a form of reward hacking. To mitigate it, we propose Rubric Dropout, which applies dropout to rubric criteria: at each training step, a random subset of each rubric's criteria is removed from the reward, so the rewarded subset of criteria is resampled at each step. All responses sampled for the same prompt are scored on the same subset, so their rewards remain comparable, and evaluation always uses the full rubric. We also introduce Block Dropout, which drops semantically similar criteria together so that a near duplicate of a dropped criterion cannot keep rewarding the same behavior. Across both benchmarks and models from 1.7B to 8B, dropout improves the gold judge's score on OOD benchmarks by up to 3 percent on HealthBench-Hard and 7 percent on ResearchQA, and it reduces both the gap between the two judges and over-crediting. It keeps in-domain reward close to the baseline, and dropout rates of 30-50 percent work best. Block Dropout improves the gold judge's OOD score further at the same dropout rate. Changing which criteria are rewarded at each step is a simple, inexpensive way to make rubric rewards harder to exploit.
Minglai Yang, Xinyu Guo, Zhengliang Shi +8
Scale AI · University of Arizona · Carnegie Mellon University +2
Rubric-based reinforcement learning extends reward-driven optimization to open-ended tasks by assigning partial credit to individual response requirements. However, rubric judges can assign a high criterion score even when the information or action it requires is absent from the response, a failure mode we term Vacuous Credit. Such awards persist after the required information is removed and can reverse the sign of a response's GRPO advantage. To address this problem, we introduce MetaRubric, which alternates evidence-aware policy optimization with response-guided rubric adaptation. We construct counterfactual counterparts by changing one task-relevant fact in each prompt. During policy optimization, credit is assigned only when the response contains sufficient evidence to satisfy the required rubric criterion. After each policy-optimization stage, current policy responses guide revisions to original and counterfactual criteria while preserving the meaning of the original prompt's initial rubric as interpreted under each prompt's facts. We also adapt criterion weights at stage boundaries to better address observed policy errors. Across multiple backbones, MetaRubric improves PubMedQA accuracy by 6.00--20.40 percentage points over static-judge GRPO, with further gains on HealthBench-Hard and two multimodal medical benchmarks.
Reinforcement learning with verifiable rewards has made post-training highly effective when correctness can be checked automatically. However, many important model behaviors require satisfying several qualitative criteria at once. Rubric-based rewards address this setting by grading prompt-specific criteria and aggregating them into a scalar reward. Yet standard static aggregations conflate a criterion's human-assigned importance with its current usefulness as an optimization signal. We show that this assumption breaks down in rubric RL: many important criteria are already saturated or currently unreachable, while criteria that distinguish rollouts are not necessarily those with the largest human weights. We introduce POW3R, a policy-aware rubric reward framework that preserves human weights and category balance as the rubric objective while adapting criterion-level reward weights during training. POW3R uses rollout-level contrast to emphasize criteria that currently separate the policy's outputs, making the GRPO reward more informative without changing the underlying evaluation target. Across three base policies on two datasets spanning multimodal and text-only settings, POW3R wins 24 of 30 base-policy/metric comparisons, improving both mean rubric reward and strict completion (the fraction of prompts whose response satisfies every required rubric criterion) over vanilla GRPO with rubric rewards, and reaches the same plateau in 2.5--4× fewer training steps. Rubric rewards should therefore distinguish what should matter in the final answer from what can teach the current policy.