Rubric-based reinforcement learning constructs rewards for aligning large language models with human preferences, typically by equally aggregating continuous scores that a judge assigns to individual criteria. However, scoring heterogeneity across criteria can distort candidate rankings. Existing work has focused on improving judges, learning criterion weights, or refining aggregation, whereas the interaction between scoring heterogeneity and ranking remains insufficiently understood. We formalize this heterogeneity as reward gain imbalance: each criterion enters the aggregated reward with its own reward gain, the gap between its expected scores on satisfied and unsatisfied responses. Counterintuitively, our experiments show that removing this imbalance, even with ground-truth labels, lowers ranking accuracy. To explain this, we derive an exact decomposition of the reward difference between any two candidates into a base satisfaction signal, a reward gain imbalance term, and a residual term, showing that reward gain imbalance can attenuate or amplify the base signal, and finding that misrankings usually concentrate on pairs where it attenuates. Guided by this analysis, we propose the adaptive Dual-Center Clipping aggregation (DCC), which bounds each criterion's contribution to the reward difference instead of simply removing the imbalance. Using only the scores of the current candidate group, it determines a clipping interval that imposes a common upper bound on per-criterion score differences between candidates, limiting the excessive influence of extreme scores while preserving continuous judgment information....
Figures & tables
Score Method
PointRubric
RubricHub-6
Raw score D
75.30 → 70.26
14.87 → 12.39
Binary score 1[D≥0]
31.41 → 30.70
2.30 → 3.01
sigmoid(D)
74.58 → 71.70
13.45 → 12.57
Table 1: The performance of Mistral-7B in terms of Exact (%), before → after giving all criteria the same reward gain with ground-truth labels.
Scoring Method
Mean
Variance
Raw score D
21.9545
86.2175
Binary score 1[D≥0]
0.7068
0.1363
Probability score sigmoid(D)
0.7071
0.1255
Table 2: Means and variances of Δk under different scoring methods.
Figure 1: Visualization of score mappings of sigmoid and DCC on two real candidate groups: dots denote scores, and the band denotes DCC’s clipping interval; DCC is rescaled to [0,1] within the group, which leaves rankings unchanged. In (a) for Mistral-7B, the sigmoid maps 23 of 24 scores to nearly 0 or 1; in (b) for Qwen3.6-27B, most scores are negative; the sigmoid maps 18 of 24 below 0.3, while DCC sets its interval from this group’s own scores, so the scores stay spread out.
PointRubric (train)
RubricHub-6
Method
Exact
Pairwise
Top-1
Exact
Pairwise
Top-1
Mistral-7B
Original D
75.30
93.49
87.05
14.87
86.28
72.21
Binary
31.41
74.11
68.07
2.30
73.81
54.14
Probability
74.58
93.45
89.93
13.45
85.97
71.68
RLVR 2
65.95
90.79
85.17
13.81
85.06
68.50
Table 3: Ranking accuracy (%) comparison of different aggregation methods on the PointRubric train split and RubricHub-6, with equal criterion weights.
Mistral-7B
Qwen3.6-27B
Method
Equal
Official
Learned
Equal
Official
Learned
Original D
75.82
75.82
76.20
81.57
80.81
81.38
Binary
30.71
32.05
32.82
56.81
58.54
62.00
Probability
74.47
74.47
75.05
81.38
80.42
81.96
RLVR 2
70.63
70.25
72.17
78.69
79.08
79.08
DCC (ours)
80.23
79.85
81.65
81.00
81.38
82.92
Table 4: Performance comparison in terms of Exact (%) on the PointRubric test split under different criterion weights: the learned weights use the judge scores and criterion types as input, and the official weights are provided by the benchmark.
Method
Overall
English
Chinese
Baseline
43.32
47.12
38.57
Original D
45.83 ± 0.39
48.42 ± 0.46
42.59 ± 0.34
Probability
48.53 ± 0.42
50.55 ± 0.45
46.00 ± 0.39
DCC (ours)
52.24 ± 0.26
54.39 ± 0.24
49.55 ± 0.30
Table 5: WritingBench scores of the base model and the fine-tuned models: the baseline is not fine-tuned and is evaluated once, and fine-tuned results are the mean ± standard deviation over three training seeds.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Mistral-7B
Qwen3.6-27B
Variant
PR
RH-6
PR
RH-6
L=0.5
49.40
12.92
83.69
42.30
L=1 (DCC)
77.22
16.11
85.37
42.12
L=2
75.30
15.04
85.37
37.17
Original D
75.30
14.87
84.89
36.99
max(D,0)
74.58
14.69
67.87
19.65
Appendix
Table 6: Evaluation results of the ablation study in terms of Exact (%), with bold marking the best result within each part. Top: the clipping interval is scaled by a factor L around its midpoint, and L=1 is DCC. Bottom: other forms of clipping, where c− and c+ denote the lower and upper centers of DCC. PR and RH-6 denote the PointRubric train split and RubricHub-6.
Judge
Data
Score
Before
After
Change
Mistral-7B
PR
Raw D
75.30
70.26
−5.04
Binary
31.41
30.70
−0.72
sigmoid(D)
74.58
71.70
−2.88
RH-6
Raw D
14.87
12.39
−2.48
Binary
2.30
3.01
+0.71
sigmoid(D)
13.45
12.57
−0.88
Appendix
Table 7: Exact (%) before and after g(⋅) . PR denotes the PointRubric train split and RH-6 denotes RubricHub-6.
Parameter
Value
Policy
Mistral-7B-Instruct-v0.3
Judge
Qwen3.6-27B
Optimizer
AdamW
Learning rate
10−6
Weight decay
0
Gradient clipping
1.0
Appendix
Table 8: GRPO training configuration.
Mistral-7B
Qwen3.6-27B
Quantity
Method
PR
RH-6
PR
RH-6
Saturated scores
Prob.
94.09
81.89
66.18
56.65
DCC
51.68
50.61
52.27
50.75
Reversed pairs
Prob.
0.00
2.82
12.77
15.19
DCC
19.15
20.16
36.17
36.29
Appendix
Table 9: Score saturation and accuracy (%) on reversed pairs for probability aggregation (Prob.) and DCC. Saturated scores have sigmoid(D) above 0.98 or below 0.02 for Prob., and lie outside the open clipping interval for DCC. In reversed pairs, the better candidate receives fewer positive judgments. PR and RH-6 denote PointRubric and RubricHub-6.
Mistral-7B
Qwen3.6-27B
Method
PR
RH-6
PR
RH-6
T=0.25
47.96
11.86
84.65
34.69
T=0.5
73.14
13.63
84.65
34.69
T=1
74.58
13.45
85.13
36.11
T=4
74.58
15.58
85.13
36.46
T=8
75.06
15.58
84.41
36.99
Appendix
Table 10: Exact (%) of probability aggregation with temperature T on the PointRubric train split (PR) and RubricHub-6 (RH-6). DCC is shown for reference.
PointRubric (train)
RubricHub-6
Width
Exact
Pairwise
Top-1
Exact
Pairwise
Top-1
Mistral-7B
L=0.5
49.40
84.69
78.98
12.92
85.10
70.34
L=1 (DCC)
77.22
94.36
90.17
16.11
86.34
73.63
L=2
75.30
93.54
87.05
15.04
86.22
72.21
Qwen3.6-27B
Appendix
Table 11: Ranking accuracy (%) when the clipping interval is scaled by a factor L around its midpoint, on the PointRubric train split and RubricHub-6 with equal criterion weights. L=1 is DCC. For each judge, the best result in each column is in bold.
Judge
Data
Pairs
Orig.
DCC
Fix / Brk
Gij>0
Mistral-7B
PR
598
99.00
99.16
1 / 0
Mistral-7B
RH-6
980
97.24
97.55
5 / 2
Qwen3.6-27B
PR
626
98.08
97.92
4 / 5
Qwen3.6-27B
RH-6
976
98.26
98.46
6 / 4
Total
3,180
16 / 11
Appendix
Table 12: Accuracy (%) of the original scores (Orig.) and DCC on required pairs, split by the sign of Gij computed from the original scores. Fix and Brk count the pairs that DCC ranks correctly and the original scores do not, and vice versa. PR denotes the PointRubric train split and RH-6 denotes RubricHub-6.
Rubrics as Rewards (RaR) extends reinforcement learning to non-verifiable domains by using an LLM judge to score responses against a structured rubric. Prior work has focused on improving the rubric or strengthening the single judge, with comparatively little attention to how different single or multiple judges might be used during RaR training. We first show that judge identity alone produces a substantial spread in scores when training Qwen3-4B-Base on RaR-Medicine and RaR-Science across five frontier and open-weight judges. Building on this, we propose RaRE (Rubrics as Rewards with Ensembles), a novel judge ensemble strategy for RaR with two mechanisms: a soft-Likert reward that uses judge logprobs to weight more confident judges more heavily, and a drop-on-disagreement mask that excludes high-disagreement rollouts from the GRPO update. We find that this design is critical: while a naive average of judge scores fails to outperform the best single judge, RaRE outperforms every single-judge RaR variant on both HealthBench and GPQA Diamond, with per-theme analysis showing that it inherits the complementary strengths of its constituent judges. Our findings establish judge ensembling as a compelling axis for improving RaR.
Michael Tian-Yue Liu, Anne Harrington, Trevor Darrell +1
Reinforcement learning with verifiable rewards has made post-training highly effective when correctness can be checked automatically. However, many important model behaviors require satisfying several qualitative criteria at once. Rubric-based rewards address this setting by grading prompt-specific criteria and aggregating them into a scalar reward. Yet standard static aggregations conflate a criterion's human-assigned importance with its current usefulness as an optimization signal. We show that this assumption breaks down in rubric RL: many important criteria are already saturated or currently unreachable, while criteria that distinguish rollouts are not necessarily those with the largest human weights. We introduce POW3R, a policy-aware rubric reward framework that preserves human weights and category balance as the rubric objective while adapting criterion-level reward weights during training. POW3R uses rollout-level contrast to emphasize criteria that currently separate the policy's outputs, making the GRPO reward more informative without changing the underlying evaluation target. Across three base policies on two datasets spanning multimodal and text-only settings, POW3R wins 24 of 30 base-policy/metric comparisons, improving both mean rubric reward and strict completion (the fraction of prompts whose response satisfies every required rubric criterion) over vanilla GRPO with rubric rewards, and reaches the same plateau in 2.5--4× fewer training steps. Rubric rewards should therefore distinguish what should matter in the final answer from what can teach the current policy.
Rubric hacking limits both performance and training robustness in rubric-based reinforcement learning. By decomposing rubrics into sub-dimensions, we reveal the Intra-Rubric Reward Divergence: as the total rubric score increases, certain sub-dimension scores systematically decrease, suggesting the model sacrifices specific quality aspects to optimize the aggregated objective. From the sub-dimension perspective, we observe two phenomena: (1) dimension imbalance, causing the policy to favor heavily covered dimensions; (2) dimension absence, leading to a systematic gap between rewards and true quality. To mitigate these issues, we propose Fair-Race Dimension Aggregation (FRDA), which has generations compete within each dimension separately to mitigate rubric imbalance, and Residual Semantic Dimension Completion (RSDC), which mines evaluation dimensions implicitly encoded in a holistic preference signal to complete rubric coverage. Experiments in chat, science, and medical domains indicate that FRDA mitigates rubric hacking with more stable improvements, while RSDC outperforms both single-reward training and naive multi-reward aggregation, yielding generally higher-quality responses.
Jianqiang Xia, Han Xu, Dianxi Shi
Shanghai Jiao Tong University · Baidu, Inc. · Intelligent Game and Decision Lab