Rubric-based reinforcement learning constructs rewards for aligning large language models with human preferences, typically by equally aggregating continuous scores that a judge assigns to individual criteria. However, scoring heterogeneity across criteria can distort candidate rankings. Existing work has focused on improving judges, learning criterion weights, or refining aggregation, whereas the interaction between scoring heterogeneity and ranking remains insufficiently understood. We formalize this heterogeneity as reward gain imbalance: each criterion enters the aggregated reward with its own reward gain, the gap between its expected scores on satisfied and unsatisfied responses. Counterintuitively, our experiments show that removing this imbalance, even with ground-truth labels, lowers ranking accuracy. To explain this, we derive an exact decomposition of the reward difference between any two candidates into a base satisfaction signal, a reward gain imbalance term, and a residual term, showing that reward gain imbalance can attenuate or amplify the base signal, and finding that misrankings usually concentrate on pairs where it attenuates. Guided by this analysis, we propose the adaptive Dual-Center Clipping aggregation (DCC), which bounds each criterion's contribution to the reward difference instead of simply removing the imbalance. Using only the scores of the current candidate group, it determines a clipping interval that imposes a common upper bound on per-criterion score differences between candidates, limiting the excessive influence of extreme scores while preserving continuous judgment information....
Figures & tables
Score Method
PointRubric
RubricHub-6
Raw score D
75.30 → 70.26
14.87 → 12.39
Binary score 1[D≥0]
31.41 → 30.70
2.30 → 3.01
sigmoid(D)
74.58 → 71.70
13.45 → 12.57
Table 1: The performance of Mistral-7B in terms of Exact (%), before → after giving all criteria the same reward gain with ground-truth labels.
Scoring Method
Mean
Variance
Raw score D
21.9545
86.2175
Binary score 1[D≥0]
0.7068
0.1363
Probability score sigmoid(D)
0.7071
0.1255
Table 2: Means and variances of Δk under different scoring methods.
Figure 1: Visualization of score mappings of sigmoid and DCC on two real candidate groups: dots denote scores, and the band denotes DCC’s clipping interval; DCC is rescaled to [0,1] within the group, which leaves rankings unchanged. In (a) for Mistral-7B, the sigmoid maps 23 of 24 scores to nearly 0 or 1; in (b) for Qwen3.6-27B, most scores are negative; the sigmoid maps 18 of 24 below 0.3, while DCC sets its interval from this group’s own scores, so the scores stay spread out.
PointRubric (train)
RubricHub-6
Method
Exact
Pairwise
Top-1
Exact
Pairwise
Top-1
Mistral-7B
Original D
75.30
93.49
87.05
14.87
86.28
72.21
Binary
31.41
74.11
68.07
2.30
73.81
54.14
Probability
74.58
93.45
89.93
13.45
85.97
71.68
RLVR 2
65.95
90.79
85.17
13.81
85.06
68.50
Table 3: Ranking accuracy (%) comparison of different aggregation methods on the PointRubric train split and RubricHub-6, with equal criterion weights.
Mistral-7B
Qwen3.6-27B
Method
Equal
Official
Learned
Equal
Official
Learned
Original D
75.82
75.82
76.20
81.57
80.81
81.38
Binary
30.71
32.05
32.82
56.81
58.54
62.00
Probability
74.47
74.47
75.05
81.38
80.42
81.96
RLVR 2
70.63
70.25
72.17
78.69
79.08
79.08
DCC (ours)
80.23
79.85
81.65
81.00
81.38
82.92
Table 4: Performance comparison in terms of Exact (%) on the PointRubric test split under different criterion weights: the learned weights use the judge scores and criterion types as input, and the official weights are provided by the benchmark.
Method
Overall
English
Chinese
Baseline
43.32
47.12
38.57
Original D
45.83 ± 0.39
48.42 ± 0.46
42.59 ± 0.34
Probability
48.53 ± 0.42
50.55 ± 0.45
46.00 ± 0.39
DCC (ours)
52.24 ± 0.26
54.39 ± 0.24
49.55 ± 0.30
Table 5: WritingBench scores of the base model and the fine-tuned models: the baseline is not fine-tuned and is evaluated once, and fine-tuned results are the mean ± standard deviation over three training seeds.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Mistral-7B
Qwen3.6-27B
Variant
PR
RH-6
PR
RH-6
L=0.5
49.40
12.92
83.69
42.30
L=1 (DCC)
77.22
16.11
85.37
42.12
L=2
75.30
15.04
85.37
37.17
Original D
75.30
14.87
84.89
36.99
max(D,0)
74.58
14.69
67.87
19.65
Appendix
Table 6: Evaluation results of the ablation study in terms of Exact (%), with bold marking the best result within each part. Top: the clipping interval is scaled by a factor L around its midpoint, and L=1 is DCC. Bottom: other forms of clipping, where c− and c+ denote the lower and upper centers of DCC. PR and RH-6 denote the PointRubric train split and RubricHub-6.
Judge
Data
Score
Before
After
Change
Mistral-7B
PR
Raw D
75.30
70.26
−5.04
Binary
31.41
30.70
−0.72
sigmoid(D)
74.58
71.70
−2.88
RH-6
Raw D
14.87
12.39
−2.48
Binary
2.30
3.01
+0.71
sigmoid(D)
13.45
12.57
−0.88
Appendix
Table 7: Exact (%) before and after g(⋅) . PR denotes the PointRubric train split and RH-6 denotes RubricHub-6.
Parameter
Value
Policy
Mistral-7B-Instruct-v0.3
Judge
Qwen3.6-27B
Optimizer
AdamW
Learning rate
10−6
Weight decay
0
Gradient clipping
1.0
Appendix
Table 8: GRPO training configuration.
Mistral-7B
Qwen3.6-27B
Quantity
Method
PR
RH-6
PR
RH-6
Saturated scores
Prob.
94.09
81.89
66.18
56.65
DCC
51.68
50.61
52.27
50.75
Reversed pairs
Prob.
0.00
2.82
12.77
15.19
DCC
19.15
20.16
36.17
36.29
Appendix
Table 9: Score saturation and accuracy (%) on reversed pairs for probability aggregation (Prob.) and DCC. Saturated scores have sigmoid(D) above 0.98 or below 0.02 for Prob., and lie outside the open clipping interval for DCC. In reversed pairs, the better candidate receives fewer positive judgments. PR and RH-6 denote PointRubric and RubricHub-6.
Mistral-7B
Qwen3.6-27B
Method
PR
RH-6
PR
RH-6
T=0.25
47.96
11.86
84.65
34.69
T=0.5
73.14
13.63
84.65
34.69
T=1
74.58
13.45
85.13
36.11
T=4
74.58
15.58
85.13
36.46
T=8
75.06
15.58
84.41
36.99
Appendix
Table 10: Exact (%) of probability aggregation with temperature T on the PointRubric train split (PR) and RubricHub-6 (RH-6). DCC is shown for reference.
PointRubric (train)
RubricHub-6
Width
Exact
Pairwise
Top-1
Exact
Pairwise
Top-1
Mistral-7B
L=0.5
49.40
84.69
78.98
12.92
85.10
70.34
L=1 (DCC)
77.22
94.36
90.17
16.11
86.34
73.63
L=2
75.30
93.54
87.05
15.04
86.22
72.21
Qwen3.6-27B
Appendix
Table 11: Ranking accuracy (%) when the clipping interval is scaled by a factor L around its midpoint, on the PointRubric train split and RubricHub-6 with equal criterion weights. L=1 is DCC. For each judge, the best result in each column is in bold.
Judge
Data
Pairs
Orig.
DCC
Fix / Brk
Gij>0
Mistral-7B
PR
598
99.00
99.16
1 / 0
Mistral-7B
RH-6
980
97.24
97.55
5 / 2
Qwen3.6-27B
PR
626
98.08
97.92
4 / 5
Qwen3.6-27B
RH-6
976
98.26
98.46
6 / 4
Total
3,180
16 / 11
Appendix
Table 12: Accuracy (%) of the original scores (Orig.) and DCC on required pairs, split by the sign of Gij computed from the original scores. Fix and Brk count the pairs that DCC ranks correctly and the original scores do not, and vice versa. PR denotes the PointRubric train split and RH-6 denotes RubricHub-6.