stat.MLNov 18, 2026

How Reward Gain Imbalance Shapes Ranking: Rethinking Rubric-Based Reward Aggregation

Authors: Zhengyuan Fan, Zhonghua Wu, Tianyu Cheng, Qun Chen

Organizations: School of Computer Science, Northwestern Polytechnical University

Abstract

Rubric-based reinforcement learning constructs rewards for aligning large language models with human preferences, typically by equally aggregating continuous scores that a judge assigns to individual criteria. However, scoring heterogeneity across criteria can distort candidate rankings. Existing work has focused on improving judges, learning criterion weights, or refining aggregation, whereas the interaction between scoring heterogeneity and ranking remains insufficiently understood. We formalize this heterogeneity as reward gain imbalance: each criterion enters the aggregated reward with its own reward gain, the gap between its expected scores on satisfied and unsatisfied responses. Counterintuitively, our experiments show that removing this imbalance, even with ground-truth labels, lowers ranking accuracy. To explain this, we derive an exact decomposition of the reward difference between any two candidates into a base satisfaction signal, a reward gain imbalance term, and a residual term, showing that reward gain imbalance can attenuate or amplify the base signal, and finding that misrankings usually concentrate on pairs where it attenuates. Guided by this analysis, we propose the adaptive Dual-Center Clipping aggregation (DCC), which bounds each criterion's contribution to the reward difference instead of simply removing the imbalance. Using only the scores of the current candidate group, it determines a clipping interval that imposes a common upper bound on per-criterion score differences between candidates, limiting the excessive influence of extreme scores while preserving continuous judgment information....

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Judging the Judges: Confidence and Disagreement in Rubrics as Rewards

    Oct 15, 2026Michael Tian-Yue Liu, Anne Harrington, Trevor Darrell +1Rubric-Based RLReward Design for RL

  2. Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR

    May 19, 2026Utkarsh Tyagi, Xingang Guo, MohammadHossein Rezaei +5Reward ModelingRubric-Based RL

  3. Intra-Rubric Reward Divergence: Mitigating Rubric Hacking through Latent Dimension Recovery and Balancing

    Oct 19, 2026Jianqiang Xia, Han Xu, Dianxi ShiReward HackingRubric-Based RL