Reward models are central to aligning large language models (LLMs) with human preferences. Yet most approaches rely on pointwise reward estimates that overlook the epistemic uncertainty in reward models arising from limited human feedback. Recent work suggests that quantifying this uncertainty can reduce the costs of human annotation via uncertainty-guided active learning and mitigate reward overoptimization in LLM post-training. However, uncertainty-aware reward models have so far been adopted without thorough comparison, leaving them poorly understood. This work introduces a unified framework, RewardUQ, to systematically evaluate uncertainty quantification for reward models. We compare common methods along standard metrics measuring accuracy and calibration, and we propose a new ranking strategy incorporating both dimensions for a simplified comparison. Our experimental results suggest that model size and initialization have the most meaningful impact on performance, and most prior work could have benefited from alternative design choices. To foster the development and evaluation of new methods and aid the deployment in downstream applications, we release our open-source framework as a Python package. Our code is available at https://github.com/lasgroup/rewarduq.
Figures & tables
Figure 1: Uncertainty-aware reward model architectures compared in this work. For a given prompt x and completion y , each model extracts an embedding z from a pretrained language model (LM) and predicts a reward r and uncertainty estimate u . Blue components indicate the parts responsible for estimating the uncertainty, while and denote trainable and frozen components, respectively.
Figure 2: Ranking scores on RewardBench across different UQ methods, training datasets, pretrained and finetuned models, and model sizes. The ranking score is defined in Equation 6 .
Figure 3: Calibration diagrams for Qwen3-0.6B (top) and Qwen3-4B (bottom) trained on UltraFeedback and evaluated on RewardBench. The predictions are well-calibrated when they agree with the actual probability per bin (i.e., on the diagonal), while the predicted upper bounds are well-calibrated when they consistently exceed the actual probability per bin (i.e., below the diagonal). The calibration metrics are defined in Equations 7 and 8 . The color intensity of each bar is proportional to the bin size. As described in Section 3.3 , the calibration diagrams for the upper and lower bounds are equivalent.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Background on our ranking score for different α . While the range is invariant of the win rate for α=0 , it has a linear dependence for α=1 as shown in Figure 4a . The inherent trade-off underlying the choice of α is shown in Figure 4b , which visualizes the weights in our ranking score in Equation 20 . For example, with α=0.2 , when the win rate increases from 0.6 to 0.8 , the confidence among true predictions is upweighted from 0.88 to 0.95 by a factor of ≈1.08 , while the confidence among false predictions is downweighted from 0.77 to 0.56 by a factor of ≈0.73 .
Figure 5: Our base metrics on RewardBench across different UQ methods, training datasets, pretrained and finetuned models, and model sizes. The accuracy metrics are defined in Equations 4 and 5 and the calibration metrics in Equations 7 and 8 .
Reinforcement learning from human feedback (RLHF) aligns large language models by training reward models on preference data and optimizing policies to maximize predicted rewards. However, this pipeline faces two fundamental challenges: (1) reward models cannot signal when their predictions are unreliable, since they usually act as deterministic point estimators; and (2) modern group-based policy optimization can amplify unreliable reward signals, as exemplified by GRPO's uniform treatment of rewards during advantage computation. As policies explore increasingly diverse responses, these two limitations create a critical vulnerability: unreliable reward estimates may be granted disproportionate influence, triggering severe reward hacking. We propose Uncertainty-Aware Reward Modeling (UARM), which equips reward models with calibrated uncertainty via quantile-based conformal prediction and reweights GRPO advantages through heteroscedastic variance decomposition. Experiments across HelpSteer, UltraFeedback, and PKU-SafeRLHF demonstrate that UARM significantly improves reward model calibration, reduces reward hacking, and enhances downstream alignment quality compared to standard GRPO and uncertainty-agnostic baselines.
Licheng Pan, Haocheng Yang, Haoxuan Li +7
Zhejiang University · Xiaohongshu Inc · National University of Singapore +1
Large Language Models (LLMs) have achieved remarkable success across diverse natural language tasks, yet the reward models employed for aligning LLMs often encounter challenges of reward hacking, where the approaches predominantly rely on directly mapping prompt-response pairs to scalar scores, which may inadvertently capture spurious correlations rather than authentic human preferences. In contrast, human evaluation employs a sophisticated process that initially weighs the relative importance of multiple high-dimensional objectives according to the prompt context, subsequently evaluating response quality through low-dimensional semantic features such as logical coherence and contextual appropriateness. Motivated by this consideration, we propose VRM, i.e., Variational Reward Modeling, a novel framework that explicitly models the evaluation process of human preference judgments by incorporating both high-dimensional objective weights and low-dimensional semantic features as latent variables, which are inferred through variational inference techniques. Additionally, we provide a theoretical analysis showing that VRM can achieve a tighter generalization error bound compared to the traditional reward model. Extensive experiments on benchmark datasets demonstrate that VRM significantly outperforms existing methods in capturing authentic human preferences.
Reward models are crucial for aligning large language models (LLMs) with human values and intentions. Existing approaches follow either Generative (GRMs) or Discriminative (DRMs) paradigms, yet both suffer from limitations: GRMs typically demand costly point-wise supervision, while DRMs produce uncalibrated relative scores that lack probabilistic interpretation. To address these challenges, we introduce a novel reward modeling paradigm: Probabilistic Reward Model (PRM). Instead of modeling reward as a deterministic scalar, our approach treats it as a random variable, learning a full probability distribution for the quality of each response. To make this paradigm practical, we present its closed-form, discrete realization: the Ordinal Probabilistic Reward Model (OPRM), which discretizes the quality score into a finite set of ordinal ratings. Building on OPRM, we propose a data-efficient training strategy called Region Flooding Tuning (RgFT). It enables rewards to better reflect absolute text quality by incorporating quality-level annotations, which guide the model to concentrate the probability mass within corresponding rating sub-regions. Experiments on various reward model benchmarks show that our method improves accuracy by \textbf{2.9%}\sim\textbf{7.4%} compared to prior reward models, demonstrating strong performance and data efficiency. Analysis of the score distribution provides evidence that our method captures not only relative rankings but also absolute quality.
Longze Chen, Qi Deng, Lu Wang +7
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Ritzz-AI +1