Reinforcement Learning from Human Feedback (RLHF) is a powerful technique for aligning large language models (LLMs) with human preference. However, it often suffers from the reward hacking issue, where policy optimization improves the proxy reward model while actually degrading performance with respect to the true human preference, due to the imperfection of the proxy. To address this, we propose Reward Model Boosting (RMB), a novel approach that enhances the robustness and reliability of the reward signal for RLHF. RMB first trains a set of reward models with a diversity-promoting regularizer. This encourages each model to learn complementary aspects of the reward landscape. Then, RMB learns a lightweight aggregator in the principle of boosting to aggregate the outputs of the diverse reward models into a more accurate and robust reward signal. Our extensive experiments demonstrate that RMB significantly improves reward accuracy on both in-distribution and out-of-distribution datasets, substantially mitigating the reward hacking issue and ultimately improving RLHF performance.
Figures & tables
Approach
Unified-Feedback
HHH-Align
MT-Bench
RewardBench
Avg.
Chat
Chat-Hard
Safety
Reasoning
Frozen RM
63.9
68.6
68.2
x
x
x
x
x
Baseline RM
68.8
70.3
69.1
64.5
95.8
37.3
59.9
64.8
Label Smooth
68.5
68.8
71.9
61.1
91.6
39.0
53.8
60.2
Margin Loss
69.6
69.8
71.0
66.4
97.2
37.5
56.8
72.7
GRM
71.5
78.7
73.0
66.8
94.1
41.9
69.5
61.5
Table 1: Preference prediction performance on all approaches learned using 40K data mode of Unified-Feedback. Results of competing approaches are directly sourced from Yang et al. (2024) . N means the number of reward models used for ensembling or boosting.
Approach
Opponent
Pairwise Preference Comparison (in %)
Pointwise Preference Comparison (in %)
Win ↑
Tie
Lose ↓
Δ
Win ↑
Tie
Lose ↓
Δ
RMB ( N=3 )
Baseline RM
55.97
3.21
40.82
↑ 15.15
44.73
23.97
31.29
↑ 13.44
Avg. Ensemble ( N=3 )
57.87
4.31
37.81
↑ 20.06
40.22
34.50
25.28
↑ 14.94
GRM
54.76
3.01
42.22
↑ 12.54
43.33
30.39
26.28
↑ 7.05
Table 2: LLM-as-judge of policies learned via RLHF.
Figure 1: Results of Best-of-N sampling experiments, where both Avg Ensemble and our RMB use three individual reward models. Figures 1(a) and 1(c) intend to show the change of the proxy reward score, as an increasing proxy reward with a decreasing/plateaued gold reward indicates reward hacking. Notice that the scores of different proxy reward models are not directly comparable. Figures 1(b) and 1(d) show the gold reward scores achieved, a higher score indicating a better reward model.
Figure 2: PPO training with reward models (RMs). Ensemble and RMB both have three RMs to aggregate.
Unified-Feedback (In-Distribution)
RewardBench (Out-of-Distribution)
Random Seeds
Individual RMs
70.71
71.34
70.52
69.85
70.30
68.45
67.66
69.97
67.11
67.24
Avg Ensemble
70.80
68.92
Our RMB
71.04
68.70
HSIC
Individual RMs
73.40
70.73
69.29
68.05
73.84
70.08
69.95
67.58
66.97
71.62
Avg Ensemble
72.04
69.40
Our RMB
74.58
71.65
Table 3: Ensemble vs Boosting with different diversity strategies. We report the performance of individual RMs (index from 0 to 4).
Figure 3: Correlation heat map among individual Reward Models
HSIC Diversity
Random Seeds Diversity
Model
Total Gain
Model
Total Gain
RM 0
29.75%
RM 1
70.13%
RM 4
27.39%
RM 0
12.39%
RM 1
26.78%
RM 4
10.99%
RM 2
8.55%
RM 2
5.82%
RM 3
7.53%
RM 3
0.66%
Table 4: Total Gain ratio of individual RMs.
Figure 4: Rank order accuracy.
Latency Type
Component/Model
Mean
p50
p90
p99
Breakdown (s/batch)
Base RMs
19.7915
19.7442
20.2839
20.3000
Avg Ensemble aggregator
5.92×10−5
5.67×10−5
6.09×10−5
1.398×10−4
RMB aggregator
1.7458×10−3
1.6997×10−3
1.9010×10−3
2.5694×10−3
End-to-End (s/batch)
Avg Ensemble
19.7915
19.7443
20.2840
20.3001
RMB
19.7932
19.7460
20.2857
20.3016
Table 5: Inference-efficiency summary.
Reward Model
Total Params.
Average
Chat
Chat-Hard
Safety
Reasoning
RMB (N=3)
6B
87.8
98.9
75.6
87.3
89.1
GPT-4o-2024-08-06
-
86.7
96.1
76.1
88.1
86.6
GPT-4-0125-preview
-
86.0
95.3
74.3
87.6
86.9
FsfairX-LLaMA3-RM
8B
84.4
99.4
65.1
86.8
86.4
Starling-RM
34B
82.7
96.9
57.2
88.2
88.5
Table 6: Results of different reward models in a budget-constrained scenario.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Basic Setups
Base models
Gemma-2b-it
Quantization for training
bf16
Fine-tuning strategy
LoRA
LoRA r
32
LoRA alpha
64
LoRA dropout
0.05
Appendix
Table 7: Key implementation details of our main experiments.
Reward model
Unified-Feedback
HHH-Align
MT-Bench
RewardBench
Avg.
Chat
Chat-Hard
Safety
Reasoning
Frozen RM
63.8
66.4
69.5
x
x
x
x
x
Baseline RM
72.1
73.4
71.2
68.2
95.5
38.0
73.8
65.3
Label Smooth
71.5
72.1
71.2
70.6
94.4
37.3
73.2
77.4
Margin Loss
72.0
75.0
72.6
70.2
95.8
38.4
73.9
72.5
GRM
73.2
79.8
73.4
70.8
97.8
42.1
77.9
65.2
Appendix
Table 8: Preference prediction performance on all Reward Models learned using 400K data mode of Unified-Feedback. Results of competing approaches are directly sourced from Yang et al. (2024) . N means the number of reward models used for ensembling or boosting.
Table 13
Approach
Opponent
Pairwise Preference Comparison (in %)
Pointwise Preference Comparison (in %)
Win ↑
Tie
Lose ↓
Δ
Win ↑
Tie
Lose ↓
Δ
RMB ( N=3 )
Baseline RM
59.54
5.42
35.04
↑ 24.50
47.28
20.19
32.53
↑ 14.75
Avg Ensemble ( N=3 )
53.15
7.93
38.92
↑ 14.23
39.40
30.91
29.69
↑ 9.71
GRM
57.38
9.37
33.25
↑ 24.13
41.46
27.44
31.10
↑ 10.36
Appendix
Table 11: LLM-as-judge (GPT-4o) of policies learned via RLHF.
RM
Reward Mean
Reward Std.
c(x) Mean
c(x) Std.
c(x) P05–P95
RM 0
2.6380
17.0647
-0.1391
17.0468
[-28.2891, 27.9354]
RM 1
0.6850
10.9031
0.0566
10.9953
[-17.9596, 18.3358]
RM 2
-0.0619
10.9822
0.0606
10.9897
[-17.8373, 17.7582]
Appendix
Table 12: Statistics of reward scores and sampled prompt-dependent constants.
Table 14: Sensitivity analysis of the λHSIC . We report the average performance of individual Reward Models, the average pairwise correlation between them, and the final RMB performance.
Regularizer
RM1
RM2
RM3
RM4
RM5
HSIC
73.40
70.73
69.29
68.05
73.84
DPP
60.58
64.93
55.20
62.74
57.19
NCL
56.02
59.45
52.87
57.40
55.16
Appendix
Table 15: Comparison of different diversity regularizers. We report the accuracy (%) of five individually trained reward models (RM1–RM5) under HSIC, DPP, and NCL regularization. HSIC consistently yields stronger RMs, while DPP and NCL significantly underperform.
Please act as an impartial judge and evaluate the quality of the responses provided by two
AI assistants to the user question displayed below. Your evaluation should consider
correctness and helpfulness. You will be given a reference answer, assistant A’s answer,
and assistant B’s answer. Your job is to evaluate which assistant’s answer is better.
Begin your evaluation by comparing both assistants’ answers with the reference answer.
Identify and correct any mistakes. Avoid any position biases and ensure that the order in
which the responses were presented does not influence your decision. Do not allow the
Appendix
Table 16: Prompt templates for LLM evaluation on text generated by RLHF optimized policy in § 3.2 from a pairwise view. This template originates from Zheng et al. (2023) ’s work, and is adapted according to Fan et al. (2026) ’s template to avoid ID and position bias, aiming to provide reliable LLM judges for pairwise comparison.
Please act as an impartial judge and evaluate the quality of the response provided by an AI assistant
to the user’s question shown below. Your evaluation should consider factors such as helpfulness,
relevance, accuracy, depth, creativity, and level of detail in the response.
Begin your evaluation with a short explanation. Be as objective as possible.
After providing your explanation, please rate the response on a scale of 1 to 10
by strictly following this format: "[[rating]]", for example: "Rating: [[5]]".
[Question]
Appendix
Table 17: Prompt templates for LLM evaluation on text generated by RLHF optimized policy in § 3.2 from a pointwise view. This template originates from Zheng et al. (2023) ’s work, aiming to provide LLM judges for single-answer grading.
Reinforcement learning from human feedback (RLHF) aligns large language models by training reward models on preference data and optimizing policies to maximize predicted rewards. However, this pipeline faces two fundamental challenges: (1) reward models cannot signal when their predictions are unreliable, since they usually act as deterministic point estimators; and (2) modern group-based policy optimization can amplify unreliable reward signals, as exemplified by GRPO's uniform treatment of rewards during advantage computation. As policies explore increasingly diverse responses, these two limitations create a critical vulnerability: unreliable reward estimates may be granted disproportionate influence, triggering severe reward hacking. We propose Uncertainty-Aware Reward Modeling (UARM), which equips reward models with calibrated uncertainty via quantile-based conformal prediction and reweights GRPO advantages through heteroscedastic variance decomposition. Experiments across HelpSteer, UltraFeedback, and PKU-SafeRLHF demonstrate that UARM significantly improves reward model calibration, reduces reward hacking, and enhances downstream alignment quality compared to standard GRPO and uncertainty-agnostic baselines.
Licheng Pan, Haocheng Yang, Haoxuan Li +7
Zhejiang University · Xiaohongshu Inc · National University of Singapore +1
Reinforcement Learning from Human Feedback (RLHF) enables powerful LLM alignment but can introduce reward hacking - models exploit spurious correlations in proxy rewards without genuine alignment. Compounding this, the objectives internalized during RLHF remain opaque, making hacking behaviors difficult to detect or correct. We introduce IR3 (Interpretable Reward Reconstruction and Rectification), a framework that reverse-engineers, interprets, and surgically repairs the implicit objectives driving RLHF-tuned models. We propose Contrastive Inverse Reinforcement Learning (C-IRL), which reconstructs the implicit reward function by contrasting paired responses from post-alignment and baseline policies to explain behavioral shifts during RLHF. We then decompose the reconstructed reward via sparse autoencoders into interpretable features, enabling identification of hacking signatures through contribution analysis. Finally, we propose mitigation strategies - clean reward optimization, adversarial shaping, constrained optimization, and feature-guided distillation - that target problematic features while preserving beneficial alignment. Experiments across multiple reward model configurations show that IR3 achieves 0.89 correlation with ground-truth rewards, identifies hacking features with over 90% precision, and significantly reduces hacking behaviors while maintaining capabilities within 3% of the original model.
Reward models (RMs) are critical to reinforcement learning from human feedback (RLHF). However, policy models can exploit biases or spurious correlations in dense RMs, causing reward hacking: RM scores rise while alignment with human preferences deteriorates, especially under distribution shift. We propose UMM-RM (Upcycle-and-Merge MoE Reward Model), which copies the feed-forward layers of an SFT checkpoint to initialize shared and routed experts, then trains the model on preference data. The always-active shared expert captures instruction-agnostic preference signals, while routed experts model fine-grained preferences across instructions or tasks. The experts are then merged into a single dense RM using learnable weights, preserving the robustness and exploitation resistance of expert diversity without MoE or ensemble inference overhead. Across multiple base models and preference datasets, UMM-RM improves preference accuracy, reduces reward hacking during PPO training, and achieves more stable preference alignment than standard dense RMs.