RMB: Reward Model Boosting Mitigates Reward Hacking

Jiabin Fan; Dezhi Ye; Yongchang Hao; Lili Mou

cs.LGSep 27, 2026

RMB: Reward Model Boosting Mitigates Reward Hacking

Authors: Jiabin Fan, Dezhi Ye, Yongchang Hao, Lili Mou

Organizations: Dept. Computing Science & Alberta Machine Intelligence Institute (Amii), University of Alberta · Tencent · Canada CIFAR AI Chair

Abstract

Reinforcement Learning from Human Feedback (RLHF) is a powerful technique for aligning large language models (LLMs) with human preference. However, it often suffers from the reward hacking issue, where policy optimization improves the proxy reward model while actually degrading performance with respect to the true human preference, due to the imperfection of the proxy. To address this, we propose Reward Model Boosting (RMB), a novel approach that enhances the robustness and reliability of the reward signal for RLHF. RMB first trains a set of reward models with a diversity-promoting regularizer. This encourages each model to learn complementary aspects of the reward landscape. Then, RMB learns a lightweight aggregator in the principle of boosting to aggregate the outputs of the diverse reward models into a more accurate and robust reward signal. Our extensive experiments demonstrate that RMB significantly improves reward accuracy on both in-distribution and out-of-distribution datasets, substantially mitigating the reward hacking issue and ultimately improving RLHF performance.

Figures & tables

Approach	Unified-Feedback	HHH-Align	MT-Bench	RewardBench
Approach	Unified-Feedback	HHH-Align	MT-Bench	Avg.	Chat	Chat-Hard	Safety	Reasoning
Frozen RM	63.9	68.6	68.2	x	x	x	x	x
Baseline RM	68.8	70.3	69.1	64.5	95.8	37.3	59.9	64.8
Label Smooth	68.5	68.8	71.9	61.1	91.6	39.0	53.8	60.2
Margin Loss	69.6	69.8	71.0	66.4	97.2	37.5	56.8	72.7
GRM	71.5	78.7	73.0	66.8	94.1	41.9	69.5	61.5

Table 1: Preference prediction performance on all approaches learned using 40K data mode of Unified-Feedback. Results of competing approaches are directly sourced from Yang et al. (2024) .

N

means the number of reward models used for ensembling or boosting.

Approach	Opponent	Pairwise Preference Comparison (in %)				Pointwise Preference Comparison (in %)
Approach	Opponent	Win $\uparrow$	Tie	Lose $\downarrow$	$\Delta$	Win $\uparrow$	Tie	Lose $\downarrow$	$\Delta$
RMB ( $N=3$ )	Baseline RM	55.97	3.21	40.82	$\uparrow$ 15.15	44.73	23.97	31.29	$\uparrow$ 13.44
	Avg. Ensemble ( $N=3$ )	57.87	4.31	37.81	$\uparrow$ 20.06	40.22	34.50	25.28	$\uparrow$ 14.94
	GRM	54.76	3.01	42.22	$\uparrow$ 12.54	43.33	30.39	26.28	$\uparrow$ 7.05

Table 2: LLM-as-judge of policies learned via RLHF.

Panel of Figure 1 — Figure 1: Results of Best-of-N sampling experiments, where both Avg Ensemble and our RMB use three individual reward models. Figures 1(a) and 1(c) intend to show the change of the proxy reward score, as an increasing proxy reward with a decreasing/plateaued gold reward indicates reward hacking. Notice that the scores of different proxy reward models are not directly comparable. Figures 1(b) and 1(d) show the gold reward scores achieved, a higher score indicating a better reward model.

Panel of Figure 2 — Figure 2: PPO training with reward models (RMs). Ensemble and RMB both have three RMs to aggregate.

		Unified-Feedback (In-Distribution)					RewardBench (Out-of-Distribution)
Random Seeds	Individual RMs	70.71	71.34	70.52	69.85	70.30	68.45	67.66	69.97	67.11	67.24
	Avg Ensemble	70.80					68.92
	Our RMB	71.04					68.70
HSIC	Individual RMs	73.40	70.73	69.29	68.05	73.84	70.08	69.95	67.58	66.97	71.62
	Avg Ensemble	72.04					69.40
	Our RMB	74.58					71.65

Table 3: Ensemble vs Boosting with different diversity strategies. We report the performance of individual RMs (index from 0 to 4).

Panel of Figure 3 — Figure 3: Correlation heat map among individual Reward Models

HSIC Diversity		Random Seeds Diversity
Model	Total Gain	Model	Total Gain
RM 0	29.75%	RM 1	70.13%
RM 4	27.39%	RM 0	12.39%
RM 1	26.78%	RM 4	10.99%
RM 2	8.55%	RM 2	5.82%
RM 3	7.53%	RM 3	0.66%

Table 4: Total Gain ratio of individual RMs.

Panel of Figure 4 — Figure 4: Rank order accuracy.

Latency Type	Component/Model	Mean	p50	p90	p99
Breakdown (s/batch)	Base RMs	$19.7915$	$19.7442$	$20.2839$	$20.3000$
	Avg Ensemble aggregator	$5.92\!\times\!10^{-5}$	$5.67\!\times\!10^{-5}$	$6.09\!\times\!10^{-5}$	$1.398\!\times\!10^{-4}$
	RMB aggregator	$1.7458\!\times\!10^{-3}$	$1.6997\!\times\!10^{-3}$	$1.9010\!\times\!10^{-3}$	$2.5694\!\times\!10^{-3}$
End-to-End (s/batch)	Avg Ensemble	$19.7915$	$19.7443$	$20.2840$	$20.3001$
End-to-End (s/batch)	RMB	$19.7932$	$19.7460$	$20.2857$	$20.3016$

Table 5: Inference-efficiency summary.

Reward Model	Total Params.	Average	Chat	Chat-Hard	Safety	Reasoning
RMB (N=3)	6B	87.8	98.9	75.6	87.3	89.1
GPT-4o-2024-08-06	-	86.7	96.1	76.1	88.1	86.6
GPT-4-0125-preview	-	86.0	95.3	74.3	87.6	86.9
FsfairX-LLaMA3-RM	8B	84.4	99.4	65.1	86.8	86.4
Starling-RM	34B	82.7	96.9	57.2	88.2	88.5

Table 6: Results of different reward models in a budget-constrained scenario.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Basic Setups
Base models	Gemma-2b-it
Quantization for training	bf16
Fine-tuning strategy	LoRA
LoRA $r$	32
LoRA alpha	64
LoRA dropout	0.05

Appendix

Table 7: Key implementation details of our main experiments.

Reward model	Unified-Feedback	HHH-Align	MT-Bench	RewardBench
Reward model	Unified-Feedback	HHH-Align	MT-Bench	Avg.	Chat	Chat-Hard	Safety	Reasoning
Frozen RM	63.8	66.4	69.5	x	x	x	x	x
Baseline RM	72.1	73.4	71.2	68.2	95.5	38.0	73.8	65.3
Label Smooth	71.5	72.1	71.2	70.6	94.4	37.3	73.2	77.4
Margin Loss	72.0	75.0	72.6	70.2	95.8	38.4	73.9	72.5
GRM	73.2	79.8	73.4	70.8	97.8	42.1	77.9	65.2

Appendix

Table 8: Preference prediction performance on all Reward Models learned using 400K data mode of Unified-Feedback. Results of competing approaches are directly sourced from Yang et al. (2024) .

N

means the number of reward models used for ensembling or boosting.

Table 13

Approach	Opponent	Pairwise Preference Comparison (in %)				Pointwise Preference Comparison (in %)
Approach	Opponent	Win $\uparrow$	Tie	Lose $\downarrow$	$\Delta$	Win $\uparrow$	Tie	Lose $\downarrow$	$\Delta$
RMB ( $N=3$ )	Baseline RM	59.54	5.42	35.04	$\uparrow$ 24.50	47.28	20.19	32.53	$\uparrow$ 14.75
	Avg Ensemble ( $N=3$ )	53.15	7.93	38.92	$\uparrow$ 14.23	39.40	30.91	29.69	$\uparrow$ 9.71
	GRM	57.38	9.37	33.25	$\uparrow$ 24.13	41.46	27.44	31.10	$\uparrow$ 10.36

Appendix

Table 11: LLM-as-judge (GPT-4o) of policies learned via RLHF.

RM	Reward Mean	Reward Std.	$c(x)$ Mean	$c(x)$ Std.	$c(x)$ P05–P95
RM 0	2.6380	17.0647	-0.1391	17.0468	[-28.2891, 27.9354]
RM 1	0.6850	10.9031	0.0566	10.9953	[-17.9596, 18.3358]
RM 2	-0.0619	10.9822	0.0606	10.9897	[-17.8373, 17.7582]

Appendix

Table 12: Statistics of reward scores and sampled prompt-dependent constants.

Figure 5: Performance–training efficiency trade-off.

$\lambda_{\text{HSIC}}$	Avg. Indiv. RM Perf.	Avg. Pairwise Corr.	RMB Perf.
0.001	$70.7\pm 0.8$	92%	71.7
0.01	$71.3\pm 1.1$	86%	72.8
0.1	$71.1\pm 2.5$	73%	74.6
1	$68.8\pm 3.0$	59%	71.2
10	$61.7\pm 3.7$	51%	66.5

Appendix

Table 14: Sensitivity analysis of the

\lambda_{\text{HSIC}}

. We report the average performance of individual Reward Models, the average pairwise correlation between them, and the final RMB performance.

Regularizer	RM1	RM2	RM3	RM4	RM5
HSIC	73.40	70.73	69.29	68.05	73.84
DPP	60.58	64.93	55.20	62.74	57.19
NCL	56.02	59.45	52.87	57.40	55.16

Appendix

Table 15: Comparison of different diversity regularizers. We report the accuracy (%) of five individually trained reward models (RM1–RM5) under HSIC, DPP, and NCL regularization. HSIC consistently yields stronger RMs, while DPP and NCL significantly underperform.

Please act as an impartial judge and evaluate the quality of the responses provided by two

AI assistants to the user question displayed below. Your evaluation should consider

correctness and helpfulness. You will be given a reference answer, assistant A’s answer,

and assistant B’s answer. Your job is to evaluate which assistant’s answer is better.

Begin your evaluation by comparing both assistants’ answers with the reference answer.

Identify and correct any mistakes. Avoid any position biases and ensure that the order in

which the responses were presented does not influence your decision. Do not allow the

Appendix

Table 16: Prompt templates for LLM evaluation on text generated by RLHF optimized policy in § 3.2 from a pairwise view. This template originates from Zheng et al. (2023) ’s work, and is adapted according to Fan et al. (2026) ’s template to avoid ID and position bias, aiming to provide reliable LLM judges for pairwise comparison.

Please act as an impartial judge and evaluate the quality of the response provided by an AI assistant

to the user’s question shown below. Your evaluation should consider factors such as helpfulness,

relevance, accuracy, depth, creativity, and level of detail in the response.

Begin your evaluation with a short explanation. Be as objective as possible.

After providing your explanation, please rate the response on a scale of 1 to 10

by strictly following this format: "[[rating]]", for example: "Rating: [[5]]".

[Question]

Appendix

Table 17: Prompt templates for LLM evaluation on text generated by RLHF optimized policy in § 3.2 from a pointwise view. This template originates from Zheng et al. (2023) ’s work, aiming to provide LLM judges for single-answer grading.

Explore similar work

Uncertainty-Aware Reward Modeling for Stable RLHF
Jun 18, 2026Licheng Pan, Haocheng Yang, Haoxuan Li +7Reward Modeling Reward Hacking
Zhejiang University · Xiaohongshu Inc · National University of Singapore +1
IR $^3$ : Contrastive Inverse Reinforcement Learning for Interpretable Detection and Mitigation of Reward Hacking
Feb 23, 2026Mohammad Beigi, Ming Jin, Junshan Zhang +3Reward Hacking Reinforcement Learning
UC Davis · Virginia Tech · Salesforce +1
UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hacking
Nov 30, 2025Lingling Fu, Yongfu XueReward Modeling Reward Hacking
Guangxi University

Jun 18, 2026J/K move · Enter open · S save

Uncertainty-Aware Reward Modeling for Stable RLHF

Licheng Pan, Haocheng Yang, Haoxuan Li, Yichen Sun+6

Zhejiang University · Xiaohongshu Inc · National University of Singapore+1

Reward Modeling Reward Hacking Group Relative Policy Optimization RL from Human Feedback

Open paper AskPDF arXiv

Reinforcement learning from human feedback (RLHF) aligns large language models by training reward models on preference data and optimizing policies to maximize predicted rewards. However, this pipeline faces two fundamental challenges: (1) reward models cannot signal when their predictions are unreliable, since they usually act as deterministic point estimators; and (2) modern group-based policy optimization can amplify unreliable reward signals, as exemplified by GRPO's uniform treatment of rewards during advantage computation. As policies explore increasingly diverse responses, these two limitations create a critical vulnerability: unreliable reward estimates may be granted disproportionate influence, triggering severe reward hacking. We propose Uncertainty-Aware Reward Modeling (UARM), which equips reward models with calibrated uncertainty via quantile-based conformal prediction and reweights GRPO advantages through heteroscedastic variance decomposition. Experiments across HelpSteer, UltraFeedback, and PKU-SafeRLHF demonstrate that UARM significantly improves reward model calibration, reduces reward hacking, and enhances downstream alignment quality compared to standard GRPO and uncertainty-agnostic baselines.

Abstract

Explore similar work

Uncertainty-Aware Reward Modeling for Stable RLHF

IR3^33: Contrastive Inverse Reinforcement Learning for Interpretable Detection and Mitigation of Reward Hacking

UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hacking

IR $^3$ : Contrastive Inverse Reinforcement Learning for Interpretable Detection and Mitigation of Reward Hacking