cs.LGFeb 26, 2025

Reward Shaping to Mitigate Reward Hacking in RLHF

Authors: Jiayi Fu, Xuandong Zhao, Chengyuan Yao, Heng Wang, Qi Han, Yanghua Xiao, Yuxia Wang

Organizations: INSAIT, Sofia University “St. Kliment Ohridski” · UC Berkeley · StepFun · Fudan University

Abstract

Reinforcement learning from human feedback (RLHF) is widely used to align large language models with human preferences, but it remains vulnerable to reward hacking, whereby a policy exploits flaws in the reward function at the expense of the intended behavior. Reward shaping can stabilize training and partially mitigate reward hacking, yet its design principles remain underexplored. Our systematic study of common shaping methods suggests that shaped rewards should be bounded and should rise rapidly before gradually saturating. We introduce Preference as Reward (PAR), which applies a sigmoid to centered rewards to produce a signal that reflects the reward model's latent preferences. We show that PAR has a variance-reduction property that stabilizes training and substantially widens the window for effective early stopping. Using Proximal Policy Optimization (PPO) and Gemma2-2B, PAR consistently outperforms the other shaping methods evaluated and demonstrates high data efficiency and robustness. Comparisons with unshaped rewards across four base models and four reinforcement learning algorithms show that PAR is effective across models and particularly beneficial for actor--critic methods when value estimates become unstable. The code is available at https://github.com/PorUna-byte/PAR.

Figures & tables

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. RMB: Reward Model Boosting Mitigates Reward Hacking

    Sep 27, 2026Jiabin Fan, Dezhi Ye, Yongchang Hao +1Reward HackingRL from Human Feedback