cs.LGFeb 26, 2025

Reward Shaping to Mitigate Reward Hacking in RLHF

Authors: Jiayi Fu, Xuandong Zhao, Chengyuan Yao, Heng Wang, Qi Han, Yanghua Xiao, Yuxia Wang

Organizations: INSAIT, Sofia University “St. Kliment Ohridski” · UC Berkeley · StepFun · Fudan University

Abstract

Reinforcement learning from human feedback (RLHF) is widely used to align large language models with human preferences, but it remains vulnerable to reward hacking, whereby a policy exploits flaws in the reward function at the expense of the intended behavior. Reward shaping can stabilize training and partially mitigate reward hacking, yet its design principles remain underexplored. Our systematic study of common shaping methods suggests that shaped rewards should be bounded and should rise rapidly before gradually saturating. We introduce Preference as Reward (PAR), which applies a sigmoid to centered rewards to produce a signal that reflects the reward model's latent preferences. We show that PAR has a variance-reduction property that stabilizes training and substantially widens the window for effective early stopping. Using Proximal Policy Optimization (PPO) and Gemma2-2B, PAR consistently outperforms the other shaping methods evaluated and demonstrates high data efficiency and robustness. Comparisons with unshaped rewards across four base models and four reinforcement learning algorithms show that PAR is effective across models and particularly beneficial for actor--critic methods when value estimates become unstable. The code is available at https://github.com/PorUna-byte/PAR.

Figures & tables

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Jul 28, 2026

Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback

Reinforcement Learning from Human Feedback (RLHF) is the standard approach for aligning large language models with human preferences, but its quality is limited by static, task-agnostic reward models. This mismatch leads to sparse learning signals and suboptimal alignment. We introduce MeRLa (Meta-Learned Reward Shaping), a principled framework that meta-learns a task-aware shaping function Φ(x,y;φ)Φ(x,y;φ) across auxiliary tasks before RLHF training. The learned shaping produces a composite reward that preserves policy optimality while providing task-specific learning signals. Our meta-objective combines task discrimination, entropy regularization, and potential-based conservation for stable convergence. We provide theoretical guarantees for policy invariance, analyze representation drift sensitivity, and formally address incentive misalignment from entropy maximization. Experiments on LLaMA-3-8B across four benchmarks show consistent improvements over PPO, DPO, GRPO, and DAPO, achieving a 90.8% length-controlled win rate on AlpacaEval 2.0 and a score of 9.14 on MT-Bench, with 41% less training instability. MeRLa retains its benefits when combined with process-based and rubric-based enhanced rewards.
Sep 27, 2026

RMB: Reward Model Boosting Mitigates Reward Hacking

Reinforcement Learning from Human Feedback (RLHF) is a powerful technique for aligning large language models (LLMs) with human preference. However, it often suffers from the reward hacking issue, where policy optimization improves the proxy reward model while actually degrading performance with respect to the true human preference, due to the imperfection of the proxy. To address this, we propose Reward Model Boosting (RMB), a novel approach that enhances the robustness and reliability of the reward signal for RLHF. RMB first trains a set of reward models with a diversity-promoting regularizer. This encourages each model to learn complementary aspects of the reward landscape. Then, RMB learns a lightweight aggregator in the principle of boosting to aggregate the outputs of the diverse reward models into a more accurate and robust reward signal. Our extensive experiments demonstrate that RMB significantly improves reward accuracy on both in-distribution and out-of-distribution datasets, substantially mitigating the reward hacking issue and ultimately improving RLHF performance.
Feb 20, 2026

Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards

Reinforcement Learning from Human Feedback (RLHF) or Verifiable Rewards (RLVR) are two key steps in the post-training of modern Language Models (LMs). A common problem is reward hacking, where the policy may exploit inaccuracies of the reward and learn an unintended behavior. Most previous works address this by limiting the policy update with a Kullback-Leibler (KL) penalty towards a reference model. We propose a different framing: Train the LM in a way that biases policy updates towards regions in which the reward is more accurate. First, we derive a theoretical connection between the accuracy of a reward model and the flatness of an optimum at convergence. Gradient regularization (GR) can then be used to bias training to flatter regions and thereby maintain reward model accuracy. We confirm these results by showing that the gradient norm and reward accuracy are empirically correlated in RLHF. We then empirically show that Reference Resets of the KL penalty find flatter regions with a higher reward accuracy. We further improve on this by proposing to use explicit GR with an efficient finite-difference estimate. Empirically, GR performs better than a KL penalty across a diverse set of RL experiments with LMs. GR achieves a higher GPT-judged win-rate in RLHF, avoids overly focusing on the format in rule-based math rewards, and prevents hacking the judge in LLM-as-a-Judge math tasks.