Reward Shaping to Mitigate Reward Hacking in RLHF
Organizations: INSAIT, Sofia University “St. Kliment Ohridski” · UC Berkeley · StepFun · Fudan University
Abstract
Reinforcement learning from human feedback (RLHF) is widely used to align large language models with human preferences, but it remains vulnerable to reward hacking, whereby a policy exploits flaws in the reward function at the expense of the intended behavior. Reward shaping can stabilize training and partially mitigate reward hacking, yet its design principles remain underexplored. Our systematic study of common shaping methods suggests that shaped rewards should be bounded and should rise rapidly before gradually saturating. We introduce Preference as Reward (PAR), which applies a sigmoid to centered rewards to produce a signal that reflects the reward model's latent preferences. We show that PAR has a variance-reduction property that stabilizes training and substantially widens the window for effective early stopping. Using Proximal Policy Optimization (PPO) and Gemma2-2B, PAR consistently outperforms the other shaping methods evaluated and demonstrates high data efficiency and robustness. Comparisons with unshaped rewards across four base models and four reinforcement learning algorithms show that PAR is effective across models and particularly beneficial for actor--critic methods when value estimates become unstable. The code is available at https://github.com/PorUna-byte/PAR.
Figures & tables
| Method | AlpacaEval 2.0 | MT-Bench | |||||
|---|---|---|---|---|---|---|---|
| LC Win Rate (%) | Win Rate (%) | Length | T1 | T2 | Overall | ||
| SFT | 50.000 | 50.000 | 899 | 5.150 | 3.975 | 4.563 | |
| PPO training | Vanilla | 0.100 | 0.370 | 2008 | 2.150 | 1.700 | 1.925 |
| WARM | 60.670 | 63.170 | 1073 | 5.525 | 3.938 | 4.731 | |
| ODIN | 0.000 | 0.000 | 3672 | 1.375 | 1.338 | 1.356 | |
| Reg | 0.000 | 0.000 | 1868 | 1.513 | 1.388 | 1.450 | |
| Policy | vs. other annotator | vs. GPT-5-nano | |||
|---|---|---|---|---|---|
| Rater | win rate | Agr. | Cohen’s | Agr. | Cohen’s |
| Annotator 1 | 70.0% | 78.0% | 0.527 | 75.5% | 0.451 |
| Annotator 2 | 71.5% | 78.0% | 0.527 | 75.0% | 0.451 |
| GPT-5-nano | 68.5% | – | – | – | – |
| Symbol | Meaning |
|---|---|
| Dataset | |
| Prompt, chosen response, rejected response in Dataset | |
| Policy model | |
| Reference model, also the SFT model | |
| Reward model | |
| Critic model |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Algorithm | Base Model | Policy Learning Rate | Critic Learning Rate | Epochs |
|---|---|---|---|---|
| PPO | Gemma2-2B | 0.5 | ||
| Gemma2-9B | 0.1 | |||
| Llama3.1-8B | 0.1 | |||
| Qwen3-4B-Base | 0.1 | |||
| GRPO | Gemma2-2B | - | 0.2 | |
| Gemma2-9B | - | 0.1 |
Reward Shaping
Reward shaping designs intermediate or modified feedback to make reinforcement learning more informative, efficient, and aligned with task goals.
See all 150 papers