Reinforcement learning from human feedback (RLHF) is widely used to align large language models with human preferences, but it remains vulnerable to reward hacking, whereby a policy exploits flaws in the reward function at the expense of the intended behavior. Reward shaping can stabilize training and partially mitigate reward hacking, yet its design principles remain underexplored. Our systematic study of common shaping methods suggests that shaped rewards should be bounded and should rise rapidly before gradually saturating. We introduce Preference as Reward (PAR), which applies a sigmoid to centered rewards to produce a signal that reflects the reward model's latent preferences. We show that PAR has a variance-reduction property that stabilizes training and substantially widens the window for effective early stopping. Using Proximal Policy Optimization (PPO) and Gemma2-2B, PAR consistently outperforms the other shaping methods evaluated and demonstrates high data efficiency and robustness. Comparisons with unshaped rewards across four base models and four reinforcement learning algorithms show that PAR is effective across models and particularly beneficial for actor--critic methods when value estimates become unstable. The code is available at https://github.com/PorUna-byte/PAR.
Figures & tables
Figure 1: RLHF training with reward shaping. The reward model assigns proxy rewards to policy responses, which are then optionally transformed using reference rewards (dashed box) before policy optimization. PAR applies a sigmoid transformation to the centered proxy reward.
Figure 2: Loss curves from PPO training show that PAR exhibits greater stability, particularly in critic loss, compared to Vanilla training. This stability is attributed to PAR’s bounded RL reward.
Figure 3: PPO training curves over two epochs. ceil5.0 denotes rRL=min(r,5.0) , and kl0.1 denotes a KL-penalty coefficient of β=0.1 . Restricting excessive reward optimization mitigates reward hacking and supports the use of bounded RL rewards.
Figure 4: Validation of the rapid-growth and gradual-saturation principle using centered and uncentered reward transformations.
Figure 5: (Zoom in to view clearly) PPO training curves for different reward hacking mitigation methods using Gemma2-2B and UltraFeedback-Binarized. Solid lines denote proxy reward, and dashed lines denote test-set win rate against the SFT model. Vanilla PPO exhibits substantial reward hacking. In this setting, WARM, Minmax, and PAR mitigate the decline in win rate, with PAR achieving the highest win rate at the end of training.
Method
AlpacaEval 2.0
MT-Bench
LC Win Rate (%) ↑
Win Rate (%) ↑
Length ↓
T1 ↑
T2 ↑
Overall ↑
SFT
50.000
50.000
899
5.150
3.975
4.563
PPO training
Vanilla
0.100
0.370
2008
2.150
1.700
1.925
WARM
60.670
63.170
1073
5.525
3.938
4.731
ODIN
0.000
0.000
3672
1.375
1.338
1.356
Reg
0.000
0.000
1868
1.513
1.388
1.450
Table 1: Benchmark results for checkpoints selected after one epoch of PPO training; the SFT model is trained for two epochs, and the winrate is calculated against the SFT model. PAR achieves the highest AlpacaEval 2.0 win rates and MT-Bench scores among the shaping methods.
Figure 6: PPO training curves on the HH-RLHF dataset with Gemma2-2B as the base model.
Figure 7: PPO training curves on the HH-RLHF dataset with Gemma2-9B as the base model.
Policy
vs. other annotator
vs. GPT-5-nano
Rater
win rate
Agr.
Cohen’s κ
Agr.
Cohen’s κ
Annotator 1
70.0%
78.0%
0.527
75.5%
0.451
Annotator 2
71.5%
78.0%
0.527
75.0%
0.451
GPT-5-nano
68.5%
–
–
–
–
Table 2: Human evaluation of PPO with PAR on the HH-RLHF dataset, using Gemma2-2B as the base model. Ties count as half a win when calculating win rates and as a separate category when calculating agreement metrics.
Symbol
Meaning
D
Dataset
x,yw,yl∼D
Prompt, chosen response, rejected response in Dataset
πθ
Policy model
πref
Reference model, also the SFT model
rϕ
Reward model
Vα
Critic model
Table 3: Summary of notations.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Algorithm
Base Model
Policy Learning Rate
Critic Learning Rate
Epochs
PPO
Gemma2-2B
6×10−7
6×10−6
0.5
Gemma2-9B
3×10−7
3×10−6
0.1
Llama3.1-8B
2×10−7
2×10−6
0.1
Qwen3-4B-Base
2×10−7
2×10−6
0.1
GRPO
Gemma2-2B
2×10−6
-
0.2
Gemma2-9B
3×10−7
-
0.1
Appendix
Table 4: Learning rates and training durations for each base-model and optimization-algorithm combination on HH-RLHF.
Figure 8: System prompt for win-rate evaluation
Figure 9: PPO training curves with different numbers of reference rewards. For example, PAR5 denotes rRL=51∑m=15σ(r−rrefm) . A single reference reward achieves a win rate comparable to that obtained with additional references.
Figure 10: PPO training curves over two epochs for PAR, Minmax, and WARM. PAR maintains a higher and more stable win rate under prolonged optimization.
Figure 11: Relationship between latent preference score and the win rate.
Figure 12: PPO training curves on the HH-RLHF dataset with Llama3.1-8B as the base model.
Figure 13: PPO training curves on the HH-RLHF dataset with Qwen3-4B-Base as the base model.
Figure 14: A2C training curves on the HH-RLHF dataset with Gemma2-2B as the base model.
Figure 15: GRPO training curves on the HH-RLHF dataset with Gemma2-2B as the base model.
Figure 16: A2C training curves on the HH-RLHF dataset with Gemma2-9B as the base model.
Figure 17: GRPO training curves on the HH-RLHF dataset with Gemma2-9B as the base model.
Figure 18: A2C training curves on the HH-RLHF dataset with Llama3.1-8B as the base model.
Figure 19: GRPO training curves on the HH-RLHF dataset with Llama-3.1-8B as the base model.
Figure 20: A2C training curves on the HH-RLHF dataset with Qwen3-4B-Base as the base model.
Figure 21: GRPO training curves on the HH-RLHF dataset with Qwen3-4B-Base as the base model.
Figure 22: Representative examples of reward hacking during vanilla PPO training. The first response exhibits incoherent keyword stuffing, whereas the second is verbose and fails to address the requested price comparison.
Reinforcement Learning from Human Feedback (RLHF) is the standard approach for aligning large language models with human preferences, but its quality is limited by static, task-agnostic reward models. This mismatch leads to sparse learning signals and suboptimal alignment. We introduce MeRLa (Meta-Learned Reward Shaping), a principled framework that meta-learns a task-aware shaping function Φ(x,y;φ) across auxiliary tasks before RLHF training. The learned shaping produces a composite reward that preserves policy optimality while providing task-specific learning signals. Our meta-objective combines task discrimination, entropy regularization, and potential-based conservation for stable convergence. We provide theoretical guarantees for policy invariance, analyze representation drift sensitivity, and formally address incentive misalignment from entropy maximization. Experiments on LLaMA-3-8B across four benchmarks show consistent improvements over PPO, DPO, GRPO, and DAPO, achieving a 90.8% length-controlled win rate on AlpacaEval 2.0 and a score of 9.14 on MT-Bench, with 41% less training instability. MeRLa retains its benefits when combined with process-based and rubric-based enhanced rewards.
Reinforcement Learning from Human Feedback (RLHF) is a powerful technique for aligning large language models (LLMs) with human preference. However, it often suffers from the reward hacking issue, where policy optimization improves the proxy reward model while actually degrading performance with respect to the true human preference, due to the imperfection of the proxy. To address this, we propose Reward Model Boosting (RMB), a novel approach that enhances the robustness and reliability of the reward signal for RLHF. RMB first trains a set of reward models with a diversity-promoting regularizer. This encourages each model to learn complementary aspects of the reward landscape. Then, RMB learns a lightweight aggregator in the principle of boosting to aggregate the outputs of the diverse reward models into a more accurate and robust reward signal. Our extensive experiments demonstrate that RMB significantly improves reward accuracy on both in-distribution and out-of-distribution datasets, substantially mitigating the reward hacking issue and ultimately improving RLHF performance.
Jiabin Fan, Dezhi Ye, Yongchang Hao +1
Dept. Computing Science & Alberta Machine Intelligence Institute (Amii), University of Alberta · Tencent · Canada CIFAR AI Chair
Reinforcement Learning from Human Feedback (RLHF) or Verifiable Rewards (RLVR) are two key steps in the post-training of modern Language Models (LMs). A common problem is reward hacking, where the policy may exploit inaccuracies of the reward and learn an unintended behavior. Most previous works address this by limiting the policy update with a Kullback-Leibler (KL) penalty towards a reference model. We propose a different framing: Train the LM in a way that biases policy updates towards regions in which the reward is more accurate. First, we derive a theoretical connection between the accuracy of a reward model and the flatness of an optimum at convergence. Gradient regularization (GR) can then be used to bias training to flatter regions and thereby maintain reward model accuracy. We confirm these results by showing that the gradient norm and reward accuracy are empirically correlated in RLHF. We then empirically show that Reference Resets of the KL penalty find flatter regions with a higher reward accuracy. We further improve on this by proposing to use explicit GR with an efficient finite-difference estimate. Empirically, GR performs better than a KL penalty across a diverse set of RL experiments with LMs. GR achieves a higher GPT-judged win-rate in RLHF, avoids overly focusing on the format in rule-based math rewards, and prevents hacking the judge in LLM-as-a-Judge math tasks.
Johannes Ackermann, Michael Noukhovitch, Takashi Ishida +1