Reward models (RMs) are critical to reinforcement learning from human feedback (RLHF). However, policy models can exploit biases or spurious correlations in dense RMs, causing reward hacking: RM scores rise while alignment with human preferences deteriorates, especially under distribution shift. We propose UMM-RM (Upcycle-and-Merge MoE Reward Model), which copies the feed-forward layers of an SFT checkpoint to initialize shared and routed experts, then trains the model on preference data. The always-active shared expert captures instruction-agnostic preference signals, while routed experts model fine-grained preferences across instructions or tasks. The experts are then merged into a single dense RM using learnable weights, preserving the robustness and exploitation resistance of expert diversity without MoE or ensemble inference overhead. Across multiple base models and preference datasets, UMM-RM improves preference accuracy, reduces reward hacking during PPO training, and achieves more stable preference alignment than standard dense RMs.
Figures & tables
Figure 1: Compared with the dense reward model, UMM-RM more effectively mitigates reward hacking. The dashed curves denote the reward scores assigned by the proxy reward model, while the solid curves represent the gold reward scores evaluated by the LLaMA2-13B reward model.
Figure 2: Comparison of PPO training reward scores between UMM-RM and reward models based on different ensemble strategies built on the TinyLlama-1.1B backbone.
Models
Anthropic
WebGPT
Harmless
Helpful
Dense RM
51.2 ± 0.3
44.6 ± 0.2
52.2 ± 0.7
Mean (ensemble RM)
57.1 ± 0.3
55.0 ± 0.3
51.4 ± 0.2
WCO (ensemble RM)
55.4 ± 0.6
54.8 ± 0.2
60.6 ± 0.3
UWO (ensemble RM)
58.0 ± 0.4
54.6 ± 0.5
59.6 ± 0.4
UnMerged MoE (2-experts)
59.8 ± 0.2
53.8 ± 0.2
57.0 ± 0.3
Table 1: Preference accuracy (%) of different reward modeling methods on TinyLlama-1.1B. Results are reported as mean ± std. over 3 random seeds.
Models
Anthropic
WebGPT
Harmless
Helpful
Base Model: Qwen2.5-0.5B
Dense RM
38.6
65.8
57.2
UMM-RM (2-experts)
48.8
66.8
58.2
UMM-RM (4-experts)
50.2
66.0
57.8
UMM-RM (6-experts)
50.8
67.2
58.4
Table 2: Comparison of accuracy (%) on preference tasks across different reward models using Qwen2.5-0.5B and Pythia-1.4B.
Figure 3: Comparison of PPO training reward scores between MoE reward models with different numbers of activated experts and UMM-RM based on the TinyLlama-1.1B backbone.
Figure 4: Proxy (left) and gold (right) reward scores during PPO training for UMM-RM with a TinyLlama-1.1B backbone and shared-expert weights α∈{0.1,0.5,0.75,0.9} .
Models
Opponent
AlpacaFarm
Win ↑
Tie
Lose ↓
UMM-RM(2-experts)
Dense RM
28.9
44.0
27.1
UMM-RM(4-experts)
30.6
45.5
23.9
UMM-RM(6-experts)
36.4
42.3
21.3
UMM-RM(2-experts)
SFT Model
32.3
39.9
27.8
UMM-RM(4-experts)
33.7
37.5
28.8
Table 3: End-to-end generation quality evaluation of policy models trained with PPO under different reward model methods.
Method
Train
Latency
Throughput
Slowdown
Dense RM
∼ 1.0 h
382.2 ± 1.1
83.72
1.00×
Unmerged MoE
∼ 1.5 h
914.4 ± 4.3
35.00
2.39×
Ensemble RM (4)
∼ 4.0 h
1543.5 ± 3.7
20.73
4.04×
UMM-RM
∼ 2.0 h
381.9 ± 1.1
83.78
1.00×
Table 4: Training and inference efficiency comparison of different reward modeling methods based on TinyLlama-1.1B. Slowdown is measured relative to Dense RM.
Beijing University of Posts and Telecommunications · National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China