The Reward Model Selection Crisis in Personalized Alignment

Fady Rezk; Yuangang Pan; Chuan-Sheng Foo; Xun Xu; Nancy Chen; Henry Gouk; Timothy Hospedales

cs.AIDec 28, 2025

The Reward Model Selection Crisis in Personalized Alignment

Authors: Fady Rezk, Yuangang Pan, Chuan-Sheng Foo, Xun Xu, Nancy Chen, Henry Gouk, Timothy Hospedales

Organizations: School of Informatics, University of Edinburgh, UK · Agency for Science, Technology and Research (A*STAR), Singapore · Samsung AI Research Center, Cambridge, UK

Abstract

Personalized alignment from preference data has focused primarily on improving personal reward model (RM) accuracy, with the implicit assumption that better preference ranking translates to better personalized behavior. However, in deployment, computational constraints necessitate inference-time adaptation such as reward-guided decoding (RGD) rather than per-user policy fine-tuning. This creates a critical but overlooked requirement: reward models must not only rank preferences accurately but also effectively guide generation. We demonstrate that standard RM accuracy fails catastrophically as a selection criterion for deployment-ready personalized rewards. We introduce policy accuracy; a metric quantifying whether RGD-adapted LLMs correctly discriminate between preferred and dispreferred responses and show that upstream RM accuracy correlates only weakly with downstream policy accuracy (Kendall's tau = 0.08--0.31). More critically, we introduce Pref-LaMP the first personalized alignment benchmark with ground-truth user completions, enabling direct behavioural evaluation. On Pref-LaMP, we expose a complete decoupling between discriminative ranking and generation metrics: methods with 20-point RM accuracy differences produce almost identical output quality, and methods with high ranking accuracy can fail to generate behaviorally aligned responses. These findings reveal that the field has been optimizing for proxy metrics that do not predict deployment performance, and that current personalized alignment methods fail to operationalize preferences into behavioral adaptation under realistic deployment constraints. In contrast, we find simple in-context learning (ICL) to be highly effective - dominating all reward-guided methods for models $\geq$ 3B parameters, achieving $\sim$ 3 point ROUGE-1 gains over the best reward method at 7B scale.

Figures & tables

Negative	R-1	R-L	BERTScore
Mined	24.6	22.2	60.0
Random	2 4.7	2 4.4	43.7
$\Delta$	+19.9	+17.8	+16.3

Table 1: Similarity of rejected titles to the ground-truth chosen title on the Pref-LaMP evaluation split, under hard-negative mining versus a random permutation of titles within the split. The training split agrees to within

0.01

on every metric.

Dataset	Reward Model	Base Policies
TLDR	SmolLM2-180M	180M, 360M, 1.7B
PRISM	Qwen2.5-0.5B	0.5B, 1.5B, 3B, 7B
Pref-LaMP	Qwen2.5-0.5B	0.5B, 1.5B, 3B, 7B

Table 2: Model configurations.

			Policy Accuracy
	Method	RM Accuracy	135M	360M	1.7B
Personalized	LoRE	82.56 $\pm$ 1.12	81.09 $\pm$ 0.76	81.09 $\pm$ 0.89	81.01 $\pm$ 1.10
	MPU	76.76 $\pm$ 0.26	49.96 $\pm$ 3.24	54.36 $\pm$ 7.36	55.12 $\pm$ 8.07
	MPU-Avg	77.83 $\pm$ 0.19	52.91 $\pm$ 4.01	52.96 $\pm$ 3.97	52.94 $\pm$ 3.98
	P-DPO	-	-	-	-
	PAL	87.00 $\pm$ 0.05	-	-	-

Table 3: Policy accuracy and reward model performance on TLDR + SmolLM. Base reward model is SmolLM2-135M. Top: Personal rewards. Mid: Global alignment.

	Personalized							Global
	LoRE	LoRE-Alt	MPU	MPU-Avg	PAL	PReF	VPL	Global RM	Global RM-v2
RM Acc.	$65.95^{\pm 4.57}$	$67.15^{\pm 0.42}$	$52.76^{\pm 5.40}$	$60.43^{\pm 1.25}$	$70.74^{\pm 0.42}$	$74.34^{\pm 2.20}$	$68.35^{\pm 2.59}$	$\textbf{77.94}^{\pm 3.55}$	$62.11^{\pm 4.15}$

Table 4: Reward model accuracies of various personal alignment methods using Qwen2.5-0.5B backbone on PRISM dataset. All personal alignment methods underperform non-personal Global Reward Model V1.

Figure 1: RM vs Policy Accuracy Correlation on PRISM across model scales. Correlations remain consistently weak: Pearson

r

ranges from 0.30–0.48, Spearman

\rho

from 0.25–0.43, and Kendall

\tau

from 0.17–0.29. While correlations slightly strengthen with scale, they remain far below what would be needed for RM accuracy to reliably predict policy performance. Notably, methods with similar RM accuracy can have substantially different policy accuracy and vice-versa, demonstrating that the standard RM metric is not a reliable proxy for deployment performance.

	Global			Personalized
Acc	GenARM	Global RM	Global RM-v2	LoRE	LoRE-Alt	MPU	MPU-Avg	PAL	PReF	VPL	ICL
RM	$90.04^{\pm 0.69}$	$92.24^{\pm 0.25}$	$92.26^{\pm 0.23}$	$91.97^{\pm 0.10}$	$92.11^{\pm 0.18}$	$86.15^{\pm 0.39}$	$88.02^{\pm 0.40}$	$53.77^{\pm 0.28}$	$63.79^{\pm 0.88}$	$39.65^{\pm 17.76}$	NA
Policy	$76.80^{\pm 8.19}$	$66.96^{\pm 0.00}$	$84.60^{\pm 1.74}$	$67.17^{\pm 0.36}$	$67.63^{\pm 0.63}$	$67.04^{\pm 0.20}$	$64.32^{\pm 4.63}$	–	$67.13^{\pm 0.29}$	$67.19^{\pm 0.40}$	$80.20^{\pm 0.69}$

Table 5: Pref-LaMP preference ranking accuracy for RMs and adapted policies (Qwen2.5-3B). RM personalization does not clearly outperform global RMs for either upstream RM or downstream policy preference ranking.

Figure 2: Generation quality (ROUGE-1, ROUGE-L, BERTScore-F1) under RGD across model scales. At 0.5B-1.5B, personalized RMs marginally improve over zero-shot; at 3B-7B, ICL baselines dominate all reward-guided methods.

	Method	0.5B	1.5B	3B	7B
Global	GenARM	100.0%	100.0%	100.0%	99.2%
	Global RM	92.4%	92.1%	91.6%	92.1%
	Global RM-v2	99.0%	98.0%	92.4%	90.8%
Personalized	LoRE	92.5%	91.7%	92.2%	92.8%
	LoRE-Alt	95.0%	94.5%	94.7%	95.5%
	MPU	84.3%	84.2%	85.0%	85.7%

Table 6: Win rate: fraction of examples where each RM judges its own RGD output as better than zero-shot. High win rates reveal severe reward hacking—methods claim near-perfect improvement despite ground-truth metrics showing minimal or negative gains (Fig 2).

Figure 3: Pref-LAMP generation quality. ROUGE-1 vs. number of in-context examples. ICL-RAG consistently outperforms random ICL, with performance scaling with both model size and context length.

Scale	RM Acc	Policy Acc	Win Rate
Kendall’s $\tau$ with ROUGE-1
0.5B	$-0.262$	$-0.266$	$-0.286$
1.5B	$-0.118$	$-0.347$	$-0.269$
3B	$-0.118$	$-0.294$	$-0.149$
7B	$-0.233$	$-0.308$	$-0.262$

Table 7: Correlations between metrics and generation quality (ROUGE-1) on Pref-LaMP. All correlations are negligible to negative across model scales, demonstrating no standard metric predicts behavioral alignment.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	LoRE	PReF	VPL	Global RM	Global RM-v2	MPU	MPU-Avg	LoRE-Alt
RM Accuracy	65.95 $\pm$ 4.57	74.34 $\pm$ 2.20	68.35 $\pm$ 2.59	77.94 $\pm$ 3.55	62.11 $\pm$ 4.15	52.76 $\pm$ 5.40	60.43 $\pm$ 1.25	67.15 $\pm$ 0.42
Policy accuracies
0.5B	56.83 $\pm$ 0.00	57.31 $\pm$ 1.66	66.43 $\pm$ 3.24	64.51 $\pm$ 0.83	62.59 $\pm$ 5.04	59.47 $\pm$ 2.91	57.55 $\pm$ 3.81	58.51 $\pm$ 0.83
1.5B	57.07 $\pm$ 0.42	58.99 $\pm$ 1.90	65.71 $\pm$ 1.81	64.51 $\pm$ 0.83	62.35 $\pm$ 4.63	60.43 $\pm$ 2.88	58.27 $\pm$ 2.49	57.55 $\pm$ 0.72
3B	57.07 $\pm$ 0.42	57.31 $\pm$ 0.42	64.03 $\pm$ 3.81	64.51 $\pm$ 0.83	61.39 $\pm$ 3.69	57.55 $\pm$ 2.88	56.12 $\pm$ 2.59	56.59 $\pm$ 1.81
7B	57.55 $\pm$ 0.00	59.23 $\pm$ 2.53	63.79 $\pm$ 2.31	64.51 $\pm$ 0.83	61.87 $\pm$ 3.81	58.03 $\pm$ 2.91	56.35 $\pm$ 2.53	57.07 $\pm$ 0.83

Appendix

Table 8: Reward model accuracy and policy accuracies per model and method for the PRISM dataset.

	LoRE	LoRE-Alt	MPU	MPU-Avg	PAL	PReF	VPL	Global RM	Global RM2
RM Accuracy	$56.53$ $\pm 0.15$	$60.33$ $\pm 0.57$	$49.66$ $\pm 1.13$	$49.07$ $\pm 1.46$	$62.51$ $\pm 0.30$	$62.73$ $\pm 0.64$	$60.31$ $\pm 2.98$	$64.51$ $\pm 0.41$	$59.18$ $\pm 0.74$

Appendix

Table 9: Test set accuracy evaluated across all samples from unseen users. Note: The results reported in the main text use only a single test sample per user due to computational constraints.

Method	RM Acc	Policy Acc	ROUGE-1	ROUGE-L	BertScore-F1
Reward Models + ARGS
GenARM	90.04 $\pm$ 0.69	76.99 $\pm$ 8.05	39.84 $\pm$ 1.06	34.02 $\pm$ 1.19	68.55 $\pm$ 0.52
Global RM	92.24 $\pm$ 0.25	64.95 $\pm$ 0.00	40.27 $\pm$ 0.03	34.66 $\pm$ 0.02	67.87 $\pm$ 0.00
Global RM-v2	92.26 $\pm$ 0.23	85.44 $\pm$ 1.67	30.64 $\pm$ 0.26	25.72 $\pm$ 0.06	66.93 $\pm$ 0.42
LoRE	91.97 $\pm$ 0.10	65.01 $\pm$ 0.17	40.23 $\pm$ 0.03	34.63 $\pm$ 0.05	67.87 $\pm$ 0.01
LoRE-Alt	92.11 $\pm$ 0.18	65.58 $\pm$ 0.56	40.23 $\pm$ 0.00	34.63 $\pm$ 0.05	67.85 $\pm$ 0.01

Appendix

Table 10: ROUGE scores for Qwen2.5-0.5B-Instruct (mean

\pm

std across seeds).

Method	RM Acc	Policy Acc	ROUGE-1	ROUGE-L	BertScore-F1
Reward Models + ARGS
GenARM	90.04 $\pm$ 0.69	76.36 $\pm$ 8.58	42.28 $\pm$ 0.95	36.22 $\pm$ 1.08	69.59 $\pm$ 0.36
Global RM	92.24 $\pm$ 0.25	64.84 $\pm$ 0.04	44.41 $\pm$ 0.00	38.64 $\pm$ 0.02	69.76 $\pm$ 0.00
Global RM-v2	92.26 $\pm$ 0.23	84.90 $\pm$ 1.88	31.92 $\pm$ 0.32	26.86 $\pm$ 0.11	67.64 $\pm$ 0.36
LoRE	91.97 $\pm$ 0.10	64.97 $\pm$ 0.25	44.37 $\pm$ 0.01	38.64 $\pm$ 0.03	69.75 $\pm$ 0.01
LoRE-Alt	92.11 $\pm$ 0.18	65.30 $\pm$ 0.42	44.38 $\pm$ 0.02	38.66 $\pm$ 0.05	69.76 $\pm$ 0.01

Appendix

Table 11: ROUGE scores for Qwen2.5-1.5B-Instruct (mean

\pm

std across seeds).

Method	RM Acc	Policy Acc	ROUGE-1	ROUGE-L	BertScore-F1
Reward Models + ARGS
GenARM	90.04 $\pm$ 0.69	76.80 $\pm$ 8.19	41.55 $\pm$ 1.56	35.28 $\pm$ 1.66	69.63 $\pm$ 0.27
Global RM	92.24 $\pm$ 0.25	66.96 $\pm$ 0.00	43.62 $\pm$ 0.01	37.80 $\pm$ 0.00	69.95 $\pm$ 0.01
Global RM-v2	92.26 $\pm$ 0.23	84.60 $\pm$ 1.74	34.73 $\pm$ 0.63	28.58 $\pm$ 0.56	66.95 $\pm$ 0.20
LoRE	91.97 $\pm$ 0.10	67.17 $\pm$ 0.36	43.63 $\pm$ 0.01	37.81 $\pm$ 0.03	69.96 $\pm$ 0.00
LoRE-Alt	92.11 $\pm$ 0.18	67.63 $\pm$ 0.63	43.66 $\pm$ 0.05	37.83 $\pm$ 0.05	69.97 $\pm$ 0.03

Appendix

Table 12: ROUGE scores for Qwen2.5-3B-Instruct (mean

\pm

std across seeds).

Method	RM Acc	Policy Acc	ROUGE-1	ROUGE-L	BertScore-F1
Reward Models + ARGS
GenARM	90.04 $\pm$ 0.69	77.55 $\pm$ 7.57	43.28 $\pm$ 0.94	37.01 $\pm$ 1.10	69.85 $\pm$ 0.50
Global RM	92.24 $\pm$ 0.25	69.54 $\pm$ 0.00	44.81 $\pm$ 0.01	38.87 $\pm$ 0.02	70.31 $\pm$ 0.00
Global RM-v2	92.26 $\pm$ 0.23	85.08 $\pm$ 1.67	35.58 $\pm$ 1.64	30.38 $\pm$ 1.29	67.26 $\pm$ 0.66
LoRE	91.97 $\pm$ 0.10	69.73 $\pm$ 0.33	44.83 $\pm$ 0.02	38.89 $\pm$ 0.01	70.31 $\pm$ 0.01
LoRE-Alt	92.11 $\pm$ 0.18	69.90 $\pm$ 0.36	44.83 $\pm$ 0.04	38.88 $\pm$ 0.00	70.32 $\pm$ 0.01

Appendix

Table 13: ROUGE scores for Qwen2.5-7B-Instruct (mean

\pm

std across seeds).

Retrieval	Format	3B	7B
Random	Preference pairs	44.75 $\pm$ 0.29	46.61 $\pm$ 0.33
Random	Chosen only	45.13 $\pm$ 0.14	46.50 $\pm$ 0.14
RAG	Preference pairs	46.15 $\pm$ 0.03	48.10 $\pm$ 0.01
RAG	Chosen only	45.62 $\pm$ 0.02	47.26 $\pm$ 0.02

Appendix

Table 14: ROUGE-1 on LaMP-5 for the demonstration-format ablation at

k=8

. Preference pairs shows each demonstration with both the preferred and the dispreferred response; chosen only shows the preferred response alone. Each entry averages three repetitions; standard deviations are at most

0.003

for random selection and at most

0.0003

for retrieval. Best per column in bold.

Personalized Language Model Alignment

Personalized language model alignment adapts model behavior to individual preferences while preserving safety, accuracy, and other shared constraints.

See all 61 papers

Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
Apr 8, 2026Qiyao Ma, Dechen Gao, Rui Cai +4Personalized Reward Modeling Reward Model Evaluation
University of California, Davis
Rethinking Personalized Generation: Test-Time Alignment via Factorized Ranking Models
Sep 28, 2026Qiyao Ma, Junshan Zhang, Zhe ZhaoLearning to Rank LLM Reranking
University of California, Davis
RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences
May 3, 2026Yangyang Zhou, Yi-Chen LiReward Modeling LLM Evaluation
Beijing University of Posts and Telecommunications · National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China

Apr 8, 2026J/K move · Enter open · S save

Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization

Qiyao Ma, Dechen Gao, Rui Cai, Boqi Zhao+3

University of California, Davis

Personalized Reward Modeling Reward Model Evaluation User Preference Modeling Pairwise Preference Evaluation

Open paper AskPDF arXiv

Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values. While benchmarks for general response quality are prevalent, evaluating how well reward models account for individual user preferences remains an open challenge. To bridge this gap, we introduce Personalized RewardBench, a novel benchmark designed to rigorously assess reward models' capacity to model personalized preferences. We construct chosen and rejected response pairs based on strict adherence to (or violation of) user-specific rubrics, ensuring that preference distinctions are uniquely tailored to the individual. In particular, human evaluations confirm that the primary discriminative factor between pairs is strictly personal preference, with both responses maintaining high general quality (e.g., correctness, relevance and helpfulness). Extensive testing reveals that existing state-of-the-art reward models struggle significantly with personalization, peaking at an accuracy of just 75.94%. Crucially, because an effective reward model benchmark should predict a reward model's performance on downstream tasks, we conduct experiments demonstrating that our benchmark exhibits a significantly higher correlation with downstream performance in both Best-of-N (BoN) sampling and Proximal Policy Optimization (PPO) compared to existing baselines. These findings establish Personalized RewardBench as a robust and accurate proxy for evaluating reward models' performance in downstream applications.

Abstract

Personalized Language Model Alignment

Explore similar work

Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization

Rethinking Personalized Generation: Test-Time Alignment via Factorized Ranking Models

RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences