The Reward Model Selection Crisis in Personalized Alignment

Fady Rezk; Yuangang Pan; Chuan-Sheng Foo; Xun Xu; Nancy Chen; Henry Gouk; Timothy Hospedales

cs.AIDec 28, 2025

The Reward Model Selection Crisis in Personalized Alignment

Authors: Fady Rezk, Yuangang Pan, Chuan-Sheng Foo, Xun Xu, Nancy Chen, Henry Gouk, Timothy Hospedales

Organizations: School of Informatics, University of Edinburgh, UK · Agency for Science, Technology and Research (A*STAR), Singapore · Samsung AI Research Center, Cambridge, UK

Abstract

Personalized alignment from preference data has focused primarily on improving personal reward model (RM) accuracy, with the implicit assumption that better preference ranking translates to better personalized behavior. However, in deployment, computational constraints necessitate inference-time adaptation such as reward-guided decoding (RGD) rather than per-user policy fine-tuning. This creates a critical but overlooked requirement: reward models must not only rank preferences accurately but also effectively guide generation. We demonstrate that standard RM accuracy fails catastrophically as a selection criterion for deployment-ready personalized rewards. We introduce policy accuracy; a metric quantifying whether RGD-adapted LLMs correctly discriminate between preferred and dispreferred responses and show that upstream RM accuracy correlates only weakly with downstream policy accuracy (Kendall's tau = 0.08--0.31). More critically, we introduce Pref-LaMP the first personalized alignment benchmark with ground-truth user completions, enabling direct behavioural evaluation. On Pref-LaMP, we expose a complete decoupling between discriminative ranking and generation metrics: methods with 20-point RM accuracy differences produce almost identical output quality, and methods with high ranking accuracy can fail to generate behaviorally aligned responses. These findings reveal that the field has been optimizing for proxy metrics that do not predict deployment performance, and that current personalized alignment methods fail to operationalize preferences into behavioral adaptation under realistic deployment constraints. In contrast, we find simple in-context learning (ICL) to be highly effective - dominating all reward-guided methods for models $\geq$ 3B parameters, achieving $\sim$ 3 point ROUGE-1 gains over the best reward method at 7B scale.

Figures & tables

Negative	R-1	R-L	BERTScore
Mined	24.6	22.2	60.0
Random	2 4.7	2 4.4	43.7
$\Delta$	+19.9	+17.8	+16.3

Table 1: Similarity of rejected titles to the ground-truth chosen title on the Pref-LaMP evaluation split, under hard-negative mining versus a random permutation of titles within the split. The training split agrees to within

0.01

on every metric.

Dataset	Reward Model	Base Policies
TLDR	SmolLM2-180M	180M, 360M, 1.7B
PRISM	Qwen2.5-0.5B	0.5B, 1.5B, 3B, 7B
Pref-LaMP	Qwen2.5-0.5B	0.5B, 1.5B, 3B, 7B

Table 2: Model configurations.

			Policy Accuracy
	Method	RM Accuracy	135M	360M	1.7B
Personalized	LoRE	82.56 $\pm$ 1.12	81.09 $\pm$ 0.76	81.09 $\pm$ 0.89	81.01 $\pm$ 1.10
	MPU	76.76 $\pm$ 0.26	49.96 $\pm$ 3.24	54.36 $\pm$ 7.36	55.12 $\pm$ 8.07
	MPU-Avg	77.83 $\pm$ 0.19	52.91 $\pm$ 4.01	52.96 $\pm$ 3.97	52.94 $\pm$ 3.98
	P-DPO	-	-	-	-
	PAL	87.00 $\pm$ 0.05	-	-	-

Table 3: Policy accuracy and reward model performance on TLDR + SmolLM. Base reward model is SmolLM2-135M. Top: Personal rewards. Mid: Global alignment.

	Personalized							Global
	LoRE	LoRE-Alt	MPU	MPU-Avg	PAL	PReF	VPL	Global RM	Global RM-v2
RM Acc.	$65.95^{\pm 4.57}$	$67.15^{\pm 0.42}$	$52.76^{\pm 5.40}$	$60.43^{\pm 1.25}$	$70.74^{\pm 0.42}$	$74.34^{\pm 2.20}$	$68.35^{\pm 2.59}$	$\textbf{77.94}^{\pm 3.55}$	$62.11^{\pm 4.15}$

Table 4: Reward model accuracies of various personal alignment methods using Qwen2.5-0.5B backbone on PRISM dataset. All personal alignment methods underperform non-personal Global Reward Model V1.

Figure 1: RM vs Policy Accuracy Correlation on PRISM across model scales. Correlations remain consistently weak: Pearson

r

ranges from 0.30–0.48, Spearman

\rho

from 0.25–0.43, and Kendall

\tau

from 0.17–0.29. While correlations slightly strengthen with scale, they remain far below what would be needed for RM accuracy to reliably predict policy performance. Notably, methods with similar RM accuracy can have substantially different policy accuracy and vice-versa, demonstrating that the standard RM metric is not a reliable proxy for deployment performance.

	Global			Personalized
Acc	GenARM	Global RM	Global RM-v2	LoRE	LoRE-Alt	MPU	MPU-Avg	PAL	PReF	VPL	ICL
RM	$90.04^{\pm 0.69}$	$92.24^{\pm 0.25}$	$92.26^{\pm 0.23}$	$91.97^{\pm 0.10}$	$92.11^{\pm 0.18}$	$86.15^{\pm 0.39}$	$88.02^{\pm 0.40}$	$53.77^{\pm 0.28}$	$63.79^{\pm 0.88}$	$39.65^{\pm 17.76}$	NA
Policy	$76.80^{\pm 8.19}$	$66.96^{\pm 0.00}$	$84.60^{\pm 1.74}$	$67.17^{\pm 0.36}$	$67.63^{\pm 0.63}$	$67.04^{\pm 0.20}$	$64.32^{\pm 4.63}$	–	$67.13^{\pm 0.29}$	$67.19^{\pm 0.40}$	$80.20^{\pm 0.69}$

Table 5: Pref-LaMP preference ranking accuracy for RMs and adapted policies (Qwen2.5-3B). RM personalization does not clearly outperform global RMs for either upstream RM or downstream policy preference ranking.

Figure 2: Generation quality (ROUGE-1, ROUGE-L, BERTScore-F1) under RGD across model scales. At 0.5B-1.5B, personalized RMs marginally improve over zero-shot; at 3B-7B, ICL baselines dominate all reward-guided methods.

	Method	0.5B	1.5B	3B	7B
Global	GenARM	100.0%	100.0%	100.0%	99.2%
	Global RM	92.4%	92.1%	91.6%	92.1%
	Global RM-v2	99.0%	98.0%	92.4%	90.8%
Personalized	LoRE	92.5%	91.7%	92.2%	92.8%
	LoRE-Alt	95.0%	94.5%	94.7%	95.5%
	MPU	84.3%	84.2%	85.0%	85.7%

Table 6: Win rate: fraction of examples where each RM judges its own RGD output as better than zero-shot. High win rates reveal severe reward hacking—methods claim near-perfect improvement despite ground-truth metrics showing minimal or negative gains (Fig 2).

Figure 3: Pref-LAMP generation quality. ROUGE-1 vs. number of in-context examples. ICL-RAG consistently outperforms random ICL, with performance scaling with both model size and context length.

Scale	RM Acc	Policy Acc	Win Rate
Kendall’s $\tau$ with ROUGE-1
0.5B	$-0.262$	$-0.266$	$-0.286$
1.5B	$-0.118$	$-0.347$	$-0.269$
3B	$-0.118$	$-0.294$	$-0.149$
7B	$-0.233$	$-0.308$	$-0.262$

Table 7: Correlations between metrics and generation quality (ROUGE-1) on Pref-LaMP. All correlations are negligible to negative across model scales, demonstrating no standard metric predicts behavioral alignment.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	LoRE	PReF	VPL	Global RM	Global RM-v2	MPU	MPU-Avg	LoRE-Alt
RM Accuracy	65.95 $\pm$ 4.57	74.34 $\pm$ 2.20	68.35 $\pm$ 2.59	77.94 $\pm$ 3.55	62.11 $\pm$ 4.15	52.76 $\pm$ 5.40	60.43 $\pm$ 1.25	67.15 $\pm$ 0.42
Policy accuracies
0.5B	56.83 $\pm$ 0.00	57.31 $\pm$ 1.66	66.43 $\pm$ 3.24	64.51 $\pm$ 0.83	62.59 $\pm$ 5.04	59.47 $\pm$ 2.91	57.55 $\pm$ 3.81	58.51 $\pm$ 0.83
1.5B	57.07 $\pm$ 0.42	58.99 $\pm$ 1.90	65.71 $\pm$ 1.81	64.51 $\pm$ 0.83	62.35 $\pm$ 4.63	60.43 $\pm$ 2.88	58.27 $\pm$ 2.49	57.55 $\pm$ 0.72
3B	57.07 $\pm$ 0.42	57.31 $\pm$ 0.42	64.03 $\pm$ 3.81	64.51 $\pm$ 0.83	61.39 $\pm$ 3.69	57.55 $\pm$ 2.88	56.12 $\pm$ 2.59	56.59 $\pm$ 1.81
7B	57.55 $\pm$ 0.00	59.23 $\pm$ 2.53	63.79 $\pm$ 2.31	64.51 $\pm$ 0.83	61.87 $\pm$ 3.81	58.03 $\pm$ 2.91	56.35 $\pm$ 2.53	57.07 $\pm$ 0.83

Appendix

Table 8: Reward model accuracy and policy accuracies per model and method for the PRISM dataset.

	LoRE	LoRE-Alt	MPU	MPU-Avg	PAL	PReF	VPL	Global RM	Global RM2
RM Accuracy	$56.53$ $\pm 0.15$	$60.33$ $\pm 0.57$	$49.66$ $\pm 1.13$	$49.07$ $\pm 1.46$	$62.51$ $\pm 0.30$	$62.73$ $\pm 0.64$	$60.31$ $\pm 2.98$	$64.51$ $\pm 0.41$	$59.18$ $\pm 0.74$

Appendix

Table 9: Test set accuracy evaluated across all samples from unseen users. Note: The results reported in the main text use only a single test sample per user due to computational constraints.

Method	RM Acc	Policy Acc	ROUGE-1	ROUGE-L	BertScore-F1
Reward Models + ARGS
GenARM	90.04 $\pm$ 0.69	76.99 $\pm$ 8.05	39.84 $\pm$ 1.06	34.02 $\pm$ 1.19	68.55 $\pm$ 0.52
Global RM	92.24 $\pm$ 0.25	64.95 $\pm$ 0.00	40.27 $\pm$ 0.03	34.66 $\pm$ 0.02	67.87 $\pm$ 0.00
Global RM-v2	92.26 $\pm$ 0.23	85.44 $\pm$ 1.67	30.64 $\pm$ 0.26	25.72 $\pm$ 0.06	66.93 $\pm$ 0.42
LoRE	91.97 $\pm$ 0.10	65.01 $\pm$ 0.17	40.23 $\pm$ 0.03	34.63 $\pm$ 0.05	67.87 $\pm$ 0.01
LoRE-Alt	92.11 $\pm$ 0.18	65.58 $\pm$ 0.56	40.23 $\pm$ 0.00	34.63 $\pm$ 0.05	67.85 $\pm$ 0.01

Appendix

Table 10: ROUGE scores for Qwen2.5-0.5B-Instruct (mean

\pm

std across seeds).

Method	RM Acc	Policy Acc	ROUGE-1	ROUGE-L	BertScore-F1
Reward Models + ARGS
GenARM	90.04 $\pm$ 0.69	76.36 $\pm$ 8.58	42.28 $\pm$ 0.95	36.22 $\pm$ 1.08	69.59 $\pm$ 0.36
Global RM	92.24 $\pm$ 0.25	64.84 $\pm$ 0.04	44.41 $\pm$ 0.00	38.64 $\pm$ 0.02	69.76 $\pm$ 0.00
Global RM-v2	92.26 $\pm$ 0.23	84.90 $\pm$ 1.88	31.92 $\pm$ 0.32	26.86 $\pm$ 0.11	67.64 $\pm$ 0.36
LoRE	91.97 $\pm$ 0.10	64.97 $\pm$ 0.25	44.37 $\pm$ 0.01	38.64 $\pm$ 0.03	69.75 $\pm$ 0.01
LoRE-Alt	92.11 $\pm$ 0.18	65.30 $\pm$ 0.42	44.38 $\pm$ 0.02	38.66 $\pm$ 0.05	69.76 $\pm$ 0.01

Appendix

Table 11: ROUGE scores for Qwen2.5-1.5B-Instruct (mean

\pm

std across seeds).

Method	RM Acc	Policy Acc	ROUGE-1	ROUGE-L	BertScore-F1
Reward Models + ARGS
GenARM	90.04 $\pm$ 0.69	76.80 $\pm$ 8.19	41.55 $\pm$ 1.56	35.28 $\pm$ 1.66	69.63 $\pm$ 0.27
Global RM	92.24 $\pm$ 0.25	66.96 $\pm$ 0.00	43.62 $\pm$ 0.01	37.80 $\pm$ 0.00	69.95 $\pm$ 0.01
Global RM-v2	92.26 $\pm$ 0.23	84.60 $\pm$ 1.74	34.73 $\pm$ 0.63	28.58 $\pm$ 0.56	66.95 $\pm$ 0.20
LoRE	91.97 $\pm$ 0.10	67.17 $\pm$ 0.36	43.63 $\pm$ 0.01	37.81 $\pm$ 0.03	69.96 $\pm$ 0.00
LoRE-Alt	92.11 $\pm$ 0.18	67.63 $\pm$ 0.63	43.66 $\pm$ 0.05	37.83 $\pm$ 0.05	69.97 $\pm$ 0.03

Appendix

Table 12: ROUGE scores for Qwen2.5-3B-Instruct (mean

\pm

std across seeds).

Method	RM Acc	Policy Acc	ROUGE-1	ROUGE-L	BertScore-F1
Reward Models + ARGS
GenARM	90.04 $\pm$ 0.69	77.55 $\pm$ 7.57	43.28 $\pm$ 0.94	37.01 $\pm$ 1.10	69.85 $\pm$ 0.50
Global RM	92.24 $\pm$ 0.25	69.54 $\pm$ 0.00	44.81 $\pm$ 0.01	38.87 $\pm$ 0.02	70.31 $\pm$ 0.00
Global RM-v2	92.26 $\pm$ 0.23	85.08 $\pm$ 1.67	35.58 $\pm$ 1.64	30.38 $\pm$ 1.29	67.26 $\pm$ 0.66
LoRE	91.97 $\pm$ 0.10	69.73 $\pm$ 0.33	44.83 $\pm$ 0.02	38.89 $\pm$ 0.01	70.31 $\pm$ 0.01
LoRE-Alt	92.11 $\pm$ 0.18	69.90 $\pm$ 0.36	44.83 $\pm$ 0.04	38.88 $\pm$ 0.00	70.32 $\pm$ 0.01

Appendix

Table 13: ROUGE scores for Qwen2.5-7B-Instruct (mean

\pm

std across seeds).

Retrieval	Format	3B	7B
Random	Preference pairs	44.75 $\pm$ 0.29	46.61 $\pm$ 0.33
Random	Chosen only	45.13 $\pm$ 0.14	46.50 $\pm$ 0.14
RAG	Preference pairs	46.15 $\pm$ 0.03	48.10 $\pm$ 0.01
RAG	Chosen only	45.62 $\pm$ 0.02	47.26 $\pm$ 0.02

Appendix

Table 14: ROUGE-1 on LaMP-5 for the demonstration-format ablation at

k=8

. Preference pairs shows each demonstration with both the preferred and the dispreferred response; chosen only shows the preferred response alone. Each entry averages three repetitions; standard deviations are at most

0.003

for random selection and at most

0.0003

for retrieval. Best per column in bold.

Personalized Language Model Alignment

Personalized language model alignment adapts model behavior to individual preferences while preserving safety, accuracy, and other shared constraints.

See all 60 papers

Apr 8, 2026

Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization

Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values. While benchmarks for general response quality are prevalent, evaluating how well reward models account for individual user preferences remains an open challenge. To bridge this gap, we introduce Personalized RewardBench, a novel benchmark designed to rigorously assess reward models' capacity to model personalized preferences. We construct chosen and rejected response pairs based on strict adherence to (or violation of) user-specific rubrics, ensuring that preference distinctions are uniquely tailored to the individual. In particular, human evaluations confirm that the primary discriminative factor between pairs is strictly personal preference, with both responses maintaining high general quality (e.g., correctness, relevance and helpfulness). Extensive testing reveals that existing state-of-the-art reward models struggle significantly with personalization, peaking at an accuracy of just 75.94%. Crucially, because an effective reward model benchmark should predict a reward model's performance on downstream tasks, we conduct experiments demonstrating that our benchmark exhibits a significantly higher correlation with downstream performance in both Best-of-N (BoN) sampling and Proximal Policy Optimization (PPO) compared to existing baselines. These findings establish Personalized RewardBench as a robust and accurate proxy for evaluating reward models' performance in downstream applications.

Qiyao Ma, Dechen Gao, Rui Cai +4

University of California, Davis

Personalized Reward Modeling Reward Model Evaluation User Preference Modeling Pairwise Preference Evaluation

Ask PDF

Sep 28, 2026

Rethinking Personalized Generation: Test-Time Alignment via Factorized Ranking Models

Aligning large language models (LLMs) to diverse user preferences is fundamentally hindered by standard alignment paradigms that optimize for monolithic users. In this work, empirical studies are first used to reveal the existence of a massive, untapped performance headroom for personalized generation through test-time alignment. We demonstrate that personalized generation is uniquely suited for test-time scaling methods like Best-of-N (BoN) because it can be viewed primarily as a candidate matching problem rather than a generator capability bottleneck. While reward models could in principle exploit this headroom, they are poorly calibrated for personalization, and their billion-parameter scale makes scoring large candidate pools prohibitively expensive. To overcome this limitation, we propose a parameter-efficient framework utilizing million-parameter scale multi-layer perceptron (MLP) ranking models. Our personalized ranking model directly reuses the internal embeddings of the base generator with minimal overhead. By scaling train-time data to provide fine-grained personalized preferences, this million-parameter ranking model accurately scores large candidate pools and can seamlessly guide generation to reduce the cost of materializing N candidates. Extensive experiments on nine datasets spanning three personalized generation settings show that our personalized ranking model effectively exploits the discovered headroom, outperforming billion-parameter generalist reward models on every dataset, with under 0.4% of their parameters and four orders of magnitude lower scoring latency.

Qiyao Ma, Junshan Zhang, Zhe Zhao

University of California, Davis

Learning to Rank LLM Reranking Personalized Language Models Inference-Time Language Model Alignment Text Generation

Ask PDF

May 3, 2026

RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences

Reinforcement Learning from Human Feedback has become the standard paradigm for language model alignment, where reward models directly determine alignment effectiveness. In this work, we focus on how to evaluate the generalizability of reward models. By "generalizability", we mean the ability of RMs to correctly rank responses to align with diverse user preferences. However, existing reward model benchmarks are typically designed around a universal preference, failing to assess this generalization. To address this critical gap, we introduce RMGAP, a benchmark comprising 1,097 instances across Chat, Writing, Reasoning, and Safety domains. Since different users exhibit diverse preferences for the same task, we first generate four distinct responses with different linguistic profiles for each collected prompt. However, the original prompt set lacks the specificity to convey different preferences. We therefore construct tailored prompts by contrasting these candidates and designing scenarios in which one response becomes the uniquely appropriate choice. Moreover, we observe that users often express the same preference using different phrasings, and thus extend each prompt with two paraphrased variants. Our evaluation of 24 state-of-the-art RMs reveals their substantial limitations: even the best RM achieves only 49.27% Best-of-N accuracy, highlighting considerable room for improvement in reward model generalization. Related data and code are available at https://github.com/nanzhi84/RMGAP.

Yangyang Zhou, Yi-Chen Li

Beijing University of Posts and Telecommunications · National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China

Reward Modeling LLM Evaluation Preference Modeling Reward Model Evaluation

Ask PDF

Abstract

Personalized Language Model Alignment

Explore similar work

Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization

Rethinking Personalized Generation: Test-Time Alignment via Factorized Ranking Models

RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences