Personalized reward models require user-specific preference comparisons, but collecting such feedback is costly and provides only limited supervision for each user. At the same time, large pools of responses can be generated without user identities or preference labels, yet existing personalized reward models leave this data unused. Directly learning from these responses is difficult because pairwise objectives constrain relative scores within a prompt, whereas pseudo-labeling individual responses requires scores that are comparable across prompts for the same user. We propose PerSPL (Personalized Semi-supervised Preference Learning), a backbone-agnostic framework that uses pair-mean score centering to stabilize this cross-prompt score coordinate and iteratively turns unlabeled responses into user-specific training signals. Against seven representative baselines across four benchmarks---PersonalLLM, HelpSteer2, Reddit TLDR, and Chatbot Arena---PerSPL achieves the highest reported mean accuracy among the evaluated methods for both seen and unseen users. The two latter benchmarks provide real-user preference comparisons in summarization and open-domain chatbot interaction, respectively. These results indicate that unlabeled responses can provide additional supervision for personalized reward modeling when per-user preference labels are scarce.
Figures & tables
Figure 1: Overview of PerSPL . A small set of labeled comparisons identifies each user’s preferences, while pair-mean score centering makes reward scores more comparable across prompts. A calibrator fitted separately for each user assigns pseudo-labels to confident responses from a shared unpaired pool and discards ambiguous ones. The reward model is then trained iteratively on both labeled and pseudo-labeled data.
PersonalLLM
HelpSteer2
Reddit TLDR
Chatbot Arena
αdiv=0.1
αdiv=0.1
100 examples/seen user
150 examples/seen user
25/12 labels
Method
Seen
Unseen
Seen
Unseen
Seen
Unseen
Seen
Unseen
Seen
Unseen
Skywork V2
81.6 ± 1.6
82.4 ± 1.4
67.0 ± 0.3
67.4 ± 0.3
64.5 ± 0.8
64.6 ± 0.8
64.5 ± 0.8
64.6 ± 0.8
80.3 ± 1.4
76.9 ± 1.4
BT
88.8 ± 3.2
89.0 ± 3.8
63.0 ± 3.7
63.2 ± 3.8
63.2 ± 4.8
64.3 ± 3.8
64.4 ± 3.5
65.2 ± 3.1
80.3 ± 2.0
78.1 ± 3.0
VPL
89.6 ± 1.2
89.8 ± 1.2
59.6 ± 0.7
59.6 ± 0.6
57.0 ± 0.8
56.9 ± 1.3
57.6 ± 1.5
57.0 ± 1.6
64.9 ± 3.1
61.9 ± 6.3
GPO
89.1 ± 6.7
89.6 ± 6.8
65.8 ± 2.6
66.4 ± 2.3
63.8 ± 2.5
64.6 ± 2.7
65.7 ± 1.7
65.9 ± 1.3
79.4 ± 1.7
77.7 ± 1.4
Table 1: Full baseline comparison on PersonalLLM, HelpSteer2, Reddit TLDR, and Chatbot Arena. We report pairwise preference accuracy (%) on seen and unseen users as mean ± std over five matched seeds. PersonalLLM and HelpSteer2 use αdiv=0.1 ; Reddit TLDR uses 100 or 150 labeled examples per seen user; Chatbot Arena uses 25/12 labeled pairs per seen/unseen user. On Chatbot Arena, VPL/PAL use their released pooled-pair evaluators; the remaining methods macro-average user accuracies. Best per column is in bold.
HelpSteer2
Reddit TLDR
αdiv=0.1
αdiv=0.01
100 examples/seen user
150 examples/seen user
Objective
Seen
Unseen
Seen
Unseen
Seen
Unseen
Seen
Unseen
PerSPL ( Llabel+Lunlabel+Lreg )
71.6 ± 0.3
71.2 ± 0.8
66.0 ± 0.4
65.0 ± 0.7
70.7 ± 1.2
70.1 ± 0.6
70.9 ± 0.9
70.3 ± 0.7
Llabel+Lunlabel ( λr=0 )
66.5 ± 0.5
69.7 ± 0.9
62.5 ± 1.2
63.2 ± 0.5
63.7 ± 1.1
68.6 ± 0.7
62.5 ± 2.1
62.5 ± 1.4
Llabel+Lreg ( K=1 )
70.9 ± 0.6
71.0 ± 0.5
65.5 ± 0.4
64.6 ± 0.8
69.4 ± 1.4
69.5 ± 1.0
69.1 ± 1.0
70.3 ± 0.7
Llabel only ( λr=0,K=1 )
70.9 ± 0.6
71.1 ± 0.7
65.4 ± 0.3
64.5 ± 0.9
69.5 ± 1.3
69.7 ± 1.1
69.0 ± 1.3
69.6 ± 0.7
Table 2: Component ablations on HelpSteer2 ( αdiv∈{0.1,0.01} ) and Reddit TLDR ( 100 or 150 labeled examples per seen user), forming a regularization × unpaired-training factorial comparison. Each variant changes only λr , K , or both, in both training phases; all other hyperparameters equal those of the full model. We report seen and unseen accuracy (%), mean ± std over five seeds. Best per column in bold.
Figure 2: Reward distributions of chosen and rejected responses on PersonalLLM ( αdiv=0.01 ), pooled over all 100 seen users and evaluated on their held-out test pairs ( 1,000 per user). Both panels use a labeled-only reward head trained identically except for pair-mean regularization ( λr=0 vs. λr=0.01 ). Without regularization, the chosen and rejected distributions overlap substantially. With λr=0.01 , they are centered near zero and show less overlap. Appendix B provides the corresponding score-space analysis and regularization-path diagnostics.
Figure 3: Performance under preference noise and varying label budgets on HelpSteer2 ( αdiv=0.01 ). PerSPL is compared with the strongest baseline, MRM. (a) Test accuracy as the label flip ratio increases. (b) Test accuracy with 50 – 200 labeled pairs per seen user. (c) Test accuracy with 10 – 50 labeled pairs per unseen user while fixing the seen-user budget at 50 . Shaded bands show ±1 standard deviation over five seeds.
OOD fraction ρ
PerSPL
Gain over labeled-only [95% CI]
0
84.70±1.09
+3.21 [2.58, 3.82]
0.25
84.55±1.02
+3.07 [2.31, 3.73]
0.50
83.63±1.12
+2.14 [1.32, 2.90]
0.75
82.71±0.89
+1.22 [0.57, 1.84]
1
82.15±0.67
+0.66 [0.10, 1.18]
Table 3: Controlled generator shift on PersonalLLM with a 0.2 symmetric label-flip rate (overall accuracy, ten matched seeds). The matched regularized labeled-only reference is 81.49±0.79 .
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Preference source
Users (seen/unseen)
Main setting
Labeled pairs (seen/unseen)
∣DU∣
PersonalLLM
reward-model scores
100/100
αdiv=0.1
45/9
5,000
HelpSteer2
human attribute ratings
50/50
αdiv=0.1
50/10
50,000
Reddit TLDR
human comparisons
20/20
100 or 150 labels
100 or 150/50
149,632
Chatbot Arena
human comparisons
19/19
active users
25/12
3,166
Appendix
Table 4: Dataset and evaluation settings for the primary results in Table 1 . Label budgets distinguish the histories of seen users from the support sets used to adapt unseen users.
Setting
B
η
α
τ
PersonalLLM ( αdiv=0.1 )
2
0.03
0.6
0.9
HelpSteer2 ( αdiv=0.1 )
5
0.01
0.7
0.6
Reddit TLDR (100/seen)
2
0.03
0.7
0.8
Reddit TLDR (150/seen)
2
0.03
0.7
0.8
Chatbot Arena (25/12)
2
0.3
0.8
0.8
Appendix
Table 5: PerSPL hyperparameters for the main results in Table 1 . Here B is the basis rank, η is the peak learning rate, α is the labeled-loss weight in Eq. ( 7 ), and τ is the pseudo-label confidence threshold. Parameters shared by all rows are given in the text.
Figure 4: Direct visualization of Theorem 3 along the population regularization path. On log–log axes, the two nonaligned settings have a locally linear single-response accuracy gain (left; slope 1 ), while the paired BT excess risk is locally quadratic (right; slope 2 ). The direction-preserving control has exactly zero accuracy gain and is therefore shown only in the risk panel. Here λ⋆ is the coefficient whose population direction is single-response optimal; only the small- λ neighborhood covered by the theorem is plotted.
Dataset
λr
Primary alignment
Disjoint alignment
Var(m) reduction per unit λr
HelpSteer2
10−8
402.3±115.7
393.6±108.0
4265.8±1339.4
HelpSteer2
10−7
396.1±115.6
387.1±108.1
4203.4±1345.2
Reddit TL;DR
10−8
178.5±136.9
184.7±115.9
1821.3±1384.0
Reddit TL;DR
10−7
173.7±137.6
179.8±118.2
1782.5±1390.6
Appendix
Table 6: Matched one-sided differences at λr=0 for the actual fixed-budget Adam training path. The primary and disjoint columns estimate AK,Adam0+ on two held-out splits; the final column reports the reduction in held-out within-user pair-mean variance per unit λr . Entries are means ± paired 95% t intervals over five seeds. Positive values denote improvement.
Figure 5: Real-data regularization paths for labeled-only reward heads. Panels (a) and (b) report the reductions in held-out cross-prompt logistic loss and paired BT loss, respectively, relative to the matched λr=0 model. Solid lines show means and shaded regions show paired 95% t intervals over five seeds. Positive values denote lower loss.
αdiv=0.001
αdiv=0.01
αdiv=0.1
Method
Seen
Unseen
Overall
Seen
Unseen
Overall
Seen
Unseen
Overall
Skywork-Reward V2
76.9 ± 1.1
76.8 ± 2.6
76.8 ± 1.7
77.8 ± 2.4
76.1 ± 1.9
77.0 ± 1.7
81.6 ± 1.6
82.4 ± 1.4
82.0 ± 1.2
BT
81.7 ± 3.8
81.8 ± 4.0
81.7 ± 3.8
82.5 ± 4.5
81.6 ± 4.2
82.0 ± 4.2
88.8 ± 3.2
89.0 ± 3.8
88.9 ± 3.4
VPL
82.8 ± 1.7
82.6 ± 0.9
82.7 ± 1.2
83.6 ± 2.1
82.9 ± 2.0
83.3 ± 1.8
89.6 ± 1.2
89.8 ± 1.2
89.7 ± 1.1
GPO
85.4 ± 2.2
84.3 ± 2.9
84.9 ± 2.5
79.4 ± 16.1
77.6 ± 15.1
78.5 ± 15.6
89.1 ± 6.7
89.6 ± 6.8
89.4 ± 6.7
PAL
81.0 ± 0.8
80.5 ± 0.7
80.8 ± 0.6
80.5 ± 2.8
79.7 ± 2.0
80.1 ± 1.9
84.7 ± 1.4
85.2 ± 1.5
85.0 ± 1.1
Appendix
Table 7: Full PersonalLLM results across αdiv∈{0.001,0.01,0.1} ( 45 pairs/seen user). Mean ± std over five seeds.
αdiv=0.01
αdiv=0.1
Method
Seen
Unseen
Overall
Seen
Unseen
Overall
Skywork-Reward V2
62.3 ± 0.6
62.6 ± 0.2
62.5 ± 0.3
67.0 ± 0.3
67.4 ± 0.3
67.2 ± 0.1
BT
53.1 ± 4.4
53.0 ± 4.5
53.1 ± 4.4
63.0 ± 3.7
63.2 ± 3.8
63.1 ± 3.7
VPL
55.4 ± 1.5
55.3 ± 1.3
55.4 ± 1.4
59.6 ± 0.7
59.6 ± 0.6
59.6 ± 0.6
GPO
60.3 ± 2.1
59.8 ± 2.0
60.1 ± 2.0
65.8 ± 2.6
66.4 ± 2.3
66.1 ± 2.4
PAL
54.7 ± 1.2
54.3 ± 0.8
54.5 ± 1.0
57.1 ± 1.4
56.9 ± 1.0
57.0 ± 1.2
Appendix
Table 8: Full HelpSteer2 results across αdiv∈{0.01,0.1} ( 50 pairs/seen user). Mean ± std over five seeds.
100 examples/seen user
150 examples/seen user
Method
Seen
Unseen
Overall
Seen
Unseen
Overall
Skywork-Reward V1
62.6 ± 0.5
62.6 ± 0.9
62.6 ± 0.9
62.6 ± 0.5
62.6 ± 0.9
62.6 ± 0.9
Skywork-Reward V2
64.5 ± 0.8
64.6 ± 0.8
64.5 ± 0.0
64.5 ± 0.8
64.6 ± 0.8
64.5 ± 0.0
BT
63.2 ± 4.8
64.3 ± 3.8
63.8 ± 4.2
64.4 ± 3.5
65.2 ± 3.1
64.8 ± 3.3
VPL
57.0 ± 0.8
56.9 ± 1.3
56.9 ± 0.5
57.6 ± 1.5
57.0 ± 1.6
57.3 ± 0.7
GPO
63.8 ± 2.5
64.6 ± 2.7
64.2 ± 2.6
65.7 ± 1.7
65.9 ± 1.3
65.8 ± 1.4
Appendix
Table 9: Full Reddit TLDR results at 100 and 150 labeled examples/seen user. Mean ± std over five seeds. SynthesizeMe variants are inference-time-personalization methods (no in-domain training); MRM is reported with both Skywork-Reward V1 and V2 backbones.
Method
Seen
Unseen
Overall
Skywork-Reward V2
80.33±1.40
76.89±1.40
78.61±0.00
BT
80.25±2.02
78.12±2.95
79.19±1.12
VPL
64.93±3.09
61.93±6.26
63.43±2.23
GPO
79.44±1.66
77.68±1.39
78.56±1.12
PAL
58.40±2.98
58.36±3.04
58.38±1.50
LoRe
81.27±2.08
78.38±2.05
79.83±1.00
Appendix
Table 10: Full Chatbot Arena results. We report pairwise accuracy (%, mean ± std over five matched seeds); Overall is the per-seed average of seen and unseen accuracy. VPL/PAL use their released pooled-pair evaluators; the other methods macro-average users. All five prespecified seeds are retained.
Setting
Seen
Unseen
Overall
HelpSteer2 αdiv=0.1
+0.91 [0.62, 1.21]
+0.35 [0.04, 0.65]
+0.62 [0.34, 0.93]
HelpSteer2 αdiv=0.01
+1.22 [0.57, 2.01]
+0.37 [ −0.43 , 1.18]
+0.81 [0.11, 1.62]
Reddit, 100/seen
+0.78 [0.28, 1.35]
+0.38 [0.15, 0.60]
+0.56 [0.39, 0.76]
Reddit, 150/seen
+0.91 [0.31, 1.51]
+0.74 [0.40, 1.08]
+0.81 [0.48, 1.05]
Chatbot Arena, 25/12
+1.72 [0.14, 3.45]
+2.66 [1.13, 4.28]
+2.19 [0.79, 3.20]
Appendix
Table 11: PerSPL minus the baseline with the highest overall mean in each setting, measured in accuracy percentage points with descriptive paired-bootstrap 95% confidence intervals over five matched seeds.
Setting
Seen
Unseen
Labeled only
80.31±1.37
82.66±0.67
ρ=0
84.56±1.32
84.83±1.07
ρ=0.25
84.44±1.14
84.66±1.14
ρ=0.50
83.55±1.15
83.71±1.38
ρ=0.75
82.58±1.00
82.83±1.19
ρ=1
81.85±0.86
82.45±0.84
Appendix
Table 12: Seen- and unseen-user breakdown for the controlled generator-shift study in Table 3 . Overall results and paired gains are reported in the main table.
Figure 6: Hyperparameter sensitivity of PerSPL on HelpSteer2 ( αdiv=0.01 ). Stars mark the main configuration ( τ=0.6 , η=0.01 , α=0.7 ). Overall accuracy changes little across the tested τ and α values and is stable near the main learning rate.
Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values. While benchmarks for general response quality are prevalent, evaluating how well reward models account for individual user preferences remains an open challenge. To bridge this gap, we introduce Personalized RewardBench, a novel benchmark designed to rigorously assess reward models' capacity to model personalized preferences. We construct chosen and rejected response pairs based on strict adherence to (or violation of) user-specific rubrics, ensuring that preference distinctions are uniquely tailored to the individual. In particular, human evaluations confirm that the primary discriminative factor between pairs is strictly personal preference, with both responses maintaining high general quality (e.g., correctness, relevance and helpfulness). Extensive testing reveals that existing state-of-the-art reward models struggle significantly with personalization, peaking at an accuracy of just 75.94%. Crucially, because an effective reward model benchmark should predict a reward model's performance on downstream tasks, we conduct experiments demonstrating that our benchmark exhibits a significantly higher correlation with downstream performance in both Best-of-N (BoN) sampling and Proximal Policy Optimization (PPO) compared to existing baselines. These findings establish Personalized RewardBench as a robust and accurate proxy for evaluating reward models' performance in downstream applications.
Large language models (LLMs) are typically aligned with population-level preferences, despite substantial variation across individual users. We introduce POPI, a user-level personalization framework that separates the problem into two components connected by a natural-language interface: a shared inference model that distills heterogeneous user signals into a concise preference summary, and a shared generator that conditions on this summary to produce personalized responses. Both components are trained under a unified preference-optimization objective, with reinforcement learning handling the non-differentiable inference step. This objective decomposes into generator approximation error and summary informativeness, revealing how a single loss simultaneously drives accurate generation and informative summarization. Because the interface is natural language, learned summaries can be inferred once per user and reused across different generators -- including frozen, black-box commercial APIs. Across four personalization benchmarks, POPI generally improves personalization quality while reducing context overhead by up to an order of magnitude.
Yizhuo Chen, Xin Liu, Ruijie Wang +7
University of Illinois Urbana-Champaign · Amazon · University of Notre Dame
Large language models can answer the same question in different styles, but users' preferred styles are hidden and vary across users and questions. We study inference-time personalization through online selection from a fixed set of prompts, each representing a persona. Users arrive sequentially, each asks a short sequence of questions, and the system displays two responses per question and observes one pairwise choice. We formulate this interaction as a variant of the contextual dueling bandit problem in which the learner starts without preference data, adapts within each session, and exploits information shared across users. The objective is to minimize cumulative strong regret, summing the utility gaps of both displayed responses relative to the user's best response among the available options. We make two contributions. First, we introduce the problem and release a reproducible preference benchmark and dataset built around a tutoring case study, containing educational questions, LLM-generated answers in six teaching styles, and pairwise judgments from LLM-simulated students with hidden preference profiles. We fit Bradley-Terry models to these judgments to obtain reference utilities for regret evaluation. Second, we propose a baseline that learns a mixture prior over users from completed sessions and adapts to each new user within the session. On students and topics never used in development, it removes 44-48% of random pairs' excess regret and achieves over half the performance of a per-user linear oracle fitted in hindsight. We release code, prompts, and the benchmark.