PerSPL: Personalized Semi-supervised Preference Learning

Xiaoan Xu; Tingzhou Wei; Juncheng Dong; Korel Gundem; Ethan X. Fang; Zhengling Qi

cs.LGOct 21, 2026

PerSPL: Personalized Semi-supervised Preference Learning

Authors: Xiaoan Xu, Tingzhou Wei, Juncheng Dong, Korel Gundem, Ethan X. Fang, Zhengling Qi

Organizations: Duke University · George Washington University

Abstract

Personalized reward models require user-specific preference comparisons, but collecting such feedback is costly and provides only limited supervision for each user. At the same time, large pools of responses can be generated without user identities or preference labels, yet existing personalized reward models leave this data unused. Directly learning from these responses is difficult because pairwise objectives constrain relative scores within a prompt, whereas pseudo-labeling individual responses requires scores that are comparable across prompts for the same user. We propose PerSPL (Personalized Semi-supervised Preference Learning), a backbone-agnostic framework that uses pair-mean score centering to stabilize this cross-prompt score coordinate and iteratively turns unlabeled responses into user-specific training signals. Against seven representative baselines across four benchmarks---PersonalLLM, HelpSteer2, Reddit TLDR, and Chatbot Arena---PerSPL achieves the highest reported mean accuracy among the evaluated methods for both seen and unseen users. The two latter benchmarks provide real-user preference comparisons in summarization and open-domain chatbot interaction, respectively. These results indicate that unlabeled responses can provide additional supervision for personalized reward modeling when per-user preference labels are scarce.

Figures & tables

Figure 1: Overview of PerSPL . A small set of labeled comparisons identifies each user’s preferences, while pair-mean score centering makes reward scores more comparable across prompts. A calibrator fitted separately for each user assigns pseudo-labels to confident responses from a shared unpaired pool and discards ambiguous ones. The reward model is then trained iteratively on both labeled and pseudo-labeled data.

	PersonalLLM		HelpSteer2		Reddit TLDR				Chatbot Arena
	$\alpha_{\rm div}{=}0.1$		$\alpha_{\rm div}{=}0.1$		$100$ examples/seen user		$150$ examples/seen user		$25/12$ labels
Method	Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen
Skywork V2	81.6 $\pm$ 1.6	82.4 $\pm$ 1.4	67.0 $\pm$ 0.3	67.4 $\pm$ 0.3	64.5 $\pm$ 0.8	64.6 $\pm$ 0.8	64.5 $\pm$ 0.8	64.6 $\pm$ 0.8	80.3 $\pm$ 1.4	76.9 $\pm$ 1.4
BT	88.8 $\pm$ 3.2	89.0 $\pm$ 3.8	63.0 $\pm$ 3.7	63.2 $\pm$ 3.8	63.2 $\pm$ 4.8	64.3 $\pm$ 3.8	64.4 $\pm$ 3.5	65.2 $\pm$ 3.1	80.3 $\pm$ 2.0	78.1 $\pm$ 3.0
VPL	89.6 $\pm$ 1.2	89.8 $\pm$ 1.2	59.6 $\pm$ 0.7	59.6 $\pm$ 0.6	57.0 $\pm$ 0.8	56.9 $\pm$ 1.3	57.6 $\pm$ 1.5	57.0 $\pm$ 1.6	64.9 $\pm$ 3.1	61.9 $\pm$ 6.3
GPO	89.1 $\pm$ 6.7	89.6 $\pm$ 6.8	65.8 $\pm$ 2.6	66.4 $\pm$ 2.3	63.8 $\pm$ 2.5	64.6 $\pm$ 2.7	65.7 $\pm$ 1.7	65.9 $\pm$ 1.3	79.4 $\pm$ 1.7	77.7 $\pm$ 1.4

Table 1: Full baseline comparison on PersonalLLM, HelpSteer2, Reddit TLDR, and Chatbot Arena. We report pairwise preference accuracy (%) on seen and unseen users as mean

\pm

std over five matched seeds. PersonalLLM and HelpSteer2 use

\alpha_{\rm div}{=}0.1

; Reddit TLDR uses

100

150

labeled examples per seen user; Chatbot Arena uses

25/12

labeled pairs per seen/unseen user. On Chatbot Arena, VPL/PAL use their released pooled-pair evaluators; the remaining methods macro-average user accuracies. Best per column is in bold.

	HelpSteer2				Reddit TLDR
	$\alpha_{\rm div}{=}0.1$		$\alpha_{\rm div}{=}0.01$		$100$ examples/seen user		$150$ examples/seen user
Objective	Seen	Unseen	Seen	Unseen	Seen	Unseen	Seen	Unseen
PerSPL ( $\mathcal{L}_{\rm label}+\mathcal{L}_{\rm unlabel}+\mathcal{L}_{\rm reg}$ )	71.6 $\pm$ 0.3	71.2 $\pm$ 0.8	66.0 $\pm$ 0.4	65.0 $\pm$ 0.7	70.7 $\pm$ 1.2	70.1 $\pm$ 0.6	70.9 $\pm$ 0.9	70.3 $\pm$ 0.7
$\mathcal{L}_{\rm label}+\mathcal{L}_{\rm unlabel}$ ( $\lambda_{r}{=}0$ )	66.5 $\pm$ 0.5	69.7 $\pm$ 0.9	62.5 $\pm$ 1.2	63.2 $\pm$ 0.5	63.7 $\pm$ 1.1	68.6 $\pm$ 0.7	62.5 $\pm$ 2.1	62.5 $\pm$ 1.4
$\mathcal{L}_{\rm label}+\mathcal{L}_{\rm reg}$ ( $K{=}1$ )	70.9 $\pm$ 0.6	71.0 $\pm$ 0.5	65.5 $\pm$ 0.4	64.6 $\pm$ 0.8	69.4 $\pm$ 1.4	69.5 $\pm$ 1.0	69.1 $\pm$ 1.0	70.3 $\pm$ 0.7
$\mathcal{L}_{\rm label}$ only ( $\lambda_{r}{=}0,K{=}1$ )	70.9 $\pm$ 0.6	71.1 $\pm$ 0.7	65.4 $\pm$ 0.3	64.5 $\pm$ 0.9	69.5 $\pm$ 1.3	69.7 $\pm$ 1.1	69.0 $\pm$ 1.3	69.6 $\pm$ 0.7

Table 2: Component ablations on HelpSteer2 (

\alpha_{\rm div}\in\{0.1,0.01\}

) and Reddit TLDR (

100

150

labeled examples per seen user), forming a regularization

\times

unpaired-training factorial comparison. Each variant changes only

\lambda_{r}

K

, or both, in both training phases; all other hyperparameters equal those of the full model. We report seen and unseen accuracy (%), mean

\pm

std over five seeds. Best per column in bold.

Figure 2: Reward distributions of chosen and rejected responses on PersonalLLM (

\alpha_{\rm div}=0.01

), pooled over all

100

seen users and evaluated on their held-out test pairs (

1{,}000

per user). Both panels use a labeled-only reward head trained identically except for pair-mean regularization (

\lambda_{r}=0

vs.

\lambda_{r}=0.01

). Without regularization, the chosen and rejected distributions overlap substantially. With

\lambda_{r}=0.01

, they are centered near zero and show less overlap. Appendix B provides the corresponding score-space analysis and regularization-path diagnostics.

Figure 3: Performance under preference noise and varying label budgets on HelpSteer2 (

\alpha_{\rm div}=0.01

). PerSPL is compared with the strongest baseline, MRM. (a) Test accuracy as the label flip ratio increases. (b) Test accuracy with

50

–

200

labeled pairs per seen user. (c) Test accuracy with

10

–

50

labeled pairs per unseen user while fixing the seen-user budget at

50

. Shaded bands show

\pm 1

standard deviation over five seeds.

OOD fraction $\rho$	PerSPL	Gain over labeled-only [95% CI]
0	$84.70\pm 1.09$	$+3.21$ [2.58, 3.82]
0.25	$84.55\pm 1.02$	$+3.07$ [2.31, 3.73]
0.50	$83.63\pm 1.12$	$+2.14$ [1.32, 2.90]
0.75	$82.71\pm 0.89$	$+1.22$ [0.57, 1.84]
1	$82.15\pm 0.67$	$+0.66$ [0.10, 1.18]

Table 3: Controlled generator shift on PersonalLLM with a 0.2 symmetric label-flip rate (overall accuracy, ten matched seeds). The matched regularized labeled-only reference is

81.49\pm 0.79

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Preference source	Users (seen/unseen)	Main setting	Labeled pairs (seen/unseen)	$\|\mathcal{D}_{U}\|$
PersonalLLM	reward-model scores	$100/100$	$\alpha_{\rm div}=0.1$	$45/9$	$5{,}000$
HelpSteer2	human attribute ratings	$50/50$	$\alpha_{\rm div}=0.1$	$50/10$	$50{,}000$
Reddit TLDR	human comparisons	$20/20$	$100$ or $150$ labels	$100$ or $150/50$	$149{,}632$
Chatbot Arena	human comparisons	$19/19$	active users	$25/12$	$3{,}166$

Appendix

Table 4: Dataset and evaluation settings for the primary results in Table 1 . Label budgets distinguish the histories of seen users from the support sets used to adapt unseen users.

Setting	$B$	$\eta$	$\alpha$	$\tau$
PersonalLLM ( $\alpha_{\rm div}=0.1$ )	2	0.03	0.6	0.9
HelpSteer2 ( $\alpha_{\rm div}=0.1$ )	5	0.01	0.7	0.6
Reddit TLDR (100/seen)	2	0.03	0.7	0.8
Reddit TLDR (150/seen)	2	0.03	0.7	0.8
Chatbot Arena (25/12)	2	0.3	0.8	0.8

Appendix

Table 5: PerSPL hyperparameters for the main results in Table 1 . Here

B

is the basis rank,

\eta

is the peak learning rate,

\alpha

is the labeled-loss weight in Eq. ( 7 ), and

\tau

is the pseudo-label confidence threshold. Parameters shared by all rows are given in the text.

Figure 4: Direct visualization of Theorem 3 along the population regularization path. On log–log axes, the two nonaligned settings have a locally linear single-response accuracy gain (left; slope

1

), while the paired BT excess risk is locally quadratic (right; slope

2

). The direction-preserving control has exactly zero accuracy gain and is therefore shown only in the risk panel. Here

\lambda^{\star}

is the coefficient whose population direction is single-response optimal; only the small-

\lambda

neighborhood covered by the theorem is plotted.

Dataset	$\lambda_{r}$	Primary alignment	Disjoint alignment	$\operatorname{Var}(m)$ reduction per unit $\lambda_{r}$
HelpSteer2	$10^{-8}$	$402.3\pm 115.7$	$393.6\pm 108.0$	$4265.8\pm 1339.4$
HelpSteer2	$10^{-7}$	$396.1\pm 115.6$	$387.1\pm 108.1$	$4203.4\pm 1345.2$
Reddit TL;DR	$10^{-8}$	$178.5\pm 136.9$	$184.7\pm 115.9$	$1821.3\pm 1384.0$
Reddit TL;DR	$10^{-7}$	$173.7\pm 137.6$	$179.8\pm 118.2$	$1782.5\pm 1390.6$

Appendix

Table 6: Matched one-sided differences at

\lambda_{r}=0

for the actual fixed-budget Adam training path. The primary and disjoint columns estimate

\widehat{A}_{K,\mathrm{Adam}}^{0+}

on two held-out splits; the final column reports the reduction in held-out within-user pair-mean variance per unit

\lambda_{r}

. Entries are means

\pm

paired 95%

t

intervals over five seeds. Positive values denote improvement.

Figure 5: Real-data regularization paths for labeled-only reward heads. Panels (a) and (b) report the reductions in held-out cross-prompt logistic loss and paired BT loss, respectively, relative to the matched

\lambda_{r}=0

model. Solid lines show means and shaded regions show paired 95%

t

intervals over five seeds. Positive values denote lower loss.

	$\alpha_{\rm div}{=}0.001$			$\alpha_{\rm div}{=}0.01$			$\alpha_{\rm div}{=}0.1$
Method	Seen	Unseen	Overall	Seen	Unseen	Overall	Seen	Unseen	Overall
Skywork-Reward V2	76.9 $\pm$ 1.1	76.8 $\pm$ 2.6	76.8 $\pm$ 1.7	77.8 $\pm$ 2.4	76.1 $\pm$ 1.9	77.0 $\pm$ 1.7	81.6 $\pm$ 1.6	82.4 $\pm$ 1.4	82.0 $\pm$ 1.2
BT	81.7 $\pm$ 3.8	81.8 $\pm$ 4.0	81.7 $\pm$ 3.8	82.5 $\pm$ 4.5	81.6 $\pm$ 4.2	82.0 $\pm$ 4.2	88.8 $\pm$ 3.2	89.0 $\pm$ 3.8	88.9 $\pm$ 3.4
VPL	82.8 $\pm$ 1.7	82.6 $\pm$ 0.9	82.7 $\pm$ 1.2	83.6 $\pm$ 2.1	82.9 $\pm$ 2.0	83.3 $\pm$ 1.8	89.6 $\pm$ 1.2	89.8 $\pm$ 1.2	89.7 $\pm$ 1.1
GPO	85.4 $\pm$ 2.2	84.3 $\pm$ 2.9	84.9 $\pm$ 2.5	79.4 $\pm$ 16.1	77.6 $\pm$ 15.1	78.5 $\pm$ 15.6	89.1 $\pm$ 6.7	89.6 $\pm$ 6.8	89.4 $\pm$ 6.7
PAL	81.0 $\pm$ 0.8	80.5 $\pm$ 0.7	80.8 $\pm$ 0.6	80.5 $\pm$ 2.8	79.7 $\pm$ 2.0	80.1 $\pm$ 1.9	84.7 $\pm$ 1.4	85.2 $\pm$ 1.5	85.0 $\pm$ 1.1

Appendix

Table 7: Full PersonalLLM results across

\alpha_{\rm div}\in\{0.001,0.01,0.1\}

(

45

pairs/seen user). Mean

\pm

std over five seeds.

	$\alpha_{\rm div}{=}0.01$			$\alpha_{\rm div}{=}0.1$
Method	Seen	Unseen	Overall	Seen	Unseen	Overall
Skywork-Reward V2	62.3 $\pm$ 0.6	62.6 $\pm$ 0.2	62.5 $\pm$ 0.3	67.0 $\pm$ 0.3	67.4 $\pm$ 0.3	67.2 $\pm$ 0.1
BT	53.1 $\pm$ 4.4	53.0 $\pm$ 4.5	53.1 $\pm$ 4.4	63.0 $\pm$ 3.7	63.2 $\pm$ 3.8	63.1 $\pm$ 3.7
VPL	55.4 $\pm$ 1.5	55.3 $\pm$ 1.3	55.4 $\pm$ 1.4	59.6 $\pm$ 0.7	59.6 $\pm$ 0.6	59.6 $\pm$ 0.6
GPO	60.3 $\pm$ 2.1	59.8 $\pm$ 2.0	60.1 $\pm$ 2.0	65.8 $\pm$ 2.6	66.4 $\pm$ 2.3	66.1 $\pm$ 2.4
PAL	54.7 $\pm$ 1.2	54.3 $\pm$ 0.8	54.5 $\pm$ 1.0	57.1 $\pm$ 1.4	56.9 $\pm$ 1.0	57.0 $\pm$ 1.2

Appendix

Table 8: Full HelpSteer2 results across

\alpha_{\rm div}\in\{0.01,0.1\}

(

50

pairs/seen user). Mean

\pm

std over five seeds.

	$100$ examples/seen user			$150$ examples/seen user
Method	Seen	Unseen	Overall	Seen	Unseen	Overall
Skywork-Reward V1	62.6 $\pm$ 0.5	62.6 $\pm$ 0.9	62.6 $\pm$ 0.9	62.6 $\pm$ 0.5	62.6 $\pm$ 0.9	62.6 $\pm$ 0.9
Skywork-Reward V2	64.5 $\pm$ 0.8	64.6 $\pm$ 0.8	64.5 $\pm$ 0.0	64.5 $\pm$ 0.8	64.6 $\pm$ 0.8	64.5 $\pm$ 0.0
BT	63.2 $\pm$ 4.8	64.3 $\pm$ 3.8	63.8 $\pm$ 4.2	64.4 $\pm$ 3.5	65.2 $\pm$ 3.1	64.8 $\pm$ 3.3
VPL	57.0 $\pm$ 0.8	56.9 $\pm$ 1.3	56.9 $\pm$ 0.5	57.6 $\pm$ 1.5	57.0 $\pm$ 1.6	57.3 $\pm$ 0.7
GPO	63.8 $\pm$ 2.5	64.6 $\pm$ 2.7	64.2 $\pm$ 2.6	65.7 $\pm$ 1.7	65.9 $\pm$ 1.3	65.8 $\pm$ 1.4

Appendix

Table 9: Full Reddit TLDR results at

100

and

150

labeled examples/seen user. Mean

\pm

std over five seeds. SynthesizeMe variants are inference-time-personalization methods (no in-domain training); MRM is reported with both Skywork-Reward V1 and V2 backbones.

Method	Seen	Unseen	Overall
Skywork-Reward V2	$80.33\pm 1.40$	$76.89\pm 1.40$	$78.61\pm 0.00$
BT	$80.25\pm 2.02$	$78.12\pm 2.95$	$79.19\pm 1.12$
VPL	$64.93\pm 3.09$	$61.93\pm 6.26$	$63.43\pm 2.23$
GPO	$79.44\pm 1.66$	$77.68\pm 1.39$	$78.56\pm 1.12$
PAL	$58.40\pm 2.98$	$58.36\pm 3.04$	$58.38\pm 1.50$
LoRe	$81.27\pm 2.08$	$78.38\pm 2.05$	$79.83\pm 1.00$

Appendix

Table 10: Full Chatbot Arena results. We report pairwise accuracy (%, mean

\pm

std over five matched seeds); Overall is the per-seed average of seen and unseen accuracy. VPL/PAL use their released pooled-pair evaluators; the other methods macro-average users. All five prespecified seeds are retained.

Setting	Seen	Unseen	Overall
HelpSteer2 $\alpha_{\rm div}=0.1$	$+0.91$ [0.62, 1.21]	$+0.35$ [0.04, 0.65]	$+0.62$ [0.34, 0.93]
HelpSteer2 $\alpha_{\rm div}=0.01$	$+1.22$ [0.57, 2.01]	$+0.37$ [ $-0.43$ , 1.18]	$+0.81$ [0.11, 1.62]
Reddit, 100/seen	$+0.78$ [0.28, 1.35]	$+0.38$ [0.15, 0.60]	$+0.56$ [0.39, 0.76]
Reddit, 150/seen	$+0.91$ [0.31, 1.51]	$+0.74$ [0.40, 1.08]	$+0.81$ [0.48, 1.05]
Chatbot Arena, 25/12	$+1.72$ [0.14, 3.45]	$+2.66$ [1.13, 4.28]	$+2.19$ [0.79, 3.20]

Appendix

Table 11: PerSPL minus the baseline with the highest overall mean in each setting, measured in accuracy percentage points with descriptive paired-bootstrap 95% confidence intervals over five matched seeds.

Setting	Seen	Unseen
Labeled only	$80.31\pm 1.37$	$82.66\pm 0.67$
$\rho=0$	$84.56\pm 1.32$	$84.83\pm 1.07$
$\rho=0.25$	$84.44\pm 1.14$	$84.66\pm 1.14$
$\rho=0.50$	$83.55\pm 1.15$	$83.71\pm 1.38$
$\rho=0.75$	$82.58\pm 1.00$	$82.83\pm 1.19$
$\rho=1$	$81.85\pm 0.86$	$82.45\pm 0.84$

Appendix

Table 12: Seen- and unseen-user breakdown for the controlled generator-shift study in Table 3 . Overall results and paired gains are reported in the main table.

Figure 6: Hyperparameter sensitivity of PerSPL on HelpSteer2 (

\alpha_{\rm div}=0.01

). Stars mark the main configuration (

\tau=0.6

\eta=0.01

\alpha=0.7

). Overall accuracy changes little across the tested

\tau

and

\alpha

values and is stable near the main learning rate.

Personalized Reward Modeling

See all 10 papers

Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
Apr 8, 2026Qiyao Ma, Dechen Gao, Rui Cai +4Personalized Reward Modeling Reward Model Evaluation
University of California, Davis
POPI: Personalizing LLMs via Optimized Natural Language Preference Inference
Oct 17, 2025Yizhuo Chen, Xin Liu, Ruijie Wang +7LLM Personalization Personalized Language Models
University of Illinois Urbana-Champaign · Amazon · University of Notre Dame
Online LLM Personalization from Pairwise Preferences Across Users
Oct 12, 2026Romain Puech, Georgy Salakhutdinov, Dimitris BertsimasEducational QA LLM Personalization
Massachusetts Institute of Technology

Apr 8, 2026J/K move · Enter open · S save

Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization

Qiyao Ma, Dechen Gao, Rui Cai, Boqi Zhao+3

University of California, Davis

Personalized Reward Modeling Reward Model Evaluation User Preference Modeling Pairwise Preference Evaluation

Open paper AskPDF arXiv

Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values. While benchmarks for general response quality are prevalent, evaluating how well reward models account for individual user preferences remains an open challenge. To bridge this gap, we introduce Personalized RewardBench, a novel benchmark designed to rigorously assess reward models' capacity to model personalized preferences. We construct chosen and rejected response pairs based on strict adherence to (or violation of) user-specific rubrics, ensuring that preference distinctions are uniquely tailored to the individual. In particular, human evaluations confirm that the primary discriminative factor between pairs is strictly personal preference, with both responses maintaining high general quality (e.g., correctness, relevance and helpfulness). Extensive testing reveals that existing state-of-the-art reward models struggle significantly with personalization, peaking at an accuracy of just 75.94%. Crucially, because an effective reward model benchmark should predict a reward model's performance on downstream tasks, we conduct experiments demonstrating that our benchmark exhibits a significantly higher correlation with downstream performance in both Best-of-N (BoN) sampling and Proximal Policy Optimization (PPO) compared to existing baselines. These findings establish Personalized RewardBench as a robust and accurate proxy for evaluating reward models' performance in downstream applications.

Abstract

Personalized Reward Modeling

Explore similar work

Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization

POPI: Personalizing LLMs via Optimized Natural Language Preference Inference

Online LLM Personalization from Pairwise Preferences Across Users