Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedback

Gihoon Kim; Euntai Kim

cs.LGMar 13, 2026

Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedback

Authors: Gihoon Kim, Euntai Kim

Organizations: Yonsei University · Korea Institute of Science and Technology

Abstract

Reinforcement Learning from Human Feedback (RLHF) is a widely used approach to align large-scale AI systems with human values. However, RLHF typically assumes a single, universal reward, which overlooks diverse preferences and limits personalization. Variational Preference Learning (VPL) seeks to address this by introducing user-specific latent variables. Despite its promise, we found that VPL suffers from posterior collapse. While this phenomenon is well known in VAEs, it has not previously been identified in preference learning frameworks. Under sparse preference data and with overly expressive decoders, VPL may cause latent variables to be ignored, reverting to a single-reward model. To overcome this limitation, we propose Swap-guided Preference Learning (SPL). The key idea is to construct fictitious swap annotators and use the mirroring property of their preferences to guide the encoder. SPL introduces three components: (1) swap-guided base regularization, (2) Preferential Inverse Autoregressive Flow (P-IAF), and (3) adaptive latent conditioning. Experiments show that SPL mitigates collapse, enriches user-specific latents, and improves preference prediction. Our code and data are available at https://github.com/cobang0111/SPL

Figures & tables

Figure 1: Overview of SPL. We propose Swap-guided Preference Learning (SPL), a new framework for personalized alignment. RLHF ( Ouyang et al., 2022 ) cannot adequately reflect user diversity. To overcome this limitation, VPL ( Poddar et al., 2024 ) encodes text data consisting of a prompt $x$ and response $y$ into a single latent embedding. However, this encoding process is highly prone to collapse. In contrast, SPL leverages the structural properties of preference data through guiding mechanisms and a Preferential Inverse Autoregressive flow, allowing the latent space to capture user-specific characteristics.

Panel of Figure 2 — Figure 2: Posterior collapse in Variational Preference Learning. We visualize latent embeddings ${\bm{z}}$ from the VPL encoder using 2D UMAP ( McInnes et al., 2018 ) . Each point denotes a user, colored by their preference type. (a) User preference types are distinctly separated, indicating non-collapse. (b), (c) Latent collapse occurs, making preference types indistinguishable.

Panel of Figure 3 — Figure 3: Differences in posterior distribution between original and swapped inputs. We test how the VPL encoder’s posterior responds when each preference pair is inverted to simulate a user with opposite choices, using the simple dataset Pets . (a) Average RMSE between original and swapped inputs across posterior mean ${\bm{\mu}}$ and log-variance $\bm{\ell}$ . Collapse appears in Llama-3.1-8B (orange), where both parameters remain unchanged, whereas Llama-3.2-3B (green) shows distinct behavior. (b) Plot ${\bm{\mu}}$ vs. ${\bm{\mu}}_{\text{swap}}$ for Llama-3.2-3B; ${\bm{\mu}}+{\bm{\mu}}_{\text{swap}}$ is in the lower panel. Initially, the curves are similar, but their difference grows and stabilizes as learning continues, resulting in a sign-reversal.

Figure 4: Preference encoding process of SPL

Model	$\beta$	Method	UF-P-2		UF-P-4
Model	$\beta$	Method	Acc. [%]	AU [%]	Acc. [%]	AU [%]
Llama-3.2-3B	$3\times 10^{-7}$	VPL	62.04 $\pm$ 0.16	0.00 $\pm$ 0.00	56.91 $\pm$ 0.12	0.00 $\pm$ 0.00
	$3\times 10^{-7}$	SPL	62.59 $\pm$ 0.37	73.05 $\pm$ 4.69	61.52 $\pm$ 0.27	76.89 $\pm$ 7.89
	$3\times 10^{-6}$	VPL	62.37 $\pm$ 0.15	88.22 $\pm$ 7.72	57.03 $\pm$ 0.10	0.00 $\pm$ 0.00
	$3\times 10^{-6}$	SPL	63.28 $\pm$ 0.13	93.07 $\pm$ 3.18	61.56 $\pm$ 0.03	82.32 $\pm$ 3.18
	$3\times 10^{-5}$	VPL	62.29 $\pm$ 0.19	14.03 $\pm$ 6.93	57.00 $\pm$ 0.07	0.00 $\pm$ 0.00

Table 1: Accuracy and active units across

\beta

Model	Method	Pets	UF-P-2	UF-P-4
Llama-3.2-3B	BTL	57.48 $\pm$ 2.37	62.25 $\pm$ 0.03	57.07 $\pm$ 0.01
	DPL	62.02 $\pm$ 1.92	62.22 $\pm$ 0.03	57.04 $\pm$ 0.05
	VPL	99.67 $\pm$ 0.38	62.37 $\pm$ 0.15	57.03 $\pm$ 0.10
	SPL (Ours)	100.0 $\pm$ 0.00	63.28 $\pm$ 0.13	61.56 $\pm$ 0.03
Llama-3.1-8B	BTL	60.74 $\pm$ 0.49	62.59 $\pm$ 0.04	57.40 $\pm$ 0.28
	DPL	61.03 $\pm$ 0.25	62.74 $\pm$ 0.03	57.66 $\pm$ 0.14

Table 2: Preference-prediction accuracy (%) compared with baselines

Model	Method	sample/s	GPU hour	peak memory
Llama-3.2-3B	VPL	6.070	13.363	6.25GB
Llama-3.2-3B	SPL	5.952	13.590	6.65GB
Llama-3.1-8B	VPL	3.370	23.791	11.37GB
Llama-3.1-8B	SPL	3.355	23.945	11.76GB

Table 3: Training computation and memory costs on UF-P-4

Panel of Figure 5 — Figure 5: Latent embeddings learned on the UF-P dataset. We visualize latent embeddings ${\bm{z}}$ from baselines and SPL (Ours) encoder using 2D t-SNE ( Maaten and Hinton, 2008 ) . Each point denotes a user, colored by their preference type. Compared to the VPL, SPL yields much clearer separation in the latent space.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Evidence of posterior collapse in preference learning

Panel of Figure 7 — Figure 7: Details and training dynamics of the SPL decoder

Model	Method	UF-P-2	UF-P-4
Llama-3.2-3B	VPL	62.37 $\pm$ 0.15	57.03 $\pm$ 0.10
	VPL-IAF	62.35 $\pm$ 0.11	58.01 $\pm$ 0.43
	SPL-IAF	63.09 $\pm$ 0.14	59.50 $\pm$ 0.07
	SPL (Ours)	63.28 $\pm$ 0.13	61.56 $\pm$ 0.03
Llama-3.1-8B	VPL	62.66 $\pm$ 0.23	57.14 $\pm$ 0.05
	VPL-IAF	62.82 $\pm$ 0.13	58.65 $\pm$ 0.09

Appendix

Table 4: Preference-prediction accuracy (%)

Panel of Figure 8 — Figure 8: Latent embeddings learned on the UF-P-4 dataset. We visualize 2D UMAP of latent embeddings ${\bm{z}}$ from VPL, SPL, and their IAF variants (VPL-IAF and SPL-IAF), using Llama-3.1-8B to compare P-IAF against a standard IAF.

$\mathcal{L}_{\text{guide}}$	P-IAF	$z$ cond.	Acc. [%]	Active Units [%]
✓			57.18	0.00
	✓		59.10	11.03
		✓	56.95	0.00
✓		✓	56.87	0.00
✓	✓		62.14	94.24
	✓	✓	62.08	93.07

Appendix

Table 5: Effect of each component

Model	Method	Accuracy [%]	Active Units [%]
Llama-3.2-3B	BTL	57.07	-
	VPL	57.03	0.00
	VPL w/ $\mathcal{L}_{guide}$	57.56	45.61
	VPL-IAF	58.01	39.45
	SPL (Ours)	61.56	82.32

Appendix

Table 6: Effect of the swap-guided base regularization

Model	Method	Accuracy [%]	Active Units [%]
Llama-3.2-3B	SPL	61.41	82.62
	SPL w/o z cond.	58.08	80.37
	SPL w/o $\mathcal{L}_{\text{guide}}$ , z cond.	56.92	0.00
	VPL	56.98	0.00

Appendix

Table 7: SPL accuracy and active units with noise on UF-P-4

$K$	Accuracy [%]	Active Units [%]
1	60.58	91.60
2	62.21	96.19
4	61.92	93.16

Appendix

Table 8: Effect of P-IAF depth

K

on UF-P-4

Model	Method	Accuracy [%]	Active Units [%]
Llama-3.2-3B	BTL	56.94	-
	VPL	56.92	31.35
	SPL (Ours)	58.12	61.13

Appendix

Table 9: Accuracy and active units with fewer preference pairs

$\lambda$	Accuracy [%]	Active Units [%]
$1.0\times 10^{-3}$	61.19 $\pm$ 0.40	81.05 $\pm$ 4.98
$1.0\times 10^{-4}$	60.90 $\pm$ 0.20	79.33 $\pm$ 0.69
$1.0\times 10^{-5}$	61.56 $\pm$ 0.03	82.32 $\pm$ 3.18
$1.0\times 10^{-6}$	59.74 $\pm$ 0.09	72.01 $\pm$ 1.62
w/o $\mathcal{L}_{guide}$	58.53 $\pm$ 0.13	78.58 $\pm$ 1.59

Appendix

Table 10: Effect of guidance loss weight

\lambda

$\eta$	Accuracy [%]	Active Units [%]
$0.10$	61.56 $\pm$ 0.03	82.32 $\pm$ 3.18
$0.25$	61.49 $\pm$ 0.11	83.11 $\pm$ 2.50
$0.50$	61.18 $\pm$ 0.18	78.22 $\pm$ 0.98
$1.00$	61.44 $\pm$ 0.06	83.50 $\pm$ 2.98

Appendix

Table 11: Effect of guidance balancing weight

\eta

Hyperparameter	Value
Token embedding dimension	3072 ( Llama-3.2-3B-instruct ), 4096 ( Llama-3.1-8B-instruct )
Max length	1024
Max preference pairs per sample $n$	8
survey size for UF-P	16
Token data type	bfloat16
Training samples in dataset	4,000 (for Pets ), 55,636 (for UF-P-2 ), 111,272 (for UF-P-4 )

Appendix

Table 12: Hyperparameters for data generation

Hyperparameter	Value
Encoder input dimension	3072 ( Llama-3.2-3B-instruct ), 4096 ( Llama-3.1-8B-instruct )
Latent dimension $d$	1024
Learning rate	1.0 $\times$ 10 -4
Learning rate scheduler	cosine with 3% warm-up steps
Epoch	2
P-IAF flow step $K$	2

Appendix

Table 13: Hyperparameters for experiments

Notation	Meaning	Notation	Meaning
Indices & counts		Embeddings, latents & contexts
$i$	preference-pair index	${\bm{e}}_{w}$	embedding of $y_{w}$
$n$	number of preference-pair in sample	${\bm{e}}_{l}$	embedding of $y_{l}$
$N$	total number of preference-pair	${\bm{z}}$	latent embedding
$j$	dimension index	${\bm{\mu}}$	mean
$k$	flow step	$\bm{\sigma}$	standard deviation

Appendix

Table 14: Notations

Preference Learning

Preference learning infers people’s values and trade-offs from choices, comparisons, rankings, and feedback to guide predictions and decisions.

See all 150 papers

Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferences
May 29, 2026Jabin Koo, Hoyoung Kim, Minwoo Jang +1LLM Alignment Preference Alignment
Department of CSE, POSTECH, Pohang, Republic of Korea · National AI Research Lab, Seoul, Republic of Korea · Graduate School of AI, POSTECH, Pohang, Republic of Korea
A Regret Minimization Framework on Preference Learning in Large Language Models
Jun 8, 2026Suhwan Kim, Taehyun Cho, Geon-Hyeong Kim +4Regret Minimization in RL LLM Alignment
Seoul National University, Seoul, South Korea · LG AI Research, Seoul, South Korea · UNIST, Ulsan, South Korea +1
PerSPL: Personalized Semi-supervised Preference Learning
Oct 21, 2026Xiaoan Xu, Tingzhou Wei, Juncheng Dong +3Personalized Reward Modeling User Preference Modeling
Duke University · George Washington University

May 29, 2026J/K move · Enter open · S save

Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferences

Jabin Koo, Hoyoung Kim, Minwoo Jang, Jungseul Ok

Department of CSE, POSTECH, Pohang, Republic of Korea · National AI Research Lab, Seoul, Republic of Korea · Graduate School of AI, POSTECH, Pohang, Republic of Korea

LLM Alignment Preference Alignment User Preference Modeling Preference Learning Federated Learning

Open paper AskPDF arXiv

Federated Learning (FL) offers a privacy-preserving pathway for aligning Large Language Models (LLMs); however, existing frameworks typically enforce a monolithic reward model, inevitably averaging out inherently conflicting user preferences (e.g., helpfulness vs. harmlessness). While Variational Preference Learning (VPL) offers a pathway to personalization, adapting it to decentralized settings presents a fundamental challenge: posterior collapse driven by severe local data scarcity and heterogeneity. In this paper, we propose Federated Variational Preference Alignment with Gumbel-Softmax Prior (FedVPA-GP), a framework designed to disentangle diverse preferences without compromising privacy. To stabilize variational inference, we introduce a Federated Mixture Prior that enables clients to leverage the aggregate population distribution as a dynamic prior. Furthermore, we incorporate an Orthogonal Loss that explicitly enforces the separation of preference prototypes in the latent space. Experiments on the HH-RLHF dataset demonstrate that FedVPA-GP significantly outperforms monolithic baselines, successfully disentangling conflicting user intents and enabling dynamic preference switching.

Abstract

Preference Learning

Explore similar work

Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferences

A Regret Minimization Framework on Preference Learning in Large Language Models

PerSPL: Personalized Semi-supervised Preference Learning