FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment

Kewen Zhu; Liping Yi; Zhiming Zhao; Zhuang Qi; Han Yu; Qinghua Hu

cs.LGMar 20, 2026

FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment

Authors: Kewen Zhu, Liping Yi, Zhiming Zhao, Zhuang Qi, Han Yu, Qinghua Hu

Organizations: College of Intelligence and Computing, Tianjin University, Tianjin, China · College of Computing and Data Science, Nanyang Technological University, Singapore

Abstract

Aligning large language models (LLMs) with human preferences in federated learning (FL) is challenging due to decentralized, privacy-sensitive, and highly non-IID preference data. Direct Preference Optimization (DPO) offers an efficient alternative to reinforcement learning with human feedback (RLHF), but its direct application in FL suffers from severe performance degradation under non-IID data and limited generalization of implicit rewards. To bridge this gap, we propose FedPDPO (Federated Personalized Direct Preference Optimization), a personalized federated framework for preference alignment of LLMs. It adopts a parameter-efficient fine-tuning architecture where each client maintains a frozen pretrained LLM backbone augmented with a Low-Rank Adaptation (LoRA) adapter, enabling communication-efficient aggregation. To address non-IID heterogeneity, we devise (1) the globally shared LoRA adapter with the personalized client-specific LLM head. Moreover, we introduce (2) a personalized DPO training strategy with a client-specific explicit reward head to complement implicit rewards and further alleviate non-IID heterogeneity, and (3) a bottleneck adapter to balance global and local features. We provide theoretical analysis establishing the probabilistic foundation and soundness. Extensive experiments on multiple preference datasets demonstrate state-of-the-art performance, achieving up to 4.80% average accuracy improvements in federated intra-domain and cross-domain settings.

Figures & tables

Figure 1: Accuracy comparison of DPO under IID and non-IID settings on the code vulnerability dataset.

Figure 2: Overview of FedPDPO . Each client fine-tunes and shares a LoRA adapter $(\boldsymbol{A},\boldsymbol{B})$ on a frozen backbone $\boldsymbol{W}_{0}$ while locally training personalized modules: a bottleneck adapter $\boldsymbol{M}$ , local LLM and reward heads $(\boldsymbol{h}_{1},\boldsymbol{h}_{2})$ with an effective PDPO training strategy.

Figure 3: Test accuracy varies as communication rounds in intra-domain FL settings with the IMDB dataset and 10 clients.

Figure 4: Test accuracy varies as communication rounds in cross-domain FL settings with 3 domains of datasets assigned to 3 clients.

Method	Intra-Domain				Cross-Domain
Method	IMDB (3)	IMDB (10)	Code-V. (3)	Code-V. (11)	WebGPT	PyDPO	UltraF.	Avg.
FedAvg ${}_{\text{(+PPO)}}$	80.70 ±0.24	70.89 ±0.28	91.40 ±0.20	78.32 ±0.24	47.93 ±0.22	80.12 ±0.15	63.89 ±0.29	63.98
FedAvg ${}_{\text{(+DPO)}}$	82.73 ±0.22	76.45 ±0.23	93.49 ±0.17	84.09 ±0.19	52.92 ±0.16	85.53 ±0.09	66.45 ±0.24	68.30
Per-FedAvg ${}_{\text{(+PPO)}}$	81.36 ±0.28	73.12 ±0.27	92.11 ±0.24	79.68 ±0.25	49.84 ±0.28	81.76 ±0.24	64.67 ±0.32	65.42
Per-FedAvg ${}_{\text{(+DPO)}}$	83.40 ±0.25	78.83 ±0.22	94.21 ±0.21	85.44 ±0.19	54.96 ±0.23	87.36 ±0.18	67.48 ±0.27	69.93
FedAMP ${}_{\text{(+PPO)}}$	80.41 ±0.30	71.45 ±0.29	91.20 ±0.26	81.23 ±0.23	49.01 ±0.26	78.43 ±0.27	63.98 ±0.35	63.81

Table 1: Test accuracy (%) under intra-domain settings with 3, 10, or 11 clients and cross-domain settings with distinct datasets per client (3 clients). Code-V.: Code-Vulnerability-Security, UltraF.: UltraFeedback. Bold and underline denote the best and second-best results.

Method	WebGPT	PyDPO	UltraF.	Avg.
FedAvg ${}_{\text{(+PPO)}}$	50.85 ±0.24	89.12 ±0.19	69.54 ±0.21	69.84
FedAvg ${}_{\text{(+DPO)}}$	54.12 ±0.18	91.80 ±0.12	71.28 ±0.22	72.40
Per-FedAvg ${}_{\text{(+PPO)}}$	52.34 ±0.27	89.86 ±0.22	70.15 ±0.25	70.78
Per-FedAvg ${}_{\text{(+DPO)}}$	55.67 ±0.21	92.38 ±0.17	72.41 ±0.23	73.49
FedAMP ${}_{\text{(+PPO)}}$	51.58 ±0.25	88.45 ±0.24	69.12 ±0.28	69.72
FedAMP ${}_{\text{(+DPO)}}$	54.92 ±0.20	91.23 ±0.19	71.56 ±0.24	72.57

Table 2: Test accuracy (%) under cross-domain settings with TinyLlama-1.1B on distinct datasets per client (3 clients).

Case	Accuracy (%)
( A1 ): w/o Bottleneck Adapter	82.44 ±0.18
( A2 ): w/o Reward Head	81.94 ±0.21
( A3 ): Full FedPDPO	85.23 ±0.14

Table 3: Ablation study of FedPDPO on the IMDB dataset with 10 clients under reward-margin–based non-IID partitioning.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Description
Sets and Data
$N$	Number of clients
$\mathcal{D}_{i}$	Private preference dataset of client $i$
$\|\mathcal{D}_{i}\|$	Size of dataset $\mathcal{D}_{i}$
$(x,y_{w},y_{l})$	Preference triple: prompt, chosen response, rejected response
Model Parameters

Appendix

Table 4: Notation Table

Panel of Figure 5 — Figure 5: An example preference triple from the IMDB dataset. The prompt is a movie review prefix, and the model must learn to prefer the chosen response (positive sentiment, blue) over the rejected response (negative sentiment, red).

Figure 6: Distribution of reward margins across the IMDB dataset and their non-IID partition into three subsets (easy, medium, hard).

Component	DistilGPT2	TinyLlama-1.1B
Backbone dimension ( $d$ )	768	2048
Number of layers	6	22
Backbone parameters	82M	1.1B
Frozen during training	✓	✓
LoRA rank ( $r$ )	8	8
LoRA $\alpha$	16	16

Appendix

Table 5: Model architecture configurations for FedPDPO .

Rank	IMDB-3	IMDB-10	Code-3	Code-11	Cross	Avg
4	86.24	83.69	96.98	92.99	72.70	86.52
8	87.41	85.23	96.92	92.80	73.44	87.16
16	87.92	84.19	96.58	94.12	73.36	86.83
32	87.68	87.54	97.24	95.26	74.23	88.39
64	87.66	87.44	96.98	96.21	74.51	88.56

Appendix

Table 6: Test accuracy (%) under varying LoRA ranks across five benchmark settings.

Figure 7: Average accuracy (%) varies with LoRA rank across five benchmark settings.

Dim	IMDB-3	IMDB-10	Code-3	Code-11	Cross	Avg
64	86.56	81.96	96.23	92.80	72.40	85.99
128	87.16	82.59	96.81	93.13	72.07	86.35
256	87.41	85.23	96.92	92.80	73.44	87.16
512	87.08	84.01	96.36	92.97	71.83	86.45

Appendix

Table 7: Test accuracy (%) under varying reward head dimensions across five benchmark settings.

Figure 8: Average accuracy (%) varies with reward head dimension across five benchmark settings.

Dim	IMDB-3	IMDB-10	Code-3	Code-11	Cross	Avg
128	86.64	86.23	96.03	93.35	72.11	86.91
256	87.41	85.23	96.92	92.80	73.44	87.16
512	87.07	85.02	96.52	93.91	72.24	86.95
1024	86.48	86.43	96.35	94.08	71.76	87.02

Appendix

Table 8: Test accuracy (%) under varying bottleneck adapter dimensions across five benchmark settings.

Figure 9: Average accuracy (%) varies with bottleneck adapter dimension across five benchmark settings.

$\beta$	IMDB-3	IMDB-10	Code-3	Code-11	Cross	Avg
0.01	85.65	82.55	95.94	91.85	71.67	85.53
0.05	86.84	83.96	96.37	92.23	72.04	86.29
0.10	87.41	85.23	96.92	92.80	73.44	87.16
0.50	87.25	83.46	97.33	94.12	71.76	86.78

Appendix

Table 9: Test accuracy (%) under varying DPO temperature

\beta

across five benchmark settings.

Figure 10: Average accuracy (%) varies with DPO temperature

\beta

across five benchmark settings.

Method	IMDB				Code-Vulnerability-Security
Method	Client0	Client1	Client2	Avg	Client0	Client1	Client2	Avg
FedAvg ${}_{\text{(+PPO)}}$	93.87 ±0.20	80.94 ±0.24	67.28 ±0.27	80.70	96.43 ±0.17	91.26 ±0.20	86.52 ±0.24	91.40
FedAvg ${}_{\text{(+DPO)}}$	95.54 ±0.18	83.19 ±0.22	69.47 ±0.25	82.73	98.21 ±0.14	93.45 ±0.17	88.80 ±0.21	93.49
Per-FedAvg ${}_{\text{(+PPO)}}$	94.56 ±0.24	81.08 ±0.28	68.45 ±0.33	81.36	95.84 ±0.21	92.37 ±0.24	88.12 ±0.31	92.11
Per-FedAvg ${}_{\text{(+DPO)}}$	96.13 ±0.21	83.33 ±0.25	70.73 ±0.30	83.40	97.62 ±0.18	94.64 ±0.21	90.36 ±0.27	94.21
FedAMP ${}_{\text{(+PPO)}}$	93.21 ±0.23	80.12 ±0.30	67.89 ±0.36	80.41	95.67 ±0.22	90.58 ±0.26	87.34 ±0.33	91.20

Appendix

Table 10: Per-client test accuracy (%) under intra-domain settings with 3 clients. Left: Reward-Margin Partition (IMDB). Right: Language-Based Partition (Code-Vulnerability-Security). Results are mean

\pm

std over 5 runs.

Personalized Language Model Alignment

Personalized language model alignment adapts model behavior to individual preferences while preserving safety, accuracy, and other shared constraints.

See all 54 papers

Personalized Federated Direct Preference Optimization via LoRA Adapters
Oct 9, 2026Zhanhong Jiang, Soumik SarkarPersonalized Language Model Alignment Direct Preference Optimization
Translational AI Center, Iowa State University, Ames, IA 50011 USA · Department of Mechanical Engineering, Iowa State University, Ames, IA 50011 USA
Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferences
May 29, 2026Jabin Koo, Hoyoung Kim, Minwoo Jang +1LLM Alignment Preference Alignment
Department of CSE, POSTECH, Pohang, Republic of Korea · National AI Research Lab, Seoul, Republic of Korea · Graduate School of AI, POSTECH, Pohang, Republic of Korea
APPA: Adaptive Preference Pluralistic Alignment for Fair Federated RLHF of LLMs
Apr 5, 2026Mahmoud Srewa, Tianyu Zhao, Salma ElmalakiLLM Alignment Pluralistic Alignment
Department of Electrical Engineering and Computer Science, University of California, Irvine, Irvine, CA 92697, USA

Oct 9, 2026J/K move · Enter open · S save

Personalized Federated Direct Preference Optimization via LoRA Adapters

Zhanhong Jiang, Soumik Sarkar

Translational AI Center, Iowa State University, Ames, IA 50011 USA · Department of Mechanical Engineering, Iowa State University, Ames, IA 50011 USA

Personalized Language Model Alignment Direct Preference Optimization Federated Learning Federated Fine-Tuning

Open paper AskPDF arXiv

Federated Direct Preference Optimization (FedDPO) enables privacy-preserving alignment of language models across distributed agents without centralizing sensitive preference data. However, existing FedDPO analysis assumes all clients converge to a single shared policy, which is fundamentally at odds with the heterogeneous preference distributions present in practice. We propose Personalized FedDPO (pFedDPO), in which each client maintains a small private LoRA adapter atop a shared global base model. In pFedDPO, the base model is synchronized via FedAvg each round, while LoRA adapters remain local and capture client-specific preference patterns. We prove that this decomposition replaces the gradient diversity

κ^2

in the FedDPO bound with a residual heterogeneity

κ^2_{\mathrm{eff}}

, which is no larger than

κ^2

when adapters absorb client-specific directions, at the cost of an additive adapter-bias term that vanishes as adapters converge. Experiments on preference datasets show that pFedDPO consistently attains a lower gradient norm than vanilla FedDPO while remaining competitive in training loss and per-client reward accuracy.

Abstract

Personalized Language Model Alignment

Explore similar work

Personalized Federated Direct Preference Optimization via LoRA Adapters

Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferences

APPA: Adaptive Preference Pluralistic Alignment for Fair Federated RLHF of LLMs