FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment
Authors: Kewen Zhu, Liping Yi, Zhiming Zhao, Zhuang Qi, Han Yu, Qinghua Hu
Organizations: College of Intelligence and Computing, Tianjin University, Tianjin, China · College of Computing and Data Science, Nanyang Technological University, Singapore
Aligning large language models (LLMs) with human preferences in federated learning (FL) is challenging due to decentralized, privacy-sensitive, and highly non-IID preference data. Direct Preference Optimization (DPO) offers an efficient alternative to reinforcement learning with human feedback (RLHF), but its direct application in FL suffers from severe performance degradation under non-IID data and limited generalization of implicit rewards. To bridge this gap, we propose FedPDPO (Federated Personalized Direct Preference Optimization), a personalized federated framework for preference alignment of LLMs. It adopts a parameter-efficient fine-tuning architecture where each client maintains a frozen pretrained LLM backbone augmented with a Low-Rank Adaptation (LoRA) adapter, enabling communication-efficient aggregation. To address non-IID heterogeneity, we devise (1) the globally shared LoRA adapter with the personalized client-specific LLM head. Moreover, we introduce (2) a personalized DPO training strategy with a client-specific explicit reward head to complement implicit rewards and further alleviate non-IID heterogeneity, and (3) a bottleneck adapter to balance global and local features. We provide theoretical analysis establishing the probabilistic foundation and soundness. Extensive experiments on multiple preference datasets demonstrate state-of-the-art performance, achieving up to 4.80% average accuracy improvements in federated intra-domain and cross-domain settings.
Figures & tables
Figure 1: Accuracy comparison of DPO under IID and non-IID settings on the code vulnerability dataset.
Figure 2: Overview of FedPDPO . Each client fine-tunes and shares a LoRA adapter (A,B) on a frozen backbone W0 while locally training personalized modules: a bottleneck adapter M , local LLM and reward heads (h1,h2) with an effective PDPO training strategy.
Figure 3: Test accuracy varies as communication rounds in intra-domain FL settings with the IMDB dataset and 10 clients.
Figure 4: Test accuracy varies as communication rounds in cross-domain FL settings with 3 domains of datasets assigned to 3 clients.
Method
Intra-Domain
Cross-Domain
IMDB (3)
IMDB (10)
Code-V. (3)
Code-V. (11)
WebGPT
PyDPO
UltraF.
Avg.
FedAvg (+PPO)
80.70 ±0.24
70.89 ±0.28
91.40 ±0.20
78.32 ±0.24
47.93 ±0.22
80.12 ±0.15
63.89 ±0.29
63.98
FedAvg (+DPO)
82.73 ±0.22
76.45 ±0.23
93.49 ±0.17
84.09 ±0.19
52.92 ±0.16
85.53 ±0.09
66.45 ±0.24
68.30
Per-FedAvg (+PPO)
81.36 ±0.28
73.12 ±0.27
92.11 ±0.24
79.68 ±0.25
49.84 ±0.28
81.76 ±0.24
64.67 ±0.32
65.42
Per-FedAvg (+DPO)
83.40 ±0.25
78.83 ±0.22
94.21 ±0.21
85.44 ±0.19
54.96 ±0.23
87.36 ±0.18
67.48 ±0.27
69.93
FedAMP (+PPO)
80.41 ±0.30
71.45 ±0.29
91.20 ±0.26
81.23 ±0.23
49.01 ±0.26
78.43 ±0.27
63.98 ±0.35
63.81
Table 1: Test accuracy (%) under intra-domain settings with 3, 10, or 11 clients and cross-domain settings with distinct datasets per client (3 clients). Code-V.: Code-Vulnerability-Security, UltraF.: UltraFeedback. Bold and underline denote the best and second-best results.
Method
WebGPT
PyDPO
UltraF.
Avg.
FedAvg (+PPO)
50.85 ±0.24
89.12 ±0.19
69.54 ±0.21
69.84
FedAvg (+DPO)
54.12 ±0.18
91.80 ±0.12
71.28 ±0.22
72.40
Per-FedAvg (+PPO)
52.34 ±0.27
89.86 ±0.22
70.15 ±0.25
70.78
Per-FedAvg (+DPO)
55.67 ±0.21
92.38 ±0.17
72.41 ±0.23
73.49
FedAMP (+PPO)
51.58 ±0.25
88.45 ±0.24
69.12 ±0.28
69.72
FedAMP (+DPO)
54.92 ±0.20
91.23 ±0.19
71.56 ±0.24
72.57
Table 2: Test accuracy (%) under cross-domain settings with TinyLlama-1.1B on distinct datasets per client (3 clients).
Case
Accuracy (%)
( A1 ): w/o Bottleneck Adapter
82.44 ±0.18
( A2 ): w/o Reward Head
81.94 ±0.21
( A3 ): Full FedPDPO
85.23 ±0.14
Table 3: Ablation study of FedPDPO on the IMDB dataset with 10 clients under reward-margin–based non-IID partitioning.
Figure 5: An example preference triple from the IMDB dataset. The prompt is a movie review prefix, and the model must learn to prefer the chosen response (positive sentiment, blue) over the rejected response (negative sentiment, red).
Figure 6: Distribution of reward margins across the IMDB dataset and their non-IID partition into three subsets (easy, medium, hard).
Component
DistilGPT2
TinyLlama-1.1B
Backbone dimension ( d )
768
2048
Number of layers
6
22
Backbone parameters
82M
1.1B
Frozen during training
✓
✓
LoRA rank ( r )
8
8
LoRA α
16
16
Appendix
Table 5: Model architecture configurations for FedPDPO .
Rank
IMDB-3
IMDB-10
Code-3
Code-11
Cross
Avg
4
86.24
83.69
96.98
92.99
72.70
86.52
8
87.41
85.23
96.92
92.80
73.44
87.16
16
87.92
84.19
96.58
94.12
73.36
86.83
32
87.68
87.54
97.24
95.26
74.23
88.39
64
87.66
87.44
96.98
96.21
74.51
88.56
Appendix
Table 6: Test accuracy (%) under varying LoRA ranks across five benchmark settings.
Figure 7: Average accuracy (%) varies with LoRA rank across five benchmark settings.
Dim
IMDB-3
IMDB-10
Code-3
Code-11
Cross
Avg
64
86.56
81.96
96.23
92.80
72.40
85.99
128
87.16
82.59
96.81
93.13
72.07
86.35
256
87.41
85.23
96.92
92.80
73.44
87.16
512
87.08
84.01
96.36
92.97
71.83
86.45
Appendix
Table 7: Test accuracy (%) under varying reward head dimensions across five benchmark settings.
Figure 8: Average accuracy (%) varies with reward head dimension across five benchmark settings.
Dim
IMDB-3
IMDB-10
Code-3
Code-11
Cross
Avg
128
86.64
86.23
96.03
93.35
72.11
86.91
256
87.41
85.23
96.92
92.80
73.44
87.16
512
87.07
85.02
96.52
93.91
72.24
86.95
1024
86.48
86.43
96.35
94.08
71.76
87.02
Appendix
Table 8: Test accuracy (%) under varying bottleneck adapter dimensions across five benchmark settings.
Figure 9: Average accuracy (%) varies with bottleneck adapter dimension across five benchmark settings.
β
IMDB-3
IMDB-10
Code-3
Code-11
Cross
Avg
0.01
85.65
82.55
95.94
91.85
71.67
85.53
0.05
86.84
83.96
96.37
92.23
72.04
86.29
0.10
87.41
85.23
96.92
92.80
73.44
87.16
0.50
87.25
83.46
97.33
94.12
71.76
86.78
Appendix
Table 9: Test accuracy (%) under varying DPO temperature β across five benchmark settings.
Figure 10: Average accuracy (%) varies with DPO temperature β across five benchmark settings.
Method
IMDB
Code-Vulnerability-Security
Client0
Client1
Client2
Avg
Client0
Client1
Client2
Avg
FedAvg (+PPO)
93.87 ±0.20
80.94 ±0.24
67.28 ±0.27
80.70
96.43 ±0.17
91.26 ±0.20
86.52 ±0.24
91.40
FedAvg (+DPO)
95.54 ±0.18
83.19 ±0.22
69.47 ±0.25
82.73
98.21 ±0.14
93.45 ±0.17
88.80 ±0.21
93.49
Per-FedAvg (+PPO)
94.56 ±0.24
81.08 ±0.28
68.45 ±0.33
81.36
95.84 ±0.21
92.37 ±0.24
88.12 ±0.31
92.11
Per-FedAvg (+DPO)
96.13 ±0.21
83.33 ±0.25
70.73 ±0.30
83.40
97.62 ±0.18
94.64 ±0.21
90.36 ±0.27
94.21
FedAMP (+PPO)
93.21 ±0.23
80.12 ±0.30
67.89 ±0.36
80.41
95.67 ±0.22
90.58 ±0.26
87.34 ±0.33
91.20
Appendix
Table 10: Per-client test accuracy (%) under intra-domain settings with 3 clients. Left: Reward-Margin Partition (IMDB). Right: Language-Based Partition (Code-Vulnerability-Security). Results are mean ± std over 5 runs.
Federated Direct Preference Optimization (FedDPO) enables privacy-preserving alignment of language models across distributed agents without centralizing sensitive preference data. However, existing FedDPO analysis assumes all clients converge to a single shared policy, which is fundamentally at odds with the heterogeneous preference distributions present in practice. We propose Personalized FedDPO (pFedDPO), in which each client maintains a small private LoRA adapter atop a shared global base model. In pFedDPO, the base model is synchronized via FedAvg each round, while LoRA adapters remain local and capture client-specific preference patterns. We prove that this decomposition replaces the gradient diversity κ2 in the FedDPO bound with a residual heterogeneity κeff2, which is no larger than κ2 when adapters absorb client-specific directions, at the cost of an additive adapter-bias term that vanishes as adapters converge. Experiments on preference datasets show that pFedDPO consistently attains a lower gradient norm than vanilla FedDPO while remaining competitive in training loss and per-client reward accuracy.
Zhanhong Jiang, Soumik Sarkar
Translational AI Center, Iowa State University, Ames, IA 50011 USA · Department of Mechanical Engineering, Iowa State University, Ames, IA 50011 USA
Federated Learning (FL) offers a privacy-preserving pathway for aligning Large Language Models (LLMs); however, existing frameworks typically enforce a monolithic reward model, inevitably averaging out inherently conflicting user preferences (e.g., helpfulness vs. harmlessness). While Variational Preference Learning (VPL) offers a pathway to personalization, adapting it to decentralized settings presents a fundamental challenge: posterior collapse driven by severe local data scarcity and heterogeneity. In this paper, we propose Federated Variational Preference Alignment with Gumbel-Softmax Prior (FedVPA-GP), a framework designed to disentangle diverse preferences without compromising privacy. To stabilize variational inference, we introduce a Federated Mixture Prior that enables clients to leverage the aggregate population distribution as a dynamic prior. Furthermore, we incorporate an Orthogonal Loss that explicitly enforces the separation of preference prototypes in the latent space. Experiments on the HH-RLHF dataset demonstrate that FedVPA-GP significantly outperforms monolithic baselines, successfully disentangling conflicting user intents and enabling dynamic preference switching.
Jabin Koo, Hoyoung Kim, Minwoo Jang +1
Department of CSE, POSTECH, Pohang, Republic of Korea · National AI Research Lab, Seoul, Republic of Korea · Graduate School of AI, POSTECH, Pohang, Republic of Korea
Aligning large language models (LLMs) with diverse human preferences requires pluralistic alignment, where a single model must respect the values of multiple distinct groups simultaneously. In federated reinforcement learning from human feedback (FedRLHF), these groups align a shared policy without centralizing preference data, which makes fair reward aggregation essential. Existing aggregation methods exhibit clear trade offs: average based aggregation systematically under aligns worst performing groups, while min aggregation prioritizes worst group performance at the cost of overall alignment. We propose APPA, an Adaptive Preference Pluralistic Alignment framework that dynamically reweights group level rewards based on historical alignment rewards. Our approach prioritizes under aligned groups without degrading well aligned ones, while requiring no access to raw preference data. Integrated into a proximal policy optimization (PPO) based FedRLHF pipeline and evaluated on GLOBALQA and OQA across three model families (Gemma 2 2B, Llama 3.2 3B, Qwen3 0.6B), APPA achieves strong fairness alignment trade offs, improving worst group alignment by up to 28% over average aggregation while maintaining higher overall alignment than min aggregation across most configurations.
Mahmoud Srewa, Tianyu Zhao, Salma Elmalaki
Department of Electrical Engineering and Computer Science, University of California, Irvine, Irvine, CA 92697, USA