Personalized Federated Direct Preference Optimization via LoRA Adapters

Zhanhong Jiang; Soumik Sarkar

cs.LGOct 9, 2026

Personalized Federated Direct Preference Optimization via LoRA Adapters

Authors: Zhanhong Jiang, Soumik Sarkar

Organizations: Translational AI Center, Iowa State University, Ames, IA 50011 USA · Department of Mechanical Engineering, Iowa State University, Ames, IA 50011 USA

Abstract

Federated Direct Preference Optimization (FedDPO) enables privacy-preserving alignment of language models across distributed agents without centralizing sensitive preference data. However, existing FedDPO analysis assumes all clients converge to a single shared policy, which is fundamentally at odds with the heterogeneous preference distributions present in practice. We propose Personalized FedDPO (pFedDPO), in which each client maintains a small private LoRA adapter atop a shared global base model. In pFedDPO, the base model is synchronized via FedAvg each round, while LoRA adapters remain local and capture client-specific preference patterns. We prove that this decomposition replaces the gradient diversity $κ^2$ in the FedDPO bound with a residual heterogeneity $κ^2_{\mathrm{eff}}$ , which is no larger than $κ^2$ when adapters absorb client-specific directions, at the cost of an additive adapter-bias term that vanishes as adapters converge. Experiments on preference datasets show that pFedDPO consistently attains a lower gradient norm than vanilla FedDPO while remaining competitive in training loss and per-client reward accuracy.

Figures & tables

Figure 1: Illustration of the pFedDPO framework. The server maintains a shared global base model $\theta$ . Each client $i$ receives $\theta$ from the server, combines it with its private LoRA adapter $\phi_{i}$ (never shared), and performs local DPO updates on its private dataset $\mathcal{D}_{i}$ . Only the base model updates $\Delta\theta_{i}$ are uploaded to the server for FedAvg aggregation, while adapters and data remain local. This design is intended to reduce effective gradient diversity ( $\kappa^{2}_{\text{eff}}\leq\kappa^{2}$ under the conditions in the Appendix) and enables personalization without additional communication overhead.

Figure 2: Comparison of baselines with SHP data.

Figure 3: Per-client preference accuracy with SHP data.

Figure 4: Low rank

r

ablation with SHP data.

Figure 5: Gradient diversity

\kappa^{2}

ablation with SHP data.

Figure 6: Participation

S

ablation with SHP data.

Method	Base Model	Personalization	Conv. Theory	Comm. Vol.	Feedback	Aggregation
( Jiang 2026 )	Full model	✗ (shared policy)	✓ rate bound ( $\kappa^{2}$ floor)	$d$ per client	Paired	FedAvg (full model)
( Spadea and Seneviratne 2025 )	Full model (Alpaca-7B)	✗ (uniform)	✗ (empirical only)	$d$ per client	KTO: single; DPO: paired	FedAvg (full model)
( Zhu et al. 2026 )	Shared LoRA + bottleneck adapter	✓ (private LLM head + reward head)	~ (probabilistic soundness only)	$\mathcal{O}((m+n)r)$ per client (LoRA dims)	Paired + explicit reward	FedAvg on shared modules; heads private
pFedDPO (ours)	Base + private LoRA per client	✓ (private $\phi_{i}$ , never shared)	✓ rate bound ( $\kappa^{2}_{\text{eff}}$ + $C\varepsilon_{\phi}$ )	$d$ per client (same as FedDPO)	Paired	FedAvg on base only; adapter stays local

Table 1: Comparison of federated fine-tuning methods. ✓=yes; ✗=no;

\sim

=partial. Comm.Vol.=per-round communication volume per selected client. pFedDPO transmits

d

-dimensional base updates (same as FedDPO), not LoRA dimensions. Its distinguishing property is a rate-explicit bound capturing how private adapters reduce

\kappa^{2}

\kappa^{2}_{\text{eff}}

Hyperparameter	Value
Model	DistilGPT-2 ( $\sim 82$ M)
# Agents (N)	5
Training samples per agent	120
Inverse temperature $\beta$	0.2
Optimizer	AdamW
Learning rate $\eta$	$1e^{-4}$

Table 2: Hyperparameter Setup.

Method	Grad Norm ( $\downarrow$ )	Train Loss ( $\downarrow$ )	Reward Acc. ( $\uparrow$ )	Empirical $\kappa^{2}$ ( $\downarrow$ , final)
Centralized DPO	$\sim$ 0.70	$\sim$ 0.10 (unstable)	$\sim$ 0.75	N/A (no agents)
FedDPO (S=5) ( Jiang 2026 )	$\sim$ 0.2	$\sim$ 0.05	$\sim$ 0.82	Not measured
pFedDPO r=8 (ours, default)	$\sim$ 0.02	$\sim$ 0.06	$\sim$ 0.80	$\sim$ 150–200 (from $\sim$ 1500)

Table 3: Comparison of DPO variants. Results at round 80 on SHP. Centralized DPO performs worst in this regime: with batch size 4 it receives far less gradient work per round than distributed methods (effective batch

5\times 4=20

for FedDPO), consistent with the computation-dominated analysis in ( Jiang 2026 ) .

\kappa^{2}

is measured on base model gradients only.

Figure 7: Comparison of baselines with HH-RLHF data.

Figure 8: Gradient diversity

\kappa^{2}

ablation with HH-RLHF data.

Figure 9: Per-client preference accuracy with HH-RLHF data.

Figure 10: Low rank

r

ablation with HH-RLHF data.

Figure 11: Participation

S

ablation with HH-RLHF data.

Personalized Language Model Alignment

Personalized language model alignment adapts model behavior to individual preferences while preserving safety, accuracy, and other shared constraints.

See all 54 papers

FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment
Mar 20, 2026Kewen Zhu, Liping Yi, Zhiming Zhao +3Personalized Language Model Alignment Direct Preference Optimization
College of Intelligence and Computing, Tianjin University, Tianjin, China · College of Computing and Data Science, Nanyang Technological University, Singapore
Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferences
May 29, 2026Jabin Koo, Hoyoung Kim, Minwoo Jang +1LLM Alignment Preference Alignment
Department of CSE, POSTECH, Pohang, Republic of Korea · National AI Research Lab, Seoul, Republic of Korea · Graduate School of AI, POSTECH, Pohang, Republic of Korea
Distributed Direct Preference Optimization
May 20, 2026Zhanhong JiangDirect Preference Optimization Non-IID Federated Learning
Translational AI Center, Iowa State University, Ames, USA.

Mar 20, 2026J/K move · Enter open · S save

FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment

Kewen Zhu, Liping Yi, Zhiming Zhao, Zhuang Qi+2

College of Intelligence and Computing, Tianjin University, Tianjin, China · College of Computing and Data Science, Nanyang Technological University, Singapore

Personalized Language Model Alignment Direct Preference Optimization Federated Fine-Tuning Non-IID Federated Learning

Open paper AskPDF arXiv

Aligning large language models (LLMs) with human preferences in federated learning (FL) is challenging due to decentralized, privacy-sensitive, and highly non-IID preference data. Direct Preference Optimization (DPO) offers an efficient alternative to reinforcement learning with human feedback (RLHF), but its direct application in FL suffers from severe performance degradation under non-IID data and limited generalization of implicit rewards. To bridge this gap, we propose FedPDPO (Federated Personalized Direct Preference Optimization), a personalized federated framework for preference alignment of LLMs. It adopts a parameter-efficient fine-tuning architecture where each client maintains a frozen pretrained LLM backbone augmented with a Low-Rank Adaptation (LoRA) adapter, enabling communication-efficient aggregation. To address non-IID heterogeneity, we devise (1) the globally shared LoRA adapter with the personalized client-specific LLM head. Moreover, we introduce (2) a personalized DPO training strategy with a client-specific explicit reward head to complement implicit rewards and further alleviate non-IID heterogeneity, and (3) a bottleneck adapter to balance global and local features. We provide theoretical analysis establishing the probabilistic foundation and soundness. Extensive experiments on multiple preference datasets demonstrate state-of-the-art performance, achieving up to 4.80% average accuracy improvements in federated intra-domain and cross-domain settings.

Abstract

Personalized Language Model Alignment

Explore similar work

FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment

Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferences

Distributed Direct Preference Optimization