Aligning large language models (LLMs) with diverse human preferences requires pluralistic alignment, where a single model must respect the values of multiple distinct groups simultaneously. In federated reinforcement learning from human feedback (FedRLHF), these groups align a shared policy without centralizing preference data, which makes fair reward aggregation essential. Existing aggregation methods exhibit clear trade offs: average based aggregation systematically under aligns worst performing groups, while min aggregation prioritizes worst group performance at the cost of overall alignment. We propose APPA, an Adaptive Preference Pluralistic Alignment framework that dynamically reweights group level rewards based on historical alignment rewards. Our approach prioritizes under aligned groups without degrading well aligned ones, while requiring no access to raw preference data. Integrated into a proximal policy optimization (PPO) based FedRLHF pipeline and evaluated on GLOBALQA and OQA across three model families (Gemma 2 2B, Llama 3.2 3B, Qwen3 0.6B), APPA achieves strong fairness alignment trade offs, improving worst group alignment by up to 28% over average aggregation while maintaining higher overall alignment than min aggregation across most configurations.
Figures & tables
Figure 1: Overview of A PP A: fedrlhf for pluralistic alignment of group preferences. At each ppo iteration, the server distributes rollouts to federated groups; each group scores responses using its local PluralLLM module and returns group-specific rewards. The server aggregates these rewards via the A PP A adaptive aggregation before updating the policy.
Property
Park et al. (2024)
A PP A (Ours)
Aggregation weight α
Single fixed scalar (manual, same for all, never updated)
Per-group αgt : automatic, data-driven, updated each PPO step
Reward source
Centralized reward model
Federated PluralLLM per group
Data sharing
Required
Not required (privacy-preserving)
Fairness-aware threshold
×
✓ ( FI threshold)
History tracking
×
EMA over group alignment history
Weight computation
Manually swept ( α∈{−∞,−1,0,1,+∞} )
Reversed softmax over EMA history (Eq. 6 )
Table 1: Key differences between Park et al. (2024) and A PP A (ours).
Figure 2: Per-group alignment score comparison across selected demographic groups (Base vs. SFT vs. PPO-Min vs. PPO-Average vs. PPO-A PP A). Top row: GLOBALQA, eight countries with diverse opinions (JS metric). Bottom row: OQA, eight US demographic groups with diverse opinions (Wasserstein metric). A wider, more uniformly filled polygon indicates higher and more equitable alignment across groups.
Figure 3: Fairness–Alignment trade-off: QA Fairness Index vs. Minimum Alignment Score (GLOBALQA). Left: DPA task using JS metric. Right: OPA task using Borda metric. Each point is a (model, aggregation strategy) pair. PPO-A PP A (red stars) generally occupies the upper-right region across the evaluated model families. Gray lines connect strategy points within each model family, illustrating the progression from Base through SFT to PPO strategies.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Model
Method
Server Agg.
FI
Avg AS
Min AS
Format Score
JS ↑
Was. ↑
Cos. ↑
JS ↑
Was. ↑
Cos. ↑
JS ↑
Was. ↑
Cos. ↑
GLOBALQA
gemma-2-2b
BASE
-
0.9389
0.9392
0.9368
0.682
0.783
0.754
0.672
0.773
0.739
0.968
SFT
-
0.9399
0.9394
0.9389
0.786
0.847
0.855
0.779
0.841
0.840
0.958
PPO/JS
MIN
0.9995
0.9997
0.9996
0.834
0.886
0.909
0.808
0.862
0.885
0.972
AVERAGE
0.9695
0.9697
0.9696
0.839
0.890
0.914
0.812
0.868
0.890
0.957
A PP A (OURS)
0.9994
0.9997
0.9995
0.861
0.912
0.916
0.843
0.898
0.891
0.963
Appendix
Table 2: Distributional Preference Alignment (DPA) (GLOBALQA + OQA, Base vs SFT vs PPO). Reward-wise QA fairness and alignment metrics.
Dataset
Model
Method
Server Agg.
FI
Avg AS
Min AS
Format Score
Bor.
Bor.
Bor.
GLOBALQA
gemma-2-2b
BASE
-
0.8265
0.461
0.434
0.999
SFT
-
0.8313
0.339
0.329
1.000
PPO/Bor.
MIN
0.8725
0.475
0.420
1.000
AVERAGE
0.8539
0.469
0.359
1.000
A PP A (OURS)
0.8911
0.511
0.461
1.000
Appendix
Table 3: Ordinal Preference Alignment (OPA) (GLOBALQA + OQA, Base vs SFT vs PPO). Reward-wise QA fairness and alignment metrics.
Hyperparameter
Value
Models
Base models
gemma-2-2b-it , Llama-3.2-3 B-Instruct ,
Qwen3-0.6B-Instruct
Precision
BF16
Data / Task
Train/valid split
80/20%
Appendix
Table 4: SFT configuration and hyperparameters.
Hyperparameter
Value
General
Policy model
gemma-2-2b-it (SFT), Llama-3.2-3B-Instruct (SFT),
Qwen3-0.6B-Instruct (SFT)
Value model
gemma-2-2b-it (SFT), Llama-3.2-3B-Instruct (SFT),
Qwen3-0.6B-Instruct (SFT)
Model / Quantization
Appendix
Table 5: PPO configuration and hyperparameters (policy and value models initialized from the respective SFT checkpoints).
Hyperparameter
Value
EMA decay ( λ )
0.8
Softmax temperature ( T )
0.1
Fairness Index threshold ( τ )
0.99
Initial historical alignment reward ( hg0 )
0 (all groups)
Alignment weight ( ω )
0.85
Min. mean threshold ( μmin )
1×10−6
Appendix
Table 6: A PP A-specific hyperparameters, fixed across all experiments.
Figure 6: Fairness-Alignment Trade-off: QA Fairness Index vs. Minimum Alignment Score (OQA). Left: DPA task using Wasserstein distance. Right: OPA task using Borda metric. Each point is a (model, aggregation strategy) pair. PPO-A PP A (red stars) generally occupies the upper-right, near-Pareto-optimal region across the evaluated model families on OQA as well. Gray lines connect strategy points within each model family, illustrating the progression from Base through SFT to PPO strategies.
Reinforcement Learning from Human Feedback (RLHF) aggregates heterogeneous preferences into a single reward model, assuming preference homogeneity. When preferences are heterogeneous, this aggregation induces a procedural fairness failure where majority preference groups dominate reward learning while minority preferences are systematically under-represented. This work defines procedural fairness in alignment as preserving distinct preference signals during reward modeling and shows that standard RLHF violates this via preference averaging. Preference-Aware RLHF (PA-RLHF) is introduced, separating optimization across preference modes at the reward learning stage. In a controlled setting, PA-RLHF improves overall alignment accuracy from 46.9% to 67.9% and reduces the fairness gap between best and worst aligned groups from 15.9 to 9.6 percentage points. These results show that procedural fairness failures in alignment can arise from structural design choices in reward learning, even in controlled, noise-free settings, with direct implications for large language models and agentic systems, where biased reward models can compound inequities across sequential decisions.
M P V S Gopinadh, Karthik Kamuju, Kummari Avinash +2
Aligning large language models (LLMs) with human preferences in federated learning (FL) is challenging due to decentralized, privacy-sensitive, and highly non-IID preference data. Direct Preference Optimization (DPO) offers an efficient alternative to reinforcement learning with human feedback (RLHF), but its direct application in FL suffers from severe performance degradation under non-IID data and limited generalization of implicit rewards. To bridge this gap, we propose FedPDPO (Federated Personalized Direct Preference Optimization), a personalized federated framework for preference alignment of LLMs. It adopts a parameter-efficient fine-tuning architecture where each client maintains a frozen pretrained LLM backbone augmented with a Low-Rank Adaptation (LoRA) adapter, enabling communication-efficient aggregation. To address non-IID heterogeneity, we devise (1) the globally shared LoRA adapter with the personalized client-specific LLM head. Moreover, we introduce (2) a personalized DPO training strategy with a client-specific explicit reward head to complement implicit rewards and further alleviate non-IID heterogeneity, and (3) a bottleneck adapter to balance global and local features. We provide theoretical analysis establishing the probabilistic foundation and soundness. Extensive experiments on multiple preference datasets demonstrate state-of-the-art performance, achieving up to 4.80% average accuracy improvements in federated intra-domain and cross-domain settings.
Kewen Zhu, Liping Yi, Zhiming Zhao +3
College of Intelligence and Computing, Tianjin University, Tianjin, China · College of Computing and Data Science, Nanyang Technological University, Singapore
Federated Learning (FL) offers a privacy-preserving pathway for aligning Large Language Models (LLMs); however, existing frameworks typically enforce a monolithic reward model, inevitably averaging out inherently conflicting user preferences (e.g., helpfulness vs. harmlessness). While Variational Preference Learning (VPL) offers a pathway to personalization, adapting it to decentralized settings presents a fundamental challenge: posterior collapse driven by severe local data scarcity and heterogeneity. In this paper, we propose Federated Variational Preference Alignment with Gumbel-Softmax Prior (FedVPA-GP), a framework designed to disentangle diverse preferences without compromising privacy. To stabilize variational inference, we introduce a Federated Mixture Prior that enables clients to leverage the aggregate population distribution as a dynamic prior. Furthermore, we incorporate an Orthogonal Loss that explicitly enforces the separation of preference prototypes in the latent space. Experiments on the HH-RLHF dataset demonstrate that FedVPA-GP significantly outperforms monolithic baselines, successfully disentangling conflicting user intents and enabling dynamic preference switching.
Jabin Koo, Hoyoung Kim, Minwoo Jang +1
Department of CSE, POSTECH, Pohang, Republic of Korea · National AI Research Lab, Seoul, Republic of Korea · Graduate School of AI, POSTECH, Pohang, Republic of Korea