Aligning large language models (LLMs) with diverse human preferences requires pluralistic alignment, where a single model must respect the values of multiple distinct groups simultaneously. In federated reinforcement learning from human feedback (FedRLHF), these groups align a shared policy without centralizing preference data, which makes fair reward aggregation essential. Existing aggregation methods exhibit clear trade offs: average based aggregation systematically under aligns worst performing groups, while min aggregation prioritizes worst group performance at the cost of overall alignment. We propose APPA, an Adaptive Preference Pluralistic Alignment framework that dynamically reweights group level rewards based on historical alignment rewards. Our approach prioritizes under aligned groups without degrading well aligned ones, while requiring no access to raw preference data. Integrated into a proximal policy optimization (PPO) based FedRLHF pipeline and evaluated on GLOBALQA and OQA across three model families (Gemma 2 2B, Llama 3.2 3B, Qwen3 0.6B), APPA achieves strong fairness alignment trade offs, improving worst group alignment by up to 28% over average aggregation while maintaining higher overall alignment than min aggregation across most configurations.
Figures & tables
Figure 1: Overview of A PP A: fedrlhf for pluralistic alignment of group preferences. At each ppo iteration, the server distributes rollouts to federated groups; each group scores responses using its local PluralLLM module and returns group-specific rewards. The server aggregates these rewards via the A PP A adaptive aggregation before updating the policy.
Property
Park et al. (2024)
A PP A (Ours)
Aggregation weight α
Single fixed scalar (manual, same for all, never updated)
Per-group αgt : automatic, data-driven, updated each PPO step
Reward source
Centralized reward model
Federated PluralLLM per group
Data sharing
Required
Not required (privacy-preserving)
Fairness-aware threshold
×
✓ ( FI threshold)
History tracking
×
EMA over group alignment history
Weight computation
Manually swept ( α∈{−∞,−1,0,1,+∞} )
Reversed softmax over EMA history (Eq. 6 )
Table 1: Key differences between Park et al. (2024) and A PP A (ours).
Figure 2: Per-group alignment score comparison across selected demographic groups (Base vs. SFT vs. PPO-Min vs. PPO-Average vs. PPO-A PP A). Top row: GLOBALQA, eight countries with diverse opinions (JS metric). Bottom row: OQA, eight US demographic groups with diverse opinions (Wasserstein metric). A wider, more uniformly filled polygon indicates higher and more equitable alignment across groups.
Figure 3: Fairness–Alignment trade-off: QA Fairness Index vs. Minimum Alignment Score (GLOBALQA). Left: DPA task using JS metric. Right: OPA task using Borda metric. Each point is a (model, aggregation strategy) pair. PPO-A PP A (red stars) generally occupies the upper-right region across the evaluated model families. Gray lines connect strategy points within each model family, illustrating the progression from Base through SFT to PPO strategies.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Model
Method
Server Agg.
FI
Avg AS
Min AS
Format Score
JS ↑
Was. ↑
Cos. ↑
JS ↑
Was. ↑
Cos. ↑
JS ↑
Was. ↑
Cos. ↑
GLOBALQA
gemma-2-2b
BASE
-
0.9389
0.9392
0.9368
0.682
0.783
0.754
0.672
0.773
0.739
0.968
SFT
-
0.9399
0.9394
0.9389
0.786
0.847
0.855
0.779
0.841
0.840
0.958
PPO/JS
MIN
0.9995
0.9997
0.9996
0.834
0.886
0.909
0.808
0.862
0.885
0.972
AVERAGE
0.9695
0.9697
0.9696
0.839
0.890
0.914
0.812
0.868
0.890
0.957
A PP A (OURS)
0.9994
0.9997
0.9995
0.861
0.912
0.916
0.843
0.898
0.891
0.963
Appendix
Table 2: Distributional Preference Alignment (DPA) (GLOBALQA + OQA, Base vs SFT vs PPO). Reward-wise QA fairness and alignment metrics.
Dataset
Model
Method
Server Agg.
FI
Avg AS
Min AS
Format Score
Bor.
Bor.
Bor.
GLOBALQA
gemma-2-2b
BASE
-
0.8265
0.461
0.434
0.999
SFT
-
0.8313
0.339
0.329
1.000
PPO/Bor.
MIN
0.8725
0.475
0.420
1.000
AVERAGE
0.8539
0.469
0.359
1.000
A PP A (OURS)
0.8911
0.511
0.461
1.000
Appendix
Table 3: Ordinal Preference Alignment (OPA) (GLOBALQA + OQA, Base vs SFT vs PPO). Reward-wise QA fairness and alignment metrics.
Hyperparameter
Value
Models
Base models
gemma-2-2b-it , Llama-3.2-3 B-Instruct ,
Qwen3-0.6B-Instruct
Precision
BF16
Data / Task
Train/valid split
80/20%
Appendix
Table 4: SFT configuration and hyperparameters.
Hyperparameter
Value
General
Policy model
gemma-2-2b-it (SFT), Llama-3.2-3B-Instruct (SFT),
Qwen3-0.6B-Instruct (SFT)
Value model
gemma-2-2b-it (SFT), Llama-3.2-3B-Instruct (SFT),
Qwen3-0.6B-Instruct (SFT)
Model / Quantization
Appendix
Table 5: PPO configuration and hyperparameters (policy and value models initialized from the respective SFT checkpoints).
Hyperparameter
Value
EMA decay ( λ )
0.8
Softmax temperature ( T )
0.1
Fairness Index threshold ( τ )
0.99
Initial historical alignment reward ( hg0 )
0 (all groups)
Alignment weight ( ω )
0.85
Min. mean threshold ( μmin )
1×10−6
Appendix
Table 6: A PP A-specific hyperparameters, fixed across all experiments.
Figure 6: Fairness-Alignment Trade-off: QA Fairness Index vs. Minimum Alignment Score (OQA). Left: DPA task using Wasserstein distance. Right: OPA task using Borda metric. Each point is a (model, aggregation strategy) pair. PPO-A PP A (red stars) generally occupies the upper-right, near-Pareto-optimal region across the evaluated model families on OQA as well. Gray lines connect strategy points within each model family, illustrating the progression from Base through SFT to PPO strategies.
College of Intelligence and Computing, Tianjin University, Tianjin, China · College of Computing and Data Science, Nanyang Technological University, Singapore
Department of CSE, POSTECH, Pohang, Republic of Korea · National AI Research Lab, Seoul, Republic of Korea · Graduate School of AI, POSTECH, Pohang, Republic of Korea