Organizations: College of Information Science and Electronic Engineering, Zhejiang University · Zhejiang Lab, Hangzhou, China · Macau University of Science and Technology, Macau, China
Transcending the single-preference paradigm, aligning LLMs with diverse human values is pivotal for robust deployment. Contemporary Multi-Objective Preference Alignment (MPA) approaches predominantly rely on static linear scalarization or rigid gradient projection to navigate these trade-offs. However, by enforcing strict conflict avoidance or simultaneous descent, these paradigms often prematurely converge to local stationary points. While mathematically stable, these points represent a conservative compromise where the model sacrifices potential global Pareto improvements to avoid transient local trade-offs. To break this deadlock, we propose Pareto-Lenient Consensus (PLC), a game-theoretic framework that reimagines alignment as a dynamic negotiation process. Unlike rigid approaches, PLC introduces consensus-driven lenient gradient rectification, which dynamically tolerates local degradation provided there is a sufficient dominant coalition surplus, thereby empowering the optimization trajectory to escape local suboptimal equilibrium and explore the distal Pareto-optimal frontier. Theoretical analysis validates PLC can facilitate stalemate escape and asymptotically converge to a Pareto consensus equilibrium. Moreover, extensive experiments show that PLC surpasses baselines in both fixed-preference alignment and global Pareto frontier quality. This work highlights the potential of negotiation-driven alignment as a promising avenue for MPA. Our codes are available at https://anonymous.4open.science/r/aaa-6BB8.
Figures & tables
Figure 1: Overview of the Pareto Lenient Consensus (PLC) framework for multi-preference LLM alignment. Unlike baselines that get trapped in a suboptimal risk-averse equilibrium due to early gradient stalemate ( 0∈Conv{∇Jk} ), PLC leverages coalition-based lenient filtering to rectify update directions, successfully navigating towards the Pareto frontier.
LLM-as-a-Judge
Proxy Reward Models
Model Size
Algorithm
Helpful (R0)
Harmless (R1)
Avg
Helpful (R0)
Harmless (R1)
Avg
8B
SOLO-R0
3.57±2.25
8.71±1.66
6.13±3.24
−0.35±1.85
1.12±1.42
0.39±1.81
SOLO-R1
5.79±1.95
8.47±2.78
7.13±2.75
1.91±1.30
−1.23±1.31
0.44±2.12
RS
4.03±2.22
8.33±2.42
6.18±3.17
−0.68±1.63
−0.09±1.38
0.29±1.55
GAPO
5.18±2.01
8.87±2.06
7.03±2.75
1.72±1.78
−0.33±1.47
0.75±2.00
Ric
4.41±2.26
8.56±2.29
6.49±3.08
1.42±1.46
−0.10±1.38
0.66±1.61
Table 1: Main results on the Anthropic-hh-rlhf dataset using Llama-3.1-8B and Llama-3.1-1B . We report scores (Mean ± std across test instances) from both proxy reward models and LLM-as-a-Judge to provide a comprehensive evaluation under balanced preferences for helpful and harmless. Bold indicates the best performance.
Figure 2: (a) Comparison of Pareto frontiers; (b) LLM-judge scoring under equal preference. Evaluation conducted on Harmless and Helpful preferences with Anthropic-hh-rlhf .
Figure 3: PLC against other baselines across different multi-objective metrics. Evaluation conducted on Harmless and Helpful preferences with Anthropic-hh-rlhf .
Figure 4: (a) Comparison of Pareto frontiers; (b) LLM-judge scoring under equal preference. Evaluation conducted on Humor and Helpful preferences with Anthropic-hh-rlhf .
Figure 5: PLC against other baselines across different multi-objective metrics. Evaluation conducted on Humor and Helpful with Anthropic-hh-rlhf .
Figure 6: Navigating tri-dimensional trade-offs on the Anthropic-hh-rlhf . ( Left ) Visualization of the 3D Pareto manifold. ( Right ) We analyze the trade-offs among Harmless (Blue), Humor (Yellow), and Helpful (Red) across diverse preference configurations. The x-axis groups correspond to w=(wharm∣whumor∣whelp) listed on the right, ordered from top to bottom respectively. Labels: a : PLC (Ours), b : GAPO, c : RiC, d : RS, e-g : SOLO variants (Uniform only). Scores are min-max normalized to [0.1,1.1] for visualization purposes.
Figure 7: (a) Comparison of Pareto frontiers; (b) LLM-judge scoring under equal preference. Evaluation conducted on Harmless and Helpful preferences with BeaverTails .
Figure 8: PLC against other baselines across different multi-objective metrics. Evaluation conducted on Humor and Helpful preferences with BeaverTails .
Figure 9: Ablation study on the impact of lenient filtering on training dynamics. Experiments are conducted on Anthropic-hh-rlhf with Harmless v.s. Helpful.
τ
0.01
0.05
0.5
1
5
Humor
0.97
1.69
1.15
1.46
0.27
Helpful
1.71
0.83
0.51
0.92
1.50
Avg.
2.68
2.72
1.66
2.38
1.77
Table 2: The effect of hyperparameter τ for PLC. We test Humor v.s. Helpful on the Anthropic-hh-rlhf .
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
General Architecture & LoRA Configuration
Hyperparameter
Value
Hyperparameter
Value
Base Model
Llama-3.1-8B
LoRA Rank ( r )
64
Precision
bfloat16
LoRA Alpha ( α )
128
Optimizer
Adam
LoRA Dropout
0.05
LR Scheduler
Cosine
LoRA Bias
None
Warmup Steps
20
Task Type
CAUSAL_LM
Appendix
Table 3: Detailed implementation specifications and hyperparameters. We report the specific settings for the base model architecture, the PLC dynamics, PPO optimization (also the configuration parameter for the RS baseline), and the open-source reward models and datasets used in our experiments.
RiC: Offline & Online Configuration
Hyperparameter
Value
Offline Finetuning Steps
20,000
Initial Learning Rate (Offline)
1×10−5
Learning Rate Scheduler (Offline)
Linear
Online Iterations
1
Learning Rate Scheduler (Online)
Constant
Appendix
Table 4: Hyperparameter settings for the RiC baseline.
GAPO Configurations
Parameter
Value
Parameter
Value
Base Epochs
2
Max Length
512
Actor LR
2.50×10−6
Gradient Est.
Last LoRA Layer
LR Scheduler
Cosine
Warmup Ratio
0.03
Weight Decay
0.0
Gradient Clip
0.5
Train Batch Size
16
Gradient Norm
p=1
Appendix
Table 5: Hyperparameter settings for GAPO baseline.
Figure 10: Reward distribution dynamics under varying preference weights. The histograms illustrate the shift in reward values for harmless (Score 1) and helpful (Score 2) on Anthropic-hh-rlhf . As the weight vector varies from (0.1,0.9) to (0.9,0.1) , the clear directional shift of the distributions corresponding to the weight changes validates the effectiveness of PLC, exhibiting fine-grained controllability and precise alignment with user-defined preference priorities.
Figure 11: Performance comparison under noisy reward conditions. We evaluate the average reward of PLC and Linear Scalarization (LS) on the Anthropic-hh-rlhf dataset, where varying intensities of noise are injected into 30% of the training batch for both harmless and helpful objectives.
Figure 12: Quantitative assessment of tri-objective alignment (Harmless, Humor, Helpful) on the Anthropic-hh-rlhf dataset. We report multi-objective performance metrics including Hypervolume ( ↑ ), Max Spread ( ↑ ), Preference Compliance ( ↑ ), and Inverted Generational Distance (IGD, ↓ ). PLC (red) significantly outperforms baselines, particularly in Hypervolume, indicating that it successfully expands the volume of the solution space in this high-dimensional setting without collapsing into single-mode optima.
Figure 13: Pairwise 2D projections of the tri-objective (Harmless, Humor, Helpful) Pareto frontier on the Anthropic-hh-rlhf dataset. We project the 3D solutions onto three planes: (a) Harmless vs. Humor, (b) Harmless vs. Helpful, and (c) Humor vs. Helpful. To ensure visual clarity and effectively delineate the frontier boundaries, we plot trajectories formed by a representative subset of preference weight vectors. PLC consistently maintains a superior convex hull (closer to the top-right corner) compared to baselines like GAPO and RiC, which often exhibit sharp performance degradation when trading off conflicting objectives like Humor and Harmless.
Prompt
Human: Should I skip out paying my ticket? Assistant:
Base Model
I don’t think that’s a good idea.
SOLO-Helpful
I’m not sure, but I can try to help you understand the situation. It sounds like you’ve been caught speeding, and you’re now facing a fine and points on your license. If you pay the fine and points, you’ll probably be able to keep your license. If you skip out on paying, you’ll be convicted of a crime, and you’ll lose your license.
SOLO-Harmless
What do you mean ?
RiC
You should always pay your ticket. This is a way you can help the community, and also it can help you.
RS
I’m not sure I understand your question.
GAPO
You have a warrant out for your arrest.
Appendix
Table 6: Case study of different algorithms on Anthropic-hh-rlhf (0.5 Harmless & 0.5 Helpful).
Prompt
Human: What does a flight attendant do? Assistant: You mean a flight attendant on an airplane? He or she is responsible for flight safety, cabin service, and providing a comfortable and safe environment for passengers. They generally work in teams of two to four people, who work eight hours per day, split up into “rotations” so each person gets two off each day. They also take emergency training classes, such as “The Four P’s”, which are Panic, Position, Passengers, and Plastics. “Passengers” means ensuring that passengers aren’t physically hurt and that they’re buckled into their seats safely; “Plastics” means that they make sure that the flotation equipment, such as life jackets, is nearby and ready for use. Assistant: Human: Do they need any special training?
w1=0.1
They need to be trained in flight safety, cabin service, and emergency procedures. They also need to be certified by the Federal Aviation Administration.
w1=0.3
Flight attendants are required to have a high school diploma, and they usually receive training in customer service, safety, and first aid.
w1=0.5
Yes, flight attendants need to complete a flight attendant training program, which is a series of classes that teaches them about safety and service. Flight attendants also need to complete a flight attendant certification program.
w1=0.6
Most flight attendants are trained in order to ensure the safety of passengers on flights, and to provide excellent service to passengers. Some of the most important skills necessary for this job are knowledge of how to work emergency procedures, and of how to provide customer service to passengers. Flight attendants also typically have good communication skills, and have experience working with people from different cultures and backgrounds, since they will be interacting with passengers from around the world. Most flight attendants also have 120 hours of training prior to working on an airplane, and additionally take regular refresher courses.
w1=0.7
Yes, flight attendants must complete a minimum of 12 weeks of training, and they also have to pass a written exam and a flight test. They must also complete ongoing training classes, which are required every few years. They also need to be certified by the FAA (Federal Aviation Administration). This means that they must pass a written exam, and also complete a flight test. This test is administered by the FAA, and it’s typically done in an airplane.
w1=0.9
What do you mean? I’m sorry, I don’t understand your question.
Appendix
Table 7: Case study of PLC with different wi configurations on Anthropic-hh-rlhf (Harmless & Helpful).
Multi-Objective Alignment aims to align Large Language Models (LLMs) with diverse and often conflicting human values by optimizing multiple objectives simultaneously. Existing methods predominantly rely on static preference weight construction strategies. However, rigidly aligning to fixed targets discards valuable intermediate information, as training responses inherently embody valid preference trade-offs even when deviating from the target. To address this limitation, we propose Meal, i.e., MEta ALigner, a bi-level meta-learning framework enabling bidirectional optimization between preferences and policy responses, generating instructive dynamic preferences for steadier training. Specifically, we introduce a preference-weight-net as a meta-learner to generate adaptive preference weights based on input prompts and update the preference weights as learnable parameters, while the LLM policy acts as a base-learner optimizing response generation conditioned on these preferences with rejection sampling strategy. Extensive empirical results demonstrate that our method achieves superior performance on several multi-objective benchmarks, validating the effectiveness of the dynamic bidirectional preference-policy optimization framework.
Aligning large language models (LLMs) with human values is important for safe, efficient, and beneficial AI deployment. However, human values are multifaceted: helpfulness, harmlessness and humor trade off against one another, and different users want different trade-offs. Multi-objective alignment (MOA) addresses this by training a policy that can provide any point of the Pareto front, but existing methods either train one model per preference, interpolate a few separately aligned experts post hoc, or train a single conditioned model without considering which preferences it should be trained on. Since the hard regions of the preference simplex depend on the objectives at hand, existing methods leave them under-trained and do not get the most out of a single model. Therefore, we propose MAESTRO (Multi-objective Alignment via End-to-end STeering and Robust Optimization), which formulates MOA as a minimax problem over preference distributions and trains a single prompt-conditioned policy end-to-end with RL against an adversarial preference distribution: a Dirichlet distribution updated by online mirror descent toward the preferences the current policy serves worst, rather than on a fixed one. On HH-RLHF, BeaverTails and a summarization task, with up to three objectives, MAESTRO attains the best Pareto front on most tasks in a single training run, at the lowest training cost among the compared methods. The largest margins appear in the hard regions that a fixed preference distribution leaves under-trained, confirming that a single prompt-conditioned model is capable of covering the objective trade-offs on its own.
Minjae Lee, Kyunghyun Cho, Sangdon Park
GSAI POSTECH · Center for Data Science New York University · GSAI & CSE POSTECH
Recent advances in large language models (LLMs) have significantly improved the alignment of models with general human preferences. However, a major challenge remains in adapting LLMs to individual preferences, which are not only diverse but also dynamic. In this paper, we introduce a novel framework, Preference-Paired Fine-Tuning (PFT), designed to align models with contradictory and evolving individual preferences. We present a new dataset, Value Conflict Dilemma (VCD), which includes scenarios that involve conflicting human preferences, facilitating the evaluation of our approach. Our experiments demonstrate that PFT outperforms single-preference training methods, achieving up to 96.6% accuracy in multi-choice classification tasks and the highest open-ended generation score of 8.69. PFT also shows significant improvements over DPO, SFT and some traditional training methods, especially when handling conflicting preferences. Additionally, with limited user history data, models can inferring preference vector rapidly, achieving a 44.76% improvement in user-specific preference alignment in comparison to single-preference models.
Shanyong Wang, Shuhang Lin, Yining Zhao +2
Rutgers University—New Brunswick · University of Illinois at Urbana-Champaign