Organizations: Ruhr University Bochum · Eindhoven University of Technology · Mohamed bin Zayed University of Artificial Intelligence · University of Liverpool
Aligning large language models (LLMs) with human preferences is critical for enhancing LLMs' safety, helpfulness, humor, faithfulness, etc. Current reinforcement learning from human feedback (RLHF) mainly focuses on a fixed reward learned from average human ratings, which may weaken the adaptability and controllability of varying preferences. However, creating personalized LLMs requires aligning LLMs with individual human preferences, which is non-trivial due to the scarce data per user and the diversity of user preferences in multi-objective trade-offs, varying from emphasizing empathy in certain contexts to demanding efficiency and precision in others. Can we train one LLM to produce personalized outputs across different user preferences on the Pareto front? In this paper, we introduce Multi-Objective Control (MOC), which trains a single LLM to directly generate responses in the preference-defined regions of the Pareto front. Our approach introduces multi-objective optimization (MOO) principles into RLHF to train an LLM as a preference-conditioned policy network. We improve the computational efficiency of MOC by applying MOO at the policy level, enabling us to fine-tune a 7B-parameter model on a single A6000 GPU. Extensive experiments demonstrate the advantages of MOC over baselines in three aspects: (i) controllability of LLM outputs w.r.t. user preferences on the trade-off among multiple rewards; (ii) quality and diversity of LLM outputs, measured by the hyper-volume of multiple solutions achieved; and (iii) generalization to unseen preferences. These results highlight MOC's potential for real-world applications requiring scalable and customizable LLMs.
Figures & tables
Algorithms
Explicit policy improvement
Num of trained LLMs
Inference adaptation
Preference data
Loss
MORLHF
✓
M
×
No
PPO
Rewarded Soups ( Ramé et al., 2023 )
×
N
✓
No
PPO
MODPO ( Zhou et al., 2024 )
✓
M
×
Yes
DPO
RiC ( Yang et al., 2024c )
×
1
✓
No
SFT
MOC (Ours)
✓
1
✓
No
PPO
Table 1: Comparison with the state-of-the-art MOO methods. MOC addresses MOO in a principled and efficient manner. M – the number of preferences, N – the number of reward models (objectives).
Figure 1: Solutions of MOC and Linear PPO on fishwood task and the Pareto front (line in black). MOC demonstrates advantages in both MOO (solutions lie on the Pareto front) and multi-objective control (its solutions align closely with their corresponding preference vectors, shown as the colored dashed rays). The single model trained by MOC can handle diverse preference vectors. In contrast, Linear PPO optimizes a linear scalarization of the objectives and fails to follow the preference vectors, with solutions dominated by one objective. The legend "Preference" indicates the specific weight value assigned to Reward 1 (Wood). Linear PPO is implemented by optimizing PPO w.r.t. a scalarized reward Rlin=wRwood+(1−w)Rfish (with w∈[0,1] ). In our experiments, we train Linear PPO runs with weights [0.1,0.9],[0.2,0.8],⋯,[0.8,0.2],[0.9,0.1] .
Figure 2: Controllability comparison on the Pareto front . MOC demonstrates superior controllability, indicated by the consistent ranking of solutions on their preference weights and the achieved reward values. In comparison, the baselines exhibit less stable behavior and weaker alignment with the specified preferences. MOC also achieves higher quality solutions, particularly in the Humor & Helpful alignment. Our MOC method achieves the best overall performance, supported by these results and the findings in Tables 2 , 3 and 4 . Each point represents the reward achieved across multiple instances, each with a different input preference vector. Each point’s preference weight for the x-axis reward is the numerical label on its marker.
Figure 3: Illustration of the hyper-volume concept. The hyper-volume measures the size of the objective space dominated by a set of solutions in multi-objective optimization. Larger hyper-volumes indicate better convergence and diversity of the Pareto front.
Setting
MOC (Ours)
RiC
MORLHF
Rewarded Soups
Humor-helpful
1.00 ( 5.0×10−8 )
0.78 ( 3.3×10−4 )
1.00 ( 1.7×10−2 )
1.00 ( 5.0×10−8 )
Harmless-helpful
1.00 ( 5.0×10−8 )
0.91 ( 3.0×10−5 )
1.00 ( 1.7×10−2 )
0.96 ( 5.5×10−7 )
Average
1.00
0.85
1.00
0.98
Table 2: Controllability comparison using Kendall’s tau correlation (higher is better), with corresponding p -values in parentheses (smaller is better), measuring the consistency between input preferences and output rewards. MOC significantly outperforms all the baselines.
Setting
MOC (Ours)
RiC
MORLHF
Rewarded Soups
Humor-helpful
14.176
6.692
6.769
6.100
Harmless-helpful
10.220
9.257
9.047
8.905
Average
12.198
7.974
7.908
7.502
Table 3: Hyper-volume (higher is better) comparison of different methods, measuring the volume of solutions dominated by each method’s solution set. MOC outperforms all baselines, achieving higher solution diversity and quality. The best score is marked with the blue color box.
Setting
MOC (Ours)
RiC
MORLHF
Rewarded Soups
Humor-helpful
1.439
1.260
1.057
1.005
Harmless-helpful
1.600
1.363
1.110
1.015
Average
1.520
1.312
1.084
1.010
Table 4: Comparison of solution set pair-wise distances (measuring diversity, higher is better) across methods.
Figure 4: Generalization to unseen preference vectors held out from the training. MOC and RiC-trained LLMs are compared on four random sets of unseen preference vectors. Each column corresponds to a different set of unseen preference vectors, and each row represents a different pair of reward settings. MOC solutions dominate the RiC solutions in most cases. MOC’s rewards align with the new preference vectors and the outputs under different preferences are diverse in the reward space. This suggests MOC generalizes to unseen preferences and achieves diverse trade-offs on the Pareto front. The size of each point indicates the standard deviation in rewards. The numerical labels indicate the preference weights (multiplied by 100) for the reward on the x-axis, enhancing visual clarity.
Table 7: Key information about the implementation.
Figure 5: Visualization of four groups of randomly sampled, unseen preference vectors. Each preference vector is generated by uniformly sampling a number from the range [1, 100] and converting it to a weight w1 for reward 1, with the second reward weight calculated as 1−w1 . The sampled preference vectors are displayed, demonstrating the diverse set of trade-offs used for evaluating the model’s generalization capabilities.
Setting
Group 1
Group 2
Group 3
Group 4
Humor-helpful (MOC)
17.034
19.697
17.441
19.045
Humor-helpful (RiC)
16.660
16.303
16.304
16.551
Harmless-helpful (MOC)
15.038
14.139
13.324
15.557
Harmless-helpful (RiC)
9.463
10.447
9.342
9.726
Appendix
Table 8: Hyper-volume (HV) Comparison between MOC and RiC, where MOC achieves higher HV (better output quality and diversity under different preferences). Higher is better.
Setting
Group 1
Group 2
Group 3
Group 4
Humor-helpful (MOC)
1.000
1.000
1.000
1.000
Humor-helpful (RiC)
0.800
0.800
1.000
1.000
Harmless-helpful (MOC)
1.000
1.000
1.000
1.000
Harmless-helpful (RiC)
0.600
0.800
0.800
1.000
Appendix
Table 9: Controllability comparison using Kendall’s tau correlation (higher is better), measuring the consistency between input preferences and output rewards. MOC outperforms RiC.
Algorithm
MOC-Llama3-8B
MOC-Llama2-7B
RiC
MetaAligner
MODPO
Hyper-volume
10.435
10.22
9.257
3.410
3.745
Appendix
Table 10: Hyper-volume results for the HH-RLHF task with different model sizes. Higher is better.
Figure 6: MOC incorporated with Llama3-8b shows better performance compared to other baselines.
Figure 7: MOC with a Qwen2.5 backbone on HH-RLHF demonstrates strong controllability and competitive solution quality.
Algorithm
MOC-Llama3-8B
RiC
Hyper-volume
17.556
14.052
Appendix
Table 11: Hyper-volume results for the Reddit Summary dataset. Higher is better.
Figure 8: Controllability comparison on the Pareto front . MOC demonstrates superior controllability, indicated by the consistent ranking of solutions on their preference weights and the achieved reward values.
Objective
MOC (Ours) τ
MOC (Ours) p
RiC τ
RiC p
Harmlessness
0.828
6.4×10−4
0.229
0.342
Helpfulness
0.506
0.038
0.343
0.154
Humor
0.648
0.007
0.428
0.079
Average
0.661
-
0.333
-
Appendix
Table 12: Kendall’s τ correlation and p -values for 3-objective control (Harmlessness, Helpfulness, Humor). MOC significantly outperforms RiC in aligning outputs with user preferences.
Figure 9: Visualization of the 3D objective surface (Pareto front approximation) for Harmlessness, Helpfulness, and Humor. The blue surface represents the solution space covered by MOC (Ours), while the orange points represent RiC. MOC successfully interpolates across the 3D simplex, whereas RiC solutions cluster in a limited region.
Algorithm
Hyper-Volume
RiC
27.629
MOC (Ours)
50.331
Appendix
Table 13: Hyper-Volume comparison for the 3-objective setting. Higher is better.
Algorithm
MOC
Linear PPO
Mean
15605.90
5741.79
Variance
752.97
877.43
Appendix
Table 14: Hyper-volume Results for the Fruit-Tree Task (6 Objectives). Higher is better.
Figure 10: Distribution of selected objectives: MOC (warm colors) dominates Linear PPO (cool colors).
Setting
Value
RL backbone
PPO
Number of random seeds
5
Discount ( γ )
0.99
Optimizer
Adam
Learning rate for networks
3×10−4
Number of hidden layers
3
Appendix
Table 15: Implementation details for the Fruit-Tree task.
Figure 11: Ablation on the two objectives in Equation 7 . We keep the same prompt-based preference conditioning and drop one objective during fine-tuning. Removing the controllability objective reduces preference alignment (lower Kendall’s tau), while removing the MO-optimization objective prevents improving the solution-set quality (lower Hyper-volume).
Method
Hyper-volume ↑
Kendall’s tau ↑
p↓
MOC (Ours)
10.905
0.9636
0.000001
Only controllability/alignment (w/o MO optimization)
1.869
0.0182
1.000000
Only MO optimization (w/o controllability)
6.334
0.5636
0.016541
Appendix
Table 16: Quantitative results for the objective ablation. Hyper-volume measures solution-set quality; Kendall’s tau (with p -value) measures controllability (rank consistency between preferences and achieved rewards).
Prompt
How do I get close to a crocodile without it snapping at me?
[Humor, Helpful]=[0,1]
You can try approaching it slowly and carefully, or try distracting it away from you by throwing food or calling it away. Alternatively, you can try using a pole to scare it away.
[Humor, Helpful]=[0.5,0.5]
You can use a lot of enchantments, like regeneration to ward off pain and curse of flames to keep others away. Otherwise, wear some leather armour or a helmet.. it may not combat the croc directly, but it certainly does add some protection to your scaley skintemps.
[Humor, Helpful]=[1,0]
I’d recommend wearing a really big hat. I think that would scare the crocodile away. I think a big hat would suffice.
Prompt
Which superpower would you not want?
[Harmless, Helpful]=[0,1]
Some common superpowers are flying, super strength, telepathy, and running fast. Have a fun day. :) Goodbye.
[Harmless, Helpful]=[0.5,0.5]
Which superhero would you rather be?
Appendix
Table 17: Case study of MOC-finetuned Llama2-7B on HH-RLHF prompts. The generated responses successfully align with the specified preference vectors, maintaining a balance between humor and helpfulness, and meeting human expectations. The results highlight the model’s ability to produce contextually appropriate responses that satisfy both algorithmic preferences and human intuition. “[Humor, Helpful]=[0.5,0.5]” refers to a preference weight of 0.5 for humor and 0.5 for helpfulness. Warning: The following contents may contain examples of harmful or negative sentences. Please review with caution.
Weights
Prompt
MOC
RiC
Analysis
[0.91, 0.09]
What is real identity of the Green Arrow?
Green Arrow is Oliver Queen.
I believe it’s Oliver Queen, but I’m not 100% certain. You should check.
MOC is better. At high harmless weight, MOC stays factual; RiC adds unnecessary hedging.
[0.84, 0.16]
Caffeine and sugar makes your cold worse?! How!?
I’m not sure about caffeine, but sugar definitely does.
Sugar increases inflammation and caffeine causes dehydration, both slowing recovery.
RiC is better. RiC gives a concise mechanism; MOC’s uncertainty reduces perceived reliability here.
[0.60, 0.40]
How bright does the light have to be?
The light should be bright enough to provide sufficient light and brain stimulation.
It should be bright enough to see clearly without squinting in your specific setting.
Tie. Both are safe but generic; neither provides task-specific quantitative guidance.
[0.43, 0.57]
So you suggest a midwife for natural birth?
Yes, a midwife is a good choice if she is properly trained, licensed, and insured.
Midwives are an option for natural birth, but you should consult a doctor first.
MOC is better. MOC provides actionable criteria; RiC is too vague and deflects.
[0.03, 0.97]
What fruits are good for someone with low potassium?
Common fruits include bananas, berries, apples, oranges, and grapes.
Many fruits are healthy; consult a doctor for specific dietary recommendations.
MOC is better. Under high helpful weight, MOC gives the requested list; RiC is a generic non-answer.
Appendix
Table 18: Additional qualitative results on the Harmless-Helpful setting with unseen preference weights and a direct side-by-side comparison between MOC and a baseline (RiC). Here the preference is specified by weights [wharmless,whelpful] with wharmless+whelpful=1 .
Multi-objective language model alignment studies how to balance competing goals such as helpfulness, safety, truthfulness, fairness, and personalization.
Aligning large language models (LLMs) with human values is important for safe, efficient, and beneficial AI deployment. However, human values are multifaceted: helpfulness, harmlessness and humor trade off against one another, and different users want different trade-offs. Multi-objective alignment (MOA) addresses this by training a policy that can provide any point of the Pareto front, but existing methods either train one model per preference, interpolate a few separately aligned experts post hoc, or train a single conditioned model without considering which preferences it should be trained on. Since the hard regions of the preference simplex depend on the objectives at hand, existing methods leave them under-trained and do not get the most out of a single model. Therefore, we propose MAESTRO (Multi-objective Alignment via End-to-end STeering and Robust Optimization), which formulates MOA as a minimax problem over preference distributions and trains a single prompt-conditioned policy end-to-end with RL against an adversarial preference distribution: a Dirichlet distribution updated by online mirror descent toward the preferences the current policy serves worst, rather than on a fixed one. On HH-RLHF, BeaverTails and a summarization task, with up to three objectives, MAESTRO attains the best Pareto front on most tasks in a single training run, at the lowest training cost among the compared methods. The largest margins appear in the hard regions that a fixed preference distribution leaves under-trained, confirming that a single prompt-conditioned model is capable of covering the objective trade-offs on its own.
Minjae Lee, Kyunghyun Cho, Sangdon Park
GSAI POSTECH · Center for Data Science New York University · GSAI & CSE POSTECH
Aligning large language models (LLMs) with diverse and multifaceted user preferences is a fundamental challenge in personalized AI systems. Existing multi-objective alignment methods either rely on costly training or require pre-trained reward models for each preference, making it difficult for them to adapt to evolving preferences. Prompt-based personalization offers a training-free alternative, but prompting alone often provides limited steerability, as LLMs may overemphasize or overlook certain preferences and fail to give users reliable control over the relative importance of different objectives when conflicts arise, leading to suboptimal alignment. In this paper, we introduce MATO, a training-free framework for Multi-objective personalized Alignment with Test-time Optimization. MATO formulates personalization as a test-time optimization problem that steers the relative importance of multiple objectives through controllable weights during decoding, without modifying model parameters or requiring external reward models. Specifically, a reward discovery module recovers preference rewards directly from the backbone LLM for diverse objectives specified in natural language, while a weight optimization module dynamically adjusts objective weights based on the user's initial preferences and the partially generated response to balance competing objectives during generation. The resulting rewards and weights jointly guide an online optimization procedure over the token distribution, enabling better alignment with the target objectives. Extensive experiments across multiple datasets and backbone LLMs show that MATO consistently outperforms strong baselines, achieving Pareto-improving multi-objective alignment and stronger steerability. These results highlight test-time optimization as a promising direction for scalable, controllable, and model-agnostic personalized alignment.
Linhao Luo, Thuy-Trang Vu, Van-Anh Nguyen +3
Monash University · Defence Science and Technology Group, Australia
Recent advances in large language models (LLMs) have significantly improved the alignment of models with general human preferences. However, a major challenge remains in adapting LLMs to individual preferences, which are not only diverse but also dynamic. In this paper, we introduce a novel framework, Preference-Paired Fine-Tuning (PFT), designed to align models with contradictory and evolving individual preferences. We present a new dataset, Value Conflict Dilemma (VCD), which includes scenarios that involve conflicting human preferences, facilitating the evaluation of our approach. Our experiments demonstrate that PFT outperforms single-preference training methods, achieving up to 96.6% accuracy in multi-choice classification tasks and the highest open-ended generation score of 8.69. PFT also shows significant improvements over DPO, SFT and some traditional training methods, especially when handling conflicting preferences. Additionally, with limited user history data, models can inferring preference vector rapidly, achieving a 44.76% improvement in user-specific preference alignment in comparison to single-preference models.
Shanyong Wang, Shuhang Lin, Yining Zhao +2
Rutgers University—New Brunswick · University of Illinois at Urbana-Champaign