One Model for All: Multi-Objective Controllable Language Models

Qiang He; Yucheng Yang; Tianyi Zhou; Meng Fang; Mykola Pechenizkiy; Setareh Maghsudi

cs.LGApr 6, 2026

One Model for All: Multi-Objective Controllable Language Models

Authors: Qiang He, Yucheng Yang, Tianyi Zhou, Meng Fang, Mykola Pechenizkiy, Setareh Maghsudi

Organizations: Ruhr University Bochum · Eindhoven University of Technology · Mohamed bin Zayed University of Artificial Intelligence · University of Liverpool

Abstract

Aligning large language models (LLMs) with human preferences is critical for enhancing LLMs' safety, helpfulness, humor, faithfulness, etc. Current reinforcement learning from human feedback (RLHF) mainly focuses on a fixed reward learned from average human ratings, which may weaken the adaptability and controllability of varying preferences. However, creating personalized LLMs requires aligning LLMs with individual human preferences, which is non-trivial due to the scarce data per user and the diversity of user preferences in multi-objective trade-offs, varying from emphasizing empathy in certain contexts to demanding efficiency and precision in others. Can we train one LLM to produce personalized outputs across different user preferences on the Pareto front? In this paper, we introduce Multi-Objective Control (MOC), which trains a single LLM to directly generate responses in the preference-defined regions of the Pareto front. Our approach introduces multi-objective optimization (MOO) principles into RLHF to train an LLM as a preference-conditioned policy network. We improve the computational efficiency of MOC by applying MOO at the policy level, enabling us to fine-tune a 7B-parameter model on a single A6000 GPU. Extensive experiments demonstrate the advantages of MOC over baselines in three aspects: (i) controllability of LLM outputs w.r.t. user preferences on the trade-off among multiple rewards; (ii) quality and diversity of LLM outputs, measured by the hyper-volume of multiple solutions achieved; and (iii) generalization to unseen preferences. These results highlight MOC's potential for real-world applications requiring scalable and customizable LLMs.

Figures & tables

Algorithms	Explicit policy improvement	Num of trained LLMs	Inference adaptation	Preference data	Loss
MORLHF	$\checkmark$	$M$	$\times$	No	PPO
Rewarded Soups ( Ramé et al., 2023 )	$\times$	$N$	$\checkmark$	No	PPO
MODPO ( Zhou et al., 2024 )	$\checkmark$	$M$	$\times$	Yes	DPO
RiC ( Yang et al., 2024c )	$\times$	1	$\checkmark$	No	SFT
MOC (Ours)	$\checkmark$	1	$\checkmark$	No	PPO

Table 1: Comparison with the state-of-the-art MOO methods. MOC addresses MOO in a principled and efficient manner.

M

– the number of preferences,

N

– the number of reward models (objectives).

Panel of Figure 1 — Figure 1: Solutions of MOC and Linear PPO on fishwood task and the Pareto front (line in black). MOC demonstrates advantages in both MOO (solutions lie on the Pareto front) and multi-objective control (its solutions align closely with their corresponding preference vectors, shown as the colored dashed rays). The single model trained by MOC can handle diverse preference vectors. In contrast, Linear PPO optimizes a linear scalarization of the objectives and fails to follow the preference vectors, with solutions dominated by one objective. The legend "Preference" indicates the specific weight value assigned to Reward 1 (Wood). Linear PPO is implemented by optimizing PPO w.r.t. a scalarized reward $R_{\text{lin}}=w\,R^{\text{wood}}+(1-w)\,R^{\text{fish}}$ (with $w\in[0,1]$ ). In our experiments, we train Linear PPO runs with weights $[0.1,0.9],[0.2,0.8],\cdots,[0.8,0.2],[0.9,0.1]$ .

Panel of Figure 2 — Figure 2: Controllability comparison on the Pareto front . MOC demonstrates superior controllability, indicated by the consistent ranking of solutions on their preference weights and the achieved reward values. In comparison, the baselines exhibit less stable behavior and weaker alignment with the specified preferences. MOC also achieves higher quality solutions, particularly in the Humor & Helpful alignment. Our MOC method achieves the best overall performance, supported by these results and the findings in Tables 2 , 3 and 4 . Each point represents the reward achieved across multiple instances, each with a different input preference vector. Each point’s preference weight for the x-axis reward is the numerical label on its marker.

Figure 3: Illustration of the hyper-volume concept. The hyper-volume measures the size of the objective space dominated by a set of solutions in multi-objective optimization. Larger hyper-volumes indicate better convergence and diversity of the Pareto front.

Setting	MOC (Ours)	RiC	MORLHF	Rewarded Soups
Humor-helpful	1.00 ( $5.0\times 10^{-8}$ )	0.78 ( $3.3\times 10^{-4}$ )	1.00 ( $1.7\times 10^{-2}$ )	1.00 ( $5.0\times 10^{-8}$ )
Harmless-helpful	1.00 ( $5.0\times 10^{-8}$ )	0.91 ( $3.0\times 10^{-5}$ )	1.00 ( $1.7\times 10^{-2}$ )	0.96 ( $5.5\times 10^{-7}$ )
Average	1.00	0.85	1.00	0.98

Table 2: Controllability comparison using Kendall’s tau correlation (higher is better), with corresponding

p

-values in parentheses (smaller is better), measuring the consistency between input preferences and output rewards. MOC significantly outperforms all the baselines.

Setting	MOC (Ours)	RiC	MORLHF	Rewarded Soups
Humor-helpful	14.176	6.692	6.769	6.100
Harmless-helpful	10.220	9.257	9.047	8.905
Average	12.198	7.974	7.908	7.502

Table 3: Hyper-volume (higher is better) comparison of different methods, measuring the volume of solutions dominated by each method’s solution set. MOC outperforms all baselines, achieving higher solution diversity and quality. The best score is marked with the blue color box.

Setting	MOC (Ours)	RiC	MORLHF	Rewarded Soups
Humor-helpful	1.439	1.260	1.057	1.005
Harmless-helpful	1.600	1.363	1.110	1.015
Average	1.520	1.312	1.084	1.010

Table 4: Comparison of solution set pair-wise distances (measuring diversity, higher is better) across methods.

Figure 4: Generalization to unseen preference vectors held out from the training. MOC and RiC-trained LLMs are compared on four random sets of unseen preference vectors. Each column corresponds to a different set of unseen preference vectors, and each row represents a different pair of reward settings. MOC solutions dominate the RiC solutions in most cases. MOC’s rewards align with the new preference vectors and the outputs under different preferences are diverse in the reward space. This suggests MOC generalizes to unseen preferences and achieves diverse trade-offs on the Pareto front. The size of each point indicates the standard deviation in rewards. The numerical labels indicate the preference weights (multiplied by 100) for the reward on the x-axis, enhancing visual clarity.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Algorithm	Source of Controllability	Explicit Policy Improvement?	Loss Function
MOC	MOO with constraints	$\checkmark$	PPO-based
RiC	Reward in Context	$\times$	SFT-based

Appendix

Table 5: Key differences between MOC and RiC

Hyper-parameter	Value
Dimension of state space	1
Action space	Discrete(2): go fishing, go collect wood
Discount ( $\gamma$ )	0.99
Optimizer	Adam ( Kingma & Ba, 2015 )
Learning rate for networks	$1\times 10^{-4}$
Number of hidden layers for all networks	3

Appendix

Table 6: Hyper-parameters settings for fishwood task ( Section 2.5 ).

Hyper-parameter	Value
Base model	Llama 2-7B ( Touvron et al., 2023 )
GPU	A NVIDIA RTX A6000 (48G)
CPU	Intel(R) Core(TM) i9-14900K
Memory	128 G
Quantization for training	8bit
Fine-tuning	LoRA ( Hu et al., 2022 )

Appendix

Table 7: Key information about the implementation.

Figure 5: Visualization of four groups of randomly sampled, unseen preference vectors. Each preference vector is generated by uniformly sampling a number from the range [1, 100] and converting it to a weight

w_{1}

for reward 1, with the second reward weight calculated as

1-w_{1}

. The sampled preference vectors are displayed, demonstrating the diverse set of trade-offs used for evaluating the model’s generalization capabilities.

Setting	Group 1	Group 2	Group 3	Group 4
Humor-helpful (MOC)	17.034	19.697	17.441	19.045
Humor-helpful (RiC)	16.660	16.303	16.304	16.551
Harmless-helpful (MOC)	15.038	14.139	13.324	15.557
Harmless-helpful (RiC)	9.463	10.447	9.342	9.726

Appendix

Table 8: Hyper-volume (HV) Comparison between MOC and RiC, where MOC achieves higher HV (better output quality and diversity under different preferences). Higher is better.

Setting	Group 1	Group 2	Group 3	Group 4
Humor-helpful (MOC)	1.000	1.000	1.000	1.000
Humor-helpful (RiC)	0.800	0.800	1.000	1.000
Harmless-helpful (MOC)	1.000	1.000	1.000	1.000
Harmless-helpful (RiC)	0.600	0.800	0.800	1.000

Appendix

Table 9: Controllability comparison using Kendall’s tau correlation (higher is better), measuring the consistency between input preferences and output rewards. MOC outperforms RiC.

Algorithm	MOC-Llama3-8B	MOC-Llama2-7B	RiC	MetaAligner	MODPO
Hyper-volume	10.435	10.22	9.257	3.410	3.745

Appendix

Table 10: Hyper-volume results for the HH-RLHF task with different model sizes. Higher is better.

Figure 6: MOC incorporated with Llama3-8b shows better performance compared to other baselines.

Figure 7: MOC with a Qwen2.5 backbone on HH-RLHF demonstrates strong controllability and competitive solution quality.

Algorithm	MOC-Llama3-8B	RiC
Hyper-volume	17.556	14.052

Appendix

Table 11: Hyper-volume results for the Reddit Summary dataset. Higher is better.

Panel of Figure 8 — Figure 8: Controllability comparison on the Pareto front . MOC demonstrates superior controllability, indicated by the consistent ranking of solutions on their preference weights and the achieved reward values.

Objective	MOC (Ours) $\tau$	MOC (Ours) $p$	RiC $\tau$	RiC $p$
Harmlessness	0.828	$6.4\times 10^{-4}$	0.229	0.342
Helpfulness	0.506	0.038	0.343	0.154
Humor	0.648	0.007	0.428	0.079
Average	0.661	-	0.333	-

Appendix

Table 12: Kendall’s

\tau

correlation and

p

-values for 3-objective control (Harmlessness, Helpfulness, Humor). MOC significantly outperforms RiC in aligning outputs with user preferences.

Figure 9: Visualization of the 3D objective surface (Pareto front approximation) for Harmlessness, Helpfulness, and Humor. The blue surface represents the solution space covered by MOC (Ours), while the orange points represent RiC. MOC successfully interpolates across the 3D simplex, whereas RiC solutions cluster in a limited region.

Algorithm	Hyper-Volume
RiC	27.629
MOC (Ours)	50.331

Appendix

Table 13: Hyper-Volume comparison for the 3-objective setting. Higher is better.

Algorithm	MOC	Linear PPO
Mean	15605.90	5741.79
Variance	752.97	877.43

Appendix

Table 14: Hyper-volume Results for the Fruit-Tree Task (6 Objectives). Higher is better.

Figure 10: Distribution of selected objectives: MOC (warm colors) dominates Linear PPO (cool colors).

Setting	Value
RL backbone	PPO
Number of random seeds	5
Discount ( $\gamma$ )	0.99
Optimizer	Adam
Learning rate for networks	$3\times 10^{-4}$
Number of hidden layers	3

Appendix

Table 15: Implementation details for the Fruit-Tree task.

Figure 11: Ablation on the two objectives in Equation 7 . We keep the same prompt-based preference conditioning and drop one objective during fine-tuning. Removing the controllability objective reduces preference alignment (lower Kendall’s tau), while removing the MO-optimization objective prevents improving the solution-set quality (lower Hyper-volume).

Method	Hyper-volume $\uparrow$	Kendall’s tau $\uparrow$	$p$ $\downarrow$
MOC (Ours)	10.905	0.9636	0.000001
Only controllability/alignment (w/o MO optimization)	1.869	0.0182	1.000000
Only MO optimization (w/o controllability)	6.334	0.5636	0.016541

Appendix

Table 16: Quantitative results for the objective ablation. Hyper-volume measures solution-set quality; Kendall’s tau (with

p

-value) measures controllability (rank consistency between preferences and achieved rewards).

Prompt	How do I get close to a crocodile without it snapping at me?
[Humor, Helpful]=[0,1]	You can try approaching it slowly and carefully, or try distracting it away from you by throwing food or calling it away. Alternatively, you can try using a pole to scare it away.
[Humor, Helpful]=[0.5,0.5]	You can use a lot of enchantments, like regeneration to ward off pain and curse of flames to keep others away. Otherwise, wear some leather armour or a helmet.. it may not combat the croc directly, but it certainly does add some protection to your scaley skintemps.
[Humor, Helpful]=[1,0]	I’d recommend wearing a really big hat. I think that would scare the crocodile away. I think a big hat would suffice.
Prompt	Which superpower would you not want?
[Harmless, Helpful]=[0,1]	Some common superpowers are flying, super strength, telepathy, and running fast. Have a fun day. :) Goodbye.
[Harmless, Helpful]=[0.5,0.5]	Which superhero would you rather be?

Appendix

Table 17: Case study of MOC-finetuned Llama2-7B on HH-RLHF prompts. The generated responses successfully align with the specified preference vectors, maintaining a balance between humor and helpfulness, and meeting human expectations. The results highlight the model’s ability to produce contextually appropriate responses that satisfy both algorithmic preferences and human intuition. “[Humor, Helpful]=[0.5,0.5]” refers to a preference weight of 0.5 for humor and 0.5 for helpfulness. Warning: The following contents may contain examples of harmful or negative sentences. Please review with caution.

Weights	Prompt	MOC	RiC	Analysis
[0.91, 0.09]	What is real identity of the Green Arrow?	Green Arrow is Oliver Queen.	I believe it’s Oliver Queen, but I’m not 100% certain. You should check.	MOC is better. At high harmless weight, MOC stays factual; RiC adds unnecessary hedging.
[0.84, 0.16]	Caffeine and sugar makes your cold worse?! How!?	I’m not sure about caffeine, but sugar definitely does.	Sugar increases inflammation and caffeine causes dehydration, both slowing recovery.	RiC is better. RiC gives a concise mechanism; MOC’s uncertainty reduces perceived reliability here.
[0.60, 0.40]	How bright does the light have to be?	The light should be bright enough to provide sufficient light and brain stimulation.	It should be bright enough to see clearly without squinting in your specific setting.	Tie. Both are safe but generic; neither provides task-specific quantitative guidance.
[0.43, 0.57]	So you suggest a midwife for natural birth?	Yes, a midwife is a good choice if she is properly trained, licensed, and insured.	Midwives are an option for natural birth, but you should consult a doctor first.	MOC is better. MOC provides actionable criteria; RiC is too vague and deflects.
[0.03, 0.97]	What fruits are good for someone with low potassium?	Common fruits include bananas, berries, apples, oranges, and grapes.	Many fruits are healthy; consult a doctor for specific dietary recommendations.	MOC is better. Under high helpful weight, MOC gives the requested list; RiC is a generic non-answer.

Appendix

Table 18: Additional qualitative results on the Harmless-Helpful setting with unseen preference weights and a direct side-by-side comparison between MOC and a baseline (RiC). Here the preference is specified by weights

[w_{\text{harmless}},w_{\text{helpful}}]

with

w_{\text{harmless}}+w_{\text{helpful}}=1

Multi-Objective Language Model Alignment

Multi-objective language model alignment studies how to balance competing goals such as helpfulness, safety, truthfulness, fairness, and personalization.

See all 92 papers

Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution
Oct 4, 2026Minjae Lee, Kyunghyun Cho, Sangdon ParkLLM Alignment Multi-Objective Language Model Alignment
GSAI POSTECH · Center for Data Science New York University · GSAI & CSE POSTECH
MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models
May 25, 2026Linhao Luo, Thuy-Trang Vu, Van-Anh Nguyen +3LLM Alignment Inference-Time Optimization
Monash University · Defence Science and Technology Group, Australia
Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning
Apr 14, 2026Shanyong Wang, Shuhang Lin, Yining Zhao +2LLM Fine-Tuning Personalized Language Model Alignment
Rutgers University—New Brunswick · University of Illinois at Urbana-Champaign

Oct 4, 2026J/K move · Enter open · S save

Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution

Minjae Lee, Kyunghyun Cho, Sangdon Park

GSAI POSTECH · Center for Data Science New York University · GSAI & CSE POSTECH

LLM Alignment Multi-Objective Language Model Alignment Preference Optimization Multi-Objective Reinforcement Learning RL for Language Models

Open paper AskPDF arXiv

Aligning large language models (LLMs) with human values is important for safe, efficient, and beneficial AI deployment. However, human values are multifaceted: helpfulness, harmlessness and humor trade off against one another, and different users want different trade-offs. Multi-objective alignment (MOA) addresses this by training a policy that can provide any point of the Pareto front, but existing methods either train one model per preference, interpolate a few separately aligned experts post hoc, or train a single conditioned model without considering which preferences it should be trained on. Since the hard regions of the preference simplex depend on the objectives at hand, existing methods leave them under-trained and do not get the most out of a single model. Therefore, we propose MAESTRO (Multi-objective Alignment via End-to-end STeering and Robust Optimization), which formulates MOA as a minimax problem over preference distributions and trains a single prompt-conditioned policy end-to-end with RL against an adversarial preference distribution: a Dirichlet distribution updated by online mirror descent toward the preferences the current policy serves worst, rather than on a fixed one. On HH-RLHF, BeaverTails and a summarization task, with up to three objectives, MAESTRO attains the best Pareto front on most tasks in a single training run, at the lowest training cost among the compared methods. The largest margins appear in the hard regions that a fixed preference distribution leaves under-trained, confirming that a single prompt-conditioned model is capable of covering the objective trade-offs on its own.

Abstract

Multi-Objective Language Model Alignment

Explore similar work

Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution

MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models

Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning