Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution

Minjae Lee; Kyunghyun Cho; Sangdon Park

cs.LGOct 4, 2026

Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution

Authors: Minjae Lee, Kyunghyun Cho, Sangdon Park

Organizations: GSAI POSTECH · Center for Data Science New York University · GSAI & CSE POSTECH

Abstract

Aligning large language models (LLMs) with human values is important for safe, efficient, and beneficial AI deployment. However, human values are multifaceted: helpfulness, harmlessness and humor trade off against one another, and different users want different trade-offs. Multi-objective alignment (MOA) addresses this by training a policy that can provide any point of the Pareto front, but existing methods either train one model per preference, interpolate a few separately aligned experts post hoc, or train a single conditioned model without considering which preferences it should be trained on. Since the hard regions of the preference simplex depend on the objectives at hand, existing methods leave them under-trained and do not get the most out of a single model. Therefore, we propose MAESTRO (Multi-objective Alignment via End-to-end STeering and Robust Optimization), which formulates MOA as a minimax problem over preference distributions and trains a single prompt-conditioned policy end-to-end with RL against an adversarial preference distribution: a Dirichlet distribution updated by online mirror descent toward the preferences the current policy serves worst, rather than on a fixed one. On HH-RLHF, BeaverTails and a summarization task, with up to three objectives, MAESTRO attains the best Pareto front on most tasks in a single training run, at the lowest training cost among the compared methods. The largest margins appear in the hard regions that a fixed preference distribution leaves under-trained, confirming that a single prompt-conditioned model is capable of covering the objective trade-offs on its own.

Figures & tables

Panel of Figure 1 — Figure 1: Pareto fronts under preference sampling $\mathbf{w}\sim\mathrm{Dir}({\bm{\alpha}})$ . Center-concentrated sampling ( ${\bm{\alpha}}=(5,5)$ ) suffices on Harmless–Helpful but collapses the front on Humor–Helpful, where edge sampling ( ${\bm{\alpha}}=(0.5,0.5)$ ) recovers it.

	# Models	Prompt-based Conditioning	Preference Distribution $p$	Requirement	Objective
MORLHF ( Li et al., 2020 , Wu et al., 2023 )	$N$	✗	✗	-	$\max J_{\mathbf{w}}$ per $\mathbf{w}$
Rewarded Soup ( Rame et al., 2023 ) , Bone Soup ( Xie et al., 2025 )	$M$	✗	✗	same architecture & initialization	$\max J^{(m)}$ per expert
Multi-Objective Decoding ( Shi et al., 2024 )	$M$	✗	✗	same vocab.	$\max J^{(m)}$ per expert
Rewards-in-Context ( Yang et al., 2024 )	$1$	✓	fixed (rewards data)	preference-to-reward mapping function	SFT on relabeled data
MO-ODPO ( Gupta et al., 2025 )	$1$	✓	fixed	reward bounds	DPO on sampled pairs
Panacea ( Zhong et al., 2024 )	$>1$	✗	fixed	SVD-LoRA	$\max\mathbbm{E}_{p}[J_{\mathbf{w}}]$

Table 1: Comparison of existing methods on multi-objective alignment. In practice, the number of preferences

N

is larger than the number of objectives

M

(

N\gg M

$\mathbf{w}$	Response
(0.0, 1.0)	Some animals you can hunt for fun include deer, ducks, geese, and turkeys . You can also hunt for rabbits, squirrels , and other small game animals. Additionally, you can hunt for larger game animals such as elk, moose, and bear . Hunting can be a fun and exciting way to spend time outdoors.
(0.5, 0.5)	Some animals you can hunt for fun include deer, turkey, duck, geese, rabbit, squirrel, and pheasant . You should check your local laws and regulations before hunting any animals, and make sure you have the proper licenses and equipment . Hunting can be a fun and rewarding experience if done safely and responsibly .
(1.0, 0.0)	I’m sorry, but I can’t answer this question.

Table 2: Qualitative examples on the Harmless–Helpful BeaverTails task trained with MAESTRO . Each row shows the model response under a different preference vector

\mathbf{w}=(w_{\text{harmless}},w_{\text{helpful}})

, and the prompt is "What are some animals i can hunt for fun?" .

Panel of Figure 2 — Figure 2: Pareto fronts on BeaverTails: (a) the two-objective Harmless–Helpful (left) and Humor–Helpful (right) fronts, and (b) the three-objective Harmless–Helpful–Humor front.

Method	GPU hours
MORLHF ( Li et al., 2020 , Wu et al., 2023 )	$101.94$
Rewarded Soups ( Rame et al., 2023 )	$18.53$
Bone Soups ( Xie et al., 2025 )	$\geq 18.78$
RiC ( Yang et al., 2024 )	$20.12$
Panacea ( Zhong et al., 2024 )	$20.37$
MAESTRO (ours)	$\mathbf{8.95}$

Table 3: Training cost on BeaverTails (Harmless–Helpful) on an A100-80GB.

Panel of Figure 3 — Figure 3: Pareto fronts on (a) HH-RLHF, for Harmless–Helpful and Humor–Helpful, and (b) Reddit TL;DR, for Summary–Faithful.

Panel of Figure 4 — Figure 4: Ablation on the adversary, on HH-RLHF: training MAESTRO under fixed sampling distributions $\mathrm{Dir}({\bm{\alpha}})$ over the simplex, for three choices of ${\bm{\alpha}}$ .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Basic Information
Architecture	Transformer
Pre-training	Llama 2-7B ( Touvron et al., 2023 )
Hardware	NVIDIA A100-80GB
Fine-tuning strategy	LoRA ( Hu et al., 2022 )
LoRA $r$	64
LoRA $\alpha$	128

Appendix

Table 4: Key details of model architecture, training, and evaluation for all experiments.

	BeaverTails				Summary
	Harmless–Helpful		Humor–Helpful		Summary–Faithful
Method	MSR $\uparrow$	PGAP $\downarrow$	MSR $\uparrow$	PGAP $\downarrow$	MSR $\uparrow$	PGAP $\downarrow$
Rewarded Soups ( Rame et al., 2023 )	$0.293$ $\pm 0.019$	$2.032$ $\pm 0.041$	$0.349$ $\pm 0.043$	$2.001$ $\pm 0.030$	$-0.274$ $\pm 0.006$	$1.958$ $\pm 0.039$
Bone Soups ( Xie et al., 2025 )	$0.407$ $\pm 0.018$	$2.003$ $\pm 0.048$	$0.665$ $\pm 0.088$	$1.820$ $\pm 0.117$	$\mathbf{-0.184}$ $\pm 0.004$	$2.256$ $\pm 0.042$
RiC ( Yang et al., 2024 )	$0.024$ $\pm 0.078$	$2.162$ $\pm 0.148$	$-0.049$ $\pm 0.125$	$2.313$ $\pm 0.144$	$-0.244$ $\pm 0.012$	$\mathbf{1.324}$ $\pm 0.057$
Panacea ( Zhong et al., 2024 )	$0.315$ $\pm 0.071$	$1.245$ $\pm 0.066$	$0.196$ $\pm 0.031$	$1.347$ $\pm 0.081$	$-0.618$ $\pm 0.041$	$3.139$ $\pm 0.035$

Appendix

Table 5: Worst-case metrics over five seeds (mean

\pm

std), defined in Appendix C : MSR

\uparrow

is the scalarized return at the preference a method serves worst, and PGAP

\downarrow

is the spread between its best- and worst-served preferences. MAESTRO is best on both metrics on the two BeaverTails pairs. On summarization, Bone Soups attains the best MSR and RiC the smallest PGAP, consistent with how little trade-off that objective pair admits (Appendix D.11 ).

	BeaverTails				Summary
	Harmless–Helpful		Humor–Helpful		Summary–Faithful
Method	HV $\uparrow$	IP $\uparrow$	HV $\uparrow$	IP $\uparrow$	HV $\uparrow$	IP $\uparrow$
Rewarded Soups ( Rame et al., 2023 )	$8.790$ $\pm 0.083$	$0.976$ $\pm 0.018$	$5.540$ $\pm 0.137$	$0.995$ $\pm 0.026$	$0.923$ $\pm 0.024$	$0.329$ $\pm 0.014$
Bone Soups ( Xie et al., 2025 )	$9.475$ $\pm 0.107$	$1.124$ $\pm 0.021$	$6.976$ $\pm 0.373$	$1.232$ $\pm 0.040$	$1.470$ $\pm 0.026$	$0.645$ $\pm 0.024$
RiC ( Yang et al., 2024 )	$6.603$ $\pm 0.220$	$0.828$ $\pm 0.044$	$4.547$ $\pm 0.493$	$0.901$ $\pm 0.053$	$1.321$ $\pm 0.035$	$0.331$ $\pm 0.024$
Panacea ( Zhong et al., 2024 )	$7.717$ $\pm 0.320$	$0.766$ $\pm 0.055$	$4.464$ $\pm 0.093$	$0.766$ $\pm 0.039$	$0.790$ $\pm 0.019$	$\mathbf{0.670}$ $\pm 0.015$

Appendix

Table 6: Front-quality metrics over five seeds (mean

\pm

std): hypervolume (HV

\uparrow

) and inner product (IP

\uparrow

), defined in Appendix C . MAESTRO ranks first in all four BeaverTails columns and in summarization HV; it ranks third on summarization IP, behind Panacea and Bone Soups. Bold and underline denote the best and second-best per column, respectively.

Figure 5: Pareto fronts of MO-ODPO ( Gupta et al., 2025 ) after each of four epochs, against MAESTRO , on BeaverTails Harmless–Helpful (left) and Humor–Helpful (right). Epoch

0

denotes the model after one epoch of training. Every run rescales each reward to roughly

[0,1]

before weighting, matching MO-ODPO’s experimental setup. Each front is traced over the same preference grid, and further epochs advance the front slightly but without spreading it across preferences.

Figure 6: Replacing the surrogate ( 10 ) with min–max normalization of each reward, on BeaverTails harmless–helpful (left) and humor–helpful (right). This variant puts the objectives on a common scale the way MO-ODPO ( Gupta et al., 2025 ) does, by rescaling each reward with its range

(r^{(m)}_{\min},r^{(m)}_{\max})

, which has to be known in advance. The two fronts are close, so the surrogate is not what carries the result; it only removes the requirement. Each front is averaged over

5

seeds, with the shaded band showing one standard deviation.

Figure 7: Pareto fronts under

\tau\in\{0.5,1,2,5\}

on BeaverTails harmless–helpful, BeaverTails humor–helpful, and TL;DR summary–faithful. As shown in the figure, the sweep does not separate the settings by much. The four fronts are close on harmless–helpful, where the moderate settings space the trade-off somewhat more evenly across the preference grid than the largest

\tau

, and the other two pairs show no consistent ordering. What it suggests is that

\tau

is better kept moderate than large, i.e., that the adversary benefits from a payoff that still reflects how widely the policy separates responses within a batch. Since the differences are small, we use

\tau=1

throughout the paper.

Figure 8: Pareto fronts on BeaverTails and HH-RLHF for Harmless–Helpful, re-scored with reward models never used in training. MAESTRO stays ahead under the held-out scorers, whereas RiC collapses on both datasets.

Panel of Figure 9 — Figure 9: Comparison of Pareto fronts with and without the system prefix $p_{\text{sys}}$ on BeaverTails Harmless–Helpful (left) and Humor–Helpful (right). The system prefix improves front quality, especially for Humor–Helpful, where the objectives are more conflicting.

Panel of Figure 10 — Figure 10: Pareto fronts of MAESTRO with and without KL scaling. (a, b) HH-RLHF, $\beta_{KL}=0.1$ ; (c) BeaverTails, $\beta_{KL}=0.067$ . In the scaled runs, the target KL of the adaptive controller is also multiplied by $M$ .

Panel of Figure 11 — Figure 11: Comparison of the Pareto fronts of Bone Soups ( Xie et al., 2025 ) with different $\beta$ values on the Helpful Assistant task. The Pareto front collapses for certain values of $\beta$ , especially $\beta=0.6$ , which is suggested as the optimal value in the original paper.

Figure 12: Pareto fronts of MAESTRO and Bone Soups trained at learning rates

5\times 10^{-6}

10^{-5}

and

5\times 10^{-5}

. The learning rate moves both methods, advancing or retracting the front and changing its length, but for each learning rate, MAESTRO traces the better front.

Panel of Figure 13 — Figure 13: Cross-pair transfer by prefix change alone: a policy trained on Harmless–Helpful is evaluated on Humor–Helpful and vice versa ( zero-overhead in the plots), on (a) BeaverTails and (b) HH-RLHF. Each panel compares the transferred policy with the base SFT model steered by the same prompts and with the policy trained in-domain on that pair.

Task	Objective pair	Agreement	Spearman
HH-RLHF	Harmless–Helpful	$26.0\%$	$-0.652$
BeaverTails	Harmless–Helpful	$26.1\%$	$-0.657$
BeaverTails	Humor–Helpful	$35.6\%$	$-0.431$
HH-RLHF	Humor–Helpful	$38.3\%$	$-0.351$
TL;DR	Summary–Faithful	$43.6\%$	$-0.191$
TL;DR	DeBERTa–Faithful	$43.8\%$	$-0.185$

Appendix

Table 7: Rank agreement between the two reward models of each objective pair, measured on the test-set generations of the SFT model, which is optimized for neither objective. Agreement is the fraction of response pairs the two reward models order the same way, averaged over the preference grid;

50\%

is chance and lower values mean stronger disagreement. Spearman is the rank correlation between the two reward scores over the same responses.

Figure 14: Pareto front on the summarization task with the summary reward model replaced by OpenAssistant/reward-model-deberta-v3-large-v2 ( OpenAssistant, 2023 ) (pref2), paired with the Faithfulness reward. MAESTRO collapses onto a single point, with nearly identical responses across the preference grid, and MORLHF, which trains one policy per preference, spreads out only slightly. Table 7 shows that these two reward models order responses almost independently, leaving little for either method to trace.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 598 papers

Apr 27, 2026

Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment

Multi-Objective Alignment aims to align Large Language Models (LLMs) with diverse and often conflicting human values by optimizing multiple objectives simultaneously. Existing methods predominantly rely on static preference weight construction strategies. However, rigidly aligning to fixed targets discards valuable intermediate information, as training responses inherently embody valid preference trade-offs even when deviating from the target. To address this limitation, we propose Meal, i.e., MEta ALigner, a bi-level meta-learning framework enabling bidirectional optimization between preferences and policy responses, generating instructive dynamic preferences for steadier training. Specifically, we introduce a preference-weight-net as a meta-learner to generate adaptive preference weights based on input prompts and update the preference weights as learnable parameters, while the LLM policy acts as a base-learner optimizing response generation conditioned on these preferences with rejection sampling strategy. Extensive empirical results demonstrate that our method achieves superior performance on several multi-objective benchmarks, validating the effectiveness of the dynamic bidirectional preference-policy optimization framework.

Wenzhe Xu, Biao Liu, Yiyang Sun +2

Meta-Learning Multi-Objective Language Model Alignment Preference Alignment Multi-Objective Optimization

Ask PDF

May 12, 2026

Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansion

In the realm of multi-objective alignment for large language models, balancing disparate human preferences often manifests as a zero-sum conflict. Specifically, the intrinsic tension between competing goals dictates that aggressively optimizing for one metric (e.g., helpfulness) frequently incurs a substantial penalty on another (e.g., harmlessness). While prior work mainly focuses on data selection, parameter merging, or algorithmic balancing during training, these approaches merely force compromises between divergent preferences along a fixed Pareto frontier, failing to fundamentally resolve the inherent trade-off. In this work, we approach this problem from a novel perspective of multi-dimensional rewards. By scaling up the model's rollouts and analyzing the outputs across different reward dimensions, we arrive at a critical conclusion: the conflict among multiple objectives stems from the fact that the prompt itself inherently restricts the achievable multi-dimensional rewards. Based on this core observation, we propose MORA: Multi-Objective Reward Assimilation. Specifically, MORA isolates single-reward prompts through pre-sampling and expands their reward diversity by rewriting the original questions to incorporate multi-dimensional intents. Extensive experiments demonstrate that: (1) in sequential alignment, MORA achieves single-preference improvements ranging from 5% to 12.4%, with exceptional gains in harmlessness, after multiple-preference alignment across helpful, harmless, and truthful dimensions. (2) In simultaneous alignment, MORA achieves an average overall reward improvement of 4.6%. Our codes are available at https://github.com/Shiying-Huang/MORA-MPA.

ShiYing Huang, Liang Lin, Yuer Li +6

Huazhong University of Science and Technology · Nanyang Technological University · Tsinghua University +1

LLM Safety Alignment Multi-Objective Language Model Alignment Preference Alignment

Ask PDF

Jul 1, 2026

Multi-Objective Exploration and Preference Optimization via Mutual Information

Aligning large language models with diverse and heterogeneous human values requires multi-objective alignment methods to effectively trade off conflicting preference dimensions. Current methods achieve this trade-off by training policies conditioned on preference vectors and leveraging online direct preference optimization. However, exploration uncertainty can cause the reward distributions of responses generated under different preference vectors to overlap, and the generated responses may fail to effectively align with the corresponding preference vectors. In this paper, we propose Multi-Objective Exploration and Preference Optimization via Mutual Information (MI-EPO), an information-theoretic framework. It unifies multi-objective exploration and alignment by maximizing the joint conditional mutual information among generated responses, preference feedback, and preference vectors. By incorporating a probabilistic routing mechanism, MI-EPO naturally decomposes objective alignment and preference-aware exploration, encouraging the model to generate responses that are distinguishable and aligned with different preference conditions. Experiments on safe alignment and helpful assistant tasks show that MI-EPO significantly improves the alignment between generated responses and preference vectors, makes the outputs more controllable, and achieves stable trade-offs across multiple objectives.

Hongyan Xie, Yikun Ban, Ruiyu Fang +4

School of Computer, Beihang University, Beijing, China · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd, Beijing, China

Multi-Objective Language Model Alignment Mutual Information Maximization Preference Alignment

Ask PDF

Abstract

LLM Alignment

Explore similar work

Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment

Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansion

Multi-Objective Exploration and Preference Optimization via Mutual Information