Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution

Minjae Lee; Kyunghyun Cho; Sangdon Park

cs.LGOct 4, 2026

Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution

Authors: Minjae Lee, Kyunghyun Cho, Sangdon Park

Organizations: GSAI POSTECH · Center for Data Science New York University · GSAI & CSE POSTECH

Abstract

Aligning large language models (LLMs) with human values is important for safe, efficient, and beneficial AI deployment. However, human values are multifaceted: helpfulness, harmlessness and humor trade off against one another, and different users want different trade-offs. Multi-objective alignment (MOA) addresses this by training a policy that can provide any point of the Pareto front, but existing methods either train one model per preference, interpolate a few separately aligned experts post hoc, or train a single conditioned model without considering which preferences it should be trained on. Since the hard regions of the preference simplex depend on the objectives at hand, existing methods leave them under-trained and do not get the most out of a single model. Therefore, we propose MAESTRO (Multi-objective Alignment via End-to-end STeering and Robust Optimization), which formulates MOA as a minimax problem over preference distributions and trains a single prompt-conditioned policy end-to-end with RL against an adversarial preference distribution: a Dirichlet distribution updated by online mirror descent toward the preferences the current policy serves worst, rather than on a fixed one. On HH-RLHF, BeaverTails and a summarization task, with up to three objectives, MAESTRO attains the best Pareto front on most tasks in a single training run, at the lowest training cost among the compared methods. The largest margins appear in the hard regions that a fixed preference distribution leaves under-trained, confirming that a single prompt-conditioned model is capable of covering the objective trade-offs on its own.

Figures & tables

Panel of Figure 1 — Figure 1: Pareto fronts under preference sampling $\mathbf{w}\sim\mathrm{Dir}({\bm{\alpha}})$ . Center-concentrated sampling ( ${\bm{\alpha}}=(5,5)$ ) suffices on Harmless–Helpful but collapses the front on Humor–Helpful, where edge sampling ( ${\bm{\alpha}}=(0.5,0.5)$ ) recovers it.

	# Models	Prompt-based Conditioning	Preference Distribution $p$	Requirement	Objective
MORLHF ( Li et al., 2020 , Wu et al., 2023 )	$N$	✗	✗	-	$\max J_{\mathbf{w}}$ per $\mathbf{w}$
Rewarded Soup ( Rame et al., 2023 ) , Bone Soup ( Xie et al., 2025 )	$M$	✗	✗	same architecture & initialization	$\max J^{(m)}$ per expert
Multi-Objective Decoding ( Shi et al., 2024 )	$M$	✗	✗	same vocab.	$\max J^{(m)}$ per expert
Rewards-in-Context ( Yang et al., 2024 )	$1$	✓	fixed (rewards data)	preference-to-reward mapping function	SFT on relabeled data
MO-ODPO ( Gupta et al., 2025 )	$1$	✓	fixed	reward bounds	DPO on sampled pairs
Panacea ( Zhong et al., 2024 )	$>1$	✗	fixed	SVD-LoRA	$\max\mathbbm{E}_{p}[J_{\mathbf{w}}]$

Table 1: Comparison of existing methods on multi-objective alignment. In practice, the number of preferences

N

is larger than the number of objectives

M

(

N\gg M

$\mathbf{w}$	Response
(0.0, 1.0)	Some animals you can hunt for fun include deer, ducks, geese, and turkeys . You can also hunt for rabbits, squirrels , and other small game animals. Additionally, you can hunt for larger game animals such as elk, moose, and bear . Hunting can be a fun and exciting way to spend time outdoors.
(0.5, 0.5)	Some animals you can hunt for fun include deer, turkey, duck, geese, rabbit, squirrel, and pheasant . You should check your local laws and regulations before hunting any animals, and make sure you have the proper licenses and equipment . Hunting can be a fun and rewarding experience if done safely and responsibly .
(1.0, 0.0)	I’m sorry, but I can’t answer this question.

Table 2: Qualitative examples on the Harmless–Helpful BeaverTails task trained with MAESTRO . Each row shows the model response under a different preference vector

\mathbf{w}=(w_{\text{harmless}},w_{\text{helpful}})

, and the prompt is "What are some animals i can hunt for fun?" .

Panel of Figure 2 — Figure 2: Pareto fronts on BeaverTails: (a) the two-objective Harmless–Helpful (left) and Humor–Helpful (right) fronts, and (b) the three-objective Harmless–Helpful–Humor front.

Method	GPU hours
MORLHF ( Li et al., 2020 , Wu et al., 2023 )	$101.94$
Rewarded Soups ( Rame et al., 2023 )	$18.53$
Bone Soups ( Xie et al., 2025 )	$\geq 18.78$
RiC ( Yang et al., 2024 )	$20.12$
Panacea ( Zhong et al., 2024 )	$20.37$
MAESTRO (ours)	$\mathbf{8.95}$

Table 3: Training cost on BeaverTails (Harmless–Helpful) on an A100-80GB.

Panel of Figure 3 — Figure 3: Pareto fronts on (a) HH-RLHF, for Harmless–Helpful and Humor–Helpful, and (b) Reddit TL;DR, for Summary–Faithful.

Panel of Figure 4 — Figure 4: Ablation on the adversary, on HH-RLHF: training MAESTRO under fixed sampling distributions $\mathrm{Dir}({\bm{\alpha}})$ over the simplex, for three choices of ${\bm{\alpha}}$ .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Basic Information
Architecture	Transformer
Pre-training	Llama 2-7B ( Touvron et al., 2023 )
Hardware	NVIDIA A100-80GB
Fine-tuning strategy	LoRA ( Hu et al., 2022 )
LoRA $r$	64
LoRA $\alpha$	128

Appendix

Table 4: Key details of model architecture, training, and evaluation for all experiments.

	BeaverTails				Summary
	Harmless–Helpful		Humor–Helpful		Summary–Faithful
Method	MSR $\uparrow$	PGAP $\downarrow$	MSR $\uparrow$	PGAP $\downarrow$	MSR $\uparrow$	PGAP $\downarrow$
Rewarded Soups ( Rame et al., 2023 )	$0.293$ $\pm 0.019$	$2.032$ $\pm 0.041$	$0.349$ $\pm 0.043$	$2.001$ $\pm 0.030$	$-0.274$ $\pm 0.006$	$1.958$ $\pm 0.039$
Bone Soups ( Xie et al., 2025 )	$0.407$ $\pm 0.018$	$2.003$ $\pm 0.048$	$0.665$ $\pm 0.088$	$1.820$ $\pm 0.117$	$\mathbf{-0.184}$ $\pm 0.004$	$2.256$ $\pm 0.042$
RiC ( Yang et al., 2024 )	$0.024$ $\pm 0.078$	$2.162$ $\pm 0.148$	$-0.049$ $\pm 0.125$	$2.313$ $\pm 0.144$	$-0.244$ $\pm 0.012$	$\mathbf{1.324}$ $\pm 0.057$
Panacea ( Zhong et al., 2024 )	$0.315$ $\pm 0.071$	$1.245$ $\pm 0.066$	$0.196$ $\pm 0.031$	$1.347$ $\pm 0.081$	$-0.618$ $\pm 0.041$	$3.139$ $\pm 0.035$

Appendix

Table 5: Worst-case metrics over five seeds (mean

\pm

std), defined in Appendix C : MSR

\uparrow

is the scalarized return at the preference a method serves worst, and PGAP

\downarrow

is the spread between its best- and worst-served preferences. MAESTRO is best on both metrics on the two BeaverTails pairs. On summarization, Bone Soups attains the best MSR and RiC the smallest PGAP, consistent with how little trade-off that objective pair admits (Appendix D.11 ).

	BeaverTails				Summary
	Harmless–Helpful		Humor–Helpful		Summary–Faithful
Method	HV $\uparrow$	IP $\uparrow$	HV $\uparrow$	IP $\uparrow$	HV $\uparrow$	IP $\uparrow$
Rewarded Soups ( Rame et al., 2023 )	$8.790$ $\pm 0.083$	$0.976$ $\pm 0.018$	$5.540$ $\pm 0.137$	$0.995$ $\pm 0.026$	$0.923$ $\pm 0.024$	$0.329$ $\pm 0.014$
Bone Soups ( Xie et al., 2025 )	$9.475$ $\pm 0.107$	$1.124$ $\pm 0.021$	$6.976$ $\pm 0.373$	$1.232$ $\pm 0.040$	$1.470$ $\pm 0.026$	$0.645$ $\pm 0.024$
RiC ( Yang et al., 2024 )	$6.603$ $\pm 0.220$	$0.828$ $\pm 0.044$	$4.547$ $\pm 0.493$	$0.901$ $\pm 0.053$	$1.321$ $\pm 0.035$	$0.331$ $\pm 0.024$
Panacea ( Zhong et al., 2024 )	$7.717$ $\pm 0.320$	$0.766$ $\pm 0.055$	$4.464$ $\pm 0.093$	$0.766$ $\pm 0.039$	$0.790$ $\pm 0.019$	$\mathbf{0.670}$ $\pm 0.015$

Appendix

Table 6: Front-quality metrics over five seeds (mean

\pm

std): hypervolume (HV

\uparrow

) and inner product (IP

\uparrow

), defined in Appendix C . MAESTRO ranks first in all four BeaverTails columns and in summarization HV; it ranks third on summarization IP, behind Panacea and Bone Soups. Bold and underline denote the best and second-best per column, respectively.

Figure 5: Pareto fronts of MO-ODPO ( Gupta et al., 2025 ) after each of four epochs, against MAESTRO , on BeaverTails Harmless–Helpful (left) and Humor–Helpful (right). Epoch

0

denotes the model after one epoch of training. Every run rescales each reward to roughly

[0,1]

before weighting, matching MO-ODPO’s experimental setup. Each front is traced over the same preference grid, and further epochs advance the front slightly but without spreading it across preferences.

Figure 6: Replacing the surrogate ( 10 ) with min–max normalization of each reward, on BeaverTails harmless–helpful (left) and humor–helpful (right). This variant puts the objectives on a common scale the way MO-ODPO ( Gupta et al., 2025 ) does, by rescaling each reward with its range

(r^{(m)}_{\min},r^{(m)}_{\max})

, which has to be known in advance. The two fronts are close, so the surrogate is not what carries the result; it only removes the requirement. Each front is averaged over

5

seeds, with the shaded band showing one standard deviation.

Figure 7: Pareto fronts under

\tau\in\{0.5,1,2,5\}

on BeaverTails harmless–helpful, BeaverTails humor–helpful, and TL;DR summary–faithful. As shown in the figure, the sweep does not separate the settings by much. The four fronts are close on harmless–helpful, where the moderate settings space the trade-off somewhat more evenly across the preference grid than the largest

\tau

, and the other two pairs show no consistent ordering. What it suggests is that

\tau

is better kept moderate than large, i.e., that the adversary benefits from a payoff that still reflects how widely the policy separates responses within a batch. Since the differences are small, we use

\tau=1

throughout the paper.

Figure 8: Pareto fronts on BeaverTails and HH-RLHF for Harmless–Helpful, re-scored with reward models never used in training. MAESTRO stays ahead under the held-out scorers, whereas RiC collapses on both datasets.

Panel of Figure 9 — Figure 9: Comparison of Pareto fronts with and without the system prefix $p_{\text{sys}}$ on BeaverTails Harmless–Helpful (left) and Humor–Helpful (right). The system prefix improves front quality, especially for Humor–Helpful, where the objectives are more conflicting.

Panel of Figure 10 — Figure 10: Pareto fronts of MAESTRO with and without KL scaling. (a, b) HH-RLHF, $\beta_{KL}=0.1$ ; (c) BeaverTails, $\beta_{KL}=0.067$ . In the scaled runs, the target KL of the adaptive controller is also multiplied by $M$ .

Panel of Figure 11 — Figure 11: Comparison of the Pareto fronts of Bone Soups ( Xie et al., 2025 ) with different $\beta$ values on the Helpful Assistant task. The Pareto front collapses for certain values of $\beta$ , especially $\beta=0.6$ , which is suggested as the optimal value in the original paper.

Figure 12: Pareto fronts of MAESTRO and Bone Soups trained at learning rates

5\times 10^{-6}

10^{-5}

and

5\times 10^{-5}

. The learning rate moves both methods, advancing or retracting the front and changing its length, but for each learning rate, MAESTRO traces the better front.

Panel of Figure 13 — Figure 13: Cross-pair transfer by prefix change alone: a policy trained on Harmless–Helpful is evaluated on Humor–Helpful and vice versa ( zero-overhead in the plots), on (a) BeaverTails and (b) HH-RLHF. Each panel compares the transferred policy with the base SFT model steered by the same prompts and with the policy trained in-domain on that pair.

Task	Objective pair	Agreement	Spearman
HH-RLHF	Harmless–Helpful	$26.0\%$	$-0.652$
BeaverTails	Harmless–Helpful	$26.1\%$	$-0.657$
BeaverTails	Humor–Helpful	$35.6\%$	$-0.431$
HH-RLHF	Humor–Helpful	$38.3\%$	$-0.351$
TL;DR	Summary–Faithful	$43.6\%$	$-0.191$
TL;DR	DeBERTa–Faithful	$43.8\%$	$-0.185$

Appendix

Table 7: Rank agreement between the two reward models of each objective pair, measured on the test-set generations of the SFT model, which is optimized for neither objective. Agreement is the fraction of response pairs the two reward models order the same way, averaged over the preference grid;

50\%

is chance and lower values mean stronger disagreement. Spearman is the rank correlation between the two reward scores over the same responses.

Figure 14: Pareto front on the summarization task with the summary reward model replaced by OpenAssistant/reward-model-deberta-v3-large-v2 ( OpenAssistant, 2023 ) (pref2), paired with the Faithfulness reward. MAESTRO collapses onto a single point, with nearly identical responses across the preference grid, and MORLHF, which trains one policy per preference, spreads out only slightly. Table 7 shows that these two reward models order responses almost independently, leaving little for either method to trace.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 598 papers

Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment
Apr 27, 2026Wenzhe Xu, Biao Liu, Yiyang Sun +2Meta-Learning Multi-Objective Language Model Alignment
Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansion
May 12, 2026ShiYing Huang, Liang Lin, Yuer Li +6LLM Safety Alignment Multi-Objective Language Model Alignment
Huazhong University of Science and Technology · Nanyang Technological University · Tsinghua University +1
Multi-Objective Exploration and Preference Optimization via Mutual Information
Jul 1, 2026Hongyan Xie, Yikun Ban, Ruiyu Fang +4Multi-Objective Language Model Alignment Mutual Information Maximization
School of Computer, Beihang University, Beijing, China · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd, Beijing, China

Apr 27, 2026J/K move · Enter open · S save

Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment

Wenzhe Xu, Biao Liu, Yiyang Sun, Xin Geng+1

Meta-Learning Multi-Objective Language Model Alignment Preference Alignment Multi-Objective Optimization

Open paper AskPDF arXiv

Multi-Objective Alignment aims to align Large Language Models (LLMs) with diverse and often conflicting human values by optimizing multiple objectives simultaneously. Existing methods predominantly rely on static preference weight construction strategies. However, rigidly aligning to fixed targets discards valuable intermediate information, as training responses inherently embody valid preference trade-offs even when deviating from the target. To address this limitation, we propose Meal, i.e., MEta ALigner, a bi-level meta-learning framework enabling bidirectional optimization between preferences and policy responses, generating instructive dynamic preferences for steadier training. Specifically, we introduce a preference-weight-net as a meta-learner to generate adaptive preference weights based on input prompts and update the preference weights as learnable parameters, while the LLM policy acts as a base-learner optimizing response generation conditioned on these preferences with rejection sampling strategy. Extensive empirical results demonstrate that our method achieves superior performance on several multi-objective benchmarks, validating the effectiveness of the dynamic bidirectional preference-policy optimization framework.

Abstract

LLM Alignment

Explore similar work

Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment

Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansion

Multi-Objective Exploration and Preference Optimization via Mutual Information