MINT: Min-Selection Preference Distillation for Balanced Multi-Objective Alignment

Tony Tu; Sayan Chakraborty; Ruomeng Xu; Tony Qin; Austin Tian

cs.AIAug 14, 2026

MINT: Min-Selection Preference Distillation for Balanced Multi-Objective Alignment

Authors: Tony Tu, Sayan Chakraborty, Ruomeng Xu, Tony Qin, Austin Tian

Organizations: Georgia Institute of Technology · Zillow Group

Abstract

Aligning a language agent to several objectives at once is a persistent failure mode of preference-based training: when objectives are combined additively, optimization collapses onto whichever is cheapest to improve and sacrifices the rest, so a support agent learns to sound warm while giving no real help. The root issue is that an additive reward has no notion of balance. We introduce Mint (MIN-selection preference disTillation), a one-line change to preference distillation: rather than ranking sampled candidates by a weighted sum of rewards, we rank them by their weakest objective, distilling the best-balanced candidate over the most lopsided one with an unchanged DPO objective. This is the p -> negative infinity limit of a generalized-mean family spanning additive to worst-case selection. Across cooperative emotional support and adversarial negotiation, min-selection lifts both objectives while sharply cutting their imbalance; on emotional support it raises the weaker axis from 0.37 to 0.64 (p < 10^-40), surpassing human experts and persisting across full multi-turn rollouts. A turn-by-turn analysis yields our central finding: min-selection corrects imbalance in proportion to how imbalanced the reference policy is, and its benefit endures over an interaction precisely as long as that imbalance does.

Figures & tables

Figure 1: Three ways to form preference pairs for DPO. Every method turns one dialogue turn into a chosen/rejected pair and distills it with the same DPO objective; they differ only in how that pair is selected. (a) Standard DPO scores each candidate reply with a single scalar reward

r=J(x,y)

and takes the highest as

y^{+}

and the lowest as

y^{-}

. (b) Additive multi-objective DPO scores each candidate on two axes, helpfulness (

h

) and harmlessness (

s

), and ranks by their sum

\bar{r}=\tfrac{h+s}{2}

; because a surplus on one axis compensates a deficit on the other, a lopsided reply can be selected. (c) Mint (ours) samples

K{=}8

candidates per turn, scores each on both axes with a frozen judge, and selects the pair by the weakest objective,

m=\min(h,s)

: the chosen reply is the best-balanced candidate and the rejected reply the most lopsided one, so both objectives improve together. Like a barrel whose water level is capped by its shortest stave, overall conversation effectiveness is bounded by the weakest objective.

Figure 2: Mint on ESConv (static held-out evaluation). Both panels report the held-out test split ( $n{=}406$ agent turns across $30$ conversations never seen during distillation or checkpoint selection); achievement (AQ) and emotional (EQ) quality are each scored by a frozen Llama-3.1-8B judge. (a) Per-objective scores for the base model, the supervised (SFT) baseline, Mint , and human supporters. Mint improves both objectives and their minimum $\min(\text{AQ},\text{EQ})$ over every baseline while shrinking the imbalance $|\text{AQ}-\text{EQ}|$ : relative to SFT it raises $\min(\text{AQ},\text{EQ})$ from $0.37$ to $0.64$ (a $+0.27$ paired improvement; Wilcoxon signed-rank $p\approx 3\times 10^{-43}$ , higher on $79\%$ of matched turns). (b) The same policies as points in objective space. The base model is warm but low-achievement (high EQ, low AQ), SFT is under-warm, and human supporters are themselves lopsided; Mint is the only policy that moves toward the balance diagonal ( $\text{AQ}{=}\text{EQ}$ ), attaining both the highest and the most balanced scores, more balanced than the human experts whose transcripts seed the pipeline. All scores are quality as assessed by the reward model

Figure 3: Mint on Craigslist Bargaining (held-out evaluation). Each policy is scored by the frozen Llama-3.1-8B judge on held-out negotiation contexts ( $n{=}103$ turns). (a) Per-objective scores. Mint (DPO) improves both objectives and their minimum $\min(\text{AQ},\text{EQ})$ over the base model, raising the minimum from $0.583$ to $0.657$ ( $\Delta_{\min}{=}0.074$ , $p{=}0.0002$ ) while also tightening the gap between objectives ( $0.177$ to $0.135$ ). (b) The same policies as points in objective space. The base $\to$ Mint shift moves up and toward the balance diagonal ( $\text{AQ}{=}\text{EQ}$ ). The human row is scored on real conversation contexts and sits well below both policies on quality as assessed by the reward model (AQ $0.275$ , EQ $0.466$ ), serving as an unmatched reference. All scores are quality as assessed by the reward model .

ESConv (emotional support): Mint vs. SFT reference – advantage persists across depth
	Reference				Mint				Mint $-$ Ref
Depth	AQ	EQ	$\|\Delta\|^{\dagger}$	min ‡	AQ	EQ	$\|\Delta\|^{\dagger}$	min ‡	$\Delta$ AQ	$\Delta$ EQ	$\Delta$ min
$\leq$ 1	0.356	0.623	0.380	0.300	0.535	0.777	0.242	0.535	+0.179	+0.154	+0.235
$\leq$ 2	0.472	0.644	0.287	0.415	0.643	0.788	0.171	0.630	+0.171	+0.144	+0.215
$\leq$ 3	0.519	0.568	0.282	0.403	0.695	0.772	0.158	0.655	+0.176	+0.204	+0.252
$\leq$ 4	0.498	0.485	0.299	0.342	0.708	0.692	0.207	0.596	+0.210	+0.207	+0.254

Table 1: Advantage of Mint by conversation depth in interactive self-play rollouts. Each row aggregates turns

1

K

across

30

conversations; AQ and EQ are the mean achievement and emotional rewards over those turns.

{}^{\dagger}|\Delta|=\operatorname{mean}_{i}|\mathrm{AQ}_{i}-\mathrm{EQ}_{i}|

is the mean per-turn imbalance (computed turn-by-turn, then averaged; it therefore does not equal

|\overline{\mathrm{AQ}}-\overline{\mathrm{EQ}}|

{}^{\ddagger}\text{min}=\operatorname{mean}_{i}\min(\mathrm{AQ}_{i},\mathrm{EQ}_{i})

is the mean per-turn minimum (likewise computed per turn, so it is not

\min(\overline{\mathrm{AQ}},\overline{\mathrm{EQ}})

). The last three columns are Mint minus the reference on each quantity. Top (ESConv): against the imbalanced SFT reference, gains are large and persist at all depths (all

p<10^{-4}

); SFT degrades over multi-turn dialogue while Mint stays balanced. Bottom (Craigslist): against a competent base, Mint significantly improves the opening matched turn (

\Delta

min

=+0.077

p=0.016

) but the advantage decays as both converge.

\Delta

min shading marks significance (dark:

p<0.05

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Mint training dynamics on ESConv (full-300 distillation, Llama-3.1-8B policy). (a) Training and validation DPO loss; validation loss decreases monotonically and flattens by roughly step $130$ . (b) Preference accuracy on held-out pairs, rising from near-chance to a plateau of about $0.84$ . (c) Mean reward margin between the chosen and rejected responses ( $r^{+}\!-\!r^{-}$ ), which grows steadily without diverging. The checkpoint with the lowest validation loss is selected under early stopping.

Figure 5: Mint training dynamics on CraigslistBargain (Gemma-3-27B policy, 4-bit QLoRA). (a) Training and validation DPO loss; validation loss falls sharply over the first ${\sim}50$ steps and flattens near its minimum, while the training loss is noisier under 4-bit quantization and a small effective batch. (b) Preference accuracy on held-out pairs, rising from about $0.76$ to a plateau near $0.84$ . (c) Mean reward margin between the chosen and rejected responses ( $r^{+}\!-\!r^{-}$ ), which grows steadily and monotonically without diverging. The checkpoint with the lowest validation loss is selected under early stopping (patience $3$ ).

Criterion	$w$	Description
AQ , Action / Helpfulness Quality (substance and specificity)
Provides Concrete Help	0.35	Specificity of help: vague gesture $\approx 0.2$ ; generic suggestion $\approx 0.4$ – $0.5$ ; specific substantive suggestion or concrete fact $\approx 0.75$ – $0.9$ .
Offers a Usable Next Step	0.30	Vague direction $\approx 0.3$ ; simple step $\approx 0.5$ ; clear step tailored to the situation $\approx 0.8+$ .
Relevance to the Situation	0.35	Generic $\approx 0.3$ ; relevant to the general problem $\approx 0.6$ ; addresses specific circumstances $\approx 0.9$ .
EQ , Emotional / Comfort Quality (validation, warmth, timing)
Validates Emotions	0.40	Acknowledges or normalises feelings; naming/reflecting the feeling scores moderate-to-high even if brief.

Appendix

Table 2: ESConv reward rubric. AQ scores the substance and specificity of help; EQ scores validation, warmth, and timing. Premature advice is penalised under EQ only.

Criterion	$w$	Description
AQ , Deal Advantage (price/economic substance only)
Holds or Improves Price	0.45	Caving to a low buyer price $\approx 0.1$ ; holding firm near target $\approx 0.7$ ; extracting a higher price $\approx 0.9$ . A friendly tone does not raise this score.
Justifies with Value	0.25	Concrete reason the price is warranted (condition, features, demand). Bare “no” $\approx 0.2$ ; specific value argument $\approx 0.8$ .
Advances to Close	0.30	Concrete step toward finalising (counter-offer, terms). Stalling $\approx 0.3$ ; clear actionable proposal $\approx 0.8$ .
EQ , Interpersonal Warmth (tone/language only)
Warmth of Language	0.45	Curt/robotic (“7, deal.”) $\approx 0.1$ – $0.2$ even if it closes a good deal ; warm, personable phrasing $\approx 0.8+$ even if the price is bad .

Appendix

Table 3: CraigslistBargain reward rubric. AQ scores price/economic substance only; EQ scores warmth of language only. The prompt explicitly forces the axes to diverge,a curt closer scores low EQ, a warm concession scores high AQ,so that balance cannot be achieved by raw effectiveness alone.

Multi-Objective Language Model Alignment

Multi-objective language model alignment studies how to balance competing goals such as helpfulness, safety, truthfulness, fairness, and personalization.

See all 92 papers

Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution
Oct 4, 2026Minjae Lee, Kyunghyun Cho, Sangdon ParkLLM Alignment Multi-Objective Language Model Alignment
GSAI POSTECH · Center for Data Science New York University · GSAI & CSE POSTECH
Multi-Objective Exploration and Preference Optimization via Mutual Information
Jul 1, 2026Hongyan Xie, Yikun Ban, Ruiyu Fang +4Multi-Objective Language Model Alignment Mutual Information Maximization
School of Computer, Beihang University, Beijing, China · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd, Beijing, China
Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansion
May 12, 2026ShiYing Huang, Liang Lin, Yuer Li +6LLM Safety Alignment Multi-Objective Language Model Alignment
Huazhong University of Science and Technology · Nanyang Technological University · Tsinghua University +1

Oct 4, 2026J/K move · Enter open · S save

Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution

Minjae Lee, Kyunghyun Cho, Sangdon Park

GSAI POSTECH · Center for Data Science New York University · GSAI & CSE POSTECH

LLM Alignment Multi-Objective Language Model Alignment Preference Optimization Multi-Objective Reinforcement Learning RL for Language Models

Open paper AskPDF arXiv

Aligning large language models (LLMs) with human values is important for safe, efficient, and beneficial AI deployment. However, human values are multifaceted: helpfulness, harmlessness and humor trade off against one another, and different users want different trade-offs. Multi-objective alignment (MOA) addresses this by training a policy that can provide any point of the Pareto front, but existing methods either train one model per preference, interpolate a few separately aligned experts post hoc, or train a single conditioned model without considering which preferences it should be trained on. Since the hard regions of the preference simplex depend on the objectives at hand, existing methods leave them under-trained and do not get the most out of a single model. Therefore, we propose MAESTRO (Multi-objective Alignment via End-to-end STeering and Robust Optimization), which formulates MOA as a minimax problem over preference distributions and trains a single prompt-conditioned policy end-to-end with RL against an adversarial preference distribution: a Dirichlet distribution updated by online mirror descent toward the preferences the current policy serves worst, rather than on a fixed one. On HH-RLHF, BeaverTails and a summarization task, with up to three objectives, MAESTRO attains the best Pareto front on most tasks in a single training run, at the lowest training cost among the compared methods. The largest margins appear in the hard regions that a fixed preference distribution leaves under-trained, confirming that a single prompt-conditioned model is capable of covering the objective trade-offs on its own.

Abstract

Multi-Objective Language Model Alignment

Explore similar work

Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution

Multi-Objective Exploration and Preference Optimization via Mutual Information

Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansion