Uncovering Cross-Objective Interference in Multi-Objective Alignment

Yining Lu; Meng Jiang

cs.CLFeb 6, 2026

Uncovering Cross-Objective Interference in Multi-Objective Alignment

Authors: Yining Lu, Meng Jiang

Organizations: University of Notre Dame

Abstract

We study a persistent failure mode in multi-objective alignment for large language models (LLMs), in which scalarized training improves only some objectives while the others degrade. We formalize this phenomenon as cross-objective interference and, to our knowledge, conduct the first systematic study of scalarization algorithms for multi-objective LLM alignment. The study shows that interference is pervasive across algorithms yet strongly model-dependent. To understand how interference arises, we derive a local covariance law stating that an objective improves or degrades at first order according to the sign of the covariance between its reward and the scalarized score. We extend this law to the clipped surrogate objectives of modern reinforcement fine-tuning and show that it still holds under mild conditions. Building on this law, we propose COVariance-floor Enforced Reweighting (COVER), a one-sided controller that raises an objective's weight only when the covariance between its reward and the clipped advantage weight falls below a target. Through extensive experiments, we find that COVER can mitigate cross-objective interference while matching linear scalarization when objectives already co-improve. Finally, to explain why interference is model-dependent, we complement the local covariance law with a global convergence analysis. This analysis gives sufficient conditions for the non-convex scalarized objective to satisfy the Polyak--Łojasiewicz condition and relates interference to model geometry.

Figures & tables

Panel of Figure 1 — Figure 1: MATH-500 test performance over training on Qwen2.5-1.5B-Base and Qwen3-1.7B-Base. Accuracy, response length in tokens (lower is more concise), and clarity under REINFORCE for nine scalarization algorithms and COVER . Curves are smoothed with a centered moving average over five evaluations. On Qwen2.5-1.5B-Base, every method except COVER (ours) and PAMA (which fails to learn) improves conciseness and clarity at the expense of accuracy, a clear instance of cross-objective interference. Under the same training procedure, eight of ten methods co-improve all three objectives on Qwen3-1.7B-Base, only Linear interferes, and PAMA again fails to learn.

		Mean covariance ( $\times 10^{-2}$ )			Negative fraction			Test performance $\Delta_{m}$
Dataset	Objective	Linear	Dynamic	COVER	Linear	Dynamic	COVER	Linear	Dynamic	COVER
MATH-500	Accuracy	$2.40$ $\pm 0.09$	$2.40$ $\pm 0.08$	$2.70$ $\pm 0.16$	$0.000$ $\pm 0.000$	$0.000$ $\pm 0.000$	$0.000$ $\pm 0.000$	$+0.34$ $\pm 0.05$	$+0.34$ $\pm 0.02$	$+0.33$ $\pm 0.02$
	Clarity	$0.91$ $\pm 0.14$	$0.83$ $\pm 0.05$	$0.93$ $\pm 0.05$	$0.001$ $\pm 0.000$	$0.001$ $\pm 0.000$	$0.001$ $\pm 0.000$	$+0.96$ $\pm 0.01$	$+0.96$ $\pm 0.01$	$+0.97$ $\pm 0.00$
	Conciseness †	$1.40$ $\pm 0.20$	$1.34$ $\pm 0.05$	$1.49$ $\pm 0.09$	$0.002$ $\pm 0.000$	$0.003$ $\pm 0.000$	$0.004$ $\pm 0.001$	$-0.03$ $\pm 0.01$	$-0.05$ $\pm 0.01$	$-0.06$ $\pm 0.02$
IFEval	Constraint following	$1.72$ $\pm 0.02$	$1.75$ $\pm 0.03$	$1.81$ $\pm 0.01$	$0.005$ $\pm 0.001$	$0.004$ $\pm 0.001$	$0.004$ $\pm 0.001$	$+0.19$ $\pm 0.04$	$+0.20$ $\pm 0.03$	$+0.21$ $\pm 0.02$
	Helpfulness	$0.03$ $\pm 0.01$	$0.04$ $\pm 0.03$	$0.06$ $\pm 0.02$	$0.416$ $\pm 0.003$	$0.420$ $\pm 0.017$	$0.419$ $\pm 0.007$	$-0.04$ $\pm 0.03$	$-0.04$ $\pm 0.02$	$-0.03$ $\pm 0.03$

Table 1: Per-objective covariance and test-performance change on MATH-500 (Qwen2.5-1.5B-Base) and IFEval (Qwen2.5-1.5B-IFT). Mean covariance :

\mathbb{E}_{t,\mathbf{x}}\big[\mathrm{Cov}_{\mathbf{y}}(r_{m},s_{\mathbf{\lambda}_{t}})\big]

over training steps

t

and prompts

\mathbf{x}

, with

\mathbf{\lambda}_{t}

the method’s weights at step

t

. Negative fraction : fraction of training prompts with a negative covariance. Test performance

\Delta_{m}

: mean of the last five test evaluations minus initial performance. † Conciseness starts at its ceiling of 1.0, so

\Delta_{m}\leq 0

by design (Appendix A.1 ). Under all three methods, helpfulness has the smallest mean covariance and the largest negative fraction, and it is the only objective besides conciseness that does not improve.

Objectives	Final accuracy
Accuracy only	$0.535$ $\pm 0.017$
Clarity only	$0.134$ $\pm 0.011$
Conciseness only	$0.000$ $\pm 0.000$
All three (Linear)	$0.359$ $\pm 0.050$

Table 2: Single- vs. multi-objective training on MATH-500 with Qwen2.5-1.5B-Base. Final accuracy : mean of the last five test evaluations.

Figure 2: Test accuracy for ten methods across three models on MATH-500. Final accuracy averages the last five evaluations up to step 560, and the

\downarrow

and the number above it mark the drop, in percentage points, from the peak. Across models, COVER matches the strongest baselines in final accuracy with one of the smallest drops from the peak.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

	Accuracy			Clarity	Conciseness	Length
Objectives	Early	Final $\uparrow$	Drop $\downarrow$	Final	Final	Final
Initialization	$0.020$			$0.030$	$1.000$	$989$
Accuracy only	$0.506$ $\pm 0.016$	$0.535$ $\pm 0.017$	$0.059$ $\pm 0.014$	$0.893$ $\pm 0.026$	$0.597$ $\pm 0.019$	$505$ $\pm 8$
Clarity only	$0.416$ $\pm 0.038$	$0.134$ $\pm 0.011$	$0.349$ $\pm 0.015$	$0.948$ $\pm 0.026$	$0.007$ $\pm 0.006$	$1020$ $\pm 5$
Conciseness only	$0.003$ $\pm 0.003$	$0.000$ $\pm 0.000$	$0.153$ $\pm 0.060$	$0.000$ $\pm 0.000$	$1.000$ $\pm 0.000$	$2$ $\pm 1$
All three (Linear)	$0.465$ $\pm 0.013$	$0.359$ $\pm 0.050$	$0.141$ $\pm 0.037$	$0.992$ $\pm 0.007$	$0.969$ $\pm 0.008$	$113$ $\pm 15$

Appendix

Table 4: Single- vs. multi-objective training on MATH-500 with Qwen2.5-1.5B-Base using REINFORCE for 315 steps. Initialization : the base model before training. Early : mean test accuracy over steps 20-40, the window in which accuracy peaks under multi-objective training. Final : mean of the last five test evaluations. Drop : peak minus final accuracy, where the peak is the highest test accuracy during training. Length : mean response length in tokens. Accuracy-only training peaks at 0.59 and loses 0.059, whereas the three multi-objective methods peak at 0.50-0.51 and lose 0.135-0.157 as responses shrink to 112-120 tokens, so co-optimization both lowers the peak and more than doubles the drop. Under conciseness only, accuracy peaks at step 10, before the Early window.

Figure 4: MATH-500 test performance over REINFORCE training on Qwen2.5-1.5B-IFT. Test accuracy, response length in tokens (lower is more concise), and clarity for nine scalarization algorithms and COVER . Curves are smoothed with a centered moving average over five evaluations. Under every method except PAMA, accuracy peaks at 0.53-0.58 within the first 200 steps and then falls by 0.10-0.16, ending below its starting value (0.38-0.46 vs. 0.48). In the same runs, responses shorten from 530 to 90-340 tokens. All ten methods meet Definition A.1 , though PAMA does so only nominally, with a 0.04 fall in accuracy and almost no change in response length or clarity. Clarity also falls from its peak under Lagrangian (by 0.10) and Tchebycheff (by 0.08).

Panel of Figure 6 — Figure 6: Test performance over training at 7B-8B scale. Columns show accuracy, response length in tokens, and clarity. Row (a) shows three models trained with Dynamic on MATH-500 and row (b) three models trained with Linear on the MATH algebra problems, with Qwen3-8B-Base in both rows. On the other four models, clarity rises to 0.92 or higher and responses shorten to about 100 tokens. Over the same steps, accuracy falls from its early peak on Qwen2.5-7B-Base (0.67 to 0.51) and ends well below its starting value on Llama3.1-8B-IFT (0.50 to 0.38) and Ministral-8B-IFT (0.74 to 0.51). On DeepSeek-7B-Chat, accuracy starts low and dips only slightly (0.23 to 0.19). Qwen3-8B-Base co-improves on both datasets (accuracy 0.31 to 0.80 and 0.49 to 0.95). Its responses also shorten, from about 1900 tokens to about 650, but remain far longer than on the other four models. Interference is thus not a small-model artifact.

Panel of Figure 7 — Figure 7: Gradient alignment across objectives during multi-objective alignment. We measure pairwise cosine similarity between per-objective gradients throughout training. Negative values indicate conflicting updates, which is a standard proxy for identifying conflicting objectives in MTL ( Yu et al., 2020 ) . Across all three models, cosine similarities remain mostly non-negative and converge toward 0 as training progresses, suggesting that objectives are weakly coupled, neither strongly synergistic nor persistently antagonistic, with no conflicting behavior observed.

	Accuracy weight	Accuracy			Conciseness	Length
Target $c^{\ast}_{\text{acc}}$	Final (%)	Peak	Final $\uparrow$	Drop $\downarrow$	Final $\uparrow$	Final $\downarrow$
$0.00$ (Linear)	$33$	$0.507$ $\pm 0.018$	$0.358$ $\pm 0.033$	$0.149$ $\pm 0.032$	$0.959$ $\pm 0.017$	$116$ $\pm 11$
$0.05$	$33$	$0.520$ $\pm 0.010$	$0.369$ $\pm 0.029$	$0.151$ $\pm 0.028$	$0.975$ $\pm 0.011$	$107$ $\pm 3$
$0.10$	$34$	$0.510$ $\pm 0.044$	$0.352$ $\pm 0.018$	$0.158$ $\pm 0.042$	$0.968$ $\pm 0.006$	$108$ $\pm 6$
$0.15$	$38$	$0.517$ $\pm 0.021$	$0.351$ $\pm 0.032$	$0.165$ $\pm 0.053$	$0.979$ $\pm 0.016$	$106$ $\pm 7$
$0.20$	$46$	$0.517$ $\pm 0.012$	$0.389$ $\pm 0.017$	$0.128$ $\pm 0.017$	$0.946$ $\pm 0.010$	$118$ $\pm 11$

Appendix

Table 5: Effect of COVER ’s covariance target on accuracy on MATH-500 using Qwen2.5-1.5B-Base, extending Table 3 with additional targets. Accuracy weight : share of the total weight at the end of training (33% at the start). Final : mean of the last five test evaluations. Drop : peak minus final accuracy. Length : response length in tokens. Targets of 0.05-0.15 lie within the usual range of

\bar{c}_{\text{acc}}

(0.05-0.16), so the accuracy weight rises little and the drop stays at the level of Linear. Targets of 0.20 and above are never reached, so the weight rises on every step and the drop shrinks. Conciseness and response length stay at the level of Linear up to a target of 0.20 and worsen at 0.30 and 0.40.

Method	Helpfulness $\uparrow$	Constraint following $\uparrow$	Conciseness $\uparrow$
Linear	$0.715$ $\pm 0.032$	$0.693$ $\pm 0.039$	$0.904$ $\pm 0.027$
Dynamic	$0.718$ $\pm 0.021$	$0.707$ $\pm 0.027$	$0.899$ $\pm 0.025$
Lagrangian	$0.736$ $\pm 0.028$	$0.755$ $\pm 0.026$	$0.582$ $\pm 0.033$
Tchebycheff	$0.000$ $\pm 0.000$	$0.241$ $\pm 0.000$	$1.000$ $\pm 0.000$
PAMA	$0.969$ $\pm 0.040$	$0.258$ $\pm 0.064$	$0.068$ $\pm 0.094$
MGDA	$1.000$ $\pm 0.000$	$0.541$ $\pm 0.026$	$0.959$ $\pm 0.019$

Appendix

Table 6: Multi-objective IFEval results on Qwen2.5-1.5B-IFT (mean of the last five evaluations). Helpfulness rewards length up to 150 words, while conciseness rewards responses shorter than the running average length of the method’s own training rollouts (Appendix A.1 ). Lagrangian reaches the highest constraint following but violates its own 0.85 conciseness constraint. MGDA and FAMO saturate helpfulness at the cost of constraint following. Tchebycheff collapses to one-token responses and PAMA to maximum-length responses, so each fails on two objectives. Only Linear, Dynamic, and COVER keep all three objectives above 0.69, and these three differ by at most 0.03 on each objective.

Weights	Helpfulness $\uparrow$	Constraint following $\uparrow$	Conciseness $\uparrow$
$[.33,.34,.33]$	$0.715$ $\pm 0.032$	$0.693$ $\pm 0.039$	$0.904$ $\pm 0.027$
$[.34,.35,.31]$	$0.764$ $\pm 0.001$	$0.676$ $\pm 0.020$	$0.863$ $\pm 0.041$
$[.45,.30,.25]$	$0.954$ $\pm 0.011$	$0.626$ $\pm 0.003$	$0.923$ $\pm 0.019$
$[.55,.25,.20]$	$0.993$ $\pm 0.003$	$0.612$ $\pm 0.010$	$0.957$ $\pm 0.013$
COVER (Adaptive)	$0.729$ $\pm 0.026$	$0.714$ $\pm 0.018$	$0.890$ $\pm 0.024$

Appendix

Table 7: Static-weight ablation on IFEval, comparing fixed weight vectors against COVER on Qwen2.5-1.5B-IFT. The reported numbers are the mean of the last five evaluations. Weight vectors are ordered as helpfulness, constraint following, and conciseness. The vector

[.33,.34,.33]

is the Linear baseline, and

[.34,.35,.31]

fixes each weight at the mean of its normalized value over training under COVER . The vectors

[.45,.30,.25]

and

[.55,.25,.20]

move progressively more weight to helpfulness, which declines under Linear ( Table 1 ). Every fixed vector that departs from Linear raises helpfulness, and none raises constraint following above Linear. In contrast, COVER reaches the highest constraint following in this table, above its own average weights held fixed (0.714 vs. 0.676). This suggests that the timing of its reweighting matters, not only its average weights.

		Mean covariance	Negative fraction	Test performance						COVER
		Linear	Linear	Linear			COVER			Engaged	Final
Dataset	Objective	Linear	Linear	Start	Peak	Final	Start	Peak	Final	(%)	$\lambda_{m}$
GSM8K	Accuracy	$1.44$	$0.000$	$0.080$	$0.900$	$0.848$	$0.080$	$0.910$	$0.830$	$76.8$	$0.520$
	Clarity	$0.51$	$0.000$	$0.060$	$1.000$	$1.000$	$0.060$	$1.000$	$1.000$	$0.0$	$0.334$
	Conciseness †	$0.84$	$0.001$	$1.000$	$1.000$	$0.884$	$1.000$	$1.000$	$0.932$	$10.8$	$0.338$
MBPP	Pass rate	$0.81$	$0.000$	$0.565$	$0.565$	$0.424$	$0.565$	$0.565$	$0.475$	$100.0$	$1.005$

Appendix

Table 8: Linear and COVER on four additional datasets. GSM8K uses Qwen2.5-1.5B-Base, while MBPP, PKU-SafeRLHF, and XSum use Qwen2.5-1.5B-IFT. Mean covariance (in units of

10^{-2}

) and Negative fraction are defined as in Table 1 . Start is the first evaluation, Peak the best, and Final the mean of the last five. Start separates objectives that never rose (MBPP pass rate, XSum coverage) from those that rose and plateaued (GSM8K accuracy, XSum faithfulness). Engaged is the fraction of training steps on which COVER raised the objective’s weight, and Final

\lambda_{m}

is its unnormalized weight at the end of training (initially 0.33, or 0.5 on MBPP). † Conciseness is scored against the running average response length (Appendix A.1 ), so it starts at 1.0 and can fall even as responses shorten. Under Linear, interference appears on MBPP and XSum, where pass rate and coverage decline from their starting values, but not on GSM8K or PKU-SafeRLHF. COVER reduces the MBPP pass-rate drop from 0.141 to 0.090 but does not prevent the XSum coverage collapse.

	Steps 0-210			Full run (315 steps)
Target $c^{\ast}_{\text{acc}}$	Peak $\uparrow$	Accuracy $\uparrow$	Drop $\downarrow$	Final $\uparrow$
$0.00$ (Linear)	$0.663$ $\pm 0.006$	$0.307$ $\pm 0.171$	$0.357$ $\pm 0.169$	$0.299$ $\pm 0.212$
$0.10$	$0.643$ $\pm 0.029$	$0.457$ $\pm 0.025$	$0.187$ $\pm 0.051$	$0.449$ $\pm 0.049$
$0.20$	$0.657$ $\pm 0.015$	$0.483$ $\pm 0.035$	$0.173$ $\pm 0.032$	$0.276$ $\pm 0.247$
$0.30$	$0.653$ $\pm 0.015$	$0.433$ $\pm 0.178$	$0.220$ $\pm 0.165$	$0.453$ $\pm 0.118$
$0.40$	$0.653$ $\pm 0.006$	$0.160$ $\pm 0.277$	$0.493$ $\pm 0.280$	$0.401$ $\pm 0.126$

Appendix

Table 9: Effect of COVER ’s covariance target on accuracy at 7B on MATH-500 using Qwen2.5-7B-Base. Peak : best test accuracy up to step 210. Accuracy : test accuracy at step 210. Drop : Peak minus Accuracy. Final : mean test accuracy over the last five evaluations of the full run. Unlike at 1.5B ( Table 5 ), where the highest target works best, only the 0.10 target reliably reduces the drop and stays above Linear through the full run. This suggests that a larger model may need a more conservative target to avoid destabilizing training.

Panel of Figure 8 — Figure 8: MATH-500 test performance over GRPO training for Qwen2.5 and Qwen3 models. Under Definition A.1 , all ten methods interfere with accuracy on Qwen2.5-1.5B-Base (PAMA never learns the task), nine of ten on Qwen2.5-1.5B-IFT (all but PAMA, whose policy hardly changes), and one of ten on Qwen3-1.7B-Base. The pattern across models is the same under REINFORCE ( Figures 1 and 4 ), so interference is not specific to the unclipped objective. COVER matches Linear on every model. On the two Qwen2.5 models, Lagrangian and Tchebycheff keep higher accuracy at the cost of longer responses and lower clarity.

			Test accuracy
Objectives	Penalty $\kappa$	Gradient norm	Peak (step)	Step 25	Final $\uparrow$	Final entropy
Accuracy only	$0$	$0.43$	$0.610$ (165)	$0.440$	$0.526$	$0.11$
Accuracy only	$3\times 10^{-4}$	$3{,}160$	$0.200$ (5)	$0.000$	$0.000$	$11.93$
Accuracy only	$10^{-3}$	$10{,}971$	$0.190$ (5)	$0.020$	$0.000$	$11.93$
Accuracy only	$3\times 10^{-3}$	$28{,}495$	$0.220$ (10)	$0.000$	$0.000$	$11.93$
Accuracy only	$10^{-2}$	$109{,}646$	$0.190$ (10)	$0.000$	$0.000$	$11.93$

Appendix

Table 10: Effect of spectral decoupling on MATH-500 with Qwen2.5-1.5B-Base using REINFORCE. Spectral decoupling ( Pezeshki et al., 2021 ) adds the logit-norm penalty

\tfrac{\kappa}{2}\|\text{logits}\|^{2}

to the policy loss as a remedy for gradient starvation. We test whether this remedy removes cross-objective interference under Linear, with accuracy-only training as a control. Gradient norm : median gradient norm over the first five updates, measured before gradient-norm clipping to 1. Final : mean of the last five test evaluations. Final entropy : policy entropy on the training batch at the last update, whose maximum

\ln|\mathcal{X}|=\ln 151936=11.93

is that of the uniform distribution over the vocabulary. Dashes mark unavailable quantities. Because the penalty sums over the full vocabulary, even

\kappa=10^{-4}

raises the gradient norm to 1,072, about 2,500 times the 0.43 of the unpenalized accuracy-only run. Every penalized run reaches this maximum entropy, including the accuracy-only runs, which have no competing objective. The collapse therefore comes from the penalty itself, not from cross-objective interference. At every tested strength, spectral decoupling collapses the policy.

Method	Hyperparameters
Lagrangian primal-dual	On MATH-500 the primary objective is accuracy, with constraints on conciseness and clarity (targets $[0.9,0.9]$ ); on IFEval the primary objective is constraint following, with constraints on helpfulness ( $\geq 0.70$ ) and conciseness ( $\geq 0.85$ ); dual learning rate $\eta_{\lambda}=0.01$ ; KL coefficient $=0$ .
Smooth Tchebycheff	Preference weights $\lambda_{m}=[0.333,0.333,0.334]$ ; OMD learning rate $\alpha=0.3$ .
MGDA	Same shared settings, except KL coefficient $=0$ .
GradNorm	Exponent $\alpha=1.5$ ; weight learning rate $\eta_{w}=0.025$ .
Nash-MTL	Nash optimization iterations $T_{\mathrm{Nash}}=20$ ; weight learning rate $\eta_{\alpha}=0.1$ ; numerical constant $\epsilon=10^{-8}$ ; maximum weight $\alpha_{\max}=1000$ .
FAMO	Weight learning rate $\eta_{w}=0.025$ ; weight decay $\gamma=10^{-5}$ ; numerical constant $\epsilon=10^{-8}$ ; loss margin $\delta=1.0$ .

Appendix

Table 11: Hyperparameter summary for different scalarization algorithms. Unless noted, all runs use the shared policy update and compute settings listed below.

Explore similar work

Apr 27, 2026cs.LG

Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment

Multi-Objective Alignment aims to align Large Language Models (LLMs) with diverse and often conflicting human values by optimizing multiple objectives simultaneously. Existing methods predominantly rely on static preference weight construction strategies. However, rigidly aligning to fixed targets discards valuable intermediate information, as training responses inherently embody valid preference trade-offs even when deviating from the target. To address this limitation, we propose Meal, i.e., MEta ALigner, a bi-level meta-learning framework enabling bidirectional optimization between preferences and policy responses, generating instructive dynamic preferences for steadier training. Specifically, we introduce a preference-weight-net as a meta-learner to generate adaptive preference weights based on input prompts and update the preference weights as learnable parameters, while the LLM policy acts as a base-learner optimizing response generation conditioned on these preferences with rejection sampling strategy. Extensive empirical results demonstrate that our method achieves superior performance on several multi-objective benchmarks, validating the effectiveness of the dynamic bidirectional preference-policy optimization framework.

Wenzhe Xu, Biao Liu, Yiyang Sun +2

Large Language Model Alignment Bidirectional

Ask PDF

May 25, 2026cs.CL

MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models

Aligning large language models (LLMs) with diverse and multifaceted user preferences is a fundamental challenge in personalized AI systems. Existing multi-objective alignment methods either rely on costly training or require pre-trained reward models for each preference, making it difficult for them to adapt to evolving preferences. Prompt-based personalization offers a training-free alternative, but prompting alone often provides limited steerability, as LLMs may overemphasize or overlook certain preferences and fail to give users reliable control over the relative importance of different objectives when conflicts arise, leading to suboptimal alignment. In this paper, we introduce MATO, a training-free framework for Multi-objective personalized Alignment with Test-time Optimization. MATO formulates personalization as a test-time optimization problem that steers the relative importance of multiple objectives through controllable weights during decoding, without modifying model parameters or requiring external reward models. Specifically, a reward discovery module recovers preference rewards directly from the backbone LLM for diverse objectives specified in natural language, while a weight optimization module dynamically adjusts objective weights based on the user's initial preferences and the partially generated response to balance competing objectives during generation. The resulting rewards and weights jointly guide an online optimization procedure over the token distribution, enabling better alignment with the target objectives. Extensive experiments across multiple datasets and backbone LLMs show that MATO consistently outperforms strong baselines, achieving Pareto-improving multi-objective alignment and stronger steerability. These results highlight test-time optimization as a promising direction for scalable, controllable, and model-agnostic personalized alignment.

Linhao Luo, Thuy-Trang Vu, Van-Anh Nguyen +3

Monash University · Defence Science and Technology Group, Australia

Large Language Model Personalization Preference Alignment Learning Test Time

Ask PDF

May 10, 2026cs.NE

EvoPref: Multi-Objective Evolutionary Optimization Discovers Diverse LLM Alignments Beyond Gradient Descent

Gradient-based preference optimization methods for large language model (LLM) alignment suffer from preference collapse, converging to narrow behavioral modes while neglecting preference diversity. We introduce EvoPref, a multi-objective evolutionary algorithm that maintains populations of Low-Rank Adaptation (LoRA) adapters optimized across helpfulness, harmlessness, and honesty objectives using Non-dominated Sorting Genetic Algorithm II (NSGA-II) selection with archive-based diversity preservation. Our primary contribution is demonstrating that population-based methods discover substantially more diverse alignments than gradient descent. On standard benchmarks, EvoPref improves preference coverage by 18% (median 82.5% vs. 70.0% for ORPO,

p<0.001

, Wilcoxon,

n=30

) and reduces collapse rates by 47% (11.0% vs. 20.6%,

p<0.001

), while achieving competitive alignment quality (median 75.5% RewardBench vs. 75.0% for ORPO,

p<0.05

). We provide theoretical motivation extending recent multi-objective evolutionary algorithm (MOEA) runtime analysis (Dang et al., 2025) suggesting why archive-based methods escape collapse more effectively than single-trajectory optimization. Comprehensive comparisons against MOEA/D, SMS-EMOA, CMA-ES, and gradient baselines (DPO, IPO, KTO, ORPO) with rigorous statistical testing (Friedman with Holm correction, Vargha-Delaney effect sizes, median with IQR) confirm that multi-objective selection with diversity preservation is essential. This work establishes evolutionary optimization as a principled paradigm for diverse LLM alignment.

Dongxin Guo, Jikun Wu, Siu Ming Yiu

The University of Hong Kong, China · Stellaris AI Limited, China

Large Language Model Alignment Large Language Model Adaptation Evolutionary Optimization Gradient Descent Archive

Ask PDF