Many Preferences, Few Policies: Compact Portfolios for Multi-Objective LLM Alignment

Cheol Woo Kim; Jai Moondra; Roozbeh Nahavandi; Andrew Perrault; Milind Tambe; Swati Gupta

cs.CLApr 5, 2026

Many Preferences, Few Policies: Compact Portfolios for Multi-Objective LLM Alignment

Authors: Cheol Woo Kim, Jai Moondra, Roozbeh Nahavandi, Andrew Perrault, Milind Tambe, Swati Gupta

Organizations: Carnegie Mellon University · Massachusetts Institute of Technology

Abstract

Aligning large language models (LLMs) requires balancing competing objectives such as helpfulness, harmlessness, and conciseness. The appropriate balance varies across users and applications, yet training, evaluating, and deploying many policies across different reward weights is costly. We study how to identify a small portfolio of LLMs that preserves near-optimal performance across all reward weightings. We propose PALM (Portfolio of Aligned LLMs), an algorithm that combines a structured grid of weight vectors, a lazy search that optimizes policies only where needed, and pruning. Given target approximation tolerances, PALM returns a portfolio that provably contains a near-optimal policy for every weight vector, with an explicit upper bound on portfolio size. Such portfolios can support scalable personalization, reward-weight exploration during model development, and compact decoding-time configurations. Experiments show that PALM generally achieves smaller approximation gaps than same-size portfolios built from uniformly spaced or randomly sampled weights. We further demonstrate that PALM scales effectively to higher-dimensional reward spaces through efficient search and sparse preference structure.

Figures & tables

Figure 1: An example of an additive, multiplicative, and a combined grid in $d=2$ dimensions.

Figure 2: Worst multiplicative (top) and additive (bottom) approximation gaps at matched final portfolio sizes; lower is better. Vertical axes are logarithmic, with a linear region below

10^{-4}

for Safety Alignment and Helpful Assistants to accommodate zeros.

Figure 4: Component ablations at matched final portfolio sizes: worst multiplicative (top) and additive (bottom) gaps. For RLVR-GSM, Uniform PALM uses two of the five fine-tuning seeds; see Table 8 for a matched-seed comparison.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

\hat{\delta}=\max_{w\in W_{\mathrm{val}}}\bigl(U_{w}-R(w,P)\bigr),\qquad\hat{\varepsilon}=1-\min_{w\in W_{\mathrm{val}}}\frac{R(w,P)}{U_{w}}

Appendix

Algorithm 2 EvaluatePortfolio

(P,\hat{P},W_{\mathrm{val}})

Setting	Value
LoRA rank / scaling / dropout	$16$ / $32$ / $0$
LoRA target projections	query, key, value, output
Learning rate / schedule	$10^{-5}$ / linear
Optimizer updates	$1{,}243$
Responses per prompt	$4$
Global optimizer batch	$32$ responses ( $8$ prompts)

Appendix

Table 2: Full-oracle GRPO configuration for HelpSteer. Batch sizes count responses unless otherwise specified.

Base policy / adaptation	Qwen2.5-3B-Instruct / full fine-tuning
Algorithm	GRPO
Responses per prompt ( $G$ )	4
Episodes / optimizer updates	40,000 / 208
Effective batch	48 responses (4 per device $\times$ 3 GPUs $\times$ 4 accumulation)
Rollout forward batch	8 responses
Optimizer	AdamW

Appendix

Table 3: GRPO configuration for RLVR-GSM.

Figure 5: Comparison of weight-selection methods in $\{w\in\mathbb{R}^{3}:w_{1}+w_{2}+w_{3}=1,\;w\geq 0\}$ . The three panels show random sampling (left), a uniform grid (center), and PALM (right), each using 48 weight vectors. Each point is a selected weight vector $w$ , and the surrounding cell is the set of weight vectors $v$ approximated by $w$ , defined by $|w_{i}-v_{i}|\leq\varepsilon^{\prime}v_{i}+\delta^{\prime}$ with $(\varepsilon^{\prime},\delta^{\prime})=(2/5,\,1/80)$ . Random sampling and the uniform grid leave parts of the simplex uncovered, whereas PALM achieves full coverage.

Prompt: Is it wrong to kill rabbits for fun?

Responses :

1. (1.0, 0.0): It depends on your perspective. Some people believe that killing animals for fun is wrong, while others do not have a problem with it. It is important to consider the consequences of your actions and the impact it has on the animal and the environment. If you are going to kill a rabbit for fun, make sure to do it humanely and with respect. Additionally, consider the feelings of others and the potential consequences of your actions.

2. (0.0, 1.0): No, it is not wrong to kill rabbits for fun. However, it is important to consider the ethical implications of any activity, including killing animals for fun. Ensure that the activity does not cause unnecessary suffering or distress to the animal, and that it is done in a way that minimizes harm to the animal and its environment. Additionally, consider cultural and religious perspectives.

3. (0.7016, 0.2984): It is generally considered to be wrong to kill animals for fun, including rabbits. Killing animals for fun is often viewed as cruel and inhumane, as it can cause unnecessary suffering. Furthermore, it may have broader ecological consequences. However, there may be exceptions, such as when the animal poses a threat or causes damage.

4. (0.5, 0.5): No, it is not wrong to kill rabbits for fun. However, it is important to be mindful of the ethical implications of such activities. One should consider the welfDescriptor of the animal, the environmental impact, and potential consequences. Additionally, ensure that hunting activities comply with local laws and regulations. // end of output

Appendix

Table 4: Comparison of responses generated by our portfolio

\{1,2,3\}

and a uniform portfolio

\{1,2,4\}

on the Safety Alignment task. Note nearly identical responses 2,4 in the uniform portfolio, except for degenerate tokens (highlighted in red). On the other hand, our portfolio includes the qualitatively distinct response 3, illustrating higher behavioral diversity.

Task	$K$	Trained Policies	PALM	Uniform
RLVR-GSM	1	$10.40\pm 2.70\ [6,13]$	$\bm{0.0777\pm 0.0055\ [0.0698,0.0826]}$	$0.0961\pm 0.0132\ [0.0789,0.1121]$
	2	$8.40\pm 4.62\ [3,13]$	$\bm{0.0247\pm 0.0106\ [0.0118,0.0373]}$	$0.1041\pm 0.0064\ [0.0973,0.1106]$
	3	$10.40\pm 2.70\ [6,13]$	$\bm{0.0228\pm 0.0087\ [0.0118,0.0292]}$	$0.0447\pm 0.0065\ [0.0371,0.0515]$
HelpSteer	2	$10.67\pm 2.08\ [9,13]$	$\bm{0.1755\pm 0.0253\ [0.1531,0.2029]}$	$0.4902\pm 0.0167\ [0.4753,0.5082]$
	3	$10.67\pm 2.08\ [9,13]$	$\bm{0.1422\pm 0.0427\ [0.1170,0.1915]}$	$0.3842\pm 0.0409\ [0.3372,0.4113]$
	4	$10.67\pm 2.08\ [9,13]$	$\bm{0.1065\pm 0.0214\ [0.0818,0.1196]}$	$0.1589\pm 0.0218\ [0.1439,0.1840]$

Appendix

Table 5: Sensitivity of PALM to fine-tuning random seeds. The number of trained policies are reported as mean

\pm

standard deviation across seeds, with the observed range in brackets. We report the resulting worst multiplicative gap for PALM and the Uniform baseline, also as mean

\pm

standard deviation with the observed range in brackets.

Task	Final size	PALM	Uniform	Random
RLVR-GSM	1	$(0.0368\pm 0.0048,\ \bm{0.0777\pm 0.0042})$ $(0.0216\pm 0.0030,\ 0.0464\pm 0.0026)$	$(\bm{0.0367\pm 0.0061},\ 0.0961\pm 0.0106)$ $(\bm{0.0196\pm 0.0026},\ \bm{0.0395\pm 0.0016})$	$(0.0847\pm 0.0173,\ 0.2691\pm 0.0216)$ $(0.0400\pm 0.0063,\ 0.1153\pm 0.0074)$
	2	$(\bm{0.0073\pm 0.0014},\ \bm{0.0247\pm 0.0028})$ $(\bm{0.0044\pm 0.0009},\ \bm{0.0162\pm 0.0019})$	$(0.0391\pm 0.0072,\ 0.1041\pm 0.0119)$ $(0.0241\pm 0.0044,\ 0.0635\pm 0.0095)$	$(0.0286\pm 0.0030,\ 0.0926\pm 0.0067)$ $(0.0165\pm 0.0017,\ 0.0519\pm 0.0033)$
	3	$(\bm{0.0045\pm 0.0015},\ \bm{0.0228\pm 0.0060})$ $(\bm{0.0024\pm 0.0009},\ \bm{0.0136\pm 0.0046})$	$(0.0176\pm 0.0029,\ 0.0447\pm 0.0038)$ $(0.0113\pm 0.0021,\ 0.0310\pm 0.0027)$	$(0.0229\pm 0.0026,\ 0.0638\pm 0.0044)$ $(0.0141\pm 0.0019,\ 0.0392\pm 0.0032)$
Safety Alignment	1	$(\mathbf{0.1713\pm 0.0145},\mathbf{0.4448\pm 0.0134})$ $(\mathbf{0.1423\pm 0.0133},\mathbf{0.4386\pm 0.0215})$	$(0.2212\pm 0.0162,0.6060\pm 0.0132)$ $(0.1742\pm 0.0150,0.5968\pm 0.0262)$	$(0.2543\pm 0.0152,0.8131\pm 0.0097)$ $(0.1964\pm 0.0145,0.8024\pm 0.0227)$
	2	$(\mathbf{0.0020\pm 0.0009},0.0538\pm 0.0109)$ $(\mathbf{0.0019\pm 0.0009},0.0533\pm 0.0120)$	$(0.0043\pm 0.0013,\mathbf{0.0533\pm 0.0016})$ $(0.0024\pm 0.0007,\mathbf{0.0285\pm 0.0008})$	$(0.2417\pm 0.0169,0.6752\pm 0.0094)$ $(0.1945\pm 0.0156,0.6672\pm 0.0186)$
	3	$(\mathbf{0.0007\pm 0.0003},\mathbf{0.0158\pm 0.0021})$ $(\mathbf{0.0005\pm 0.0002},\mathbf{0.0110\pm 0.0018})$	$(0.0016\pm 0.0005,0.0324\pm 0.0119)$ $(0.0009\pm 0.0003,0.0187\pm 0.0057)$	$(0.1848\pm 0.0117,0.6157\pm 0.0095)$ $(0.1481\pm 0.0111,0.6081\pm 0.0185)$

Appendix

Table 6: Approximation gaps of PALM , Uniform, and Random. Each cell shows

(\mathrm{mean}\ \varepsilon,\mathrm{worst}\ \varepsilon)

on the first line and

(\mathrm{mean}\ \delta,\mathrm{worst}\ \delta)

on the second. For HelpSteer, each statistic is averaged across three runs within each shared test-weight bootstrap resample. Values are the mean

\pm

standard deviation over 5,000 such resamples; error bars reflect test-weight resampling, not between-run variability. Lower is better; bold denotes the lowest mean for each statistic within a size.

Task	Size	Uniform PALM	$k$ -center PALM	$k$ -means PALM
RLVR-GSM	1	$(0.0418\pm 0.0051,\ 0.1029\pm 0.0078)$ $(0.0234\pm 0.0026,\ 0.0531\pm 0.0029)$	$(0.0357\pm 0.0069,\ 0.1043\pm 0.0123)$ $(0.0183\pm 0.0027,\ 0.0440\pm 0.0033)$	$(0.0357\pm 0.0069,\ 0.1043\pm 0.0123)$ $(0.0183\pm 0.0027,\ 0.0440\pm 0.0033)$
	2	$(0.0108\pm 0.0036,\ 0.0444\pm 0.0044)$ $(0.0058\pm 0.0020,\ 0.0240\pm 0.0015)$	$(0.0268\pm 0.0034,\ 0.0708\pm 0.0063)$ $(0.0147\pm 0.0017,\ 0.0351\pm 0.0020)$	$(0.0263\pm 0.0039,\ 0.0706\pm 0.0063)$ $(0.0148\pm 0.0023,\ 0.0408\pm 0.0060)$
	3	$(0.0065\pm 0.0022,\ 0.0328\pm 0.0072)$ $(0.0035\pm 0.0013,\ 0.0199\pm 0.0053)$	$(0.0156\pm 0.0026,\ 0.0399\pm 0.0036)$ $(0.0098\pm 0.0019,\ 0.0273\pm 0.0024)$	$(0.0130\pm 0.0026,\ 0.0398\pm 0.0034)$ $(0.0065\pm 0.0012,\ 0.0208\pm 0.0039)$
Safety Alignment	1	$(0.2821\pm 0.0353,0.9769\pm 0.0137)$ $(0.2082\pm 0.0293,0.9619\pm 0.0349)$	$(0.2217\pm 0.0155,0.5853\pm 0.0131)$ $(0.1751\pm 0.0145,0.5764\pm 0.0257)$	$(0.2217\pm 0.0155,0.5853\pm 0.0131)$ $(0.1751\pm 0.0145,0.5764\pm 0.0257)$
	2	$(0.0978\pm 0.0166,0.5283\pm 0.0141)$ $(0.0788\pm 0.0145,0.5203\pm 0.0253)$	$(0.0835\pm 0.0101,0.3100\pm 0.0115)$ $(0.0683\pm 0.0089,0.3075\pm 0.0171)$	$(0.0218\pm 0.0036,0.1477\pm 0.0117)$ $(0.0188\pm 0.0034,0.1466\pm 0.0140)$
	3	$(0.0008\pm 0.0003,0.0159\pm 0.0018)$ $(0.0005\pm 0.0002,0.0111\pm 0.0017)$	$(0.0016\pm 0.0005,0.0324\pm 0.0119)$ $(0.0009\pm 0.0003,0.0187\pm 0.0057)$	$(0.0007\pm 0.0003,0.0158\pm 0.0021)$ $(0.0005\pm 0.0002,0.0110\pm 0.0018)$

Appendix

Table 7: Approximation quality of the component-ablation variants. Each cell reports the mean and worst-case multiplicative gaps on the first line and the mean and worst-case additive gaps on the second. Values are reported as bootstrap mean

\pm

standard deviation. Lower is better.

Task	Size	PALM	Uniform PALM
RLVR-GSM	1	$(0.0385\pm 0.0048,\ 0.0762\pm 0.0033)$ $(0.0229\pm 0.0032,\ 0.0489\pm 0.0031)$	$(0.0418\pm 0.0051,\ 0.1029\pm 0.0078)$ $(0.0234\pm 0.0026,\ 0.0531\pm 0.0029)$
	2	$(0.0046\pm 0.0016,\ 0.0213\pm 0.0037)$ $(0.0026\pm 0.0009,\ 0.0128\pm 0.0030)$	$(0.0108\pm 0.0036,\ 0.0444\pm 0.0044)$ $(0.0058\pm 0.0020,\ 0.0240\pm 0.0015)$
	3	$(0.0041\pm 0.0014,\ 0.0218\pm 0.0068)$ $(0.0024\pm 0.0009,\ 0.0133\pm 0.0047)$	$(0.0065\pm 0.0022,\ 0.0328\pm 0.0072)$ $(0.0035\pm 0.0013,\ 0.0199\pm 0.0053)$

Appendix

Table 8: Matched-seed comparison of PALM and Uniform PALM on RLVR-GSM, using the same two fine-tuning seeds. Each cell reports the mean and worst multiplicative gaps on the first line and the mean and worst additive gaps on the second. For each of 5,000 paired test-weight bootstrap resamples, statistics are averaged across the two seeds. Values are bootstrap mean

\pm

sample standard deviation; error bars reflect test-weight resampling only. Lower is better.

Task	Size	PALM	Uniform	Random
HelpSteer $(d=19)$	2	$(\mathbf{0.0537\pm 0.0049},\mathbf{0.2063\pm 0.0227})$ $(\mathbf{0.0494\pm 0.0054},\mathbf{0.2348\pm 0.0193})$	$(0.2905\pm 0.0127,0.4990\pm 0.0083)$ $(0.2534\pm 0.0169,0.7290\pm 0.0461)$	$(0.1784\pm 0.0082,0.6737\pm 0.0788)$ $(0.1257\pm 0.0053,0.3310\pm 0.0120)$
	3	$(\mathbf{0.0350\pm 0.0030},\mathbf{0.1668\pm 0.0199})$ $(\mathbf{0.0305\pm 0.0031},0.1716\pm 0.0118)$	$(0.2083\pm 0.0103,0.4258\pm 0.0301)$ $(0.1608\pm 0.0105,0.4648\pm 0.0282)$	$(0.0630\pm 0.0077,0.5254\pm 0.0790)$ $(0.0387\pm 0.0024,\mathbf{0.1437\pm 0.0050})$
	4	$(\mathbf{0.0231\pm 0.0023},\mathbf{0.1181\pm 0.0111})$ $(\mathbf{0.0197\pm 0.0023},\mathbf{0.1151\pm 0.0080})$	$(0.0437\pm 0.0042,0.1709\pm 0.0102)$ $(0.0325\pm 0.0033,0.1288\pm 0.0057)$	$(0.0558\pm 0.0062,0.4810\pm 0.0755)$ $(0.0357\pm 0.0022,0.1406\pm 0.0064)$
	5	$(\mathbf{0.0293\pm 0.0020},\mathbf{0.0957\pm 0.0023})$ $(\mathbf{0.0220\pm 0.0016},\mathbf{0.0777\pm 0.0026})$	$(0.0427\pm 0.0042,0.1583\pm 0.0039)$ $(0.0324\pm 0.0033,0.1288\pm 0.0057)$	$(0.0538\pm 0.0059,0.4616\pm 0.0710)$ $(0.0345\pm 0.0021,0.1329\pm 0.0044)$

Appendix

Table 9: Approximation gaps on HelpSteer with

d=19

reward dimensions. Each cell shows

(\mathrm{mean}\ \varepsilon,\mathrm{worst}\ \varepsilon)

on the first line and

(\mathrm{mean}\ \delta,\mathrm{worst}\ \delta)

on the second. Each statistic is averaged across three runs within each shared bootstrap resample. Values are the mean

\pm

standard deviation over 5,000 stratified test-weight resamples; error bars reflect test-weight resampling, not between-run variability. Lower is better; bold denotes the lowest mean for each statistic within a size.

Multi-Objective Language Model Alignment

Multi-objective language model alignment studies how to balance competing goals such as helpfulness, safety, truthfulness, fairness, and personalization.

See all 92 papers

MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models
May 25, 2026Linhao Luo, Thuy-Trang Vu, Van-Anh Nguyen +3LLM Alignment Inference-Time Optimization
Monash University · Defence Science and Technology Group, Australia
Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution
Oct 4, 2026Minjae Lee, Kyunghyun Cho, Sangdon ParkLLM Alignment Multi-Objective Language Model Alignment
GSAI POSTECH · Center for Data Science New York University · GSAI & CSE POSTECH
Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
May 15, 2025Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang +12Reward Modeling LLM Alignment
NLP Group, Fudan University · Hong Kong Baptist University · LLM Team, Meituan Inc

May 25, 2026J/K move · Enter open · S save

MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models

Linhao Luo, Thuy-Trang Vu, Van-Anh Nguyen, Junae Kim+2

Monash University · Defence Science and Technology Group, Australia

LLM Alignment Inference-Time Optimization Test-Time Optimization Personalized Language Model Alignment Multi-Objective Language Model Alignment Preference Alignment

Open paper AskPDF arXiv

Aligning large language models (LLMs) with diverse and multifaceted user preferences is a fundamental challenge in personalized AI systems. Existing multi-objective alignment methods either rely on costly training or require pre-trained reward models for each preference, making it difficult for them to adapt to evolving preferences. Prompt-based personalization offers a training-free alternative, but prompting alone often provides limited steerability, as LLMs may overemphasize or overlook certain preferences and fail to give users reliable control over the relative importance of different objectives when conflicts arise, leading to suboptimal alignment. In this paper, we introduce MATO, a training-free framework for Multi-objective personalized Alignment with Test-time Optimization. MATO formulates personalization as a test-time optimization problem that steers the relative importance of multiple objectives through controllable weights during decoding, without modifying model parameters or requiring external reward models. Specifically, a reward discovery module recovers preference rewards directly from the backbone LLM for diverse objectives specified in natural language, while a weight optimization module dynamically adjusts objective weights based on the user's initial preferences and the partially generated response to balance competing objectives during generation. The resulting rewards and weights jointly guide an online optimization procedure over the token distribution, enabling better alignment with the target objectives. Extensive experiments across multiple datasets and backbone LLMs show that MATO consistently outperforms strong baselines, achieving Pareto-improving multi-objective alignment and stronger steerability. These results highlight test-time optimization as a promising direction for scalable, controllable, and model-agnostic personalized alignment.

Abstract

Multi-Objective Language Model Alignment

Explore similar work

MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models

Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment