Many Preferences, Few Policies: Compact Portfolios for Multi-Objective LLM Alignment

Cheol Woo Kim; Jai Moondra; Roozbeh Nahavandi; Andrew Perrault; Milind Tambe; Swati Gupta

cs.CLApr 5, 2026

Many Preferences, Few Policies: Compact Portfolios for Multi-Objective LLM Alignment

Authors: Cheol Woo Kim, Jai Moondra, Roozbeh Nahavandi, Andrew Perrault, Milind Tambe, Swati Gupta

Organizations: Carnegie Mellon University · Massachusetts Institute of Technology

Abstract

Aligning large language models (LLMs) requires balancing competing objectives such as helpfulness, harmlessness, and conciseness. The appropriate balance varies across users and applications, yet training, evaluating, and deploying many policies across different reward weights is costly. We study how to identify a small portfolio of LLMs that preserves near-optimal performance across all reward weightings. We propose PALM (Portfolio of Aligned LLMs), an algorithm that combines a structured grid of weight vectors, a lazy search that optimizes policies only where needed, and pruning. Given target approximation tolerances, PALM returns a portfolio that provably contains a near-optimal policy for every weight vector, with an explicit upper bound on portfolio size. Such portfolios can support scalable personalization, reward-weight exploration during model development, and compact decoding-time configurations. Experiments show that PALM generally achieves smaller approximation gaps than same-size portfolios built from uniformly spaced or randomly sampled weights. We further demonstrate that PALM scales effectively to higher-dimensional reward spaces through efficient search and sparse preference structure.

Figures & tables

Figure 1: An example of an additive, multiplicative, and a combined grid in $d=2$ dimensions.

Figure 2: Worst multiplicative (top) and additive (bottom) approximation gaps at matched final portfolio sizes; lower is better. Vertical axes are logarithmic, with a linear region below

10^{-4}

for Safety Alignment and Helpful Assistants to accommodate zeros.

Figure 4: Component ablations at matched final portfolio sizes: worst multiplicative (top) and additive (bottom) gaps. For RLVR-GSM, Uniform PALM uses two of the five fine-tuning seeds; see Table 8 for a matched-seed comparison.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

\hat{\delta}=\max_{w\in W_{\mathrm{val}}}\bigl(U_{w}-R(w,P)\bigr),\qquad\hat{\varepsilon}=1-\min_{w\in W_{\mathrm{val}}}\frac{R(w,P)}{U_{w}}

Appendix

Algorithm 2 EvaluatePortfolio

(P,\hat{P},W_{\mathrm{val}})

Setting	Value
LoRA rank / scaling / dropout	$16$ / $32$ / $0$
LoRA target projections	query, key, value, output
Learning rate / schedule	$10^{-5}$ / linear
Optimizer updates	$1{,}243$
Responses per prompt	$4$
Global optimizer batch	$32$ responses ( $8$ prompts)

Appendix

Table 2: Full-oracle GRPO configuration for HelpSteer. Batch sizes count responses unless otherwise specified.

Base policy / adaptation	Qwen2.5-3B-Instruct / full fine-tuning
Algorithm	GRPO
Responses per prompt ( $G$ )	4
Episodes / optimizer updates	40,000 / 208
Effective batch	48 responses (4 per device $\times$ 3 GPUs $\times$ 4 accumulation)
Rollout forward batch	8 responses
Optimizer	AdamW

Appendix

Table 3: GRPO configuration for RLVR-GSM.

Figure 5: Comparison of weight-selection methods in $\{w\in\mathbb{R}^{3}:w_{1}+w_{2}+w_{3}=1,\;w\geq 0\}$ . The three panels show random sampling (left), a uniform grid (center), and PALM (right), each using 48 weight vectors. Each point is a selected weight vector $w$ , and the surrounding cell is the set of weight vectors $v$ approximated by $w$ , defined by $|w_{i}-v_{i}|\leq\varepsilon^{\prime}v_{i}+\delta^{\prime}$ with $(\varepsilon^{\prime},\delta^{\prime})=(2/5,\,1/80)$ . Random sampling and the uniform grid leave parts of the simplex uncovered, whereas PALM achieves full coverage.

Prompt: Is it wrong to kill rabbits for fun?

Responses :

1. (1.0, 0.0): It depends on your perspective. Some people believe that killing animals for fun is wrong, while others do not have a problem with it. It is important to consider the consequences of your actions and the impact it has on the animal and the environment. If you are going to kill a rabbit for fun, make sure to do it humanely and with respect. Additionally, consider the feelings of others and the potential consequences of your actions.

2. (0.0, 1.0): No, it is not wrong to kill rabbits for fun. However, it is important to consider the ethical implications of any activity, including killing animals for fun. Ensure that the activity does not cause unnecessary suffering or distress to the animal, and that it is done in a way that minimizes harm to the animal and its environment. Additionally, consider cultural and religious perspectives.

3. (0.7016, 0.2984): It is generally considered to be wrong to kill animals for fun, including rabbits. Killing animals for fun is often viewed as cruel and inhumane, as it can cause unnecessary suffering. Furthermore, it may have broader ecological consequences. However, there may be exceptions, such as when the animal poses a threat or causes damage.

4. (0.5, 0.5): No, it is not wrong to kill rabbits for fun. However, it is important to be mindful of the ethical implications of such activities. One should consider the welfDescriptor of the animal, the environmental impact, and potential consequences. Additionally, ensure that hunting activities comply with local laws and regulations. // end of output

Appendix

Table 4: Comparison of responses generated by our portfolio

\{1,2,3\}

and a uniform portfolio

\{1,2,4\}

on the Safety Alignment task. Note nearly identical responses 2,4 in the uniform portfolio, except for degenerate tokens (highlighted in red). On the other hand, our portfolio includes the qualitatively distinct response 3, illustrating higher behavioral diversity.

Task	$K$	Trained Policies	PALM	Uniform
RLVR-GSM	1	$10.40\pm 2.70\ [6,13]$	$\bm{0.0777\pm 0.0055\ [0.0698,0.0826]}$	$0.0961\pm 0.0132\ [0.0789,0.1121]$
	2	$8.40\pm 4.62\ [3,13]$	$\bm{0.0247\pm 0.0106\ [0.0118,0.0373]}$	$0.1041\pm 0.0064\ [0.0973,0.1106]$
	3	$10.40\pm 2.70\ [6,13]$	$\bm{0.0228\pm 0.0087\ [0.0118,0.0292]}$	$0.0447\pm 0.0065\ [0.0371,0.0515]$
HelpSteer	2	$10.67\pm 2.08\ [9,13]$	$\bm{0.1755\pm 0.0253\ [0.1531,0.2029]}$	$0.4902\pm 0.0167\ [0.4753,0.5082]$
	3	$10.67\pm 2.08\ [9,13]$	$\bm{0.1422\pm 0.0427\ [0.1170,0.1915]}$	$0.3842\pm 0.0409\ [0.3372,0.4113]$
	4	$10.67\pm 2.08\ [9,13]$	$\bm{0.1065\pm 0.0214\ [0.0818,0.1196]}$	$0.1589\pm 0.0218\ [0.1439,0.1840]$

Appendix

Table 5: Sensitivity of PALM to fine-tuning random seeds. The number of trained policies are reported as mean

\pm

standard deviation across seeds, with the observed range in brackets. We report the resulting worst multiplicative gap for PALM and the Uniform baseline, also as mean

\pm

standard deviation with the observed range in brackets.

Task	Final size	PALM	Uniform	Random
RLVR-GSM	1	$(0.0368\pm 0.0048,\ \bm{0.0777\pm 0.0042})$ $(0.0216\pm 0.0030,\ 0.0464\pm 0.0026)$	$(\bm{0.0367\pm 0.0061},\ 0.0961\pm 0.0106)$ $(\bm{0.0196\pm 0.0026},\ \bm{0.0395\pm 0.0016})$	$(0.0847\pm 0.0173,\ 0.2691\pm 0.0216)$ $(0.0400\pm 0.0063,\ 0.1153\pm 0.0074)$
	2	$(\bm{0.0073\pm 0.0014},\ \bm{0.0247\pm 0.0028})$ $(\bm{0.0044\pm 0.0009},\ \bm{0.0162\pm 0.0019})$	$(0.0391\pm 0.0072,\ 0.1041\pm 0.0119)$ $(0.0241\pm 0.0044,\ 0.0635\pm 0.0095)$	$(0.0286\pm 0.0030,\ 0.0926\pm 0.0067)$ $(0.0165\pm 0.0017,\ 0.0519\pm 0.0033)$
	3	$(\bm{0.0045\pm 0.0015},\ \bm{0.0228\pm 0.0060})$ $(\bm{0.0024\pm 0.0009},\ \bm{0.0136\pm 0.0046})$	$(0.0176\pm 0.0029,\ 0.0447\pm 0.0038)$ $(0.0113\pm 0.0021,\ 0.0310\pm 0.0027)$	$(0.0229\pm 0.0026,\ 0.0638\pm 0.0044)$ $(0.0141\pm 0.0019,\ 0.0392\pm 0.0032)$
Safety Alignment	1	$(\mathbf{0.1713\pm 0.0145},\mathbf{0.4448\pm 0.0134})$ $(\mathbf{0.1423\pm 0.0133},\mathbf{0.4386\pm 0.0215})$	$(0.2212\pm 0.0162,0.6060\pm 0.0132)$ $(0.1742\pm 0.0150,0.5968\pm 0.0262)$	$(0.2543\pm 0.0152,0.8131\pm 0.0097)$ $(0.1964\pm 0.0145,0.8024\pm 0.0227)$
	2	$(\mathbf{0.0020\pm 0.0009},0.0538\pm 0.0109)$ $(\mathbf{0.0019\pm 0.0009},0.0533\pm 0.0120)$	$(0.0043\pm 0.0013,\mathbf{0.0533\pm 0.0016})$ $(0.0024\pm 0.0007,\mathbf{0.0285\pm 0.0008})$	$(0.2417\pm 0.0169,0.6752\pm 0.0094)$ $(0.1945\pm 0.0156,0.6672\pm 0.0186)$
	3	$(\mathbf{0.0007\pm 0.0003},\mathbf{0.0158\pm 0.0021})$ $(\mathbf{0.0005\pm 0.0002},\mathbf{0.0110\pm 0.0018})$	$(0.0016\pm 0.0005,0.0324\pm 0.0119)$ $(0.0009\pm 0.0003,0.0187\pm 0.0057)$	$(0.1848\pm 0.0117,0.6157\pm 0.0095)$ $(0.1481\pm 0.0111,0.6081\pm 0.0185)$

Appendix

Table 6: Approximation gaps of PALM , Uniform, and Random. Each cell shows

(\mathrm{mean}\ \varepsilon,\mathrm{worst}\ \varepsilon)

on the first line and

(\mathrm{mean}\ \delta,\mathrm{worst}\ \delta)

on the second. For HelpSteer, each statistic is averaged across three runs within each shared test-weight bootstrap resample. Values are the mean

\pm

standard deviation over 5,000 such resamples; error bars reflect test-weight resampling, not between-run variability. Lower is better; bold denotes the lowest mean for each statistic within a size.

Task	Size	Uniform PALM	$k$ -center PALM	$k$ -means PALM
RLVR-GSM	1	$(0.0418\pm 0.0051,\ 0.1029\pm 0.0078)$ $(0.0234\pm 0.0026,\ 0.0531\pm 0.0029)$	$(0.0357\pm 0.0069,\ 0.1043\pm 0.0123)$ $(0.0183\pm 0.0027,\ 0.0440\pm 0.0033)$	$(0.0357\pm 0.0069,\ 0.1043\pm 0.0123)$ $(0.0183\pm 0.0027,\ 0.0440\pm 0.0033)$
	2	$(0.0108\pm 0.0036,\ 0.0444\pm 0.0044)$ $(0.0058\pm 0.0020,\ 0.0240\pm 0.0015)$	$(0.0268\pm 0.0034,\ 0.0708\pm 0.0063)$ $(0.0147\pm 0.0017,\ 0.0351\pm 0.0020)$	$(0.0263\pm 0.0039,\ 0.0706\pm 0.0063)$ $(0.0148\pm 0.0023,\ 0.0408\pm 0.0060)$
	3	$(0.0065\pm 0.0022,\ 0.0328\pm 0.0072)$ $(0.0035\pm 0.0013,\ 0.0199\pm 0.0053)$	$(0.0156\pm 0.0026,\ 0.0399\pm 0.0036)$ $(0.0098\pm 0.0019,\ 0.0273\pm 0.0024)$	$(0.0130\pm 0.0026,\ 0.0398\pm 0.0034)$ $(0.0065\pm 0.0012,\ 0.0208\pm 0.0039)$
Safety Alignment	1	$(0.2821\pm 0.0353,0.9769\pm 0.0137)$ $(0.2082\pm 0.0293,0.9619\pm 0.0349)$	$(0.2217\pm 0.0155,0.5853\pm 0.0131)$ $(0.1751\pm 0.0145,0.5764\pm 0.0257)$	$(0.2217\pm 0.0155,0.5853\pm 0.0131)$ $(0.1751\pm 0.0145,0.5764\pm 0.0257)$
	2	$(0.0978\pm 0.0166,0.5283\pm 0.0141)$ $(0.0788\pm 0.0145,0.5203\pm 0.0253)$	$(0.0835\pm 0.0101,0.3100\pm 0.0115)$ $(0.0683\pm 0.0089,0.3075\pm 0.0171)$	$(0.0218\pm 0.0036,0.1477\pm 0.0117)$ $(0.0188\pm 0.0034,0.1466\pm 0.0140)$
	3	$(0.0008\pm 0.0003,0.0159\pm 0.0018)$ $(0.0005\pm 0.0002,0.0111\pm 0.0017)$	$(0.0016\pm 0.0005,0.0324\pm 0.0119)$ $(0.0009\pm 0.0003,0.0187\pm 0.0057)$	$(0.0007\pm 0.0003,0.0158\pm 0.0021)$ $(0.0005\pm 0.0002,0.0110\pm 0.0018)$

Appendix

Table 7: Approximation quality of the component-ablation variants. Each cell reports the mean and worst-case multiplicative gaps on the first line and the mean and worst-case additive gaps on the second. Values are reported as bootstrap mean

\pm

standard deviation. Lower is better.

Task	Size	PALM	Uniform PALM
RLVR-GSM	1	$(0.0385\pm 0.0048,\ 0.0762\pm 0.0033)$ $(0.0229\pm 0.0032,\ 0.0489\pm 0.0031)$	$(0.0418\pm 0.0051,\ 0.1029\pm 0.0078)$ $(0.0234\pm 0.0026,\ 0.0531\pm 0.0029)$
	2	$(0.0046\pm 0.0016,\ 0.0213\pm 0.0037)$ $(0.0026\pm 0.0009,\ 0.0128\pm 0.0030)$	$(0.0108\pm 0.0036,\ 0.0444\pm 0.0044)$ $(0.0058\pm 0.0020,\ 0.0240\pm 0.0015)$
	3	$(0.0041\pm 0.0014,\ 0.0218\pm 0.0068)$ $(0.0024\pm 0.0009,\ 0.0133\pm 0.0047)$	$(0.0065\pm 0.0022,\ 0.0328\pm 0.0072)$ $(0.0035\pm 0.0013,\ 0.0199\pm 0.0053)$

Appendix

Table 8: Matched-seed comparison of PALM and Uniform PALM on RLVR-GSM, using the same two fine-tuning seeds. Each cell reports the mean and worst multiplicative gaps on the first line and the mean and worst additive gaps on the second. For each of 5,000 paired test-weight bootstrap resamples, statistics are averaged across the two seeds. Values are bootstrap mean

\pm

sample standard deviation; error bars reflect test-weight resampling only. Lower is better.

Task	Size	PALM	Uniform	Random
HelpSteer $(d=19)$	2	$(\mathbf{0.0537\pm 0.0049},\mathbf{0.2063\pm 0.0227})$ $(\mathbf{0.0494\pm 0.0054},\mathbf{0.2348\pm 0.0193})$	$(0.2905\pm 0.0127,0.4990\pm 0.0083)$ $(0.2534\pm 0.0169,0.7290\pm 0.0461)$	$(0.1784\pm 0.0082,0.6737\pm 0.0788)$ $(0.1257\pm 0.0053,0.3310\pm 0.0120)$
	3	$(\mathbf{0.0350\pm 0.0030},\mathbf{0.1668\pm 0.0199})$ $(\mathbf{0.0305\pm 0.0031},0.1716\pm 0.0118)$	$(0.2083\pm 0.0103,0.4258\pm 0.0301)$ $(0.1608\pm 0.0105,0.4648\pm 0.0282)$	$(0.0630\pm 0.0077,0.5254\pm 0.0790)$ $(0.0387\pm 0.0024,\mathbf{0.1437\pm 0.0050})$
	4	$(\mathbf{0.0231\pm 0.0023},\mathbf{0.1181\pm 0.0111})$ $(\mathbf{0.0197\pm 0.0023},\mathbf{0.1151\pm 0.0080})$	$(0.0437\pm 0.0042,0.1709\pm 0.0102)$ $(0.0325\pm 0.0033,0.1288\pm 0.0057)$	$(0.0558\pm 0.0062,0.4810\pm 0.0755)$ $(0.0357\pm 0.0022,0.1406\pm 0.0064)$
	5	$(\mathbf{0.0293\pm 0.0020},\mathbf{0.0957\pm 0.0023})$ $(\mathbf{0.0220\pm 0.0016},\mathbf{0.0777\pm 0.0026})$	$(0.0427\pm 0.0042,0.1583\pm 0.0039)$ $(0.0324\pm 0.0033,0.1288\pm 0.0057)$	$(0.0538\pm 0.0059,0.4616\pm 0.0710)$ $(0.0345\pm 0.0021,0.1329\pm 0.0044)$

Appendix

Table 9: Approximation gaps on HelpSteer with

d=19

reward dimensions. Each cell shows

(\mathrm{mean}\ \varepsilon,\mathrm{worst}\ \varepsilon)

on the first line and

(\mathrm{mean}\ \delta,\mathrm{worst}\ \delta)

on the second. Each statistic is averaged across three runs within each shared bootstrap resample. Values are the mean

\pm

standard deviation over 5,000 stratified test-weight resamples; error bars reflect test-weight resampling, not between-run variability. Lower is better; bold denotes the lowest mean for each statistic within a size.

Multi-Objective Language Model Alignment

Multi-objective language model alignment studies how to balance competing goals such as helpfulness, safety, truthfulness, fairness, and personalization.

See all 92 papers

May 25, 2026

MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models

Aligning large language models (LLMs) with diverse and multifaceted user preferences is a fundamental challenge in personalized AI systems. Existing multi-objective alignment methods either rely on costly training or require pre-trained reward models for each preference, making it difficult for them to adapt to evolving preferences. Prompt-based personalization offers a training-free alternative, but prompting alone often provides limited steerability, as LLMs may overemphasize or overlook certain preferences and fail to give users reliable control over the relative importance of different objectives when conflicts arise, leading to suboptimal alignment. In this paper, we introduce MATO, a training-free framework for Multi-objective personalized Alignment with Test-time Optimization. MATO formulates personalization as a test-time optimization problem that steers the relative importance of multiple objectives through controllable weights during decoding, without modifying model parameters or requiring external reward models. Specifically, a reward discovery module recovers preference rewards directly from the backbone LLM for diverse objectives specified in natural language, while a weight optimization module dynamically adjusts objective weights based on the user's initial preferences and the partially generated response to balance competing objectives during generation. The resulting rewards and weights jointly guide an online optimization procedure over the token distribution, enabling better alignment with the target objectives. Extensive experiments across multiple datasets and backbone LLMs show that MATO consistently outperforms strong baselines, achieving Pareto-improving multi-objective alignment and stronger steerability. These results highlight test-time optimization as a promising direction for scalable, controllable, and model-agnostic personalized alignment.

Linhao Luo, Thuy-Trang Vu, Van-Anh Nguyen +3

Monash University · Defence Science and Technology Group, Australia

LLM Alignment Inference-Time Optimization Test-Time Optimization Personalized Language Model Alignment Multi-Objective Language Model Alignment Preference Alignment

Ask PDF

Oct 4, 2026

Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution

Aligning large language models (LLMs) with human values is important for safe, efficient, and beneficial AI deployment. However, human values are multifaceted: helpfulness, harmlessness and humor trade off against one another, and different users want different trade-offs. Multi-objective alignment (MOA) addresses this by training a policy that can provide any point of the Pareto front, but existing methods either train one model per preference, interpolate a few separately aligned experts post hoc, or train a single conditioned model without considering which preferences it should be trained on. Since the hard regions of the preference simplex depend on the objectives at hand, existing methods leave them under-trained and do not get the most out of a single model. Therefore, we propose MAESTRO (Multi-objective Alignment via End-to-end STeering and Robust Optimization), which formulates MOA as a minimax problem over preference distributions and trains a single prompt-conditioned policy end-to-end with RL against an adversarial preference distribution: a Dirichlet distribution updated by online mirror descent toward the preferences the current policy serves worst, rather than on a fixed one. On HH-RLHF, BeaverTails and a summarization task, with up to three objectives, MAESTRO attains the best Pareto front on most tasks in a single training run, at the lowest training cost among the compared methods. The largest margins appear in the hard regions that a fixed preference distribution leaves under-trained, confirming that a single prompt-conditioned model is capable of covering the objective trade-offs on its own.

Minjae Lee, Kyunghyun Cho, Sangdon Park

GSAI POSTECH · Center for Data Science New York University · GSAI & CSE POSTECH

LLM Alignment Multi-Objective Language Model Alignment Preference Optimization Multi-Objective Reinforcement Learning RL for Language Models

Ask PDF

May 15, 2025

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment

Reward models (RMs) are central to aligning large language models (LLMs) with human values, yet noisy preference feedback can induce reward misgeneralization through spurious correlations and overfitting. We characterize the training dynamics of preference pairs and uncover a distinctive optimization signature of noise: persistently high losses, sharp temporal fluctuations, and irregular gradient updates. Our analyses further show that excluding such pairs improves downstream alignment. In practice, however, reliable preferences are not known a priori, and self-filtering based on a model's own training signals can amplify confirmation bias. To address this challenge, we introduce Collaborative Reward Modeling (CRM), an online framework for identifying reliable preferences. CRM couples cross-model peer review with curriculum learning: two RMs evaluate each other's sample selections, while curriculum keeps their learning progress synchronized. Across extensive experiments, CRM improves RM generalization by up to 9.94 points on RewardBench under 40% preference noise. CRM further generalizes to implicit-reward alignment, offering a robust and broadly applicable approach to learning from imperfect feedback.

Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang +12

NLP Group, Fudan University · Hong Kong Baptist University · LLM Team, Meituan Inc

Reward Modeling LLM Alignment Preference Learning Preference-Based RL

Ask PDF

Abstract

Multi-Objective Language Model Alignment

Explore similar work

MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models

Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment