Sample-efficient Personalization of Large Language Model Alignment with Multidimensional User Preference Data
Organizations: Center for Advanced Intelligence Project, RIKEN, Japan · Université de Montréal, Canada · Mila - Quebec AI Institute, Canada · Center for Advanced Intelligence Project, RIKEN ·
Abstract
Preference alignment has become a central tool to align large language models (LLMs) with human values and desired behavior, and its online variant can further advance their capabilities beyond what static data alone offer by repeatedly generating and labeling new responses with human feedback. But costly, time-consuming annotations limit its scalability, making sample efficiency essential, and existing methods typically seek to improve it by expanding coverage of the response space or selecting pairs with the highest uncertainty for comparison. Although they are effective, these methods quantify uncertainty from the perspective of the LLM, which may differ from the uncertainty perceived by the user: a pair can appear highly uncertain to the LLM even when past comparisons from the user already imply a clear ordering. In this paper, we propose multidimensional preference-guided online direct preference optimization (MPG-DPO), which leverages low-cost multidimensional evaluations from automated verifiers on the user historical comparisons to construct a personalized preference cone guiding response selection for online preference alignment. Specifically, it first discards responses that are confidently dominated under the estimated user preference, then selects two responses that are competitive under the retained preference cone yet exhibit the strongest unresolved disagreement. Theoretical analysis shows that, with high probability, the fraction of queries spent on cone-dominated responses vanishes as learning progresses. Experiments across multiple LLM families show that MPG-DPO outperforms strong contenders under matched query budgets.
Figures & tables
| LLM | Method | IMDb | TL;DR | AlpacaEval2 | Arena-Hard |
|---|---|---|---|---|---|
| Average Reward | Win Rate (%) | LC Win Rate (%) | Win Rate (%) | ||
| Qwen2.5-7B | Online DPO | 81.6 0.55 | 63.38 0.24 | 59.03 0.18 | 54.68 0.17 |
| XPO | 81.8 0.66 | 63.82 0.27 | 59.14 0.20 | 54.74 0.24 | |
| APL | 82.1 0.61 | 64.02 0.26 | 59.13 0.22 | 54.94 0.21 | |
| APL+MD | 82.3 0.63 | 64.34 0.26 | 59.41 0.27 | 55.08 0.23 | |
| MPG-DPO | 83.2 0.64 | 65.31 0.29 | 60.71 0.22 | 56.04 0.23 |
| Method Variant | IMDb | TL;DR | AlpacaEval2 | Arena-Hard |
|---|---|---|---|---|
| Average Reward | Win Rate (%) | LC Win Rate (%) | Win Rate (%) | |
| Without cone filtering | 81.9 0.62 | 64.04 0.25 | 58.88 0.23 | 54.74 0.22 |
| Without trade-off selection | 82.1 0.54 | 64.12 0.24 | 59.02 0.19 | 54.82 0.26 |
| Fixed Pareto cone | 82.2 0.59 | 64.43 0.26 | 59.23 0.31 | 55.02 0.28 |
| Full method | 83.2 0.64 | 65.31 0.29 | 60.71 0.22 | 56.04 0.23 |
| LLM | Method | IMDb | TL;DR | AlpacaEval2 | Arena-Hard |
|---|---|---|---|---|---|
| Average Reward | Win Rate (%) | LC Win Rate (%) | Win Rate (%) | ||
| Qwen2.5-7B | Scalar-Margin | 81.7 0.53 | 63.48 0.27 | 58.67 0.24 | 54.74 0.21 |
| APL+MD-Score | 81.9 0.57 | 63.87 0.23 | 59.02 0.26 | 54.82 0.26 | |
| APL+MD-Scalar | 82.1 0.59 | 64.11 0.23 | 59.21 0.23 | 54.98 0.20 | |
| MPG-DPO | 83.2 0.64 | 65.31 0.29 | 60.71 0.22 | 56.04 0.23 | |
| Mistral-7B | Scalar-Margin | 77.6 0.57 | 61.83 0.22 | 56.79 0.17 | 53.14 0.22 |
| Method | Kimi-K2.6 | DeepSeek-V4.1-Flash | Qwen3.8-27B | |||
|---|---|---|---|---|---|---|
| AlpacaEval2 | Arena-Hard | AlpacaEval2 | Arena-Hard | AlpacaEval2 | Arena-Hard | |
| Online DPO | 59.02 0.23 | 54.34 0.34 | 56.45 0.27 | 53.73 0.39 | 57.73 0.28 | 52.98 0.27 |
| XPO | 59.57 0.38 | 54.67 0.41 | 56.94 0.34 | 54.10 0.38 | 57.94 0.26 | 53.23 0.34 |
| APL | 59.08 0.28 | 54.45 0.25 | 56.75 0.23 | 53.85 0.33 | 57.93 0.32 | 53.44 0.31 |
| MPG-DPO | 60.45 0.33 | 55.98 0.36 | 57.04 0.40 | 54.31 0.38 | 59.26 0.29 | 54.33 0.33 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value | Hyperparameter | Value |
|---|---|---|---|
| LoRA rank ( ) | 32 | Epochs (query budget) | 3 |
| LoRA ( ) | 64 | Global Batch Size | 64 |
| Learning Rate | 5 10 -5 | Warm-up Ratio | 0.05 |
| Optimizer | AdamW | Max Sequence Length | 256 |
| Precision | bf16 | Gradient Clipping | 1.0 |
| LLM | Method | IMDb | TL;DR | AlpacaEval2 | Arena-Hard |
|---|---|---|---|---|---|
| Average Reward | Win Rate (%) | LC Win Rate (%) | Win Rate (%) | ||
| Qwen2.5-7B | Online DPO | 81.6 0.55 | 63.38 0.24 | 59.03 0.18 | 54.68 0.17 |
| XPO | 81.8 0.66 | 63.82 0.27 | 59.14 0.20 | 54.74 0.24 | |
| APL | 82.1 0.61 | 64.02 0.26 | 59.13 0.22 | 54.94 0.21 | |
| APL+MD | 82.8 0.66 | 64.37 0.29 | 59.51 0.27 | 55.21 0.26 | |
| MPG-DPO | 85.1 0.63 | 66.26 0.26 | 61.73 0.25 | 58.13 0.25 |