cs.LGNov 17, 2026

Sample-efficient Personalization of Large Language Model Alignment with Multidimensional User Preference Data

Authors: Zhen-Yu Zhang, Jerry Huang, Gang Niu, Takashi Ishida, Masashi Sugiyama

Organizations: Center for Advanced Intelligence Project, RIKEN, Japan · Université de Montréal, Canada · Mila - Quebec AI Institute, Canada · Center for Advanced Intelligence Project, RIKEN · 

Abstract

Preference alignment has become a central tool to align large language models (LLMs) with human values and desired behavior, and its online variant can further advance their capabilities beyond what static data alone offer by repeatedly generating and labeling new responses with human feedback. But costly, time-consuming annotations limit its scalability, making sample efficiency essential, and existing methods typically seek to improve it by expanding coverage of the response space or selecting pairs with the highest uncertainty for comparison. Although they are effective, these methods quantify uncertainty from the perspective of the LLM, which may differ from the uncertainty perceived by the user: a pair can appear highly uncertain to the LLM even when past comparisons from the user already imply a clear ordering. In this paper, we propose multidimensional preference-guided online direct preference optimization (MPG-DPO), which leverages low-cost multidimensional evaluations from automated verifiers on the user historical comparisons to construct a personalized preference cone guiding response selection for online preference alignment. Specifically, it first discards responses that are confidently dominated under the estimated user preference, then selects two responses that are competitive under the retained preference cone yet exhibit the strongest unresolved disagreement. Theoretical analysis shows that, with high probability, the fraction of queries spent on cone-dominated responses vanishes as learning progresses. Experiments across multiple LLM families show that MPG-DPO outperforms strong contenders under matched query budgets.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. A Zeroth-Order Paradigm for LLM Preference Alignment

    Sep 16, 2026Peter Chen, Xi Chen, Wotao Yin +1Pairwise Preference LearningLLM Alignment

  2. Online LLM Personalization from Pairwise Preferences Across Users

    Oct 12, 2026Romain Puech, Georgy Salakhutdinov, Dimitris BertsimasEducational QALLM Personalization

  3. MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge

    Jul 27, 2025Guangchen Lan, Sipeng Zhang, Tianle Wang +7Preference OptimizationPreference Learning for LLMs