Preference Modeling

Follow this topic Ask about this topic

Latest papers

36 papers10 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Pluralistic Alignment in Language Models: Objectives, Human References, and Preference Representations
Oct 17, 2026Meng-Chen Wu, Umesh BodhwaniPluralistic Alignment Preference Modeling
Amazon Science
On the possibilities and limitations of LLM-based multi-agent systems for multi-criteria decision-making in healthcare operations management
Oct 16, 2026Marco Boresta, Tommaso Giovannelli, Sepehr Namirad +1Preference Modeling Multi-Criteria Decision Analysis
Istituto di Analisi dei Sistemi ed Informatica (IASI), CNR, Rome, Italy · Department of Mechanical, Materials, and Industrial Engineering, University of Cincinnati, Cincinnati, OH · Department of Computer, Control, and Management Engineering (DIAG), Sapienza University of Rome, Rome, Italy
Agentic-Priors: Using Large Language Models to Address the Cold Start Problem in Preference Elicitation
Oct 15, 2026Bruno Arcanjo, Zhaoxing Li, Sebastian SteinPreference Elicitation User Preference Modeling
University of Southampton, Southampton, United Kingdom
Online LLM Personalization from Pairwise Preferences Across Users
Oct 12, 2026Romain Puech, Georgy Salakhutdinov, Dimitris BertsimasLLM Personalization User Preference Modeling
Massachusetts Institute of Technology
Covariate-dependent Joint Modeling of Multivariate Ordinal Preferences and Its Connections with Comparison Models
Oct 6, 2026Yujie Chen, Antik Chakraborty, Anindya BhadraPreference Modeling Preference Learning
Department of Statistics Purdue University
On the Complexity of Preference-Based Bandits
Sep 30, 2026Ahmed Ben Yahmed, Marc Abeille, Clément CalauzènesPairwise Preference Learning Multi-Armed Bandits
CREST, ENSAE Paris, FAIRPLAY · FAIRPLAY
Training LLM Judges from Language Feedback via Position-Selective Self-Distillation
Sep 30, 2026Ilgee Hong, Changlong Yu, Zhenghao Xu +5LLM-as-a-Judge Preference Modeling
Georgia Institute of Technology · Amazon · UC San Diego
JudgeProfile: Understanding and Steering Subjectivity in LLM Judges
Sep 29, 2026Qi Cao, Kangning Liu, Xuan Kan +10Human Preference Evaluation LLM Evaluation
Meta · University of California San Diego · The University of Hong Kong +1
PriceBench: A Diagnostic Benchmark for Price, Quality, and Brand Preferences in LLM Booking Agents
Sep 25, 2026Pavel KireyevLLM Agents LLM Agent Evaluation
Department of Management London School of Economics and Political Science
ZooWork-ShopRanker: An Open, Preference-Aligned E-Commerce Reranker
Sep 25, 2026Siqiao Xue, Shuxuan Liu, Ning HuE-Commerce Search LLM Reranking
ZooWork Team
Learning Heterogeneous Preferences
Sep 15, 2026Shiwali Mohan, Matt Hong, Dule Shu +3Pairwise Preference Learning Human Preference Modeling
Human-Centered AI Toyota Research Institute Los Altos, CA
VIBE-Bench: Evaluating Personalized Large Language Models When Profiles Don't Mean Preferences
Sep 1, 2026Yiwen Jiang, Yang Deng, Stephanie Fong +9LLM Evaluation LLM Personalization
1Monash University · 2Singapore Management University · University of Liverpool +1
AcCoRD: Evaluating User-Agent Collaboration Under Realistic User Preference Dynamics
Aug 28, 2026Tejas Srinivasan, Shikib Mehri, Nandita Shankar Naik +3LLM Agent Evaluation Human-AI Collaboration
University of Southern California · Contextual AI · Capital One +1
Multiple latent orderings better predict language model preferences
Aug 26, 2026Aviral Chawla, William H. W. Thompson, Jean-Gabriel YoungPreference Modeling Pairwise Preference Learning
Vermont Complex Systems Institute University of Vermont Burlington, Vermont 05403 · Vermont Complex Systems Institute Department of Statistics University of Vermont Burlington, Vermont 05403
MoPLEx: Estimating Plackett-Luce Mixture Models for Multi-Objective Alignment
Aug 25, 2026Dongyue Li, Ziniu Zhang, Lu Wang +1Learning to Rank Discrete Choice Modeling
Northeastern University, Boston, MA · University of Michigan, Ann Arbor, MI
LLM-Derived Preference Judgments Are Not Self-Consistent
Aug 18, 2026Matthew T. Ford, Francis Bahk, Jingjing Wang +4Preference Modeling User Preference Modeling
Cornell University · Georgia Institute of Technology
What do Reward Models Memorize?
Jul 27, 2026Ivo Verhoeven, Pushkar Mishra, Ekaterina ShutovaReward Modeling Pairwise Preference Evaluation
ILLC, University of Amsterdam, The Netherlands · Google DeepMind, London, United Kingdom
Beyond expert users: agents should help users construct preferences, not just elicit them
Jun 29, 2026Irena Saracay, Ludwig Schmidt, Carlos GuestrinAI Agent Benchmarks Recommender Systems
Stanford University
Steerable Cultural Preference Optimization of Reward Models
Jun 17, 2026Minsik Oh, Advit Deepak, Sophie Wu +2Reward Modeling Preference Modeling
1Stanford University · University of Amsterdam
DynaCF: Mitigating Shortcut Learning in Reward Models via Dynamic Counterfactual Sensitivity
Jun 8, 2026Fengyuan Liu, Yongliang Miao, Zirui He +3Pairwise Preference Learning Reward Modeling
The Chinese University of Hong Kong, Shenzhen · New Jersey Institute of Technology · Institute of Computing Technology, CAS
Provably Efficient Personalized Multi-Objective Bandits with Proactive Conversational Queries
Jun 7, 2026Linfeng Cao, Ming Shi, Ness B. ShroffMulti-Armed Bandits Combinatorial Multi-Armed Bandits
Department of Computer Science and Engineering, The Ohio State University · Department of Electrical Engineering, University at Buffalo · Department of Electrical and Computer Engineering, The Ohio State University
Support Vector Rubrics: Closing the Gap Between Self-Generated and Human Rubrics
Jun 6, 2026Mengyuan Sun, Yu Li, Zhuohao Yu +2Reward Modeling Preference Modeling
National Engineering Research Center for Software Engineering, Peking University · University of Science and Technology of China
Reward Learning from Best-of- $N$ Preference Data: Targets, Tradeoffs, and Design Principles
May 28, 2026Rattana Pukdee, Maria-Florina Balcan, Pradeep RavikumarPairwise Preference Learning Reward Modeling
Machine Learning Department Carnegie Mellon University
General Preference Reinforcement Learning
May 18, 2026Muhammad Umer, Muhammad Ahmed Mohsin, Ahsan Bilal +5Preference Modeling Multi-Objective Language Model Alignment
Stanford University · The University of Oklahoma
Learning What Evaluators Value: A Reliable Approach to Modeling Evaluator Preferences
May 15, 2026Madeline Celi Kitch, Nihar B. ShahHuman Preference Modeling Preference Learning
Carnegie Mellon University
Embeddings for Preferences, Not Semantics
May 8, 2026Carter Blair, Ariel D. Procaccia, Milind TambeText Embeddings Word Embeddings
Harvard University
RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences
May 3, 2026Yangyang Zhou, Yi-Chen LiReward Modeling LLM Evaluation
Beijing University of Posts and Telecommunications · National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China
HorizonBench: Long-Horizon Personalization with Evolving Preferences
Apr 19, 2026Shuyue Stella Li, Bhargavi Paranjape, Kerem Oktar +9Long-Context Modeling User Preference Modeling
University of Washington · Meta · OpenAI +1
ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework
Apr 8, 2026Kai Qin, Liangxin Liu, Yu Liang +7Preference Modeling Self-Reflection in Language Model Agents
Shenzhen International Graduate School, Tsinghua University · Baidu Inc., Beijing, China
CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling
Mar 9, 2026Dengcan Liu, Fengkai Yang, Xiaohan Wang +7Preference Modeling Reward Modeling
University of Science and Technology of China · Peking University · Meituan +1

Oct 17, 2026J/K move · Enter open · S save

Pluralistic Alignment in Language Models: Objectives, Human References, and Preference Representations

Meng-Chen Wu, Umesh Bodhwani

Amazon Science

Pluralistic Alignment Preference Modeling

Open paper AskPDF arXiv

Pluralistic alignment aims to represent meaningful variation in human preferences rather than collapse heterogeneous judgments into a single target. Yet existing work often mixes three distinct questions: what pluralistic behavior a model should exhibit, whose preferences define the reference, and what evidence is sufficient to demonstrate success. We organize the literature around a two-layer taxonomy. Layer 1 distinguishes representing a range of human-supported viewpoints (Overton Pluralism), adapting faithfully to a specified perspective or preference condition (Steerable Pluralism), and matching the response distribution observed in a target population (Distributional Pluralism). Layer 2 distinguishes aggregate populations, countries and regions, and subgroups as human references. Preference representation, implementation mechanism, modality, and evidence quality are treated as cross-cutting design choices rather than additional forms of pluralism. We synthesize a protocol-bounded corpus of 70 works on pluralistic alignment and neighboring areas, using the objective-by-human-reference matrix to map existing approaches and identify underexplored combinations. The coded literature concentrates on Steerable Pluralism and Distributional Pluralism, while eligible Overton Pluralism work appears only at the aggregate-population level under our stated protocol. We then connect each objective to the evidence needed to support it, including perspective validity, conditional fidelity, distributional fidelity, support preservation, and uncertainty in subgroup estimates. Together, the taxonomy and evaluation framework clarify what current methods establish, where their claims exceed their evidence, and how future systems can represent disagreement without mistaking variation for faithful representation.