Preference Learning

Follow this topic Ask about this topic

Preference learning studies how to infer what people value from pairwise comparisons, rankings, choices, behavior, and other feedback. Common approaches include Bradley–Terry and Gaussian-process models, active query selection, inverse reinforcement learning, and reward-model training with methods such as direct preference optimization. These techniques support personalized recommendations, robot and agent behavior, and multi-objective decisions where no single fixed reward captures every user’s priorities. A central challenge is that preferences can be noisy, context-dependent, and heterogeneous, so researchers must balance efficient data collection and accurate prediction against robust generalization and faithful representation of differing values.

Latest papers

149 papers19 papers in the last four weeks, up 36% on the four weeks before. 0.1% of all new papers.

Beauty in the Eye of AI: Aligning LLMs and Vision Models with Human Aesthetics in Network Visualization
Apr 3, 2026Peng Zhang, Xuefeng Li, Xiaoqi Wang +2Data Visualization LLM-as-a-Judge
Northeastern University, United States · Bosch AI Research, United States · The Ohio State University, United States
Random Is Hard to Beat: Active Selection in online DPO with Modern LLMs
Apr 3, 2026Giyeong Oh, Junghyun Lee, Jaehyun Park +3Active Preference Learning Language Model Post-Training
Seoul National University · Ewha Womans University · KAIST +1
Reinforcement Learning from Human Feedback: A Statistical Perspective
Apr 2, 2026Pangpang Liu, Chengchun Shi, Will Wei SunRL from Human Feedback LLM Alignment
Department of Biostatistics, Yale University. · Department of Statistics, London School of Economics and Political Science. · Mitch Daniels School of Business, Purdue University.
PLOT: Enhancing Preference Learning via Optimal Transport
Apr 2, 2026Liang Zhu, Yuelin Bai, Xiankun Ren +6Preference Learning LLM Alignment
Southern University of Science and Technology · Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · Chongqing University +2
DEFT: Distribution-guided Efficient Fine-Tuning for Human Alignment
Apr 2, 2026Liang Zhu, Feiteng Fang, Yuelin Bai +4LLM Alignment LLM Fine-Tuning
Southern University of Science and Technology · Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · University of Science and Technology of China +1
Active Preference Learning over Latent Preference Archetypes for Many-Objective Bayesian Optimization
Mar 30, 2026Manisha Dubey, Sebastiaan De Peuter, Wanrong Wang +1Active Preference Learning Bayesian Optimization
Department of Computer Science, University of Manchester, UK · Centre for AI in Assistive Autonomy, University of Edinburgh, Scotland, UK · Department of Computer Science, Aalto University, Espoo, Finland +2
Learning What Matters Now: Dynamic Preference Inference under Contextual Shifts
Mar 24, 2026Xianwei Cao, Dou Quan, Zhenliang Zhang +1Preference Learning Multi-Objective Reinforcement Learning
School of Artificial Intelligence, Xidian University, Xi’an, 710071, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China
PreferRec: Learning and Transferring Pareto Preferences for Multi-objective Re-ranking
Mar 23, 2026Wei Zhou, Wuyang Li, Junkai Ji +5Recommender Systems Preference Learning
Shenzhen University, Shenzhen, China · Shenzhen Technology University, Shenzhen, China
User Preference Modeling for Conversational LLM Agents: Weak Rewards from Retrieval-Augmented Interaction
Mar 21, 2026Yuren Hao, Shuhaib Mehri, ChengXiang Zhai +1User Preference Modeling Preference Learning
University of Illinois at Urbana-Champaign
Binary Classification from Coupled Pairwise Labels
Mar 20, 2026Tomoya Tate, Kosuke Sugiyama, Masato UchidaPairwise Comparison Binary Classification
Waseda University, Tokyo, Japan
MALLES: A Multi-agent LLMs-based Economic Sandbox with Consumer Preference Alignment
Mar 18, 2026Yusen Wu, Yiran Liu, Xiaotie DengMulti-Agent LLM Systems Human Behavior Simulation
Contextual Preference Distribution Learning
Mar 17, 2026Benjamin Hudson, Laurent Charlin, Emma FrejingerPreference Learning Decision-Making under Uncertainty
Mila – Quebec Artificial Intelligence Institute, Montreal, Quebec, Canada · Département d’informatique et de recherche opérationnelle (DIRO), Université de Montréal, Montreal, Quebec, Canada · HEC Montréal, Montreal, Quebec, Canada
Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedback
Mar 13, 2026Gihoon Kim, Euntai KimPreference Learning RL from Human Feedback
Yonsei University · Korea Institute of Science and Technology
Improving through Interaction: Searching Behavioral Representation Spaces with CMA-ES-IG
Mar 9, 2026Nathaniel Dennler, Zhonghao Shi, Yiran Tao +3Active Preference Learning Preference Learning
CSAIL/AeroAstro, Massachusetts Institute of Technology. · Department of Computer Science, University of Southern California.
CRED: Counterfactual Reasoning and Environment Design for Active Preference Learning
Mar 9, 2026Yi-Shiuan Tung, Gyanig Kumar, Wei Jiang +2Active Preference Learning Preference Learning
Department of Computer Science, University of Colorado Boulder
VisualDeltas: Learning Preferences from Visual Quality Perturbations
Mar 7, 2026Hailiang Huang, Yihao Liu, Shengyue Guan +2Preference Learning Multimodal Learning
Peking University, Beijing, China · Alibaba Group, Hangzhou, China · Zhejiang University, Hangzhou, China
Density-Guided Response Optimization: Community-Grounded Alignment via Implicit Acceptance Signals
Mar 3, 2026Patrick Gerard, Svitlana VolkovaPreference Alignment Preference Learning
Information Sciences Institute, University of Southern California, Marina Del Rey, California, USA · Aptima Inc., Dayton, Ohio, USA
Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferences
Feb 26, 2026Junghyun Lee, Minju Hong, Kwang-Sung Jun +2Regret Minimization in RL Preference-Based RL
KAIST AI · KAIST EE · POSTECH CSE/AI
What Is Missing: Interpretable Ratings for Large Language Model Outputs
Feb 17, 2026Nicholas Stranges, Yimin YangLLM Evaluation Language Model Post-Training
Department of Electrical and Computer Engineering, Western University, Canada
Next-token Preference: Scaling Unsupervised Reward Modeling
Feb 11, 2026Jingxuan Fan, Yueying Li, Zhenting Qi +4Reward Modeling Preference Learning
Harvard University · Cornell University · Microsoft Research +1
Towards Optimality in Active Preference Learning
Feb 2, 2026Yao Zhao, Kwang-Sung JunActive Preference Learning Active Learning
Department of Computer Science, University of Arizona, Tucson, USA.
Co-Evolving Agents: Learning from Failures as Hard Negatives
Nov 27, 2025Yeonsung Jung, Trilok Padhi, Sina Shaham +4Hard Negative Mining Self-Improving Agents
KAIST · Georgia State University · Independent Researcher +2
Adaptive Margin RLHF via Preference over Preferences
Sep 26, 2025Yaswanth Chittepu, Prasann Singhal, Greg Durrett +1Pairwise Preference Learning Reward Modeling
University of Massachusetts Amherst · University of Texas at Austin
Interactive Multi-Objective Probabilistic Preference Learning with Soft and Hard Bounds
Jun 27, 2025Edward Chen, Sang T. Truong, Natalie Dullerud +2Active Learning Multi-Criteria Decision Analysis
Department of Computer Science, Stanford University, Stanford, California, USA
MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning
May 30, 2025Jingyan Shen, Jiarui Yao, Rui Yang +5Pairwise Preference Learning Reward Modeling
New York University · University of Illinois Urbana-Champaign · Rice University
Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
May 15, 2025Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang +12Reward Modeling LLM Alignment
NLP Group, Fudan University · Hong Kong Baptist University · LLM Team, Meituan Inc
Dual-Difficulty Curriculum Learning for Direct Preference Optimization
Apr 10, 2025Mengyang Li, Haozhan Geng, Zhong Zhang +1Pairwise Preference Learning LLM Alignment
Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission Tianjin Normal University Tianjin, China
Inferring the Unspoken: Aligning Embodied Agents with Implicit Preferences
Feb 2, 2025Manjie Xu, Xinyi Yang, Wei Liang +2User Preference Modeling Preference Learning
Institute for Artificial Intelligence, Peking University · School of Psychological and Cognitive Sciences, Peking University · School of Intelligence Science and Technology, Peking University +5
A Probabilistic Approach for Model Alignment with Human Comparisons
Mar 16, 2024Junyu Cao, Mohsen BayatiHuman Preference Modeling Statistical Learning Theory
McCombs School of Business, The University of Texas at Austin · Graduate School of Business, Stanford University

Apr 3, 2026J/K move · Enter open · S save

Beauty in the Eye of AI: Aligning LLMs and Vision Models with Human Aesthetics in Network Visualization

Peng Zhang, Xuefeng Li, Xiaoqi Wang, Han-Wei Shen+1

Northeastern University, United States · Bosch AI Research, United States · The Ohio State University, United States

Data Visualization LLM-as-a-Judge Preference Learning

Open paper AskPDF arXiv

Network visualization has traditionally relied on heuristic metrics, such as stress, under the assumption that optimizing them leads to aesthetic and informative layouts. However, no single metric consistently produces the most effective results. A data-driven alternative is to learn from human preferences, where annotators select their favored visualization among multiple layouts of the same graphs. These human-preference labels can then be used to train a generative model that approximates human aesthetic preferences. However, obtaining human labels at scale is costly and time-consuming. As a result, this generative approach has so far been tested only with machine-labeled data. In this paper, we explore the use of large language models (LLMs) and vision models (VMs) as proxies for human judgment. Through a carefully designed user study involving 27 participants, we curated a large set of human preference labels. We used this data both to better understand human preferences and to bootstrap LLM/VM labelers. We show that prompt engineering that combines few-shot examples and diverse input formats, such as image embeddings, significantly improves LLM-human alignment, and additional filtering by the confidence score of the LLM pushes the alignment to human-human levels. Furthermore, we demonstrate that carefully trained VMs can achieve VM-human alignment at a level comparable to that between human annotators. Our results suggest that AI can feasibly serve as a scalable proxy for human labelers.