Preference Learning

Follow this topic Ask about this topic

Preference learning studies how to infer what people value from pairwise comparisons, rankings, choices, behavior, and other feedback. Common approaches include Bradley–Terry and Gaussian-process models, active query selection, inverse reinforcement learning, and reward-model training with methods such as direct preference optimization. These techniques support personalized recommendations, robot and agent behavior, and multi-objective decisions where no single fixed reward captures every user’s priorities. A central challenge is that preferences can be noisy, context-dependent, and heterogeneous, so researchers must balance efficient data collection and accurate prediction against robust generalization and faithful representation of differing values.

Latest papers

149 papers19 papers in the last four weeks, up 36% on the four weeks before. 0.1% of all new papers.

S $^2$ COPE: Self-Supervised Concept Discovery via Preference Learning
Jun 12, 2026Shilong Xiang, Zirui Zhang, Chengzhi MaoVision-Language Models Self-Supervised Learning
Rutgers University
PAWS: Preference Learning with Advantage-Weighted Segments
Jun 10, 2026Aleksandar Taranovic, Onur Celik, Niklas Freymuth +6Credit Assignment in RL Robotic RL
Autonomous Learning Robots, Karlsruhe Institute of Technology, Karlsruhe, Germany · Bosch Center for Artificial Intelligence, Renningen, Germany · Institute for Material Handling and Logistics (IFL), Karlsruhe Institute of Technology, Karlsruhe, Germany
Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Short
Jun 8, 2026Han Zhou, Adam X. Yang, Laurence Aitchison +2Pairwise Preference Learning Reward Modeling
University of Cambridge · 2Mistral AI · *Work done as an AI Scientist Intern at Mistral AI.
DynaCF: Mitigating Shortcut Learning in Reward Models via Dynamic Counterfactual Sensitivity
Jun 8, 2026Fengyuan Liu, Yongliang Miao, Zirui He +3Pairwise Preference Learning Reward Modeling
The Chinese University of Hong Kong, Shenzhen · New Jersey Institute of Technology · Institute of Computing Technology, CAS
Provably Efficient Personalized Multi-Objective Bandits with Proactive Conversational Queries
Jun 7, 2026Linfeng Cao, Ming Shi, Ness B. ShroffMulti-Armed Bandits Combinatorial Multi-Armed Bandits
Department of Computer Science and Engineering, The Ohio State University · Department of Electrical Engineering, University at Buffalo · Department of Electrical and Computer Engineering, The Ohio State University
Statistical Priors for Implicit Preferences: Decoupling Skill Selection as a Local Harness in Personal Agents
Jun 4, 2026Zeyu Gan, Huayi Tang, Yong LiuLLM Agents User Preference Modeling
Gaoling School of Artificial Intelligence Renmin University of China Beijing, China
Rethinking Sales Lead Scoring with LLM-based Hierarchical Preference Ranking
Jun 3, 2026Chenyu Zhang, Yiwen Liu, Yin Sun +4Learning to Rank LLM Reranking
Intelligent Business Team, Li Auto Inc.
Enhancing LLM Metacognition via Cognitive Pairwise Training
May 30, 2026Weitao Li, Hao Zhou, Xuanyu Lei +11Pairwise Preference Learning RL for Language Model Reasoning
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China · WeChat AI, Tencent Inc., China
Skill or Skip? Learning Selective Skill Invocation in Agentic Tasks via Dual-Granularity Preference Learning
May 30, 2026Chishui Chen, Jiaye Lin, Te Sun +6LLM Agent Skill Learning Tool-Using Agents
Meituan · Shanghai Jiao Tong University · Nanjing University +2
The Representation-Rationalizability Tradeoff in Reward Learning
May 29, 2026Jing Dong, Yaoliang Yu, Pascal PourpartPairwise Preference Learning Reward Modeling
Vector Institute · University of Waterloo & Vector Institute
Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferences
May 29, 2026Jabin Koo, Hoyoung Kim, Minwoo Jang +1LLM Alignment Preference Alignment
Department of CSE, POSTECH, Pohang, Republic of Korea · National AI Research Lab, Seoul, Republic of Korea · Graduate School of AI, POSTECH, Pohang, Republic of Korea
Reward Learning from Best-of- $N$ Preference Data: Targets, Tradeoffs, and Design Principles
May 28, 2026Rattana Pukdee, Maria-Florina Balcan, Pradeep RavikumarPairwise Preference Learning Reward Modeling
Machine Learning Department Carnegie Mellon University
Calibrated Preference Learning: The Case of Label Ranking
May 28, 2026Santo M. A. R. Thies, Viktor Bengs, Timo Kaufmann +2Probability Calibration Learning to Rank
Data Science and its Applications, Deutsches Forschungsinstitut für Künstliche Intelligenz (DFKI), Kaiserslautern, Germany · Munich Center for Machine Learning, Munich (MCML), Germany · Ludwig Maximilian Universität München, Munich, Germany
From Blind Guess to Informed Judgment: Teaching LLMs to Evaluate Materials by Building Knowledge-Augmented Preference Signals
May 28, 2026Yeyong Yu, Wenya Hu, Xing Wu +1LLM Evaluation Pairwise Preference Evaluation
School of Computer Engineering & Science, Shanghai University, Shanghai, 200444, China. · Center of Materials Informatics and Data Science, Materials Genome2026 Institute, Shanghai University, Shanghai, 200444, China. · Key Laboratory of Silicate Cultural Relics Conservation (Shanghai University), Ministry of Education, China. +1
Evolving to the Aesthetics of a Vision-Language Model
May 27, 2026Stephen James Krol, Jon McCormackVLM Evaluation Preference Learning
SensiLab, Monash University Melbourne, Australia
Optimising Factual Consistency in Summarisation via Preference Learning from Multiple Imperfect Metrics
May 26, 2026Yuxuan Ye, Raul Santos-Rodriguez, Edwin SimpsonFactual Consistency Evaluation Text Summarization
Intelligent System Laboratory University of Bristol United Kingdom
CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations
May 25, 2026Mike Zhang, Ali Basirat, Desmond ElliottMultilingual Language Models LLM Alignment
Department of Computer Science (DIKU), University of Copenhagen · Centre for Language Technology (CST), University of Copenhagen · Pioneer Centre for Artificial Intelligence
Active Query Synthesis for Preference Learning
May 25, 2026Namrata Nadagouda, Nauman Ahad, Maegan Tucker +1Pairwise Preference Learning Active Learning
Georgia Institute of Technology · Gauss Labs
When In-Distribution Gains Fail: Evaluating Weak-to-Strong Reward Models under Preference Shift
May 25, 2026Khoi Le, Tri Cao, Phong Nguyen +5Pairwise Preference Learning Reward Modeling
National University of Singapore · VinUniversity · Nanyang Technological University
Operationalizing Individual Fairness via Gradient Descent and Bradley-Terry Models
May 22, 2026Conlan Olson, Linjun Zhang, Zhun Deng +1Mahalanobis Distance Algorithmic Fairness
Columbia University · Rutgers University · UNC Chapel Hill +1
Beyond Scalar Objectives: Expert-Feedback-Driven Autonomous Experimentation for Scientific Discovery at the Nanoscale
May 20, 2026Ralph Bulanadi, Jefferey Baxter, Arpan Biswas +5Pairwise Preference Learning Deep Kernel Learning
Center for Nanophase Materials Sciences, Oak Ridge National Laboratory, Oak Ridge, TN 37831, USA · University of Tennessee-Oak Ridge Innovation Institute, University of Tennessee, Knoxville, TN 37996, USA · Department of Material Science and Engineering, School of Materials and Chemical Technology, Institute of Science Tokyo, Yokohama, 226-8502, Japan +3
Preferences Order, Ratings Anchor: From Fused Expert Aesthetic Ground Truth to Self-Distillation
May 19, 2026Yuanpei Zhao, Jie Lin, Chao Zhang +5Pairwise Preference Learning Personalized Image Aesthetics Assessment
1Sichuan University · 2NetEase Fuxi AI Lab · †Work done during an internship at NetEase Fuxi AI Lab. +1
CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging
May 19, 2026Zhenyu Li, Aleksandar Cvejic, Zehui Chen +1Pairwise Preference Learning Human Preference Evaluation
KAUST · USTC
DEFLECT: Temporal Counterfactual Preference Learning for Delay-Robust Asynchronous VLAs
May 19, 2026Yixiang Zhu, Yonghao Chen, Jiaxiang Zou +2Robotic Control Visuomotor Policy Learning
The Hong Kong University of Science and Technology (Guangzhou) · One Robotics
General Preference Reinforcement Learning
May 18, 2026Muhammad Umer, Muhammad Ahmed Mohsin, Ahsan Bilal +5Preference Modeling Multi-Objective Language Model Alignment
Stanford University · The University of Oklahoma
Learning What Evaluators Value: A Reliable Approach to Modeling Evaluator Preferences
May 15, 2026Madeline Celi Kitch, Nihar B. ShahHuman Preference Modeling Preference Learning
Carnegie Mellon University
AI Outperforms Humans in Personalized Image Aesthetics Assessment via LLM-Based Interviews and Semantic Feature Extraction
May 14, 2026Yoshia Abe, Tatsuya Daikoku, Yasuo KuniyoshiPersonalized Image Aesthetics Assessment Image Quality Assessment
Graduate School of Information Science and Technology The University of Tokyo Bunkyo-ku, Tokyo 113-8654, Japan
Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty?
May 12, 2026Yichen Feng, Yuetai Li, Chunjiang Liu +14Multimodal Large Language Models Image Quality Assessment
YFPO: Yoked Feature Preference Optimization with Neuron-Guided Rewards
May 12, 2026Yifan LePreference Optimization Language Model Post-Training
Independent Researcher
Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training
May 11, 2026Christian Moya, Alex Semendinger, Guang Lin +1Pairwise Preference Learning LLM Alignment
Department of Mathematics, Purdue University, West Lafayette IN, USA

Jun 12, 2026J/K move · Enter open · S save

S $^2$ COPE: Self-Supervised Concept Discovery via Preference Learning

Shilong Xiang, Zirui Zhang, Chengzhi Mao

Rutgers University

Vision-Language Models Self-Supervised Learning VLM Interpretability Preference Learning Unsupervised Visual Concept Discovery

Open paper AskPDF arXiv

Current representation learning paradigms force a fundamental compromise: self-supervised methods scale to massive datasets but yield opaque features, whereas interpretable models remain bottlenecked by the need for dense human annotation. We introduce Self-Supervised Concept discOvery via Preference lEarning (\model), a label-free framework that resolves this dilemma. Instead of treating Vision-Large-Language Models (VLLMs) as static feature extractors, \model leverages them as active participants in a self-supervised preference optimization loop. By autonomously hypothesizing, validating, and reinforcing candidate visual attributes directly from raw imagery, our framework discovers novel, structured concepts without a single label. Extensive experiments across natural, medical, and physics domains demonstrate that \model successfully extracts domain-specific concepts where standard VLLMs often fail to generate. By amortizing concept discovery directly into the VLLM backbone through our self-supervised preference objective -- rather than relying on static generation and disjoint filtering -- we achieve up to a 24-point absolute improvement in downstream top-1 classification accuracy on unseen data. Our work suggest that interpretability can emerge through a model's autonomous interaction with incidental visual structures, without any human supervision.