Preference Learning

Follow this topic Ask about this topic

Preference learning studies how to infer what people value from pairwise comparisons, rankings, choices, behavior, and other feedback. Common approaches include Bradley–Terry and Gaussian-process models, active query selection, inverse reinforcement learning, and reward-model training with methods such as direct preference optimization. These techniques support personalized recommendations, robot and agent behavior, and multi-objective decisions where no single fixed reward captures every user’s priorities. A central challenge is that preferences can be noisy, context-dependent, and heterogeneous, so researchers must balance efficient data collection and accurate prediction against robust generalization and faithful representation of differing values.

Latest papers

149 papers19 papers in the last four weeks, up 36% on the four weeks before. 0.1% of all new papers.

NanoResearch: Co-Evolving Skills, Memory, and Policy for Personalized Research Automation
May 11, 2026Jinhang Xu, Qiyuan Zhu, Yujun Wu +11Multi-Agent LLM Systems LLM Agent Skill Learning
1Shanghai Artificial Intelligence Laboratory · 2The Hong Kong University of Science and Technology · 3Peking University +4
Personalizing LLMs with Binary Feedback: A Preference-Corrected Optimization Framework
May 11, 2026Xilai Ma, Liye Zhao, Weijun Yao +3LLM Personalization User Preference Modeling
Harbin Institute of Technology, Shenzhen, China · Huawei Technologies Co., Ltd. · Nanyang Technological University
Learning the Preferences of a Learning Agent
May 9, 2026Karim Abdel Sadek, Mark Bedaywi, Rhys Gould +1Reinforcement Learning Preference Learning
University of California, Berkeley
Sufficient conditions for a Heuristic Rating Estimation Method application
May 9, 2026Jacek Szybowski, Konrad Kułakowski, Jiri MazurekPairwise Comparison Pairwise Preference Evaluation
AGH University of Science and Technology, Faculty of Applied Mathematics, Krakow, Poland · AGH University of Science and Technology, Faculty of Electrical Engineering, Automatics, Computer Science and Biomedical Engineering, Krakow, Poland · Silesian University in Opava, Department of Informatics and Mathematics, School of Business Administration in Karvina, Czech Republic
Embeddings for Preferences, Not Semantics
May 8, 2026Carter Blair, Ariel D. Procaccia, Milind TambeText Embeddings Word Embeddings
Harvard University
Response Time Enhances Alignment with Heterogeneous Preferences
May 7, 2026Federico Echenique, Alireza Fallah, Baihe Huang +1Pairwise Preference Learning LLM Alignment
Department of Economics, University of California, Berkeley · Department of Computer Science and Ken Kennedy Institute, Rice University · Departments of Electrical Engineering and Computer Sciences, University of California, Berkeley +2
SuperFace: Preference-Aligned Facial Expression Estimation Beyond Pseudo Supervision
May 7, 2026Zejian Kang, Xuanyang Xu, Wentao Yang +6Facial Expression Recognition Preference Learning
Zhejiang University · The Chinese University of Hong Kong, Shenzhen · Westlake University +3
PREFER: Personalized Review Summarization with Online Preference Learning
May 7, 2026Millend Roy, Agostino Capponi, Vineet GoyalText Summarization E-Commerce
Department of Industrial Engineering and Operations Research Columbia University New York, NY 10027
Threshold-Guided Optimization for Visual Generative Models
May 6, 2026Jinbin Bai, Yu Lei, Qingyu Shi +6Diffusion Model Alignment Preference Alignment
National University of Singapore · Collov Labs · Peking University +2
QUIVER: Cost-Aware Adaptive Preference Querying in Surrogate-Assisted Evolutionary Multi-Objective Optimization
May 5, 2026Florian A. D. BurnatSurrogate-Assisted Optimization Multi-Objective Evolutionary Optimization
University of Warwick Warwick Business School Coventry, England, UK
Anomaly-Preference Image Generation
May 4, 2026Fuyun Wang, Yuanzhi Wang, Xu Guo +6Diffusion Model Alignment Diffusion Model Sampling
Nanjing University of Science and Technology, Nanjing, China · Beijing Normal University, Beijing, China · China Academy of Space Technology, Beijing, China +1
PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning
May 1, 2026Ziqin Yuan, Ruiqi Wang, Dezhong Zhao +2Reward Modeling Mixture of Experts
1Purdue University, West Lafayette, IN, USA. · 2Beijing University of Chemical Technology, Beijing, China. · 3Indiana University Bloomington, Bloomington, IN, USA.
AesRM: Improving Video Aesthetics with Expert-Level Feedback
Apr 30, 2026Yujin Han, Yujie Wei, Yefei He +7Reward Modeling Video Quality Assessment
The University of Hong Kong · Alibaba Group · Fudan University +3
Learning from Disagreement: Clinician Overrides as Implicit Preference Signals for Clinical AI in Value-Based Care
Apr 30, 2026Prabhjot Singh, Abhishek Gupta, Chris Betz +4Clinical Decision Support Preference Learning
Altitude
Mind the Gap: Structure-Aware Consistency in Preference Learning
Apr 30, 2026Mehryar Mohri, Yutao ZhongPairwise Preference Learning LLM Alignment
Google Research, New York, NY · Courant Institute of Mathematical Sciences, New York, NY
Three Models of RLHF Annotation: Extension, Evidence, and Authority
Apr 28, 2026Steve CoyneLLM Alignment Human-in-the-Loop Annotation
University of Toronto, Canada · University of Toronto, Toronto, Canada
Pref-CTRL: Preference Driven LLM Alignment using Representation Editing
Apr 26, 2026Imranul Ashrafi, Inigo Jauregi Unanue, Massimo PiccardiLLM Alignment Representation Engineering
University of Technology Sydney, Australia
Probably Approximately Consensus: On the Learning Theory of Finding Common Ground
Apr 23, 2026Carter Blair, Ben Armstrong, Shiri Alouf-Heffetz +2User Preference Modeling Preference Learning
University of Waterloo · Tulane University · Ben Gurion University +1
Conjecture and Inquiry: Quantifying Software Performance Requirements via Interactive Retrieval-Augmented Preference Elicitation
Apr 23, 2026Shihai Wang, Tao ChenRequirements Engineering Preference Learning
School of Computer Science and Engineering, UESTC, Chengdu, China · IDEAS Lab, University of Birmingham, Birmingham, UK
Vibrotactile Preference Learning: Uncertainty-Aware Preference Learning for Personalized Vibration Feedback
Apr 22, 2026Rongtao Zhang, Xin Zhu, Masoume Pourebadi Khotbehsara +3Pairwise Preference Learning Active Preference Learning
Department of Computer Science University of Southern California Los Angeles, California, USA
DT2IT-MRM: Debiased Preference Construction and Iterative Training for Multimodal Reward Modeling
Apr 21, 2026Zhihong Zhang, Jie Zhao, Xiaojian Huang +5Multimodal Reward Modeling Preference Learning
University of Science and Technology of China
Decisive: Guiding User Decisions with Optimal Preference Elicitation from Unstructured Documents
Apr 20, 2026Akriti Jain, Anish Mulay, Divyansh Verma +3AI-Assisted Decision Making Human-AI Decision Making
Adobe Research, India · IIT Madras · Work done during internship at Adobe Research +4
Learning Unanimously Acceptable Lotteries via Queries
Apr 19, 2026Davin Choo, Paul W. Goldberg, Nicholas TehQuery Complexity Learning-Augmented Algorithms
Harvard University, USA · University of Oxford, UK
Scaffold-Conditioned Preference Triplets for Controllable Molecular Optimization with Large Language Models
Apr 14, 2026Yi Xiong, Liang Xiong, Xiaohong Ji +4Molecular Optimization Large Language Model-Guided Optimization
National Key Laboratory of Parallel and Distributed Processing College of Computer Science and Technology National University of DefenseTechnology · DP Technology
From Recency Bias to Stable Convergence Block Kaczmarz Methods for Online Preference Learning in Matchmaking Applications
Apr 11, 2026James NguyenPreference Learning Recommender Systems
Department of Mathematics, University of California, Irvine.
Efficient Personalization of Generative User Interfaces
Apr 10, 2026Yi-Hao Peng, Jeffrey P. Bigham, Jason WuPairwise Preference Learning Human Preference Modeling
Massachusetts Institute of Technology · Carnegie Mellon University · Purdue University
Relational Preference Encoding in Looped Transformer Internal States
Apr 10, 2026Jan KirinPairwise Preference Learning Reward Modeling
Explaining Neural Networks in Preference Learning: a Post-hoc Inductive Logic Programming Approach
Apr 8, 2026Daniele Fossemò, Filippo Mignosi, Giuseppe Placidi +3Neural Network Interpretability Preference Learning
Department of Information Engineering, Computer Science and Mathematics, University of L’Aquila, Italy · ICAR-CNR, Palermo, Italy · Department of Life, Health and Environmental Sciences, University of L’Aquila, Italy +2
LLM Evaluation as Tensor Completion: Low Rank Structure and Semiparametric Efficiency
Apr 7, 2026Jiachun Li, David Simchi-Levi, Will Wei SunSemiparametric Inference LLM Evaluation
Massachusetts Institute of Technology · Purdue University
AdaptFuse: Training-Free Sequential Preference Inference via Externalized Bayesian Computation
Apr 5, 2026Fangzhou Lin, Zilin Dai, Shuo Xing +8Preference Learning Large Language Model-Based Recommendation
Texas A&M University · Worcester Polytechnic Institute · Tohoku University +1

May 11, 2026J/K move · Enter open · S save

NanoResearch: Co-Evolving Skills, Memory, and Policy for Personalized Research Automation

Jinhang Xu, Qiyuan Zhu, Yujun Wu, Zirui Wang+10

1Shanghai Artificial Intelligence Laboratory · 2The Hong Kong University of Science and Technology · 3Peking University+4

Multi-Agent LLM Systems LLM Agent Skill Learning LLM Agents Preference Learning Memory-Augmented Agents Autonomous Research Agents

Open paper AskPDF arXiv

LLM-powered multi-agent systems can now automate the full research pipeline from ideation to paper writing, but a fundamental question remains: automation for whom? Researchers operate under different resource configurations, hold different methodological preferences, and target different output formats. A system that produces uniform outputs regardless of these differences will systematically under-serve every individual user, making personalization a precondition for research automation to be genuinely usable. However, achieving it requires three capabilities that current systems lack: accumulating reusable procedural knowledge across projects, retaining user-specific experience across sessions, and internalizing implicit preferences that resist explicit formalization. We propose NanoResearch, a multi-agent framework that addresses these gaps through tri-level co-evolution. A skill bank distills recurring operations into compact procedural rules reusable across projects. A memory module maintains user- and project-specific experience that grounds planning decisions in each user's research history. A label-free policy learning converts free-form feedback into persistent parameter updates of the planner, reshaping subsequent coordination. These three layers co-evolve: reliable skills produce richer memory, richer memory informs better planning, and preference internalization continuously realigns the loop to each user. Extensive experiments demonstrate that NanoResearch delivers substantial gains over state-of-the-art AI research systems, and progressively refines itself to produce better research at lower cost over successive cycles.