Pairwise Preference Learning

Follow this topic Ask about this topic

Pairwise preference learning studies how to infer what people value from choices between alternatives, rather than requiring absolute scores. Common approaches include Bradley–Terry reward models, direct preference optimization for language models, and active learning or dueling-bandit methods that select informative comparisons. The methods support applications from ranking and recommendations to AI alignment, robotics, and policy learning, including settings with noisy labels or limited annotation budgets. A central challenge is that preferences may be heterogeneous, context-dependent, or cyclic, so a single consistent utility score can miss important differences between people and situations.

Latest papers

126 papers26 papers in the last four weeks, up 136% on the four weeks before. 0.2% of all new papers.

Exploring the Limits of the Delta Learning Hypothesis: Single-Pair Generator Biases in Direct Preference Optimization
Oct 27, 2026Giuseppe Curci, Stefano Simonazzi, Andrea Molinari +1Direct Preference Optimization LLM Alignment
Okkam, Italy · Villanova.ai, Italy · University of Trento, Italy
An Interactive Evolutionary Multiobjective Optimization Algorithm Based on Interpretable Active Preference Learning
Oct 26, 2026Thomas Quadt, Boris Čule, Roy Lindelauf +2Multi-Objective Evolutionary Optimization Active Preference Learning
Department of Intelligent Systems, Tilburg University, Tilburg, NL · Data Science Center of Excellence, Faculty of Military Sciences, Breda, NL
Reward-Weighted Preference Optimization for Selective Combinatorial Optimization Problems
Oct 22, 2026Haohui Zhang, Xinyu Hu, Wouter van Heeswijk +2RL for Combinatorial Optimization Combinatorial Optimization
University of Twente, Enschede, Netherlands · University of Bath, Bath, United Kingdom · Delft University of Technology, Delft, Netherlands
PerSPL: Personalized Semi-supervised Preference Learning
Oct 21, 2026Xiaoan Xu, Tingzhou Wei, Juncheng Dong +3Personalized Reward Modeling User Preference Modeling
Duke University · George Washington University
Pairwise Learning Increases Data Efficiency in Deep Neural Networks
Oct 20, 2026Robert Bain, Marjan Rashidi, Elizabeth ChrastilData-Efficient Learning Small-Data Learning
Department of Neurobiology & Behavior, University of California, Irvine, Irvine, CA 92697, USA · Department of Cognitive Sciences, University of California, Irvine, Irvine, CA 92697, USA
Statistical Learning Theory for Direct Preference Optimization
Oct 19, 2026Joel Wills, Yuanfan Li, Yiming Ying +1Direct Preference Optimization Pairwise Preference Learning
School of Mathematics and Statistics, The University of Sydney
Bridging Exploration and Persona: Information-Directed Dueling for Test-Time Agent Personalization
Oct 17, 2026Jie Mao, Pingchen Lu, Xiangyi Wang +4LLM Personalization User Preference Modeling
The Chinese University of Hong Kong, Shenzhen · University of Copenhagen · City University of Hong Kong +1
Replicable Dueling Bandits
Oct 14, 2026Rohan Deb, Aadirupa Saha, Udaya Ghai +2Dueling Bandits Multi-Armed Bandits
University of Illinois, Urbana Champaign · University of Illinois, Chicago · Amazon +1
What Does Disagreement Mean? A Semantic Identifiability Frontier for Pairwise Preference Learning
Oct 14, 2026Manoj Saravanan, Rohit Kumar Salla, Shrikar Reddy KotaPairwise Preference Learning Annotator Disagreement
Virginia Tech, Blacksburg, VA, USA.
When Pairwise Preferences Cycle: A Reliability-Anchored Preference Walk for Video GRPO
Oct 14, 2026Xinrui Lin, Fan Li, Shumin Wang +2Pairwise Preference Learning RL for Video Generation
University of Science and Technology of China, Hefei, China · Huawei Noah’s Ark Lab
Learning Individual Scientific Taste with Privileged Hindsight Distillation
Oct 14, 2026Hanwei Fan, Yanqiao Wang, Yiqi Zhu +3AI-Assisted Scientific Research Learning Using Privileged Information
Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China · Department of Computer Science and Technology, Tsinghua University, Beijing, China · Yau Mathematical Sciences Center, Tsinghua University, Beijing, China +1
Online LLM Personalization from Pairwise Preferences Across Users
Oct 12, 2026Romain Puech, Georgy Salakhutdinov, Dimitris BertsimasEducational QA LLM Personalization
Massachusetts Institute of Technology
Can You Trust Your Preference Models? Stackelberg Games for Alignment
Oct 10, 2026Riccardo Formenti, Aditya Agrawal, Vikas GargLLM Alignment Preference-Based RL
Department of Mathematics ETH Zürich · Department of Computer Science Aalto University · YaiYai Ltd and Aalto University
Uniform Information Density-Based Preference Optimization
Oct 10, 2026Yongyuan Li, Yang XuPreference Optimization LLM Alignment
Department of Computer Science and Engineering Southern University of Science and Technology Shenzhen, Guangdong, China
EmoGavel: Grounding-Aware Reinforcement Learning with Verifiable Rewards for Multimodal Emotion Preference Learning
Oct 9, 2026Zilong Huang, Kong Aik Lee, Zezhong Jin +2Pairwise Preference Learning Reinforcement Learning with Verifiable Rewards
The Hong Kong Polytechnic University, Hong Kong SAR, China · Kyoto University, Japan
How Many Repeated Pairwise Comparisons Are Needed for Ranking under Heterogeneity?
Oct 7, 2026Shashaank Aiyer, Han ShaoPairwise Ranking Pairwise Preference Learning
University of Maryland
Covariate-dependent Joint Modeling of Multivariate Ordinal Preferences and Its Connections with Comparison Models
Oct 6, 2026Yujie Chen, Antik Chakraborty, Anindya BhadraPreference Modeling Preference Learning
Department of Statistics Purdue University
Groupwise Distortion Guarantees for Preference-Based Alignment
Oct 4, 2026Jacob Brodkey, Roberto Tamez, Aaron RothPairwise Preference Learning Social Choice Theory
University of Pennsylvania
Learning in Continuous Games from Pairwise Preference Feedback
Oct 4, 2026Anas BarakatPairwise Preference Learning Nash Equilibrium Computation
Singapore University of Technology and Design
Bidirectional Preference Synthesis: Learning Prompt-Conditioned Preferences from Boundary Failures
Oct 3, 2026Junbo Wang, Lidong Lu, Zhuoqun Li +4Preference Learning Preference Optimization
Kuaishou Technology · Nanjing University
Targeted Active Learning for Preference-Based Treatment Effects on Multivariate Outcomes
Oct 2, 2026Lola Giordani, Mathieu Even, Chloé Geoffroy +3Active Preference Learning Individualized Treatment Effect Estimation
Theremia, Paris · Paris Brain Institute (ICM), Sorbonne Université, Inserm, CNRS, AP-HP · Inria, Inserm, Université de Montpellier, France +1
On the Complexity of Preference-Based Bandits
Sep 30, 2026Ahmed Ben Yahmed, Marc Abeille, Clément CalauzènesPairwise Preference Learning Multi-Armed Bandits
CREST, ENSAE Paris, FAIRPLAY · FAIRPLAY
The Missing Coefficients: Bayesian Pairwise Merging for Model Personalization
Sep 30, 2026Yaling Shen, Tongtong Wu, Siyuan Yan +1Pairwise Preference Learning Personalized Language Models
Monash University
Optimal Design for Active Preference Learning with Biased LLM Judges
Sep 30, 2026Zhongman Du, Huiming Zhang, Haodong Zhu +1Pairwise Preference Learning LLM Alignment
Beihang University · Zhongguancun Academy · Hangzhou Innovation Institute of Beihang University
Uncertainty-Normalized Margins for Direct Preference Optimization
Sep 29, 2026Sadegh Khorasani, Petrus Mikkola, Matthias GrossglauserPairwise Preference Learning Direct Preference Optimization
School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland · University of Helsinki, Helsinki, Finland
Using Context Is Not Enough: Test-Time Training for Personalized Reward Modeling
Sep 28, 2026Bohao Wang, Xiaoyan Zhao, Yang Zhang +4Pairwise Preference Learning Reward Modeling
Zhejiang University · National University of Singapore
From Soft Targets to Reward Signals: How Assignment and Reward Objectives Interact
Sep 28, 2026Jiangtao Lin, Bangyang Wei, Siyi Liu +2Pairwise Preference Learning Reward Modeling
Tsinghua Shenzhen International Graduate School, Tsinghua University · School of Vehicle and Mobility, Tsinghua University · Tencent Holdings Limited +1
PrefLUT: Reusable and Refinable Personalized Color Editing from Pairwise Preferences
Sep 28, 2026Chuanzhi Xu, Langyi Chen, Chengkun Yue +5Pairwise Preference Learning User Preference Modeling
The University of Sydney · Indiana University · The University of Hong Kong
Axiom Satisfiability of Linear Rewards in Alignment
Sep 26, 2026Soumya Nasipuri, Sayak Ray Chowdhury, Sanjukta RoyReward Modeling Preference Alignment
Indian Statistical Institute, India · Indian Institute of Technology Kanpur, India
Toward Human-Aligned Judgement of Speech Emotion Similarity
Sep 26, 2026Yun-Shao Tsai, Yi-Cheng Lin, Chih-Kai Yang +7Speech Generation Evaluation Emotional Speech Synthesis
Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · Independent Researcher · National Taiwan University, Taiwan +2

Oct 27, 2026J/K move · Enter open · S save

Exploring the Limits of the Delta Learning Hypothesis: Single-Pair Generator Biases in Direct Preference Optimization

Giuseppe Curci, Stefano Simonazzi, Andrea Molinari, Andrea Zugarini

Okkam, Italy · Villanova.ai, Italy · University of Trento, Italy

Direct Preference Optimization LLM Alignment Pairwise Preference Learning

Open paper AskPDF arXiv

Direct Preference Optimization (DPO) typically relies on preference pairs generated from diverse model pools and ranked by an external LLM judge. The recently proposed delta-learning hypothesis suggests that comparable supervision can instead be obtained from a single pair of models by treating the larger model's response as preferred. We investigate the limits of this approach across model families and scales. We show, both theoretically and empirically, that fixed model pairs introduce systematic, generator-dependent biases that are predictive of the preference label independent of response quality. Consequently, DPO trained on single-pair data consistently underperforms multi-model preference data, despite the larger generator producing higher-quality responses, and we trace this gap to training dynamics consistent with reward hacking on these biases. We further show that simply increasing the number of generators, even without an external judge, is enough to partially mitigate this effect. Our results highlight generator diversity as a critical factor in preference-data construction.