Pairwise Preference Learning

Follow this topic Ask about this topic

Momentum

12 papers in the last four weeks, up 300% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 81

All topics

Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training
May 11, 2026Christian Moya, Alex Semendinger, Guang Lin +1Pairwise Preference Learning LLM Alignment
Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairs
May 10, 2026Yunhong Lu, Qichao Wang, Hengyuan Cao +2Pairwise Preference Learning Rectified Flow
Zhejiang University · Shanghai Institute for Advanced Study-Zhejiang University · Shanghai Institute for Mathematics and Interdisciplinary Sciences
$ξ$ -DPO: Direct Preference Optimization via Ratio Reward Margin
May 9, 2026Zhengyuan Fan, Zhonghua Wu, Yuxuan Du +1Pairwise Preference Learning Direct Preference Optimization
School of Computer Science, Northwestern Polytechnical University
Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph
May 8, 2026Ning Liu, Chuanneng Sun, Kristina Klinkner +1Pairwise Preference Learning LLM Alignment
Amazon
Response Time Enhances Alignment with Heterogeneous Preferences
May 7, 2026Federico Echenique, Alireza Fallah, Baihe Huang +1Pairwise Preference Learning LLM Alignment
Department of Economics, University of California, Berkeley · Department of Computer Science and Ken Kennedy Institute, Rice University · Departments of Electrical Engineering and Computer Sciences, University of California, Berkeley +2
A Unified Pair-GRPO Family: From Implicit to Explicit Preference Constraints for Stable and General RL Alignment
May 7, 2026Hao YuPairwise Preference Learning LLM Alignment
Department of Automation, Tsinghua University
Pair2Score: Pairwise-to-Absolute Transfer for LLM-Based Essay Scoring
May 3, 2026İbrahim Rıza Hallaç, Hasan OğulPairwise Preference Learning Pairwise Comparison
Department of Computer Science and Communication · Østfold University College, Halden 1757, Norway
Multi-User Dueling Bandits: A Fair Approach using Nash Social Welfare
May 3, 2026Maheed H. Ahmed, Mahsa GhasemiPairwise Preference Learning Multi-Armed Bandits
Electrical and Computer Engineering Purdue University
Mind the Gap: Structure-Aware Consistency in Preference Learning
Apr 30, 2026Mehryar Mohri, Yutao ZhongPairwise Preference Learning LLM Alignment
Google Research, New York, NY · Courant Institute of Mathematical Sciences, New York, NY
Learning from Noisy Preferences: A Semi-Supervised Learning Approach to Direct Preference Optimization
Apr 27, 2026Xinxin Liu, Ming Li, Zonglin Lyu +2Pairwise Preference Learning Diffusion Model Alignment
University of Central Florida
Preferences of a Voice-First Nation: Large-Scale Pairwise Evaluation and Preference Analysis for TTS in Indian Languages
Apr 23, 2026Srija Anand, Ashwin Sankar, Ishvinder Sethi +10Pairwise Preference Learning Human Preference Evaluation
Indian Institute of Technology Madras, India · AI4Bharat, India · Josh Talks, India
Vibrotactile Preference Learning: Uncertainty-Aware Preference Learning for Personalized Vibration Feedback
Apr 22, 2026Rongtao Zhang, Xin Zhu, Masoume Pourebadi Khotbehsara +3Pairwise Preference Learning Haptic Feedback
Department of Computer Science University of Southern California Los Angeles, California, USA
Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner
Apr 20, 2026Wei Chen, Yubing Wu, Junmei Yang +5Pairwise Preference Learning LLM Alignment
1South China University of Technology · 2AIP, RIKEN · 3Columbia University +1
Efficient Personalization of Generative User Interfaces
Apr 10, 2026Yi-Hao Peng, Jeffrey P. Bigham, Jason WuPairwise Preference Learning Human Preference Modeling
Massachusetts Institute of Technology · Carnegie Mellon University · Purdue University
Relational Preference Encoding in Looped Transformer Internal States
Apr 10, 2026Jan KirinPairwise Preference Learning Reward Modeling
Not All Preferences Deserve Gradients: Understanding Gradient Utility in Offline Reasoning Alignment
Feb 1, 2026Hui Wu, Hengyi Cai, Jinman Zhao +6Pairwise Preference Learning Training Data Selection
Baidu Inc. · University of Toronto · School of Intelligence Science and Technology, Peking University +1
Adaptive Margin RLHF via Preference over Preferences
Sep 26, 2025Yaswanth Chittepu, Prasann Singhal, Greg Durrett +1Pairwise Preference Learning Reward Modeling
University of Massachusetts Amherst · University of Texas at Austin
Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings
Aug 16, 2025Jenny Y. Huang, Yunyi Shen, Dennis Wei +1Pairwise Preference Learning LLM Evaluation
Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology · MIT-IBM Watson AI Lab · IBM Research
MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning
May 30, 2025Jingyan Shen, Jiarui Yao, Rui Yang +5Pairwise Preference Learning Reward Modeling
New York University · University of Illinois Urbana-Champaign · Rice University
Dual-Difficulty Curriculum Learning for Direct Preference Optimization
Apr 10, 2025Mengyang Li, Haozhan Geng, Zhong Zhang +1Pairwise Preference Learning LLM Alignment
Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission Tianjin Normal University Tianjin, China
Edu-QuRating: Multi-Dimensional Educational Data Curation with Distilled Pairwise Judgements
Date pendingOliver G. B. Garrod, Robin A. A. Ince, Meng Liu +6Language Model Pretraining Pairwise Preference Learning
Fab AI

May 11, 2026, cs.LGJ/K move · Enter open · S save

Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training

Christian Moya, Alex Semendinger, Guang Lin, Elliott Thornley

Pairwise Preference Learning LLM Alignment Preference Optimization Spurious Correlation Robustness Preference Learning

Open paper AskPDF arXiv

Preference learning methods like Direct Preference Optimization (DPO) are known to induce reliance on spurious correlations, leading to sycophancy and length bias in today's language models and potentially severe goal misgeneralization in future systems. In this work, we provide a unified theoretical analysis of this phenomenon, characterizing the mechanisms of spurious learning, its consequences on deployment, and a provable mitigation strategy. Focusing on log-linear policies, we show that standard preference-learning objectives induce reliance on spurious features at the population level through two channels: mean spurious bias and causal-spurious correlation leakage. We then show that this reliance creates an irreducible vulnerability to distribution shift: more data from the same training distribution fails to reduce the model's dependence on spurious features. To address this, we propose tie training, a data augmentation strategy using ties (equal-utility preference pairs) to introduce data-driven regularization. We demonstrate that this approach selectively reduces spurious learning without degrading causal learning. Finally, we validate our theory on log-linear models and provide empirical evidence that both the spurious learning mechanisms and the benefits of tie training persist for neural networks and large language models.

Momentum

Related topics

Latest papers 81