Pairwise Preference Learning

Follow this topic Ask about this topic

Momentum

12 papers in the last four weeks, up 300% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 81

All topics

A Markov Chain Approach to Preference Alignment
Jun 21, 2026Takuya Koriyama, Tengyuan LiangPairwise Preference Learning Pairwise Comparison
The University of Chicago
Which Pairs to Compare for LLM Post-Training?
Jun 17, 2026Jiangze Han, Vineet Goyal, Will MaPairwise Preference Learning Pairwise Comparison
Columbia University
PrefSQA: Pairwise Preference Prediction for Speech Quality Assessment and the Critical Role of High Quality Datasets
Jun 17, 2026Junyi Fan, Donald S. WilliamsonPairwise Preference Learning Pairwise Preference Evaluation
Department of Computer Science and Engineering, The Ohio State University, USA
Learning Robust Pair Confidence for Multimodal Emotion-Cause Pair Extraction
Jun 17, 2026Zhuangzhuang Pan, Ning Dong, Yingna Su +1Pairwise Preference Learning Multimodal Robustness
Institute for Advanced Studies Universiti Malaya Kuala Lumpur 50603, Malaysia · School of Information Engineering Suqian University Suqian 223800, China · Digitization Department Suzhou University of Technology Suzhou 215500, China
Surprise-Guided MergeSort: Budget-Efficient Human-in-the-Loop Ranking via Adaptive Comparison Scheduling
Jun 14, 2026Yujin Park, Haejun Chung, Ikbeom JangPairwise Preference Learning Pairwise Comparison
Hanyang University, Seoul, Republic of Korea · Hankuk University of Foreign Studies, Yongin, Republic of Korea
Boosting Direct Preference Optimization with Penalization
Jun 10, 2026Pengwei SunPairwise Preference Learning LLM Alignment
Stanford University.
Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Short
Jun 8, 2026Han Zhou, Adam X. Yang, Laurence Aitchison +2Pairwise Preference Learning Reward Modeling
University of Cambridge · 2Mistral AI · *Work done as an AI Scientist Intern at Mistral AI.
DynaCF: Mitigating Shortcut Learning in Reward Models via Dynamic Counterfactual Sensitivity
Jun 8, 2026Fengyuan Liu, Yongliang Miao, Zirui He +3Pairwise Preference Learning Reward Modeling
The Chinese University of Hong Kong, Shenzhen · New Jersey Institute of Technology · Institute of Computing Technology, CAS
Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modeling
Jun 2, 2026Yifan Wang, Jinyi Mu, Mayank Jobanputra +5Pairwise Preference Learning Reward Modeling
Saarland University · Independent Researcher · Bielefeld University +2
Follow-Your-Preference++: Rethinking Preference Alignment for Image Inpainting
Jun 2, 2026Junkun Yuan, Yutao Shen, Toru Aonishi +2Pairwise Preference Learning Reward Modeling
Zhejiang University · The University of Tokyo · Tsinghua University
Local Preferential Bayesian Optimization
Jun 1, 2026Johanna Menn, Miriam Kober, Paul Brunzema +2Pairwise Preference Learning Pairwise Preference Evaluation
Institute for Data Science in Mechanical Engineering, RWTH Aachen University, Aachen, Germany · Department of Clinical Research, University of Bern, Bern, Switzerland · Center for Reproducible Science and Research Synthesis, University of Zurich, Zurich, Switzerland +1
From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
May 31, 2026Jun-Jie Yang, Chia-Heng Hsu, Kui-Yuan Chen +1Pairwise Preference Learning Representation Learning
Department of Computer Science, National Yang Ming Chiao Tung University, Hsinchu, Taiwan
Enhancing LLM Metacognition via Cognitive Pairwise Training
May 30, 2026Weitao Li, Hao Zhou, Xuanyu Lei +11Pairwise Preference Learning RL for Language Model Reasoning
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China · WeChat AI, Tencent Inc., China
Reinforcement Learning with Pairwise Preferences in Long-Term Decision Problems
May 29, 2026Jonathan Colaço Carr, Prakash Panangaden, Doina Precup +1Pairwise Preference Learning Markov Decision Processes
Stanford University · School of Computer Science, McGill University, Montreal, Quebec, Canada · Mila - Quebec AI Institute, Montreal, Quebec, Canada +1
From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Merging
May 29, 2026Qi Sun, Siyue Zhang, Yulin Chen +3Pairwise Preference Learning Low-Rank Adaptation
NYU Shanghai · NTU · NYU +1
The Representation-Rationalizability Tradeoff in Reward Learning
May 29, 2026Jing Dong, Yaoliang Yu, Pascal PourpartPairwise Preference Learning Reward Modeling
Vector Institute · University of Waterloo & Vector Institute
Differentially Private Preference Data Synthesis for Large Language Model Alignment
May 29, 2026Fengyu Gao, Jing YangPairwise Preference Learning LLM Alignment
Department of Computer Science, University of Virginia, Charlottesville, Virginia, USA · Department of Electrical and Computer Engineering, University of Virginia, Charlottesville, Virginia, USA.
Reward Learning from Best-of- $N$ Preference Data: Targets, Tradeoffs, and Design Principles
May 28, 2026Rattana Pukdee, Maria-Florina Balcan, Pradeep RavikumarPairwise Preference Learning Reward Modeling
Machine Learning Department Carnegie Mellon University
Active Query Synthesis for Preference Learning
May 25, 2026Namrata Nadagouda, Nauman Ahad, Maegan Tucker +1Pairwise Preference Learning Active Learning
Georgia Institute of Technology · Gauss Labs
When In-Distribution Gains Fail: Evaluating Weak-to-Strong Reward Models under Preference Shift
May 25, 2026Khoi Le, Tri Cao, Phong Nguyen +5Pairwise Preference Learning Reward Modeling
National University of Singapore · VinUniversity · Nanyang Technological University
Learning Kernel-Based MDPs from Episodic Preferential Feedback
May 22, 2026Nikola Pavlovic, Sattar Vakili, Qing ZhaoPairwise Preference Learning Regret Minimization in RL
Cornell University · MediaTek Research
Convex Optimization for Alignment and Preference Learning on a Single GPU
May 22, 2026Miria Feng, Mert PilanciPairwise Preference Learning LLM Alignment
Department of Electrical Engineering, Stanford University, California, United States.
Token-weighted Direct Preference Optimization with Attention
May 21, 2026Chengyu Huang, Zhuohang Li, Sheng-Yen Chou +1Pairwise Preference Learning LLM Alignment
Cornell University · Vanderbilt University
Beyond Scalar Objectives: Expert-Feedback-Driven Autonomous Experimentation for Scientific Discovery at the Nanoscale
May 20, 2026Ralph Bulanadi, Jefferey Baxter, Arpan Biswas +5Pairwise Preference Learning Active Learning
Center for Nanophase Materials Sciences, Oak Ridge National Laboratory, Oak Ridge, TN 37831, USA · University of Tennessee-Oak Ridge Innovation Institute, University of Tennessee, Knoxville, TN 37996, USA · Department of Material Science and Engineering, School of Materials and Chemical Technology, Institute of Science Tokyo, Yokohama, 226-8502, Japan +3
Preferences Order, Ratings Anchor: From Fused Expert Aesthetic Ground Truth to Self-Distillation
May 19, 2026Yuanpei Zhao, Jie Lin, Chao Zhang +5Pairwise Preference Learning Personalized Image Aesthetics Assessment
1Sichuan University · 2NetEase Fuxi AI Lab · †Work done during an internship at NetEase Fuxi AI Lab. +1
CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging
May 19, 2026Zhenyu Li, Aleksandar Cvejic, Zehui Chen +1Pairwise Preference Learning Human Preference Evaluation
KAUST · USTC
Pairwise Preference Reward and Group-Based Diversity Enhancement for Superior Open-Ended Generation
May 18, 2026Guining Cao, Jiaxin Peng, Chu Zeng +3Pairwise Preference Learning Open-Ended Generation
Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd · School of Software and Microelectronics, Peking University · Tsinghua University
Active Learners as Efficient PRP Rerankers
May 14, 2026Jeremías Figueiredo Paschmann, Juan Kaplan, Francisco Nattero +3Pairwise Preference Learning Pairwise Comparison
ELIAS Lab, Departamento de Ingeniería, Universidad de San Andrés
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
May 12, 2026Truong Nguyen, Tien-Phat Nguyen, Linh Ngo Van +3Pairwise Preference Learning LLM Alignment
Hanoi University of Science and Technology · University of Stuttgart · German Research Center for Artificial Intelligence +3
Variance-aware Reward Modeling with Anchor Guidance
May 12, 2026Shuxing Fang, Ruijian Han, Liangyu Zhang +1Pairwise Preference Learning Reward Modeling
Department of Data Science and Artificial Intelligence, Hong Kong Polytechnic University · School of Statistics and Data Science, Shanghai University of Finance and Economics

Jun 21, 2026, cs.LGJ/K move · Enter open · S save

A Markov Chain Approach to Preference Alignment

Takuya Koriyama, Tengyuan Liang

The University of Chicago

Pairwise Preference Learning Pairwise Comparison Pairwise Preference Evaluation Preference Alignment

Open paper AskPDF arXiv

We propose Markov Chain from Human Feedback (MCHF), an elementary approach for aligning generative models from pairwise human preferences. Unlike Reinforcement Learning from Human Feedback (RLHF), which reduces comparisons to a scalar reward, and Nash Learning from Human Feedback (NLHF), which preserves pairwise utilities through a KL-regularized minimax optimization, MCHF uses pairwise preferences directly to define a transition mechanism over model outputs. Given a pairwise utility

U(x,y)

, which quantifies human preference for

y

over

x

, and a reference probability distribution

μ_{\mathsf{ref}}

, we define a Markov kernel

\mathsf{P}(x, dy)\propto \exp(U(x,y))μ_{\mathsf{ref}}(dy)

, and take the Markov chain starting from

μ_{\mathsf{ref}}

as an iterative alignment procedure. We show that MCHF converges geometrically fast to the stationary distribution, with a convergence rate governed by the seminorm

\|U\|_\oplus=\inf_{g,f\in L^\infty(μ_{\mathsf{ref}})}\|U-g\oplus f\|_\infty

, which quantifies the non-transitive structure of the pairwise utility. We further show that a mirror-descent algorithm for NLHF satisfies an analogous structure-adaptive convergence guarantee. Finally, through a perturbation analysis, we prove that when

\|U\|_\oplus

is small, MCHF and NLHF agree up to first order around an RLHF solution, which yields a unified view of reward-based, game-theoretic, and Markovian approaches to alignment. In particular, for two natural algorithms that converge to the MCHF/NLHF equilibria, we show that the first step of MCHF and NLHF recovers the RLHF solution based on the column-sum reward

\hat{f}(y)=\int μ_{\mathsf{ref}}(dx) U(x, y)

, and starting from the second iteration, both algorithms incorporate the same linear functional of the residual

U-(-\hat f)\oplus \hat f

, which captures the non-transitive structure of the pairwise utility

U

Momentum

Related topics

Latest papers 81