Preference Alignment

Follow this topic Ask about this topic

Preference alignment aims to make AI outputs reflect what people value, from helpful and safe language-model responses to personalized recommendations, generated images, and robot behavior. Common approaches learn reward models from pairwise judgments and optimize with reinforcement learning from human feedback (RLHF), or use direct preference optimization (DPO); other methods steer models at inference time or adapt them to individual users and multiple objectives. Evaluations use human judgments and task-specific benchmarks, including language-model preference tests and image-generation comparisons. A central challenge is that preferences can be noisy, conflicting, culturally varied, or change over time, while optimizing imperfect reward signals can introduce bias, reduce capabilities, or reward behavior people do not actually want.

Latest papers

109 papers8 papers in the last four weeks, down 38% on the four weeks before. 0.0% of all new papers.

Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment
May 20, 2026Zhiqin Yang, Yonggang Zhang, Wei Xue +3LLM Alignment Direct Preference Optimization
1The Hong Kong University of Science and Technology · 2LIGHTSPEED · 3Hong Kong Baptist University.
Spectral Souping: A Unified Framework for Online Preference Alignment
May 19, 2026Yinlam Chow, Guy Tennenholtz, Ted Yun +4Multi-Objective Language Model Alignment Preference Alignment
Google DeepMind · Google Research
When Preference Labels Fall Short: Aligning Diffusion Models from Real Data
May 19, 2026Weiyan Chen, Weijian Deng, Yao Xiao +5Diffusion Model Alignment Preference Alignment
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China · Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Australian National University, Canberra, Australia +2
Where Pretraining writes and Alignment reads: the asymmetry of Transformer weight space
May 15, 2026Valeria Ruscio, Eli-Shaoul Khedouri, Keiran ThompsonLanguage Model Pretraining Transformer Interpretability
Intuition Machines
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
May 12, 2026Truong Nguyen, Tien-Phat Nguyen, Linh Ngo Van +3Pairwise Preference Learning LLM Alignment
Hanoi University of Science and Technology · University of Stuttgart · German Research Center for Artificial Intelligence +3
Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansion
May 12, 2026ShiYing Huang, Liang Lin, Yuer Li +6LLM Safety Alignment Multi-Objective Language Model Alignment
Huazhong University of Science and Technology · Nanyang Technological University · Tsinghua University +1
DGPO: Beyond Pairwise Preferences with Directional Consistent Groupwise Optimization
May 11, 2026Mengyi Deng, Zhiwei Li, Xin Li +4LLM Alignment Reasoning Consistency in Language Models
1Information Hub, The Hong Kong University of Science and Technology (Guangzhou), China · 2The Hong Kong University of Science and Technology, Hong Kong SAR
ProteinOPD: Towards Effective and Efficient Preference Alignment for Protein Design
May 11, 2026Yulin Zhang, He Cao, Zihao Jiang +6Protein Language Models Multi-Teacher Knowledge Distillation
Tsinghua University · International Digital Economy Academy · Hong Kong University of2026 Science and Technology (Guangzhou) +1
EvoPref: Multi-Objective Evolutionary Optimization Discovers Diverse LLM Alignments Beyond Gradient Descent
May 10, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuMulti-Objective Evolutionary Optimization Multi-Objective Language Model Alignment
The University of Hong Kong, China · Stellaris AI Limited, China
Mechanistic Analysis of Alignment Algorithms in Language Models
May 9, 2026Aarush Sinha, Ishan Garg, Veeraraju Elluru +2LLM Interpretability Mechanistic Interpretability
University of Copenhagen · Independent · IIT Jodhpur +2
Can Revealed Preferences Clarify LLM Alignment and Steering?
May 8, 2026Khurram Yamin, Jingjing Tang, Eric Horvitz +1Language Model Steering LLM Decision-Making
1Carnegie Mellon University · 2Microsoft Research
Theoretical Limits of Language Model Alignment
May 8, 2026Lucas Monteiro Paes, Natalie Mackraz, Barry-John Theobald +1Reward Hacking LLM Alignment
Apple
Better Reward Models Are Not Always Less Biased: Social Alignment Beyond Standard Reward Benchmarks
May 6, 2026Gayane Ghazaryan, Esra DönmezReward Modeling Language Model Safety Evaluation
Institute for Natural Language Processing, University of Stuttgart · Interchange Forum for Reflecting on Intelligent Systems, University of Stuttgart
Threshold-Guided Optimization for Visual Generative Models
May 6, 2026Jinbin Bai, Yu Lei, Qingyu Shi +6Diffusion Model Alignment Preference Alignment
National University of Singapore · Collov Labs · Peking University +2
Towards General Preference Alignment: Diffusion Models at Nash Equilibrium
May 6, 2026Jiaming Hu, Jiamu Bai, Haoyu Wang +2Diffusion Model Alignment Preference Optimization
Boston University · Penn State University · University at Albany
Mind the Gap: Structure-Aware Consistency in Preference Learning
Apr 30, 2026Mehryar Mohri, Yutao ZhongPairwise Preference Learning LLM Alignment
Google Research, New York, NY · Courant Institute of Mathematical Sciences, New York, NY
Learning from Noisy Preferences: A Semi-Supervised Learning Approach to Direct Preference Optimization
Apr 27, 2026Xinxin Liu, Ming Li, Zonglin Lyu +2Pairwise Preference Learning Diffusion Model Alignment
University of Central Florida
Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment
Apr 27, 2026Wenzhe Xu, Biao Liu, Yiyang Sun +2Meta-Learning Multi-Objective Language Model Alignment
Pref-CTRL: Preference Driven LLM Alignment using Representation Editing
Apr 26, 2026Imranul Ashrafi, Inigo Jauregi Unanue, Massimo PiccardiLLM Alignment Representation Engineering
University of Technology Sydney, Australia
HeadRank: Decoding-Free Passage Reranking via Preference-Aligned Attention Heads
Apr 19, 2026Juyuan Wang, Chenxing Wang, Yuchen Fang +8Attention Head Analysis Self-Attention
Weixin Group, Tencent, China · 2South China University of Technology, Guangzhou, China
Reward-Aware Trajectory Shaping for Few-step Visual Generation
Apr 16, 2026Rui Li, Bingyu Li, Yuanzhi Liang +3Diffusion Model Alignment Diffusion Model Distillation
University of Science and Technology of China HeFei, China · TeleAI ShangHai, China
Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning
Apr 14, 2026Shanyong Wang, Shuhang Lin, Yining Zhao +2LLM Fine-Tuning Personalized Language Model Alignment
Rutgers University—New Brunswick · University of Illinois at Urbana-Champaign
Principles Do Not Apply Themselves: A Hermeneutic Perspective on AI Alignment
Apr 12, 2026Behrooz RazeghiAI Alignment LLM Alignment
School of Engineering and Applied Sciences, Harvard University
From Recency Bias to Stable Convergence Block Kaczmarz Methods for Online Preference Learning in Matchmaking Applications
Apr 11, 2026James NguyenPreference Learning Recommender Systems
Department of Mathematics, University of California, Irvine.
Multi-Faceted Self-Consistent Preference Alignment for Query Rewriting in Conversational Search
Apr 8, 2026Zhiyu Cao, Peifeng Li, Qiaoming ZhuQuery Reformulation Conversational Search
School of Computer Science and Technology, Soochow University, Suzhou, China
ICG: Improving Cover Image Generation via MLLM-based Prompting and Personalized Preference Alignment
Apr 8, 2026Zhipeng Bian, Jieming Zhu, Qijiong Liu +6Image Generation Diffusion-Based Image Generation
Huazhong University of Science and Technology · Huawei Noah’s Ark Lab · Hong Kong Polytechnic University +1
Beyond Compromise: Pareto-Lenient Consensus for Efficient Multi-Preference LLM Alignment
Apr 7, 2026Renxuan Tan, Rongpeng Li, Zhifeng Zhao +1Preference Alignment Multi-Objective Language Model Alignment
College of Information Science and Electronic Engineering, Zhejiang University · Zhejiang Lab, Hangzhou, China · Macau University of Science and Technology, Macau, China
Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelines
Apr 7, 2026Hwanjun SongLLM Alignment Preference Alignment
KAIST
Relative Density Ratio Optimization for Stable and Statistically Consistent Model Alignment
Apr 6, 2026Hiroshi Takahashi, Tomoharu Iwata, Atsutoshi Kumagai +4LLM Alignment Preference Alignment
NTT, Inc. · NTT DOCOMO, INC.
Constructive Alignment: Governing Preference Dynamics in Human-AI Interaction
Apr 1, 2026Max Kanwal, Caryn TranPreference Alignment Human-AI Interaction
Stanford University · Northwestern University

May 20, 2026J/K move · Enter open · S save

Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment

Zhiqin Yang, Yonggang Zhang, Wei Xue, Dong Fang+2

1The Hong Kong University of Science and Technology · 2LIGHTSPEED · 3Hong Kong Baptist University.

LLM Alignment Direct Preference Optimization Preference Optimization Preference Alignment RL from Human Feedback

Open paper AskPDF arXiv

Direct Preference Optimization (DPO) has emerged as a popular alternative to Reinforcement Learning from Human Feedback (RLHF), offering theoretical equivalence with simpler implementation. We prove this equivalence is conditional rather than universal, depending on an implicit assumption frequently violated in practice: the RLHF-optimal policy must prefer human-preferred responses. When this assumption fails, DPO optimizes relative advantage over the reference policy rather than absolute alignment with human preferences, leading to pathological convergence where policies decrease DPO loss while preferring dispreferred responses. We characterize when this assumption is violated, show the existence of an undesirable solution space, and prove that DPO and RLHF optimize fundamentally different objectives in such cases. To address this, we introduce Constrained Preference Optimization (CPO), augmenting RLHF with constraints for provable alignment. We further provide a geometric interpretation through soft margin ranking, revealing that DPO implements margin ranking with potentially negative targets. Our theoretical analysis establishes when DPOs' guarantees hold and provides solutions preserving simplicity with provable alignment. Comprehensive experiments on standard benchmarks demonstrate that CPO achieves state-of-the-art performance. Code is available at: https://github.com/visitworld123/CPO.