Direct Preference Optimization

Also known as DPO

Direct Preference Optimization (DPO) studies how to align a model with preferences by training it to favor chosen responses over rejected ones, usually relative to a reference model. Research extends the basic pairwise objective with adaptive margins and weighting, token- or segment-level credit, methods for noisy or conflicting labels, and online or curriculum-based data selection. DPO is used for language-model alignment in helpfulness, safety, and reasoning, and is increasingly adapted to multimodal models and diffusion-based image, video, and audio generation. A central tension is improving preference alignment and robustness without sacrificing general capabilities, diversity, or reliable performance when preferences and inputs shift.

Latest papers

162 papers24 papers in the last four weeks, up 60% on the four weeks before. 0.1% of all new papers.

Linear-DPO: Linear Direct Preference Optimization for Diffusion and Flow-Matching Generative Models
May 20, 2026Kesong Li, Yixuan Xu, Kuo-kun Tseng +3Diffusion Model Alignment Flow Matching
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China · Alibaba Group, Hangzhou, Zhejiang, China
Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment
May 20, 2026Zhiqin Yang, Yonggang Zhang, Wei Xue +3LLM Alignment Direct Preference Optimization
1The Hong Kong University of Science and Technology · 2LIGHTSPEED · 3Hong Kong Baptist University.
Distributed Direct Preference Optimization
May 20, 2026Zhanhong JiangDirect Preference Optimization Non-IID Federated Learning
Translational AI Center, Iowa State University, Ames, USA.
AMATA: Adaptive Multi-Agent Trajectory Alignment for Knowledge-Intensive Question Answering
May 17, 2026Taolin Zhang, Dongyang Li, Chen Chen +5Direct Preference Optimization Agentic RAG
School of Computer Science and Information Engineering, Hefei University of Technology · Shanghai University of Electric Power · Guangdong University of Finance and Economics +2
Improving Code Translation with Syntax-Guided and Semantic-aware Preference Optimization
May 13, 2026Yuhan Wu, Huan Zhang, Wei Cheng +3Code Translation Direct Preference Optimization
State Key Laboratory for Novel Software Technology, Nanjing University, China · National Institute of Healthcare Data Science, Nanjing University, China
BSO: Safety Alignment Is Density Ratio Matching
May 12, 2026Tien-Phat Nguyen, Truong Nguyen, Thin Nguyen +3Functional Bregman Divergences Direct Preference Optimization
Hanoi University of Science and Technology Hanoi, Vietnam · Deakin University Burwood, VIC 3125, Australia · Max Planck Research School for Intelligent Systems +2
SyncDPO: Enhancing Temporal Synchronization in Video-Audio Joint Generation via Preference Learning
May 12, 2026Xin Cheng, Xihua Wang, Ying Ba +5Audio-Video Generation Direct Preference Optimization
Renmin University of China · Westlake University
MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization
May 11, 2026Rohan Surana, Xintong Li, Sheldon Yu +7Direct Preference Optimization Active Learning
1UC San Diego · 2Adobe Research
Few-Shot Truly Benign DPO Attack for Jailbreaking LLMs
May 9, 2026Sangyeon Yoon, Wonje Jeung, Yoonjun Cho +2Direct Preference Optimization LLM Fine-Tuning
Yonsei University
$ξ$ -DPO: Direct Preference Optimization via Ratio Reward Margin
May 9, 2026Zhengyuan Fan, Zhonghua Wu, Yuxuan Du +1Pairwise Preference Learning Direct Preference Optimization
School of Computer Science, Northwestern Polytechnical University
Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph
May 8, 2026Ning Liu, Chuanneng Sun, Kristina Klinkner +1Pairwise Preference Learning LLM Alignment
Amazon
Diffusion-APO: Trajectory-Aware Direct Preference Alignment for Video Diffusion Transformers
May 8, 2026Jingyuan Zhu, Biaolong Chen, Le Zhang +3Diffusion Model Alignment Video Diffusion Models
Alibaba Group
Confidence-Aware Alignment Makes Reasoning LLMs More Reliable
May 8, 2026Kejia Chen, Jiawen Zhang, Yihong Wu +5LLM Alignment Direct Preference Optimization
Zhejiang University · Université de Montréal · Sun Yat-sen University +1
Topology-Enhanced Alignment for Large Language Models: Trajectory Topology Loss and Topological Preference Optimization
May 8, 2026Yurui Pan, Ke Xu, Bo PengLLM Alignment Direct Preference Optimization
School of Computing and Intelligent Innovation, Fudan University · School of Economics and Management, Tongji University · College of Information Technology, Shanghai Ocean University
Fusion in Your Way: Aligning Image Fusion with Heterogeneous Demands via Direct Preference Optimization
May 7, 2026Weijian Su, Songqian Zhang, Yuqi Han +3Diffusion Model Alignment Multi-Source Image Fusion
School of Computer Science and Technology, Dalian University of Technology · Key Laboratory of Social Computing and Cognitive Intelligence (Dalian University of Technology), Ministry of Education · Institute of Image Processing and Understanding, North Minzu University
Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models
May 6, 2026Huatian Zhang, Zhendong Mao, Lei Zhang +1LLM Alignment Direct Preference Optimization
University of Science and Technology of China
RLearner-LLM: Balancing Logical Grounding and Fluency in Large Language Models via Hybrid Direct Preference Optimization
May 6, 2026Qiming Bao, Juho Leinonen, Paul Denny +1LLM Grounding LLM Alignment
University of Auckland · Aalto University
Gradient-Gated DPO: Stabilizing Preference Optimization in Language Models
May 4, 2026Inoussa Mouiche, Abderaouf BahiLLM Alignment Direct Preference Optimization
WASP Lab, School of Computer Science University of Windsor, ON N9B 3P4
Efficient Preference Poisoning Attack on Offline RLHF
May 4, 2026Chenye Yang, Weiyu Xu, Lifeng LaiDirect Preference Optimization Data Poisoning Attacks
Department of Electrical and Computer Engineering, University of California, Davis, Davis, CA, USA · Department of Electrical and Computer Engineering, University of Iowa, Iowa City, IA, USA.
TUR-DPO: Topology- and Uncertainty-Aware Direct Preference Optimization
Apr 30, 2026Abdulhady Abas Abdullah, Fatemeh Daneshfar, Seyedali Mirjalili +1LLM Alignment Direct Preference Optimization
Artificial Intelligence and Innovation Centre, University of Kurdistan, Erbil, Iraq · Department of Computer Engineering, University of Kurdistan, Iran · Centre for Artificial Intelligence Research and Optimisation, Torrens University Australia, Brisbane, Australia +2
ViPO: Visual Preference Optimization at Scale
Apr 27, 2026Ming Li, Jie Wu, Justin Cui +3Diffusion Model Alignment Direct Preference Optimization
University of Central Florida · 2ByteDance Seed · ‡ This work was done during the first author’s internship at USA +1
Learning from Noisy Preferences: A Semi-Supervised Learning Approach to Direct Preference Optimization
Apr 27, 2026Xinxin Liu, Ming Li, Zonglin Lyu +2Pairwise Preference Learning Diffusion Model Alignment
University of Central Florida
MGDA-Decoupled: Geometry-Aware Multi-Objective Optimisation for DPO-based LLM Alignment
Apr 22, 2026Andor Vári-Kakas, Ji Won Park, Natasa TagasovskaDirect Preference Optimization Gradient Descent
Prescient Design, CS CoE, Genentech | Roche
HiPO: Hierarchical Preference Optimization for Adaptive Reasoning in LLMs
Apr 22, 2026Darsh Kachroo, Arjun Prasaath Anbazhagan, Adriana Caraeni +2LLM Alignment Direct Preference Optimization
Vellore Institute of Technology, Chennai · Northwestern University · University of Massachusetts Amherst +2
S2H-DPO: Hardness-Aware Preference Optimization for Vision-Language Models
Apr 20, 2026Nitish Shukla, Surgan Jandial, Arun RossDirect Preference Optimization Visual Reasoning
Michigan State University · Carnegie Mellon University
SignDPO: Multi-level Direct Preference Optimisation for Skeleton-based Gloss-free Sign Language Translation
Apr 20, 2026Muxin Pu, Xiao-Ming Wu, Mei Kuan Lim +3Sign Language Translation Direct Preference Optimization
Monash University · Nanyang Technological University
Cat-DPO: Category-Adaptive Safety Alignment
Apr 19, 2026Tiankai Yang, Yi Nian, Xinyuan Li +3LLM Alignment Direct Preference Optimization
University of Southern California · 2Northwestern University
Demystifying the unreasonable effectiveness of online alignment methods
Apr 19, 2026Enoch Hyunwook KangLLM Alignment Direct Preference Optimization
Foster School of Business University of Washington
GroupDPO: Memory-Efficient Group-Wise Direct Preference Optimization
Apr 17, 2026Jixuan Leng, Si Si, Hsiang-Fu Yu +2LLM Alignment Direct Preference Optimization
Carnegie Mellon University · Google DeepMind · Google
Pareto-Optimal Offline Reinforcement Learning via Smooth Tchebycheff Scalarization
Apr 14, 2026Aadyot Bhatnagar, Peter Mørch Groth, Sebastian Ibarraran +1Multi-Objective Reinforcement Learning Offline RL
Profluent Bio, Inc. · Stanford University

May 20, 2026J/K move · Enter open · S save

Linear-DPO: Linear Direct Preference Optimization for Diffusion and Flow-Matching Generative Models

Kesong Li, Yixuan Xu, Kuo-kun Tseng, Weiyi Lu+2

School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China · Alibaba Group, Hangzhou, Zhejiang, China

Diffusion Model Alignment Flow Matching Direct Preference Optimization T2I Generation Preference Optimization for Diffusion Models

Open paper AskPDF arXiv

Direct Preference Optimization (DPO) is successful for alignment in LLMs but still faces challenges in text-to-image generation. Existing studies are confined to denoising diffusion models while overlooking flow-matching, and suffer from an objective mismatch when applying discrete NLP-based DPO to regression-based generative tasks.\ In this paper, we derive a generalized DPO objective that covers both diffusion and flow-matching via a unified reverse-time SDE framework, and point out from a gradient perspective that the standard DPO objective is suboptimal for text-to-image generation. Consequently, we propose Linear-DPO, which replaces the aggressive sigmoid-based utility function with a sustained linear utility and incorporates an EMA-updated reference model. Qualitative and quantitative experiments on diffusion models (SD1.5, SDXL) and flow-matching model (SD3-Medium) demonstrate the superiority of our approach over existing baselines.