Direct Preference Optimization

Also known as DPO

Direct Preference Optimization (DPO) studies how to align a model with preferences by training it to favor chosen responses over rejected ones, usually relative to a reference model. Research extends the basic pairwise objective with adaptive margins and weighting, token- or segment-level credit, methods for noisy or conflicting labels, and online or curriculum-based data selection. DPO is used for language-model alignment in helpfulness, safety, and reasoning, and is increasingly adapted to multimodal models and diffusion-based image, video, and audio generation. A central tension is improving preference alignment and robustness without sacrificing general capabilities, diversity, or reliable performance when preferences and inputs shift.

Latest papers

162 papers24 papers in the last four weeks, up 60% on the four weeks before. 0.1% of all new papers.

EditCaption: Human-Refined SFT and HAE-DPO for Image Editing Instruction Synthesis
Apr 9, 2026Xiangyuan Wang, Honghao Cai, Yunhao Bai +9Direct Preference Optimization Image Captioning
Peking University · Xiaohongshu Inc. · The Chinese University of Hong Kong, Shenzhen +2
Random Is Hard to Beat: Active Selection in online DPO with Modern LLMs
Apr 3, 2026Giyeong Oh, Junghyun Lee, Jaehyun Park +3Active Preference Learning Language Model Post-Training
Seoul National University · Ewha Womans University · KAIST +1
Mitigating LLM biases toward spurious social contexts using direct preference optimization
Apr 2, 2026Hyunji Nam, Dorottya DemszkyLanguage Model Robustness Language Model Bias Evaluation
Stanford University
STAR: Mitigating Cascading Errors in Spatial Reasoning via Turn-point Alignment and Segment-level DPO
Apr 1, 2026Pukun Zhao, Longxiang Wang, Chen Chen +4Spatial Reasoning in Language Models Spatial Reasoning
Guangdong University of Finance and Economics · Chongqing University · Westlake University +1
Self-Improving Code Generation via Semantic Entropy and Behavioral Consensus
Mar 31, 2026Huan Zhang, Wei Cheng, Wei HuLanguage Model Self-Improvement LLM Code Generation
State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · State Key Laboratory for Novel Software Technology, National Institute of Healthcare Data Science, Nanjing University, Nanjing, China
Aligning Multimodal Sequential Recommendations via Robust Direct Preference Optimization with Sparse MoE
Mar 31, 2026Hejin Huang, Jusheng Zhang, Kaitong Cai +2Recommender Systems Direct Preference Optimization
Sun Yat-sen University · Snap Inc
Improving Safety Alignment via Balanced Direct Preference Optimization
Mar 24, 2026Shiji Zhao, Mengyang Wang, Shukun Xiong +7LLM Safety Alignment LLM Alignment
Institute of Artificial Intelligence, Beihang University, Beijing, China.
ACPO: Counteracting Likelihood Displacement in Vision-Language Alignment with Asymmetric Constraints
Mar 23, 2026Kaili Huang, Hongming Zhang, Rui Shen +4Vision-Language Alignment Preference Optimization
SenseTime Research
Causal Direct Preference Optimization for Distributionally Robust Generative Recommendation
Mar 21, 2026Chu Zhao, Enneng Yang, Jianzhe Zhao +1Large Language Model-Based Recommendation Distribution Shift Robustness
Northeastern University, Shenyang, China · Shenzhen Campus of Sun Yat-sen University, China
An Empirical Study of SFT-DPO Interaction and Parameterization in Small Language Models
Mar 20, 2026Yuming Feng, Christy YangDirect Preference Optimization Supervised Fine-Tuning
Department of Computer Science, Stanford University
FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment
Mar 20, 2026Kewen Zhu, Liping Yi, Zhiming Zhao +3Personalized Language Model Alignment Direct Preference Optimization
College of Intelligence and Computing, Tianjin University, Tianjin, China · College of Computing and Data Science, Nanyang Technological University, Singapore
Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimization
Mar 18, 2026Haocheng Luo, Zehang Deng, Thanh-Toan Do +3LLM Alignment Direct Preference Optimization
Monash University, Australia · Swinburne University of Technology, Australia
Do Understanding and Generation Fight? A Diagnostic Study of DPO for Unified Multimodal Models
Mar 17, 2026Abinav Rao, Sujan RachuriUnified Multimodal Models Multi-Objective Language Model Alignment
GlyphPrinter: Region-Grouped Direct Preference Optimization for Glyph-Accurate Visual Text Rendering
Mar 16, 2026Xincheng Shuai, Ziye Li, Henghui Ding +1Image Generation Direct Preference Optimization
Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, China · Generative AI Lab, College of Computing and Data Science, Nanyang Technological University, Singapore
GDPO-SR: Group Direct Preference Optimization for One-Step Generative Image Super-Resolution
Mar 16, 2026Qiaosi Yi, Shuai Li, Rongyuan Wu +3Image Super-Resolution Direct Preference Optimization
The Hong Kong Polytechnic University · OPPO Research Institute
PhysMoDPO: Physically-Plausible Humanoid Motion with Preference Optimization
Mar 13, 2026Yangsong Zhang, Anujith Muraleedharan, Rikhat Akizhanov +5Human Motion Generation Robot Motion Generation
Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI) · LIGM, École des Ponts, IP Paris, Univ Gustave Eiffel, CNRS · École polytechnique fédérale de Lausanne (EPFL)
SVLL: Staged Vision-Language Learning for Physically Grounded Embodied Task Planning
Mar 12, 2026Yuyuan Yang, Junkun Hong, Hongrong Wang +11Visual Grounding for Robotics Large Language Model-Based Robot Planning
FNii-Shenzhen, CUHKSZ · Shenzhen University · SAI, CUHKSZ +4
Risk-Controllable Multi-View Diffusion for Driving Scenario Generation
Mar 12, 2026Hongyi Lin, Wenxiu Shi, Heye Huang +5Autonomous Driving Scenario Generation Controllable Video Generation
Tsinghua University · MIT · Z-One Tech +2
V2A-DPO: Omni-Preference Optimization for Video-to-Audio Generation
Mar 11, 2026Nolan Chan, Timmy Gang, Yongqian Wang +2Audio-Video Generation Direct Preference Optimization
The Chinese University of Hong Kong, Hong Kong SAR, China · National Research Council Canada, Canada · The University of Warwick, UK +1
CompassDPO: Dynamics-Controlled Direct Preference Optimization for Robust Safety Alignment
Mar 7, 2026Jilong Liu, Yonghui Yang, Pengyang Shao +5Direct Preference Optimization LLM Safety Alignment
Hefei University of Technology · National University of Singapore
From Prompting to Preference Optimization: A Comparative Study of LLM-based Automated Essay Scoring
Mar 6, 2026Minh Hoang Nguyen, Vu Hoang Pham, Xuan Thanh Huynh +5Automated Essay Scoring Language Model Post-Training
Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam
Place-it-R1: Unlocking Environment-aware Reasoning Potential of MLLM for Video Object Insertion
Mar 6, 2026Bohai Gu, Taiyi Wu, Dazhao Du +5Video Object Insertion Video Editing
The Hong Kong University of Science and Technology · AI Technology Center, Tencent Video, Tencent · Peking University
MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization
Mar 3, 2026Ashutosh Chaubey, Jiacheng Pang, Mohammad SoleymaniMultimodal Large Language Models Multimodal Hallucination
University of Southern California
IdGlow: Dynamic Identity Modulation for Multi-Subject Generation
Feb 28, 2026Honghao Cai, Xiangyuan Wang, Jing Li +15Multi-Subject Image Generation Image Generation
Xiaohongshu Inc. · The Chinese University of Hong Kong, Shenzhen · Peking University +1
CoME: Empowering Channel-of-Mobile-Experts with Informative Hybrid-Capabilities Reasoning
Feb 27, 2026Yuxuan Liu, Weikai Xu, Kun Huang +9Mobile GUI Agents Agentic Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China · MiLM Plus, Xiaomi Inc. · Nanyang Technological University +1
SPATIALALIGN: Aligning Dynamic Spatial Relationships in Video Generation
Feb 26, 2026Fengming Liu, Tat-Jen Cham, Chuanxia ZhengT2V Generation Direct Preference Optimization
College of Computing and Data Science, Nanyang Technological University, 50 Nanyang Avenue, Singapore 639798.
$φ$ -DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Models
Feb 26, 2026Thanh-Dat Truong, Huu-Thien Tran, Jackson Cothren +2Continual Learning Direct Preference Optimization
CVIU Lab, University of Arkansas, USA · Dep. of Geosciences, University of Arkansas, USA · Carnegie Mellon University, USA
Alignment-Weighted DPO: A principled reasoning approach to improve safety alignment
Feb 24, 2026Mengxuan Hu, Vivek V. Datla, Anoop Kumar +4LLM Alignment LLM Safety Alignment
University of Virginia · Capital One
JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation
Feb 22, 2026Kai Liu, Yanhao Zheng, Kai Wang +7Audio-Video Generation Audio-Visual Alignment
Zhejiang University · University of Toronto · National University of Singapore +3
Multi-Objective Alignment of Language Models for Personalized Psychotherapy
Feb 17, 2026Mehrab Beikzadeh, Yasaman Asadollah Salmanpour, Ashima Suvarna +4Personalized Language Model Alignment Multi-Objective Language Model Alignment
Department of Computer Science, University of California, Los Angeles, CA, USA · Department of Psychology, The University of Texas at Austin, Austin, TX, USA · Department of Psychiatry, NYU Grossman School of Medicine, New York, NY, USA

Apr 9, 2026J/K move · Enter open · S save

EditCaption: Human-Refined SFT and HAE-DPO for Image Editing Instruction Synthesis

Xiangyuan Wang, Honghao Cai, Yunhao Bai, Chao Hui+8

Peking University · Xiaohongshu Inc. · The Chinese University of Hong Kong, Shenzhen+2

Direct Preference Optimization Image Captioning

Open paper AskPDF arXiv

High-quality source-target image pairs with precise editing instructions are essential for instruction-guided image editing, yet constructing such training triplets at scale remains costly. Recent pipelines often rely on vision-language models to synthesize editing instructions automatically, but we find that strong VLMs still struggle to describe visual transformations between image pairs. In particular, they exhibit three recurring failure modes: orientation inconsistency, viewpoint ambiguity, and missing fine-grained attributes. In a human evaluation on 400 image pairs, several open-source VLM baselines produce critical-error rates above 47%, making many synthesized instructions unsuitable for downstream training. To address this, we propose EditCaption, a two-stage post-training pipeline for image editing instruction synthesis. First, we construct a 100K supervised fine-tuning dataset through GLM-based auto-captioning, EditScore filtering, and human refinement. Second, we collect 10K human-annotated preference pairs, where each rejected instruction is labeled with its primary error type and severity. Based on this dataset, we propose Hardness-Adaptive Error-Aware DPO (HAE-DPO), a task-adapted DPO objective that introduces an adaptive margin based on human-labeled severity, failure-mode type, and reference-model hardness. Experiments across three benchmarks demonstrate that our 235B model with SFT+HAE-DPO achieves state-of-the-art performance among open-source and closed models, scoring 4.720 on Eval-400, 4.672 on HQ-Edit, and 4.651 on ByteMorph-Bench -- surpassing Gemini-3-Pro on all three. Human evaluation confirms critical error rates drop from 47.75% to 17.50%, with correct rates improving from 41.75% to 70.25%, surpassing Gemini-3-Pro (66.00%).