Direct Preference Optimization

Also known as DPO

Direct Preference Optimization (DPO) studies how to align a model with preferences by training it to favor chosen responses over rejected ones, usually relative to a reference model. Research extends the basic pairwise objective with adaptive margins and weighting, token- or segment-level credit, methods for noisy or conflicting labels, and online or curriculum-based data selection. DPO is used for language-model alignment in helpfulness, safety, and reasoning, and is increasingly adapted to multimodal models and diffusion-based image, video, and audio generation. A central tension is improving preference alignment and robustness without sacrificing general capabilities, diversity, or reliable performance when preferences and inputs shift.

Latest papers

162 papers24 papers in the last four weeks, up 60% on the four weeks before. 0.1% of all new papers.

Unbiased Alignment for Large Language Models with Noisy Preferences
Jul 3, 2026Jialiang Wang, Xianming Liu, Xiong Zhou +2Pairwise Preference Learning LLM Alignment
1Harbin Institute of Technology · 2City University of Hong Kong.
Learning to Watch: Active Video Anomaly Understanding via Interleaved Policy Optimization
Jul 1, 2026Mengjingcheng Mo, Jiaxu Leng, Xinbo GaoVideo Understanding Direct Preference Optimization
School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, China · Chongqing College of Artificial Intelligence, Chongqing, China
ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMs
Jun 30, 2026Zhiyuan Yao, Zheren Fu, Zhixiao Zheng +3Cross-Attention Direct Preference Optimization
University of Science and Technology of China, Hefei, China · State Key Laboratory of Communication Content Cognition, People’s Daily Online, China · Huawei Technologies Ltd., China
Reliability-Prioritized Fine-Grained Generation in Multimodal Large
Jun 28, 2026Xiaomeng Fan, Wei Wu, Yuwei Wu +9Direct Preference Optimization Multimodal Large Language Models
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology · Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University · Department of Electrical and Computer System Engineering, Monash University
Reward-Free Code Alignment from Pretrained or Fine-Tuned LLM: Unpacking the Trade-offs for Code Generation
Jun 27, 2026Sanjeepan Sivapiran, Gias UddinDirect Preference Optimization Preference Alignment
York University, Canada
Weight-Space Geometry of Offline Reasoning Training
Jun 21, 2026Aleksandr Nikolich, Igor Kiselev, Vladimir Platonov +1RL for Language Models Direct Preference Optimization
1Keenable AI · 2Accenture · 3Yandex
Beyond Uniform Forgetting: A Study of Sequential Direct Preference Optimization Across Preference Settings
Jun 18, 2026Pranav Bhandari, Nicolas Fay, Amitava Datta +2Continual Learning for LLMs Direct Preference Optimization
1Network Analysis and Social Influence Modelling (NASIM) Lab · School of Physics Maths and Computing, The University of Western Australia · School of Psychological Science, The University of Western Australia +1
Which Pairs to Compare for LLM Post-Training?
Jun 17, 2026Jiangze Han, Vineet Goyal, Will MaPairwise Preference Learning Pairwise Comparison
Columbia University
Repeated post-training is not Self-improving: Diagnosing Scientific Amnesia in Continual DPO Pipelines
Jun 17, 2026Jianzhe Lin, Fei Wang, Xiaolin Li +2Continual Learning for LLMs Direct Preference Optimization
MetaAI
When Good Verifiers Go Bad: Silent Negative Transfer in Verifier-Guided VLM Training
Jun 12, 2026Jianzhe LinVLM Evaluation VLM Adaptation
MetaAI · Work done at Meta
Direct Preference Optimization for Chatbot Fine-Tuning: An Empirical Study
Jun 11, 2026Dezhi Yu, Yvonne Qiu, ShuoJia FuLLM Alignment Direct Preference Optimization
Boosting Direct Preference Optimization with Penalization
Jun 10, 2026Pengwei SunPairwise Preference Learning LLM Alignment
Stanford University.
Training LLMs to Enforce Multi-Level Instruction Hierarchies via Gravity-Weighted Direct Preference Optimization
Jun 9, 2026Lena S. Bolliger, Lena A. JägerDirect Preference Optimization LLM Safety Alignment
Department of Computational Linguistics, University of Zurich, Switzerland
DOG-DPO:Dynamic Optimization in Geometry for Safety Alignment
Jun 4, 2026Yi Nian, Tiankai Yang, Yudi Zhang +7Direct Preference Optimization LLM Safety Alignment
University of Southern California · Iowa State University · University of Wisconsin–Madison +4
P $^2$ -DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization
Jun 2, 2026Ruipeng Zhang, Zhihao Li, Haozhang Yuan +2VLM Robustness Direct Preference Optimization
Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence, School of Computer Science & Engineering, South China University of Technology · Pazhou Lab, Guangzhou, China · Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human, Guangzhou, China
Follow-Your-Preference++: Rethinking Preference Alignment for Image Inpainting
Jun 2, 2026Junkun Yuan, Yutao Shen, Toru Aonishi +2Pairwise Preference Learning Reward Modeling
Zhejiang University · The University of Tokyo · Tsinghua University
AgentPLM: Agentic Protein Language Models with Reasoning-Augmented Decoding for Protein Sequence Design
Jun 1, 2026Sahil Rahman, Maxx Richard RahmanRL for Language Models Direct Preference Optimization
Bedford College, London, United Kingdom · Saarland University, Saarbrücken, Germany
Citation Grounding: Detecting and Reducing LLM Citation Hallucinations via Legal Citation Graphs
May 30, 2026Volodymyr OvcharovLegal Citation Verification LLM Hallucination Mitigation
LEX AI LLC, Kyiv, Ukraine
The Representation-Rationalizability Tradeoff in Reward Learning
May 29, 2026Jing Dong, Yaoliang Yu, Pascal PourpartPairwise Preference Learning Reward Modeling
Vector Institute · University of Waterloo & Vector Institute
Learning from Fine-Grained Visual Discrepancies: Mitigating Multimodal Hallucinations via In-Context Visual Contrastive Optimization
May 29, 2026Haolin Deng, Xin Zou, Zhiwei Jin +3Vision-Language Models Direct Preference Optimization
The Hong Kong University of Science and Technology (Guangzhou) · OPPO AI Center · The Hong Kong University of Science and Technology
Recovering Diversity Without Losing Alignment: A DPO Recipe for Post-Trained LLMs
May 28, 2026Vinay Samuel, Yapei Chang, Mohit IyyerLLM Alignment Direct Preference Optimization
University of Maryland, College Park
Compositional Text-to-Image Generation Via Region-aware Bimodal Direct Preference Optimization
May 27, 2026Zhuohan Liu, Wujian Peng, Yitong Chen +1Diffusion Model Guidance Direct Preference Optimization
Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University · Shanghai Collaborative Innovation Center of Intelligent Visual Computing
AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates
May 27, 2026Shaolong Chen, Madalina Ciobanu, Qingqing Mao +1LLM Alignment Direct Preference Optimization
Incept Labs Houston, TX · Incept Labs, Houston, TX · Titan Holdings, San Francisco, CA
Reasoning Matters: Mitigate Hallucination in Multimodal Large Reasoning Models via Reasoning-Conditioned Preference Optimization
May 27, 2026Jiawei Kong, Hao Fang, Shunxiang Liao +5Multimodal Grounding Direct Preference Optimization
Tsinghua Shenzhen International Graduate School, Tsinghua University · Harbin Institute of Technology, Shenzhen
Curriculum Learning for Safety Alignment
May 25, 2026Sandeep Kumar, Virginia Smith, Chhavi YadavLLM Alignment Direct Preference Optimization
Carnegie Mellon University · Simons Institute, UC Berkeley
Towards Anatomically Plausible Human Image Generation via Synthetic Localized Preferences
May 25, 2026Bao Li, Yuliang Xiu, Zhen LiuDiffusion Model Alignment Direct Preference Optimization
Westlake University · The Chinese University of Hong Kong, Shenzhen
Token-weighted Direct Preference Optimization with Attention
May 21, 2026Chengyu Huang, Zhuohang Li, Sheng-Yen Chou +1Pairwise Preference Learning LLM Alignment
Cornell University · Vanderbilt University
CrossVLA: Cross-Paradigm Post-Training and Inference Optimization for Vision-Language-Action Models
May 21, 2026Zhi LiuDirect Preference Optimization VLA Model Post-Training
Tianjin University
How Much Online RL is Enough? Informative Rollouts for Offline Preference Optimization in RLVR
May 20, 2026Richa Verma, Balaraman RavindranDirect Preference Optimization RL for Language Model Reasoning
TCS Research Department of CSE IIT Madras Chennai, India · Department of Data Science & AI, Wadhwani School of Data Science & AI IIT Madras Chennai, India
PREFINE: Preference-Based Implicit Reward and Cost Fine-Tuning for Safety Alignment
May 20, 2026Richa Verma, Bavish Kulur, Sanjay Chawla +1Direct Preference Optimization Constrained RL
TCS Research, Department of CSE, IIT Madras India · Department of Computing Science, University of Alberta Canada · Qatar Computing Research Institute, Hamad Bin Khalifa University Qatar +1

Jul 3, 2026J/K move · Enter open · S save

Unbiased Alignment for Large Language Models with Noisy Preferences

Jialiang Wang, Xianming Liu, Xiong Zhou, Hui Liu+1

1Harbin Institute of Technology · 2City University of Hong Kong.

Pairwise Preference Learning LLM Alignment Direct Preference Optimization Learning with Noisy Labels Preference Learning

Open paper AskPDF arXiv

The alignment of large language models with human preferences is commonly achieved through Reinforcement Learning from Human Feedback or Direct Preference Optimization. However, these methods are vulnerable to the significant noise prevalent in real-world preference datasets. To address this critical issue, we present a theoretical framework for unbiased alignment, introducing the Unbiased Reward Model (URM) loss and the Unbiased Direct Preference Optimization (UDPO) loss. By mathematically correcting the distortion induced by preference noise, our novel objectives enable unbiased model training directly from noisy datasets, without requiring clean ground-truth supervision. We provide rigorous theoretical analyses demonstrating that our methods are noise-tolerant, parameter downward compatible, and classification-calibrated. Comprehensive experiments across diverse datasets demonstrate that our approaches outperform state-of-the-art baselines. Code available at: https://github.com/cswjl/unbiased-alignment.