Direct Preference Optimization

Also known as DPO

Direct Preference Optimization (DPO) studies how to align a model with preferences by training it to favor chosen responses over rejected ones, usually relative to a reference model. Research extends the basic pairwise objective with adaptive margins and weighting, token- or segment-level credit, methods for noisy or conflicting labels, and online or curriculum-based data selection. DPO is used for language-model alignment in helpfulness, safety, and reasoning, and is increasingly adapted to multimodal models and diffusion-based image, video, and audio generation. A central tension is improving preference alignment and robustness without sacrificing general capabilities, diversity, or reliable performance when preferences and inputs shift.

Latest papers

162 papers24 papers in the last four weeks, up 60% on the four weeks before. 0.1% of all new papers.

Continuous-Utility Direct Preference Optimization
Jan 31, 2026Muhammad Ahmed Mohsin, Muhammad Umer, Emily FoxLLM Alignment Direct Preference Optimization
Stanford University, Stanford, California, USA
D-STEER: Two-Way Control of DPO-Induced Safety Behavior in Activation Space
Dec 3, 2025Samarth Raina, Saksham Aggarwal, Amit Dhanda +3Causal Interventions in Language Models Activation Steering
IIIT Delhi · Microsoft · Apple (USA) +2
Overcoming State Inertia: Minimally Invasive Temporal Alignment for Evolving Contexts
Dec 3, 2025Yijun LiaoLLM Alignment Direct Preference Optimization
Understanding Negative Learning in Large Language Model Fine-tuning
Nov 30, 2025Yue Wang, Qizhou Wang, Zizhuo Zhang +3LLM Fine-Tuning RL Fine-Tuning
TMLR Group, Department of Computer Science, Hong Kong Baptist University · RIKEN AIP · The University of Tokyo
Hierarchical Alignment: Block-Level LoRA-DPO for Preference Tuning in Large Language Models
Oct 14, 2025Yishen Chen, Qi Dong, Yukun ZhangLLM Alignment Preference Alignment
Fudan University Shanghai, China · The Chinese University of Hong Kong Hong Kong, China
Adaptive Margin RLHF via Preference over Preferences
Sep 26, 2025Yaswanth Chittepu, Prasann Singhal, Greg Durrett +1Pairwise Preference Learning Reward Modeling
University of Massachusetts Amherst · University of Texas at Austin
DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)
Jul 10, 2025Wenxuan Zhou, Shujian Zhang, Brice Magdalou +4Direct Preference Optimization Human Preference Modeling
Google DeepMind · CEE-M, Univ Montpellier, CNRS, INRAE, Institut Agro · Work done while at Google DeepMind +1
A Technical Survey of Reinforcement Learning Techniques for Large Language Models
Jul 5, 2025Saksham Sahai Srivastava, Vaneet AggarwalRL for Language Models LLM Alignment
University of Georgia, Athens, GA, USA · Purdue University, West Lafayette, IN, USA
InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models
May 20, 2025Yanggan Gu, Yuanyi Wang, Zhaoyi Yan +4Language Model Merging LLM Alignment
The Hong Kong Polytechnic University (PolyU) · Zhejiang University · PolyU-Daya Bay Technology and Innovation Research Institute
Reinforcement Learning from Human Feedback
Apr 16, 2025Nathan LambertLLM Alignment Direct Preference Optimization
Dual-Difficulty Curriculum Learning for Direct Preference Optimization
Apr 10, 2025Mengyang Li, Haozhan Geng, Zhong Zhang +1Pairwise Preference Learning LLM Alignment
Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission Tianjin Normal University Tianjin, China
Distributionally Robust Reinforcement Learning with Human Feedback
Mar 1, 2025Debmalya Mandal, Paulius Sasnauskas, Goran RadanovicDirect Preference Optimization LLM Fine-Tuning
Dept. of Computer Science University of Warwick, UK · Department of Computing Science University of Alberta, Edmonton, Canada · Max-Planck Institute for Software Systems, Germany

Jan 31, 2026J/K move · Enter open · S save

Continuous-Utility Direct Preference Optimization

Muhammad Ahmed Mohsin, Muhammad Umer, Emily Fox

Stanford University, Stanford, California, USA

LLM Alignment Direct Preference Optimization Preference Optimization LLM Reasoning

Open paper AskPDF arXiv

Large language model reasoning is often treated as a monolithic capability, relying on binary preference supervision that fails to capture partial progress or fine-grained reasoning quality. We introduce continuous utility direct preference optimization (CU-DPO), a framework that aligns models to a portfolio of prompt-based cognitive strategies by replacing binary labels with continuous scores that capture fine-grained reasoning quality. We prove that learning with K strategies yields a Theta(K log K) improvement in sample complexity over binary preferences and that DPO converges to the entropy-regularized utility-maximizing policy. To exploit this signal, we propose a two-stage pipeline: (i) strategy selection, which optimizes the model to choose the best strategy via best-vs-all comparisons, and (ii) execution refinement, which trains correct execution using margin-stratified pairs. The framework is domain-agnostic: any task admitting cognitively distinct solution strategies and a decomposable continuous utility signal can be incorporated into the portfolio. On mathematical reasoning benchmarks, CU-DPO improves strategy selection accuracy from 35-46% to 68-78% across seven base models, yielding downstream reasoning gains of up to +6.6 points on in-distribution datasets with effective out-of-distribution transfer. CU-DPO demonstrates consistent gains on code generation and causal reasoning benchmarks, confirming generalization beyond the mathematical domain.