Direct Preference Optimization

Follow this field Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

8 new papers

A weekly snapshot of new work published in Direct Preference Optimization.

Period ending 2026-09-14

15 new papers

A weekly snapshot of new work published in Direct Preference Optimization.

Period ending 2026-09-07

14 new papers

A weekly snapshot of new work published in Direct Preference Optimization.

Inside this field

Focused directions

Focused topic

Direct Preference Optimization

Within Direct Preference Optimization

85 papersExplore

Focused topic

Pairwise Preferences

Within Direct Preference Optimization

66 papersExplore

Focused topic

Preference Alignment

Within Direct Preference Optimization

58 papersExplore

Focused topic

Preference Learning

Within Direct Preference Optimization

53 papersExplore

Focused topic

Model Preference

Within Direct Preference Optimization

30 papersExplore

Focused topic

Robust Preference Optimization

Within Direct Preference Optimization

21 papersExplore

289 papers

Latest in Direct Preference Optimization

Browse all topics →

Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment
May 20, 2026Zhiqin Yang, Yonggang Zhang, Wei Xue +3Direct Preference Optimization Reinforcement Learning From Human Feedback
Distributed Direct Preference Optimization
May 20, 2026Zhanhong JiangDirect Preference Optimization Federated Reinforcement Learning
Spectral Souping: A Unified Framework for Online Preference Alignment
May 19, 2026Yinlam Chow, Guy Tennenholtz, Ted Yun +4Preference Alignment Large Language Model Adaptation
When Preference Labels Fall Short: Aligning Diffusion Models from Real Data
May 19, 2026Weiyan Chen, Weijian Deng, Yao Xiao +5Preference Alignment Generative Models
Efficient Elicitation of Collective Disagreements
May 19, 2026Mohamed Ouaguenouni, Felipe Garrido-Lucero, Umberto Grandi +2Disagreement Pairwise Preferences
PrefBench: Evaluating Zero-Shot LLM Agents in Hidden-Preference Personalized Pricing Negotiations
May 19, 2026Yingjie LeiNegotiation Agentic Commerce
Progressive Autonomy as Preference Learning: A Formalization of Trust Calibration for Agentic Tool Use
May 18, 2026Changkun OuAgentic Tool Use Preference Learning
GAMMA: Global Bit Allocation for Mixed-Precision Models under Arbitrary Budgets
May 18, 2026Zhangyang Yao, Haiyan Zhao, Haoyu Wang +3Mixed-Precision Quantization Token Budgets
From Volume to Value: Preference-Aligned Memory Construction for On-Device RAG
May 18, 2026Changmin Lee, Jaemin Kim, Taesik GongRetrieval-Augmented Generation Systems Preference Alignment
Pairwise Preference Reward and Group-Based Diversity Enhancement for Superior Open-Ended Generation
May 18, 2026Guining Cao, Jiaxin Peng, Chu Zeng +3Preference Alignment Diversity
Learning What Evaluators Value: A Reliable Approach to Modeling Evaluator Preferences
May 15, 2026Madeline Celi Kitch, Nihar B. ShahPairwise Preferences Mismatch
Contexting as Recommendation: Evolutionary Collaborative Filtering for Context Engineering
May 15, 2026Jiachen Zhu, Zhuoying Ou, Congmin Zheng +9Collaborative Filtering Recommendation
Generalized Priority-Aware Shapley Value
May 14, 2026Kiljae Lee, Ziqi Liu, Weijing Tang +1Shapley Value Prioritization
Direct Preference Optimization for English-Mandarin Code-Switching Speech Recognition in Audio LLMs
May 13, 2026Trung Nguyen Quang, Cheng Yi Lewis Won, Minh Duc Pham +3Code-Switching Large Audio Language Models
Reducing Hallucination in Vision-Language Models via Stage-wise Preference Optimization under Distribution Shift
May 13, 2026Qinwu XuLarge Language Model Hallucination Hallucination Mitigation
Probing Persona-Dependent Preferences in Language Models
May 13, 2026Oscar Gilg, Pierre Beckmann, Daniel Paleka +1Pairwise Preferences
Improving Code Translation with Syntax-Guided and Semantic-aware Preference Optimization
May 13, 2026Yuhan Wu, Huan Zhang, Wei Cheng +3Code Generation Robust Preference Optimization
Learning to Decide with AI Assistance under Human-Alignment
May 12, 2026Nina Corvelo Benz, Eleni Straitouri, Manuel Gomez-RodriguezArtificial Intelligence Alignment Preference Learning
Semantic Reward Collapse and the Preservation of Epistemic Integrity in Adaptive AI Systems
May 12, 2026William ParrisEpistemic Actions Robust Preference Optimization
BSO: Safety Alignment Is Density Ratio Matching
May 12, 2026Tien-Phat Nguyen, Truong Nguyen, Thin Nguyen +3Safety Alignment Large Language Model Alignment
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
May 12, 2026Truong Nguyen, Tien-Phat Nguyen, Linh Ngo Van +3Direct Preference Optimization Token-Level Uncertainty
SyncDPO: Enhancing Temporal Synchronization in Video-Audio Joint Generation via Preference Learning
May 12, 2026Xin Cheng, Xihua Wang, Ying Ba +5Joint Audio-Video Generation Spatio-Temporal Coherence
Variance-aware Reward Modeling with Anchor Guidance
May 12, 2026Shuxing Fang, Ruijian Han, Liangyu Zhang +1Process Reward Modeling Pairwise Preferences
Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansion
May 12, 2026ShiYing Huang, Liang Lin, Yuer Li +6Preference Alignment Ceiling
Macro: Enhancing Multilingual Counterfactual Explanations through Alignment-as-Preference Optimization
May 12, 2026Yilong Wang, Qianli Wang, Bohao Chu +3Counterfactual Explanation Counterfactual Reasoning
A Theory of Time-Sensitive Language Generation: Sparse Hallucination Beats Mode Collapse
May 11, 2026Atul Ganju, Travis McVoy, Shaddin Dughmi +1Deadlines Large Language Model Hallucination
When to Ask a Question: Understanding Communication Strategies in Generative AI Tools
May 11, 2026Charlotte Park, Kate Donahue, Manish RaghavanGenerative Artificial Intelligence Elicitation
Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training
May 11, 2026Christian Moya, Alex Semendinger, Guang Lin +1Preference Learning Direct Preference Optimization
DGPO: Beyond Pairwise Preferences with Directional Consistent Groupwise Optimization
May 11, 2026Mengyi Deng, Zhiwei Li, Xin Li +4Direct Preference Optimization Group Relative Policy Optimization
MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization
May 11, 2026Rohan Surana, Xintong Li, Sheldon Yu +7Direct Preference Optimization Multi-Objective Policy Optimization
ProteinOPD: Towards Effective and Efficient Preference Alignment for Protein Design
May 11, 2026Yulin Zhang, He Cao, Zihao Jiang +6Protein Design Preference Alignment
Learning the Preferences of a Learning Agent
May 9, 2026Karim Abdel Sadek, Mark Bedaywi, Rhys Gould +1Preference Learning Entropy Regularized Reinforcement Learning
Few-Shot Truly Benign DPO Attack for Jailbreaking LLMs
May 9, 2026Sangyeon Yoon, Wonje Jeung, Yoonjun Cho +2Benign Refusals
Compressed Video Aggregator: Content-driven Module for Efficient Micro-Video Recommendation
May 9, 2026Yang Xiao, Huiyuan Chen, Kaiyuan Deng +5Videos Visual Embeddings
Mechanistic Analysis of Alignment Algorithms in Language Models
May 9, 2026Aarush Sinha, Ishan Garg, Veeraraju Elluru +2Large Language Model Alignment Latent Representation Alignment
$ξ$ -DPO: Direct Preference Optimization via Ratio Reward Margin
May 9, 2026Zhengyuan Fan, Zhonghua Wu, Yuxuan Du +1Direct Preference Optimization Bias Learning
Embeddings for Preferences, Not Semantics
May 8, 2026Carter Blair, Ariel D. Procaccia, Milind TambePairwise Preferences Domain-Invariant Text Embeddings
Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph
May 8, 2026Ning Liu, Chuanneng Sun, Kristina Klinkner +1Direct Preference Optimization Pairwise Preferences
Nash without Numbers: A Social Choice Approach to Mixed Equilibria in Context-Ordinal Games
May 8, 2026Ian Gemp, Crystal Qian, Marc Lanctot +1Nash Equilibrium Equilibrium
Diffusion-APO: Trajectory-Aware Direct Preference Alignment for Video Diffusion Transformers
May 8, 2026Jingyuan Zhu, Biaolong Chen, Le Zhang +3Pre-Trained Video Diffusion Models Robust Preference Optimization
Confidence-Aware Alignment Makes Reasoning LLMs More Reliable
May 8, 2026Kejia Chen, Jiawen Zhang, Yihong Wu +5Large Language Model Reasoning Robust Preference Optimization
Goal-Conditioned Supervised Learning for LLM Fine-Tuning
May 8, 2026Shijun Li, Kaiwen Dong, Xiang Gao +1Large Language Model Fine-Tuning Large Language Model Alignment
Multi-Objective Constraint Inference using Inverse reinforcement learning
May 7, 2026Syed Ihtesham Hussain Shah, Floris den Hengst, Aneta Lisowska +1Constrained Reinforcement Learning Inference-Time Control
Preference Instability in Reward Models: Detection and Mitigation via Sparse Autoencoders
May 7, 2026Shunchang Liu, Xin Chen, Belen Martin Urcelay +1Preference Learning Process Reward Modeling
Arena as Offline Reward: Efficient Fine-Grained Preference Optimization for Diffusion Models
May 7, 2026Zhikai Li, Yue Zhao, Edward Zhongwei Zhang +4Robust Preference Optimization Direct Preference Optimization
Fusion in Your Way: Aligning Image Fusion with Heterogeneous Demands via Direct Preference Optimization
May 7, 2026Weijian Su, Songqian Zhang, Yuqi Han +3Visible Image Fusion Infrared-Visible Image Fusion
PREFER: Personalized Review Summarization with Online Preference Learning
May 7, 2026Millend Roy, Agostino Capponi, Vineet GoyalSummarization Preference Learning
Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization
May 6, 2026Xin Yu, Liuchen Liao, Yiwen Zhang +3Unsupervised On-Policy Self-Distillation On-Policy Distillation
Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models
May 6, 2026Huatian Zhang, Zhendong Mao, Lei Zhang +1Large Language Model Uncertainty Direct Preference Optimization
Threshold-Guided Optimization for Visual Generative Models
May 6, 2026Jinbin Bai, Yu Lei, Qingyu Shi +6Preference Alignment Training-Free Reward Alignment
RLearner-LLM: Balancing Logical Grounding and Fluency in Large Language Models via Hybrid Direct Preference Optimization
May 6, 2026Qiming Bao, Juho Leinonen, Paul Denny +1Large Language Model Alignment Large Language Models(Llms
Towards General Preference Alignment: Diffusion Models at Nash Equilibrium
May 6, 2026Jiaming Hu, Jiamu Bai, Haoyu Wang +2Preference Alignment Direct Preference Optimization
From Language to Logic: A Theoretical Architecture for VLM-Grounded Safe Navigation
May 5, 2026Kristy Sakano, Kalonji Harrington, Mumu XuVision-Language Navigation Object Goal Navigation
U-Define: Designing User Workflows for Hard and Soft Constraints in LLM-Based Planning
May 4, 2026Christine P Lee, Xinyu Jessica Wang, Aws Albarghouthi +2Workflow Automation and Optimization Structural Constraints
Gradient-Gated DPO: Stabilizing Preference Optimization in Language Models
May 4, 2026Inoussa Mouiche, Abderaouf BahiDirect Preference Optimization
Efficient Preference Poisoning Attack on Offline RLHF
May 4, 2026Chenye Yang, Weiyu Xu, Lifeng LaiReinforcement Learning From Human Feedback Direct Preference Optimization
VLA-ATTC: Adaptive Test-Time Compute for VLA Models with Relative Action Critic Model
May 2, 2026Wenhao Li, Xiu Su, Yichao Cao +5Vision-Language-Action Models Critic-Free Reinforcement Learning
Online Self-Calibration Against Hallucination in Vision-Language Models
May 1, 2026Minghui Chen, Chenxu Yang, Hengjie Zhu +3Large Vision Language Models Large Language Model Hallucination
TUR-DPO: Topology- and Uncertainty-Aware Direct Preference Optimization
Apr 30, 2026Abdulhady Abas Abdullah, Fatemeh Daneshfar, Seyedali Mirjalili +1Direct Preference Optimization Large Language Model Post-Training
Mind the Gap: Structure-Aware Consistency in Preference Learning
Apr 30, 2026Mehryar Mohri, Yutao ZhongLarge Language Model Alignment Direct Preference Optimization

May 20, 2026 · cs.AIJ/K move · Enter open · S save

Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment

Zhiqin Yang, Yonggang Zhang, Wei Xue, Dong Fang+2

1The Hong Kong University of Science and Technology · 2LIGHTSPEED · 3Hong Kong Baptist University.

Direct Preference Optimization (DPO) has emerged as a popular alternative to Reinforcement Learning from Human Feedback (RLHF), offering theoretical equivalence with simpler implementation. We prove this equivalence is conditional rather than universal, depending on an implicit assumption frequently violated in practice: the RLHF-optimal policy must prefer human-preferred responses. When this assumption fails, DPO optimizes relative advantage over the reference policy rather than absolute alignment with human preferences, leading to pathological convergence where policies decrease DPO loss while preferring dispreferred responses. We characterize when this assumption is violated, show the existence of an undesirable solution space, and prove that DPO and RLHF optimize fundamentally different objectives in such cases. To address this, we introduce Constrained Preference Optimization (CPO), augmenting RLHF with constraints for provable alignment. We further provide a geometric interpretation through soft margin ranking, revealing that DPO implements margin ranking with potentially negative targets. Our theoretical analysis establishes when DPOs' guarantees hold and provides solutions preserving simplicity with provable alignment. Comprehensive experiments on standard benchmarks demonstrate that CPO achieves state-of-the-art performance. Code is available at: https://github.com/visitworld123/CPO.

Direct Preference Optimization Reinforcement Learning From Human Feedback Feature Alignment Equivalence

Open paper Ask about it PDF