Direct Preference Optimization

Also known as DPO

Overview

Direct Preference Optimization (DPO) studies how to align a model with preferences by training it to favor chosen responses over rejected ones, usually relative to a reference model. Research extends the basic pairwise objective with adaptive margins and weighting, token- or segment-level credit, methods for noisy or conflicting labels, and online or curriculum-based data selection. DPO is used for language-model alignment in helpfulness, safety, and reasoning, and is increasingly adapted to multimodal models and diffusion-based image, video, and audio generation. A central tension is improving preference alignment and robustness without sacrificing general capabilities, diversity, or reliable performance when preferences and inputs shift.

Latest papers

CardsList

174 papers23 papers in the last four weeks, up 28% on the four weeks before. 0.2% of all new papers.

  1. On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinforcement Learning

    Nov 6, 2026Haoyu Wang, Ru Wang, Yuanyuan Wang +2Test-Time RLRL for Language Model Reasoning

  2. When Scores Conflict with Preferences: Pre-Alignment Diagnosis and Adaptive Correction of Constraint Drift in DPO

    Nov 5, 2026Ping Liu, Yan YanDirect Preference OptimizationPreference Optimization

  3. Objective-Conditioned Active Preference Learning for Multi-Objective Alignment

    Nov 3, 2026Duc Anh Vu, Thong Nguyen, Hoang Pham +2LLM AlignmentMulti-Objective Language Model Alignment

  4. TopoDPO: Dense Preference Alignment for Structure-Sensitive Aerial Referring Segmentation

    Nov 1, 2026Huaiyuan Peng, Jisheng Dang, Wencan Zhang +6Referring Image SegmentationRemote Sensing Image Segmentation

  5. SecPO: Principled Adversarial Training for Prompt Injection Security

    Oct 30, 2026Sizhe Chen, Yibo Peng, Jaewon Chang +2Prompt Injection DefensePrompt Injection Attacks on AI Agents

  6. Reliability-Weighted Direct Preference Optimization: Penalized Weights for Robust Preference Learning

    Oct 30, 2026Bin Luo, Param Venkat Vivek Kesireddy, Anil VallepuDirect Preference OptimizationPreference Learning

  7. Aligning Large Language Models for Safety in African Languages: A Case Study of Hausa

    Oct 27, 2026Valdini Douglace Lemofouet, Blessing Ngozi Uzor, Paula Chikaodinaka Anyanwu +4LLM Safety AlignmentLanguage Model Safety Evaluation

  8. On the Sparsity of Direct Preference Optimization: Weight Disentanglement in the NTK Regime

    Oct 27, 2026Kensuke Sasaki, Issei SatoDirect Preference OptimizationLLM Alignment

  9. Exploring the Limits of the Delta Learning Hypothesis: Single-Pair Generator Biases in Direct Preference Optimization

    Oct 27, 2026Giuseppe Curci, Stefano Simonazzi, Andrea Molinari +1Direct Preference OptimizationLLM Alignment

  10. Aligning Large Language Models with Human Feedback: Mathematical Foundations and Algorithm Design

    Oct 22, 2026Siliang Zeng, Luca Viano, Chenliang Li +6LLM AlignmentRL from Human Feedback

  11. Stochastic Pruning of the DPO Computation Graph: 1.4x Faster Training and Lower Peak Memory

    Oct 21, 2026Yubing Wu, Wei Chen, Junmei Yang +2Direct Preference OptimizationMemory-Efficient Training

  12. ECHO: Emergent Common-Mode Hidden Offsets Drive Vocabulary Collapse in Preference Optimization

    Oct 21, 2026Yun Li, Ehsan Javanmardi, Simon Thompson +7Direct Preference OptimizationFine-Tuning Language Models

  13. Answering in the User's Variety: Calibrated Accommodation Pairs for Post-Training

    Oct 20, 2026Zhizhuo Kou, Heming Li, Jian Yang +4Dialect Bias in Language ModelsLanguage Model Post-Training

  14. TPO: Tri-level Distributionally Robust Learning for OOD Direct Preference Optimization

    Oct 20, 2026Chengtao Jian, Kai YangLLM AlignmentDirect Preference Optimization

  15. The Shape of the DPO Hyperparameter Landscape: What to Measure, How It Moves with Scale, and When Not to Transfer It

    Oct 20, 2026Hannan Mahadik, David Salinas, Jacek Golebiowski +1Direct Preference OptimizationHyperparameter Optimization

  16. Statistical Learning Theory for Direct Preference Optimization

    Oct 19, 2026Joel Wills, Yuanfan Li, Yiming Ying +1Direct Preference OptimizationPairwise Preference Learning

  17. Over-Defense in Coding Agents Begins at Silent Forks

    Oct 18, 2026Hanjun Luo, Xuecheng Liu, Xiucheng Zhang +5AI Coding AgentsRL for Code Generation

  18. Robust Direct Preference Optimization under Preference Label-Flip Attacks

    Oct 18, 2026Chenye Yang, Weiyu Xu, Lifeng LaiDirect Preference OptimizationAdversarial Robustness

  19. Manacá-1B-Instruct-2: Reproducible Safety Alignment for a Native Brazilian-Portuguese LLM. Negative DPO Results, a Controlled Composition Scan, and a Silent Failure Mode of the LLM Judge

    Oct 14, 2026Bruno Leonardo Santos Menezes, Carlos Leonardo Souza Cardoso, Fabio André Machado PortoPortuguese Language ModelsLLM Safety Alignment

  20. DPOison-VL: Clean Preference Poisoning in Vision-Language Direct Preference Optimization

    Oct 13, 2026Chengtao Jian, Kai YangData Poisoning AttacksAdversarial Attacks on VLMs

  21. Personalized Federated Direct Preference Optimization via LoRA Adapters

    Oct 9, 2026Zhanhong Jiang, Soumik SarkarPersonalized Language Model AlignmentDirect Preference Optimization

  22. CM-DPO: Constraint-Margin Direct Preference Optimization for LLM Planning

    Oct 6, 2026Rabimba Karanjai, Qun Gu, Hemanth Hegadehalli Madhavarao +8Direct Preference OptimizationLLM Planning

  23. LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization

    Oct 6, 2026Yang Qu, Yusheng Han, Chengjia Feng +1Direct Preference OptimizationLLM Alignment

  24. Geometry-Aware Preference Optimization for Text-to-Image Diffusion Models

    Oct 4, 2026Lei Wang, Zhen Wang, Yuexiang Xie +1Preference Optimization for Diffusion ModelsDiffusion Model Alignment

  25. Hallucination Reduction for LLM-Based Audio Understanding via Multimodal Direct Preference Optimization

    Oct 2, 2026Bebe Cosgrove, Aaron Isidore Grace, Weiran WangAudio UnderstandingAudio-Language Models

  26. GAW-PO: Preference Optimization with Gradient-Aligned Token Weights

    Oct 1, 2026Andreea Dutulescu, Stefan Ruseti, Mihai Masala +2LLM AlignmentDirect Preference Optimization

  27. Smaller Models, Better Rejects: Preference Distillation Scaling

    Sep 30, 2026Rui Cai, Wenhui Zhu, Xiwen Chen +15Direct Preference OptimizationKnowledge Distillation