Direct Preference Optimization

Also known as DPO

Direct Preference Optimization (DPO) studies how to align a model with preferences by training it to favor chosen responses over rejected ones, usually relative to a reference model. Research extends the basic pairwise objective with adaptive margins and weighting, token- or segment-level credit, methods for noisy or conflicting labels, and online or curriculum-based data selection. DPO is used for language-model alignment in helpfulness, safety, and reasoning, and is increasingly adapted to multimodal models and diffusion-based image, video, and audio generation. A central tension is improving preference alignment and robustness without sacrificing general capabilities, diversity, or reliable performance when preferences and inputs shift.

Latest papers

162 papers24 papers in the last four weeks, up 60% on the four weeks before. 0.1% of all new papers.

Preference Optimization Improves Definition Generation: Semantic Change in EU Institutional Sustainable Finance Disclosures
Oct 23, 2026Mariia Fedorova, Andrey Kutuzov, Gustavo Cadenas DelascioSemantic Change Detection Text Generation
University of Oslo
Aligning Large Language Models with Human Feedback: Mathematical Foundations and Algorithm Design
Oct 22, 2026Siliang Zeng, Luca Viano, Chenliang Li +6LLM Alignment RL from Human Feedback
University of Minnesota · EPFL · Texas A&M University +2
Stochastic Pruning of the DPO Computation Graph: 1.4x Faster Training and Lower Peak Memory
Oct 21, 2026Yubing Wu, Wei Chen, Junmei Yang +2Direct Preference Optimization Memory-Efficient Training
South China University of Technology · Shenzhen Technology University
ECHO: Emergent Common-Mode Hidden Offsets Drive Vocabulary Collapse in Preference Optimization
Oct 21, 2026Yun Li, Ehsan Javanmardi, Simon Thompson +7Direct Preference Optimization Fine-Tuning Language Models
Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan · TIER IV, Inc., Tokyo, Japan · Huawei
Answering in the User's Variety: Calibrated Accommodation Pairs for Post-Training
Oct 20, 2026Zhizhuo Kou, Heming Li, Jian Yang +4Dialect Bias in Language Models Language Model Post-Training
1The Hong Kong University of Science and Technology · 2The Hong Kong University of Science and Technology (Guangzhou) · Peking University
TPO: Tri-level Distributionally Robust Learning for OOD Direct Preference Optimization
Oct 20, 2026Chengtao Jian, Kai YangLLM Alignment Direct Preference Optimization
School of Computer Science and Technology, Tongji University, Shanghai, China · Shenzhen Loop Area Institute, Shenzhen, China · Shanghai Research Institute for Intelligent Autonomous Systems, Shanghai, China
The Shape of the DPO Hyperparameter Landscape: What to Measure, How It Moves with Scale, and When Not to Transfer It
Oct 20, 2026Hannan Mahadik, David Salinas, Jacek Golebiowski +1Direct Preference Optimization Hyperparameter Optimization
ELLIS Institute Tübingen, Germany · ELLIS Institute Tübingen & Prior Labs, Germany · distil labs, Germany
Statistical Learning Theory for Direct Preference Optimization
Oct 19, 2026Joel Wills, Yuanfan Li, Yiming Ying +1Direct Preference Optimization Pairwise Preference Learning
School of Mathematics and Statistics, The University of Sydney
Over-Defense in Coding Agents Begins at Silent Forks
Oct 18, 2026Hanjun Luo, Xuecheng Liu, Xiucheng Zhang +5AI Coding Agents RL for Code Generation
New York University · New York University Abu Dhabi · Carnegie Mellon University +3
Robust Direct Preference Optimization under Preference Label-Flip Attacks
Oct 18, 2026Chenye Yang, Weiyu Xu, Lifeng LaiDirect Preference Optimization Adversarial Robustness
Department of Electrical and Computer Engineering, University of California, Davis, Davis, CA 95616, USA · Department of Electrical and Computer Engineering, University of Iowa, Iowa City, IA 52242, USA
Manacá-1B-Instruct-2: Reproducible Safety Alignment for a Native Brazilian-Portuguese LLM. Negative DPO Results, a Controlled Composition Scan, and a Silent Failure Mode of the LLM Judge
Oct 14, 2026Bruno Leonardo Santos Menezes, Carlos Leonardo Souza Cardoso, Fabio André Machado PortoPortuguese Language Models LLM Safety Alignment
National Laboratory for Scientific Computing (LNCC), Institute of Artificial Intelligence, Petrópolis, Brazil
DPOison-VL: Clean Preference Poisoning in Vision-Language Direct Preference Optimization
Oct 13, 2026Chengtao Jian, Kai YangData Poisoning Attacks Adversarial Attacks on VLMs
School of Computer Science and Technology, Tongji University, China · Shenzhen Loop Area Institute, China · Shanghai Research Institute for Intelligent Autonomous Systems, China
Personalized Federated Direct Preference Optimization via LoRA Adapters
Oct 9, 2026Zhanhong Jiang, Soumik SarkarPersonalized Language Model Alignment Direct Preference Optimization
Translational AI Center, Iowa State University, Ames, IA 50011 USA · Department of Mechanical Engineering, Iowa State University, Ames, IA 50011 USA
CM-DPO: Constraint-Margin Direct Preference Optimization for LLM Planning
Oct 6, 2026Rabimba Karanjai, Qun Gu, Hemanth Hegadehalli Madhavarao +8Direct Preference Optimization LLM Planning
PayPal AI Lab
LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization
Oct 6, 2026Yang Qu, Yusheng Han, Chengjia Feng +1Direct Preference Optimization LLM Alignment
Northeastern University
Geometry-Aware Preference Optimization for Text-to-Image Diffusion Models
Oct 4, 2026Lei Wang, Zhen Wang, Yuexiang Xie +1Preference Optimization for Diffusion Models Diffusion Model Alignment
Sun Yat-Sen University, Guangzhou, China · Alibaba Group, Hangzhou, China
Hallucination Reduction for LLM-Based Audio Understanding via Multimodal Direct Preference Optimization
Oct 2, 2026Bebe Cosgrove, Aaron Isidore Grace, Weiran WangAudio Understanding Audio-Language Models
Smith College, Northampton, USA · University of Iowa, Iowa City, USA
GAW-PO: Preference Optimization with Gradient-Aligned Token Weights
Oct 1, 2026Andreea Dutulescu, Stefan Ruseti, Mihai Masala +2LLM Alignment Direct Preference Optimization
National University of Science and Technology POLITEHNICA Bucharest · NVIDIA
Smaller Models, Better Rejects: Preference Distillation Scaling
Sep 30, 2026Rui Cai, Wenhui Zhu, Xiwen Chen +15Direct Preference Optimization Knowledge Distillation
AI Agentic Modeling and Foundation Team, LinkedIn · University of California, Davis · Arizona State University +3
Uncertainty-Normalized Margins for Direct Preference Optimization
Sep 29, 2026Sadegh Khorasani, Petrus Mikkola, Matthias GrossglauserPairwise Preference Learning Direct Preference Optimization
School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland · University of Helsinki, Helsinki, Finland
GaugeVLM: Structuring Spatial Supervision with Measured Geometric Interventions
Sep 29, 2026Hongbo Wang, Zihan Lin, Wenkui Yang +5Visual Spatial Reasoning Direct Preference Optimization
NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences +2
Beyond Binary Preferences: Graded Preference Optimization for Limb-Motion Captioning
Sep 29, 2026Yanan Wang, Tingsong Li, Kaixun Jiang +3Direct Preference Optimization Video-Language Model Evaluation
Zhejiang University · Alibaba Token Hub, Alibaba Group · University of Science and Technology of China +2
BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment
Sep 28, 2026Antonio Ferrara, Alberto Rumi, Francesco BonchiLLM Alignment Debiased ML
Intesa Sanpaolo AI Research
FestDPO: Few-step Generator Alignment with Direct Preference Optimization
Sep 28, 2026Jaewoo Lee, Kyuil Sim, Hyeongyu Kang +3Direct Preference Optimization T2I Generation
KAIST · MongooseAI · Omelet
Masking Frequent Tokens Sharpens Direct Preference Optimization
Sep 26, 2026Harshvardhan Saini, Samyak Jha, Yiming Tang +1LLM Alignment Direct Preference Optimization
Indian Institute of Technology Dhanbad · National University of Singapore
Contamination, Prior, or Evidence? Decomposing and Training Evidence Use in Whole-Slide Vision-Language Models
Sep 26, 2026Wenhao Zhang, Zhongliang Zhou, Shiyuan Zhang +6Pathology VQA Medical VLMs
University of Virginia, Charlottesville · Merck & Co., Inc.
Which Objectives Need a Dial? Predicting Objective Conflict and Covering Trade-offs in Steerable Pluralistic Alignment
Sep 22, 2026David Tsoi, Esra DönmezDirect Preference Optimization Multi-Objective Language Model Alignment
Department of Computer Science, University of Stuttgart · Institute for Natural Language Processing, University of Stuttgart · Interchange Forum for Reflecting on Intelligent Systems, University of Stuttgart
Interactive TTS: Dynamic Speaking Style Adaptation for Expressive Speech Synthesis
Sep 22, 2026Wenjie Tian, Kangxiang Xia, Jingbin Hu +7TTS Synthesis Direct Preference Optimization
Alibaba ATH Token Foundry
Style-Debiased DPO: Updating LLM Knowledge with Factuality-Aware Synthetic Preference Data
Sep 15, 2026Takayuki Yamamoto, Daisuke KawaharaKnowledge Editing Direct Preference Optimization
Waseda University
Beyond Safe Answers: Segment-Aware Listwise Alignment for Reasoning Safety in Large Reasoning Models
Sep 14, 2026JungMin Yun, Junehyoung Kwon, Hayeong Ryu +3LLM Alignment Direct Preference Optimization
Department of Artificial Intelligence, Chung-Ang University · Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University

Oct 23, 2026J/K move · Enter open · S save

Preference Optimization Improves Definition Generation: Semantic Change in EU Institutional Sustainable Finance Disclosures

Mariia Fedorova, Andrey Kutuzov, Gustavo Cadenas Delascio

University of Oslo

Semantic Change Detection Text Generation Direct Preference Optimization

Open paper AskPDF arXiv

Dictionary-like definition generators are known to be useful for lexical semantic change detection (LSCD). However, they suffer from overly granular sense distribution, producing excessive number of definitions for one and the same lexical sense. In this work, we show that fine-tuning with Direct Preference Optimization (DPO) helps to overcome this problem. We test our approach both quantitatively on a curated academic LSCD benchmark and qualitatively on real-world data for a practical task of auditing diachronic changes in sustainability terminology within project documentation of EU development finance. All models, code and predictions produced in this study are publicly available at https://github.com/ltgoslo/dpo-for-definition-generation.