Direct Preference Optimization

Also known as DPO

Direct Preference Optimization (DPO) studies how to align a model with preferences by training it to favor chosen responses over rejected ones, usually relative to a reference model. Research extends the basic pairwise objective with adaptive margins and weighting, token- or segment-level credit, methods for noisy or conflicting labels, and online or curriculum-based data selection. DPO is used for language-model alignment in helpfulness, safety, and reasoning, and is increasingly adapted to multimodal models and diffusion-based image, video, and audio generation. A central tension is improving preference alignment and robustness without sacrificing general capabilities, diversity, or reliable performance when preferences and inputs shift.

Latest papers

162 papers24 papers in the last four weeks, up 60% on the four weeks before. 0.1% of all new papers.

Bridging Data, Reasoning, and Alignment: A Unified Framework for Context-Aware Instruction-Following TTS
Sep 13, 2026Jingbin Hu, Luyu Wang, Wenjie Tian +7Text-to-Speech Speech Language Models
Shenzhen Pimei Technology Co., Ltd., Guangdong, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China · State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China
Direct Preference Density Alignment for Conversational Audio Equalization
Sep 11, 2026Ioannis Stylianou, Sven Ewan Shepstone, Jon Francombe +2RL for Language Models LLM Alignment
Department of Electronic Systems Aalborg University Aalborg, Denmark · Bang & Olufsen A/S Struer, Denmark
LOCUS: Task-Aware Low-Rank Post-Training for Token-Efficient Language Generation
Sep 10, 2026Dongfang ZhaoDirect Preference Optimization Controllable Text Generation
Preference Optimization with LALM Feedback for Continuous Autoregressive Non-Verbal Vocalization Generation
Sep 10, 2026Jingbin Hu, Qirui Zhan, Yuang Cao +7Nonverbal Vocalization Generation Direct Preference Optimization
Shenzhen Pimei Technology Co., Ltd., Guangdong, China · Yutu Zhineng, Beijing, China · State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China +1
You Can't Prefer Emotions You Don't Sample: Intensity Undershoot in DPO-Tuned LLMs
Sep 7, 2026Hyunwoo Kim, Usama KhalidDirect Preference Optimization Valence-Arousal Modeling
Department of Computer Software Hanyang University, Seoul
Beyond Single-Negative Preference: Multi-Negative DPO for LLM-Centric Historical Entity Linking
Sep 7, 2026Tien Nam Nguyen, Emanuela Boros, Ahmed Hamdi +3LLM Alignment Direct Preference Optimization
L3i, University of La Rochelle, La Rochelle, France · University of Toulouse, IRIT, Toulouse, France · University of Innsbruck, Austria +1
DIPLOMAT: Dialogue-Span-Aware Direct Preference Optimization for Polite Persuasive Workplace Negotiation Dialogues
Sep 6, 2026Bibhuti Jha, Rishikant Chigrupaatii, Priyanshu Priya +1Automated Negotiation Direct Preference Optimization
Department of Computer Science and Engineering, Indian Institute of Technology Patna, India · GREYC Laboratory, University of Caen Normandy, France
CopyShield: A Cross-Level Benchmark of Copyright Defenses in LLMs
Sep 1, 2026Maryam Alshehyari, Dushyant Singh Chauhan, Samuele Poppi +3LLM Safety Benchmarks Direct Preference Optimization
Computing and Mathematical Science Division Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)
PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization
Aug 31, 2026Boryeong Cho, Sumyeong Ahn, Se-Young YunPairwise Preference Learning Direct Preference Optimization
KAIST AI · KENTECH
Disentangling Optimization Scale from Preference Scale in DPO
Aug 27, 2026Ivan KruzhilovDirect Preference Optimization Preference Learning
Preference Optimization for Non-Verbal Vocalization Synthesis
Aug 25, 2026Haoyang Li, Chenglin Xu, Junchuan Zhao +4Nonverbal Vocalization Generation Preference Optimization
Nanyang Technological University, Singapore · LIGHTSPEED, Singapore · National University of Singapore, Singapore +2
FAR-DPO: Feasibility-Aware and Robust Direct Preference Optimization for Cyclic Peptide Design
Aug 20, 2026Guofeng Zhang, Rong Han, Xiaoyu Wang +4Molecular Generation Constrained Generative Modeling
Beijing University of Posts and Telecommunications, Beijing, China · Shenzhen University, Shenzhen, China · Tsinghua University, Beijing, China
PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment
Aug 20, 2026Jiawei Feng, Jiancan Wu, Xingyu Zhu +3Multimodal Large Language Models Vision-Language Alignment
University of Science and Technology of China Hefei, China · National University of Singapore Kent Ridge, Singapore
BICPO-VLA: Behavior-Identified Continuation Preference Optimization for Smooth Asynchronous Vision-Language-Action Control
Aug 14, 2026Ming Shang, Yuchen Huang, Jiaoyang Chen +8Vision-Language-Action Models Efficient VLA Model Inference
Beihang University, Beijing, China · Beijing Institute of Technology, Beijing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China +4
Context Blindness in DPO: Mitigating Object Hallucination in MLLMs via Context-Calibrated Preference Optimization
Aug 12, 2026Byungoh Ko, Jinyoung Park, Jongha Kim +3Direct Preference Optimization Object Hallucination in VLMs
Korea University, Seoul, Republic of Korea · KAIST, Daejeon, Republic of Korea
Reinforcing Step-level Reasoning for Effective Self-Correction in LLMs
Aug 12, 2026Vu Duc Anh, Nhat M. Hoang, Do Xuan Long +3LLM Self-Correction Direct Preference Optimization
Nanyang Technological University, Singapore · National University of Singapore · Institute for Infocomm Research (I2R), A*STAR +1
Se-DPO: Self-Evolving Token Credit for Direct Preference Optimization
Aug 10, 2026Wenxiao Zhao, Shu Wang, Ying Nian WuLLM Alignment Direct Preference Optimization
University of California, Los Angeles · Shanghai AI Laboratory
Linguistically-Aligned and Visually-Grounded Preference Optimization for Clinically-Augmented Medical Report Generation
Aug 9, 2026Qiang Hu, Yuxuan Luo, Yingjie Guo +4Radiology Report Generation Medical Report Generation
Huazhong University of Science and Technology
SAGA: Score-Weighted Adaptive Generation Alignment for Low-Resource Nordic Language Models
Aug 6, 2026Hoda Fakharzadehjahromy, Emil Wiman, Andreas Bueff +2LLM Alignment Direct Preference Optimization
Linköping University · University of Iceland
Positive-Unlabeled Preference Optimization For Chest X-ray Report Generation
Aug 5, 2026Yuta Kobayashi, Pradyun Ramesh, Muhammad Ahmed Chaudhry +5Radiology Report Generation Radiology VLMs
Columbia University · Stanford University · Emory University
AeroDPO: Unleashing Lightweight UAV Navigation with High-Fidelity Perception and Automated Preference Optimization
Aug 2, 2026Peng Xu, Chengcheng Wang, Shaohua WanCollision Avoidance Direct Preference Optimization
University of Electronic Science and Technology of China, Chengdu, China · 2Shenzhen Institute for Advanced Study, UESTC, Shenzhen, China
Rolling With Resistance: Preference-Optimized LLM Counselors Can Trade Goal Persistence for Relational Attunement in Motivational Interviewing
Jul 30, 2026Weiying Chen, Junlong Shen, Zhexuan TangDirect Preference Optimization Preference Optimization
University of Alberta · University of Shanghai for Science and Technology
TELLER: Dual-Path Iterative Preference Optimization for Table Entity Linking
Jul 29, 2026Yixin Peng, Kehao Li, Stefan DeckerDirect Preference Optimization Preference Optimization
RWTH Aachen, Ahornstraße 55, 52074 Aachen, Germany
SciForma: Structure-Faithful Generation of Scientific Diagrams
Jul 20, 2026Yuxuan Luo, Peng Zhang, Xinjie Zhang +3Scientific Visualization Direct Preference Optimization
Wangxuan Institute of Computer Technology, Peking University, China · State Key Lab of CAD & CG, Zhejiang University, China · Microsoft Research Asia
When Physical Preferences Meet Semantic Constraints: Physical and Semantic Direct Preference Optimization for Text-to-Video Generation
Jul 18, 2026Siwei Meng, Yawei Luo, Shu Zhang +1Physical Consistency in Video Generation Direct Preference Optimization
Computer Science and Engineering · University of Nevada, Reno · Reno, NV, USA +5
TD-DPO: Difference-Aware Preference Optimization for Mitigating Sycophancy in Clinical Autism Intervention Dialogue
Jul 17, 2026Shuzhong Lai, Junhong Lai, Chenxi Li +5LLM Sycophancy Direct Preference Optimization
1Nanhu Brain-Computer Interface Institute · 2MOE Frontiers Science Center for Brain and Brain-Machine Integration, Zhejiang University · College of Computer Science and Technology, Zhejiang University +3
Direct Image-to-Modern Vietnamese Translation of Han-Nom Manuscripts via Multimodal RLHF Preference Alignment
Jul 13, 2026Thi Kim Trang Vo, Nghia Hieu Nguyen, Ha Minh TanDirect Preference Optimization Preference Optimization
University of Information Technology, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam
Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories
Jul 12, 2026Yixiong Chen, Alan YuilleDirect Preference Optimization LLM Agents
Johns Hopkins University
Metadata-Free Meta-Reweighted Direct Preference Optimization under Noisy Preference Labels
Jul 9, 2026Hua Qu, Yifan Li, Xiaodong YuanLLM Alignment Meta-Learning
Xi’an Jiaotong University
D2PO: Optimizing Diffusion Samplers via Dynamic Preference
Jul 7, 2026Jinkyu Kim, Jinyoung Choi, Bohyung HanDiffusion Model Alignment Direct Preference Optimization
1ECE & 2IPAI, Seoul National University, Korea · 3AIGS, Ulsan National Institute of Science and Technology, Korea

Sep 13, 2026J/K move · Enter open · S save

Bridging Data, Reasoning, and Alignment: A Unified Framework for Context-Aware Instruction-Following TTS

Jingbin Hu, Luyu Wang, Wenjie Tian, Kangxiang Xia+6

Shenzhen Pimei Technology Co., Ltd., Guangdong, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China · State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China

Text-to-Speech Speech Language Models Direct Preference Optimization

Open paper AskPDF arXiv

The ISCSLP 2026 CoT-TTS Challenge requires TTS systems to generate Chain-of-Thought (CoT) reasoning from dialogue history before synthesizing contextually appropriate speech. While the official baseline establishes a unified architecture, it remains constrained by limited contextual comprehension, weak instruction fidelity, and suboptimal audio quality. We present a systematic optimization pipeline to address these limitations. First, we develop a data process framework that cleans raw data via FullSubNet denoising, Qwen3-ASR re-transcription, and Qwen3.5-35B-A3B-based history-CoT consistency analysis, while distilling 545K high-fidelity instruction samples using Qwen3-TTS and Seed-VC under strict quality filtration. Second, we propose a Context-Aware Direct Preference Optimization (CA-DPO) method. By employing a cascaded filtering strategy, ASR prescreening, LLM tournament ranking, and speaker similarity verification, we obtain high-confidence preference pairs that significantly enhance holistic ``Context

\rightarrow

CoT

\rightarrow

Speech'' consistency during DPO training. Third, we establish an evaluation method featuring a 500-sample test set and an LLM-as-Judge framework to independently assess reasoning and execution fidelity. Experiments demonstrate that our system significantly outperforms the baseline across all objective and subjective metrics, validating our data governance and alignment strategies.