Backtranslation Augmented Direct Preference Optimization

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, up 75% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 50

All topics

Uncertainty-Normalized Margins for Direct Preference Optimization
Sep 29, 2026Sadegh Khorasani, Petrus Mikkola, Matthias GrossglauserBacktranslation Augmented Direct Preference Optimization Preference Alignment Learning
School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland · University of Helsinki, Helsinki, Finland
BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment
Sep 28, 2026Antonio Ferrara, Alberto Rumi, Francesco BonchiLarge Language Model Alignment Backtranslation Augmented Direct Preference Optimization
Intesa Sanpaolo AI Research
Latent evolving World Action Model
Sep 23, 2026Xueji Fang, Boqiang Duan, Hua Wu +2Action Generation World Models
Zhejiang University · Westlake University · Baidu Inc.
Style-Debiased DPO: Updating LLM Knowledge with Factuality-Aware Synthetic Preference Data
Sep 15, 2026Takayuki Yamamoto, Daisuke KawaharaBacktranslation Augmented Direct Preference Optimization Paraphrasing
Waseda University
Beyond Safe Answers: Segment-Aware Listwise Alignment for Reasoning Safety in Large Reasoning Models
Sep 14, 2026JungMin Yun, Junehyoung Kwon, Hayeong Ryu +3Large Reasoning Models Reasoning Skills
Department of Artificial Intelligence, Chung-Ang University · Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University
Direct Preference Density Alignment for Conversational Audio Equalization
Sep 14, 2026Ioannis Stylianou, Sven Ewan Shepstone, Jon Francombe +2Backtranslation Augmented Direct Preference Optimization Large Language Model Alignment
Department of Electronic Systems Aalborg University Aalborg, Denmark · Bang & Olufsen A/S Struer, Denmark
Direct Diversity Optimization for Diverse Successful Trajectories in Preference Post-Training
Sep 9, 2026Junwon Ko, Dong-Jae Lee, Minchan Kwon +2Post-Training Backtranslation Augmented Direct Preference Optimization
School of Electrical Engineering, KAIST Daejeon, Republic of Korea
PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization
Aug 31, 2026Boryeong Cho, Sumyeong Ahn, Se-Young YunBacktranslation Augmented Direct Preference Optimization Preference Learning
KAIST AI · KENTECH
SSPO: Structure-Aware Similarity-Weighted Preference Optimization for Neural Combinatorial Optimization
Aug 12, 2026Yuanyu Li, Jintao Xu, Zijiang Liu +6Neural Combinatorial Optimization Backtranslation Augmented Direct Preference Optimization
Supply Chain Tech Team Y, JD.com, Beijing, China · Department of Industrial Engineering, Tsinghua University, Beijing, China · Faculty of Engineering and Faculty of Business and Economics, The University of Hong Kong, Hong Kong, China
Context Blindness in DPO: Mitigating Object Hallucination in MLLMs via Context-Calibrated Preference Optimization
Aug 12, 2026Byungoh Ko, Jinyoung Park, Jongha Kim +3Hallucination Mitigation Object Hallucination
Korea University, Seoul, Republic of Korea · KAIST, Daejeon, Republic of Korea
Se-DPO: Self-Evolving Token Credit for Direct Preference Optimization
Aug 10, 2026Wenxiao Zhao, Shu Wang, Ying Nian WuBacktranslation Augmented Direct Preference Optimization Credit Assignment
University of California, Los Angeles · Shanghai AI Laboratory
Linguistically-Aligned and Visually-Grounded Preference Optimization for Clinically-Augmented Medical Report Generation
Aug 9, 2026Qiang Hu, Yuxuan Luo, Yingjie Guo +4Medical Report Generation Multimodal Clinical Data
Huazhong University of Science and Technology
Learning When to Trust via Selective Context Preference Optimization
Aug 6, 2026Xian Sun, Wei Chow, Yingshuo Wang +4Large Language Model Reliability AI Trustworthiness
1Duke University · 2National University of Singapore · 3UC Berkeley +3
Stuck on "A": Diagnosing and Repairing Interface Injury in Attention-to-KDA Linearization of a 0.6B Language Model
Aug 3, 2026Ronglong BaoDataset Distillation Qwen3
DT-Project
Rolling With Resistance: Preference-Optimized LLM Counselors Can Trade Goal Persistence for Relational Attunement in Motivational Interviewing
Jul 30, 2026Weiying Chen, Junlong Shen, Zhexuan TangIntrinsic Motivation Counseling
University of Alberta · University of Shanghai for Science and Technology
TELLER: Dual-Path Iterative Preference Optimization for Table Entity Linking
Jul 29, 2026Yixin Peng, Kehao Li, Stefan DeckerEntities Backtranslation Augmented Direct Preference Optimization
RWTH Aachen, Ahornstraße 55, 52074 Aachen, Germany
Ontology-Amplified Distillation and Contextuality Auditing for Sovereign Enterprise Language Models: A Combined Proof-of-Mechanism and Negative-Results Method Study
Jul 11, 2026Thanh Luong TuanModel Auditing Symbolic Distillation
Foundation AgenticOS (FAOS)
Breaking the Quality--Intelligibility Trade-off in Streaming Target Speaker Extraction via Deep-Feature-Anchored Preference Optimization
Jul 11, 2026Shuhai Peng, Jinjiang Liu, Hui Lu +5Target Speaker Extraction Backtranslation Augmented Direct Preference Optimization
Tsinghua University · The Chinese University of Hong Kong · SenseTime
Metadata-Free Meta-Reweighted Direct Preference Optimization under Noisy Preference Labels
Jul 9, 2026Hua Qu, Yifan Li, Xiaodong YuanBacktranslation Augmented Direct Preference Optimization Meta-Learning
Xi’an Jiaotong University
Unbiased Alignment for Large Language Models with Noisy Preferences
Jul 3, 2026Jialiang Wang, Xianming Liu, Xiong Zhou +2Backtranslation Augmented Direct Preference Optimization Reinforcement Learning From Human Feedback
1Harbin Institute of Technology · 2City University of Hong Kong.
Distributionally Robust Listwise Preference Optimization
Jul 2, 2026Xudong Wu, Jian Qian, Pangpang Liu +2Sequential Preference Optimization Backtranslation Augmented Direct Preference Optimization
The University of Hong Kong · Hong Kong SAR · Yale University +3
Weight-Space Geometry of Offline Reasoning Training
Jun 21, 2026Aleksandr Nikolich, Igor Kiselev, Vladimir Platonov +1Weight-Space Backtranslation Augmented Direct Preference Optimization
1Keenable AI · 2Accenture · 3Yandex
Beyond Uniform Forgetting: A Study of Sequential Direct Preference Optimization Across Preference Settings
Jun 18, 2026Pranav Bhandari, Nicolas Fay, Amitava Datta +2Backtranslation Augmented Direct Preference Optimization Preference Alignment Learning
1Network Analysis and Social Influence Modelling (NASIM) Lab · School of Physics Maths and Computing, The University of Western Australia · School of Psychological Science, The University of Western Australia +1
Boosting Direct Preference Optimization with Penalization
Jun 10, 2026Pengwei SunBacktranslation Augmented Direct Preference Optimization Penalties
Stanford University.
Training LLMs to Enforce Multi-Level Instruction Hierarchies via Gravity-Weighted Direct Preference Optimization
Jun 9, 2026Lena S. Bolliger, Lena A. JägerBacktranslation Augmented Direct Preference Optimization Instruction-Tuned Models
Department of Computational Linguistics, University of Zurich, Switzerland
AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates
May 27, 2026Shaolong Chen, Madalina Ciobanu, Qingqing Mao +1Backtranslation Augmented Direct Preference Optimization Large Language Model Adaptation
Incept Labs Houston, TX · Incept Labs, Houston, TX · Titan Holdings, San Francisco, CA
Reasoning Matters: Mitigate Hallucination in Multimodal Large Reasoning Models via Reasoning-Conditioned Preference Optimization
May 27, 2026Jiawei Kong, Hao Fang, Shunxiang Liao +5Multimodal Reasoning Hallucination Mitigation
Tsinghua Shenzhen International Graduate School, Tsinghua University · Harbin Institute of Technology, Shenzhen
Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models
May 26, 2026Austin Wang, Jiaqi Han, Stefano Ermon +1Diffusion Alignment Text-To-Image Diffusion Models
Caltech · Stanford University
Token-weighted Direct Preference Optimization with Attention
May 21, 2026Chengyu Huang, Zhuohang Li, Sheng-Yen Chou +1Backtranslation Augmented Direct Preference Optimization Reward Functions
Cornell University · Vanderbilt University
Linear-DPO: Linear Direct Preference Optimization for Diffusion and Flow-Matching Generative Models
May 20, 2026Kesong Li, Yixuan Xu, Kuo-kun Tseng +3Text-To-Image Diffusion Models Diffusion Models
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China · Alibaba Group, Hangzhou, Zhejiang, China
Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment
May 20, 2026Zhiqin Yang, Yonggang Zhang, Wei Xue +3Backtranslation Augmented Direct Preference Optimization Reinforcement Learning From Human Feedback
1The Hong Kong University of Science and Technology · 2LIGHTSPEED · 3Hong Kong Baptist University.
Distributed Direct Preference Optimization
May 20, 2026Zhanhong JiangBacktranslation Augmented Direct Preference Optimization Decentralized Learning
Translational AI Center, Iowa State University, Ames, USA.
Reducing Hallucination in Vision-Language Models via Stage-wise Preference Optimization under Distribution Shift
May 13, 2026Qinwu XuHallucination Mitigation Citation Hallucination
Meta AI
BSO: Safety Alignment Is Density Ratio Matching
May 12, 2026Tien-Phat Nguyen, Truong Nguyen, Thin Nguyen +3Safety Alignment Large Language Model Alignment
Hanoi University of Science and Technology Hanoi, Vietnam · Deakin University Burwood, VIC 3125, Australia · Max Planck Research School for Intelligent Systems +2
DGPO: Beyond Pairwise Preferences with Directional Consistent Groupwise Optimization
May 11, 2026Mengyi Deng, Zhiwei Li, Xin Li +4Backtranslation Augmented Direct Preference Optimization Preference Alignment Learning
1Information Hub, The Hong Kong University of Science and Technology (Guangzhou), China · 2The Hong Kong University of Science and Technology, Hong Kong SAR
MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization
May 11, 2026Rohan Surana, Xintong Li, Sheldon Yu +7Backtranslation Augmented Direct Preference Optimization Frictive Policy Optimization
1UC San Diego · 2Adobe Research
$ξ$ -DPO: Direct Preference Optimization via Ratio Reward Margin
May 9, 2026Zhengyuan Fan, Zhonghua Wu, Yuxuan Du +1Backtranslation Augmented Direct Preference Optimization
School of Computer Science, Northwestern Polytechnical University
Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph
May 8, 2026Ning Liu, Chuanneng Sun, Kristina Klinkner +1Backtranslation Augmented Direct Preference Optimization Preference Alignment Learning
Amazon
RLearner-LLM: Balancing Logical Grounding and Fluency in Large Language Models via Hybrid Direct Preference Optimization
May 6, 2026Qiming Bao, Juho Leinonen, Paul Denny +1Large Language Model Alignment Large Language Models(Llms
University of Auckland · Aalto University
Gradient-Gated DPO: Stabilizing Preference Optimization in Language Models
May 4, 2026Inoussa Mouiche, Abderaouf BahiBacktranslation Augmented Direct Preference Optimization
WASP Lab, School of Computer Science University of Windsor, ON N9B 3P4
Efficient Preference Poisoning Attack on Offline RLHF
May 4, 2026Chenye Yang, Weiyu Xu, Lifeng LaiBacktranslation Augmented Direct Preference Optimization Reinforcement Learning From Human Feedback
Department of Electrical and Computer Engineering, University of California, Davis, Davis, CA, USA · Department of Electrical and Computer Engineering, University of Iowa, Iowa City, IA, USA.
Backtranslation Augmented Direct Preference Optimization for Neural Machine Translation
Apr 28, 2026Mehrdad Ghassabi, Spehr Rajabi, Hamidreza Baradaran Kashani +3Neural Machine Translation Backtranslation Augmented Direct Preference Optimization
Faculty of Computer Engineering University of Isfahan Isfahan, Iran · School of Computer Science University of Windsor Windsor, Canada
Learning from Noisy Preferences: A Semi-Supervised Learning Approach to Direct Preference Optimization
Apr 27, 2026Xinxin Liu, Ming Li, Zonglin Lyu +2Backtranslation Augmented Direct Preference Optimization Diffusion Alignment
University of Central Florida
Aligning with Your Own Voice: Self-Corrected Preference Learning for Hallucination Mitigation in LVLMs
Apr 27, 2026Byeonggeuk Lim, JungMin Yun, Junehyoung Kwon +2Hallucination Mitigation Preference Datasets
Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University · Department of Artificial Intelligence, Chung-Ang University
SignDPO: Multi-level Direct Preference Optimisation for Skeleton-based Gloss-free Sign Language Translation
Apr 20, 2026Muxin Pu, Xiao-Ming Wu, Mei Kuan Lim +3Gloss-Free Sign Language Translation Sign Language Translation
Monash University · Nanyang Technological University
GroupDPO: Memory-Efficient Group-Wise Direct Preference Optimization
Apr 17, 2026Jixuan Leng, Si Si, Hsiang-Fu Yu +2Backtranslation Augmented Direct Preference Optimization
Carnegie Mellon University · Google DeepMind · Google
Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervision
Dec 29, 2025Dohyun Kim, Seungwoo Lyu, Seung Wook Kim +1Diffusion Alignment Backtranslation Augmented Direct Preference Optimization
Dept. of CSE, Korea University · 2NVIDIA
Multi-dimensional Preference Alignment by Conditioning Reward Itself
Dec 11, 2025Jiho Jang, Jinyoung Kim, Kyungjune Baek +1Diffusion Alignment Backtranslation Augmented Direct Preference Optimization
Seoul National University · Sejong University
DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)
Jul 10, 2025Wenxuan Zhou, Shujian Zhang, Brice Magdalou +4Backtranslation Augmented Direct Preference Optimization Nonconvex
Google DeepMind · CEE-M, Univ Montpellier, CNRS, INRAE, Institut Agro · Work done while at Google DeepMind +1
Beyond Single-Negative Preference: Multi-Negative DPO for LLM-Centric Historical Entity Linking
Date pendingTien Nam Nguyen, Emanuela Boros, Ahmed Hamdi +3Backtranslation Augmented Direct Preference Optimization History
L3i, University of La Rochelle, La Rochelle, France · University of Toulouse, IRIT, Toulouse, France · University of Innsbruck, Austria +1

Sep 29, 2026, cs.LGJ/K move · Enter open · S save

Uncertainty-Normalized Margins for Direct Preference Optimization

Sadegh Khorasani, Petrus Mikkola, Matthias Grossglauser

School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland · University of Helsinki, Helsinki, Finland

Backtranslation Augmented Direct Preference Optimization Preference Alignment Learning

Open paper AskPDF arXiv

Direct preference optimization (DPO) models binary preferences through a Bradley-Terry model with a common noise scale, without explicitly accounting for preference strength or prompt-dependent uncertainty from human feedback. We introduce uncertainty-normalized margin DPO (UNM-DPO), which combines strength-dependent margins with a learned prompt scale. Motivated by a heteroskedastic Bradley-Terry model, we develop two training objectives. Both compare the implicit rewards of preferred and rejected responses, derived from response log-probability ratios to a reference policy. Advantage-only (AO) divides this reward difference by the prompt scale before subtracting the margin; whole-residual (WR) subtracts the margin before dividing by the scale. For the WR comparison model, we establish a necessary and sufficient condition under which known margins make the prompt scale identifiable. We introduce a practical procedure for learning the scale. Building on WR, we introduce ULNM-DPO-WR, which normalizes each response's implicit reward by its length. We evaluate our methods against DPO and related baselines on HelpSteer2 and HelpSteer3, using the Skywork reward model as a judge. With Llama-3.1-8B-Instruct, ULNM-DPO-WR achieves tie-adjusted win rates against matched DPO of 68.00% and 65.31% on evaluation panels, with higher mean rewards and shorter responses on average. On AlpacaEval with a GPT-4.1 judge and GPT-4-Turbo reference answers, the same 8B policy achieves a length-controlled win rate of 21.62%, compared with 16.39% for DPO and 15.30% for SimPO. These results demonstrate the potential of combining preference-strength margins, learned prompt scales, and length normalization for policy optimization.

Momentum

Related topics

Latest papers 50