Preference Optimization

Follow this topic Ask about this topic

Preference optimization uses comparisons, rankings, or desirability signals to steer models toward outputs people or task-specific evaluators prefer. Common approaches include reinforcement learning from human feedback, Direct Preference Optimization (DPO) and variants such as KTO, as well as listwise, group-wise, and token-level objectives; newer methods also use online feedback, synthetic preferences, and task-specific rewards. Work spans language-model alignment on instruction-following and reasoning benchmarks, as well as image, audio, recommendation, translation, and robotic systems. A central challenge is that preferences can be noisy, inconsistent, or specific to a user or context, while optimizing proxy rewards can cause unwanted behavior or reduce the model’s existing capabilities.

Latest papers

155 papers24 papers in the last four weeks, up 41% on the four weeks before. 0.1% of all new papers.

Selective Tie Learning: Mitigating Spurious Correlations in Preference Optimization
Oct 26, 2026Christian Moya, Purav Matlia, Elliott Thornley +1Preference Optimization Spurious Correlation Robustness
Purdue University · National University of Singapore
RubricOpt: Self-Revision of Semantic Rubrics for Preference Optimization
Oct 21, 2026Wei Chen, Yubing Wu, Shigui Li +7Rubric-Guided Generation Preference Optimization
South China University of Technology · LinkSprite · University of Waterloo
TeachBack: Learning to Teach via Bidirectional Preference Optimization for On-Policy Self-Distillation
Oct 21, 2026Linfei Ji, Jiayun Li, Haozhao Wang +5Language Model Distillation On-Policy Self-Distillation
Huazhong University of Science and Technology · Tsinghua University · Harbin Institute of Technology +1
Divergence based Ranked Preference Alignment for LLMs
Oct 18, 2026Rajdeep Haldar, Yue Xing, Qifan SongLLM Alignment Preference Alignment
Purdue University · Michigan State University
SteadyThought: Mitigating LLM Under-Thinking via Thought-Level Preference Optimization
Oct 18, 2026Zhenyue Peng, Lin Yang, Jie Wang +7Efficient Language Model Reasoning Adaptive Reasoning in Language Models
Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China · University of Science and Technology of China, Hefei 230026, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China +3
Qwen-MT-Lumen at WMT2026: Instruction-Specialized Translation via Quality-Aware Training and On-Policy Distillation
Oct 14, 2026Hao Wang, Yangyang Liu, Xiaohu Zhao +11Machine Translation Preference Optimization
Alibaba Token Hub
Do Factuality Methods Make LLMs More Factual? Reproducibility and Fair Comparison
Oct 13, 2026Pratuat Amatya, Vinay Setty, V Venktesh +1LLM Alignment Factual Knowledge in Language Models
Factiverse AS · University of Stavanger · Stockholm University +1
Prompt or Policy? Optimizing Physician Slate Presentations with LLM User Simulators
Oct 13, 2026Andrei Niculae, Adrian Cosma, Emilian RadoiAutomatic Prompt Optimization Large Language Model-Based Recommendation
National University of Science and Technology POLITEHNICA Bucharest, Romania · IDSIA, Dalle Molle Institute for Artificial Intelligence, Lugano, Switzerland
FrameLite: Decision-Oriented Aesthetic Cropping with a Compact Vision-Language Model
Oct 13, 2026Jianze Li, Kai Liu, Ziqing Zhang +4Image Editing Vision-Language Models
Shanghai Jiao Tong University · Institute of Media Technology and Experience Design, Huawei
Uniform Information Density-Based Preference Optimization
Oct 10, 2026Yongyuan Li, Yang XuPreference Optimization LLM Alignment
Department of Computer Science and Engineering Southern University of Science and Technology Shenzhen, Guangdong, China
Evolving Creatives with Clicks: CTR-Driven Advertising image Editing via Attribute Disentanglement and Preference Alignment
Oct 9, 2026Xu Zhang, Haohan Wang, Zheng Zhang +9Image Editing Text-Guided Image Editing
TMCC, College of Computer Science, Nankai University · VCIP, Computer Science, Nankai University
Emo-BPO: Emotion Bidirectional Preference Optimization for Diffusion-based Emotional TTS
Oct 9, 2026Jiacheng Shi, Hongfei Du, Xinyuan Song +3Preference Optimization for Diffusion Models Emotional Speech Synthesis
College of William & Mary, USA · Emory University, USA · George Mason University, USA
Successive Training Stages and Large Language Model Persuasion: Effects of Misalignment, Supervised Fine-Tuning, and Preference Optimization
Oct 7, 2026Antony Dalmiere, Pascal Marchand, Guillaume Auriol +1Persuasion in Language Models LLM Post-Training
LAAS-TRUST · LAAS-TRUST, INSA Toulouse · LAAS-TSF, LAAS
RobotAPO: Adversarial Physics Preference Optimization for Robotic Manipulation Video Generation
Oct 7, 2026Kerui Li, Zhe Jing, Chenyi Huang +4Physical Consistency in Video Generation Robotic Manipulation
Institute of Automation, Chinese Academy of Sciences · GigaAI · Beijing Institute of Technology
AuraSE: Low-Hallucination Generative Speech Enhancement via Multimodal Flow Matching and Inference Policy Optimization
Oct 5, 2026Yingda Shen, Yao Qian, Yuxuan Hu +7Inference-Time Optimization Audio Diffusion Models
The Chinese University of Hong Kong, Shenzhen · Microsoft Research
Geometry-Aware Preference Optimization for Text-to-Image Diffusion Models
Oct 4, 2026Lei Wang, Zhen Wang, Yuexiang Xie +1Preference Optimization for Diffusion Models Diffusion Model Alignment
Sun Yat-Sen University, Guangzhou, China · Alibaba Group, Hangzhou, China
Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution
Oct 4, 2026Minjae Lee, Kyunghyun Cho, Sangdon ParkLLM Alignment Multi-Objective Language Model Alignment
GSAI POSTECH · Center for Data Science New York University · GSAI & CSE POSTECH
Bidirectional Preference Synthesis: Learning Prompt-Conditioned Preferences from Boundary Failures
Oct 3, 2026Junbo Wang, Lidong Lu, Zhuoqun Li +4Preference Learning Preference Optimization
Kuaishou Technology · Nanjing University
Exploration-Preserving Policy Optimization
Oct 2, 2026Hangzhan jin, Mohammad Hamdaqa, Doina PrecupRL for Language Model Reasoning RL Exploration
Mila - Quebec AI Institute · Polytechnique Montr´eal · McGill University +2
Rubric-Based Optimization for Text-to-Music Generation
Oct 2, 2026Ping Wang, Guang Yang, Shao-Rong Su +3Text-to-Music Generation Rubric-Based RL
Paul G. Allen School of Computer Science & Engineering, University of Washington · Department of Electrical and Computer Engineering, University of Washington · Allen Institute for AI
PsyEvo: A Personalized Counseling Agent That Self-Evolves at Test Time
Oct 2, 2026Yuting Yan, Shihao Xu, Junhao Yu +6LLM Agent Self-Improvement Mental Health Counseling
Lyncia Lab · Tianqiao and Chrissy Chen Institute
Mitigating Social Sycophancy via Pluralistic Preference Optimization
Oct 1, 2026Stephane Hatgis-Kessell, Myra Cheng, Xiaoxuan Hou +4LLM Sycophancy Preference Optimization
Department of Computer Science, Stanford University · Amazon · Department of Computer Science, University of Washington
GAW-PO: Preference Optimization with Gradient-Aligned Token Weights
Oct 1, 2026Andreea Dutulescu, Stefan Ruseti, Mihai Masala +2LLM Alignment Direct Preference Optimization
National University of Science and Technology POLITEHNICA Bucharest · NVIDIA
Uncertainty-Normalized Margins for Direct Preference Optimization
Sep 29, 2026Sadegh Khorasani, Petrus Mikkola, Matthias GrossglauserPairwise Preference Learning Direct Preference Optimization
School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland · University of Helsinki, Helsinki, Finland
Preference-Guided Adaptation for Open-Vocabulary Semantic Segmentation via Prompt Disagreement
Sep 28, 2026Hyun-Kurl Jang, Jihun Kim, Kuk-Jin YoonWeakly Supervised Semantic Segmentation Preference Optimization
Visual Intelligence Lab. KAIST
Agentic Multi-Turn Reasoning: A Fairness Approach
Sep 27, 2026Thanh-Dat Truong, Sankalp Pandey, Hugh Churchill +3Preference Optimization Agentic Reasoning
CVIU Lab, University of Arkansas, USA · Dep. of Physics, University of Arkansas, USA · Dep. of Geosciences, University of Arkansas, USA +1
DimPO: Dimensionality Reduction for Attention using Preference Optimization
Sep 26, 2026Vojtěch Lanz, Yufei Cui, Prasanna ParthasarathiTransformer Attention LLM Compression
Charles University, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics · Noah’s Ark Lab, Huawei Technologies Ltd.
A Zeroth-Order Paradigm for LLM Preference Alignment
Sep 16, 2026Peter Chen, Xi Chen, Wotao Yin +1Pairwise Preference Learning LLM Alignment
Department of Electrical Engineering and Computer Sciences (EECS) University of California, Berkeley Berkeley, CA 94720, USA · Stern School of Business New York University New York, NY 10012, USA · Decision Intelligence Lab (Seattle) DAMO Academy, Alibaba Group U.S. Bellevue, WA 98004, USA +1
StalePO: Anchored Token-Level Preference Optimization using Legacy Post-Edits in Machine Translation
Sep 14, 2026Rohit Dhaipule, Sukhdeep Singh Kharbanda, Prasanth Bathala +2Preference Optimization Language Model Post-Training
Translation Services, Amazon
Direct Preference Density Alignment for Conversational Audio Equalization
Sep 11, 2026Ioannis Stylianou, Sven Ewan Shepstone, Jon Francombe +2RL for Language Models LLM Alignment
Department of Electronic Systems Aalborg University Aalborg, Denmark · Bang & Olufsen A/S Struer, Denmark

Oct 26, 2026J/K move · Enter open · S save

Selective Tie Learning: Mitigating Spurious Correlations in Preference Optimization

Christian Moya, Purav Matlia, Elliott Thornley, Guang Lin

Purdue University · National University of Singapore

Preference Optimization Spurious Correlation Robustness Preference Learning Data Selection

Open paper AskPDF arXiv

Preference optimization can learn spurious correlations, and augmenting the preference data with ties (equally preferred responses) can reduce spurious reliance. However, ties differ in their corrective effect, making their selection for augmentation important. For log-linear policies, we derive each tie's influence on spurious reliance and show that taking the largest predicted reductions is first-order optimal among subsets of equal size. Computing these influences requires knowledge of the spurious features and the curvature of the training objective, which is often unavailable or expensive to estimate. To this end, we explore using the magnitude of a tie's implicit reward margin and characterize the conditions under which it is a useful proxy for predicted influence. To test the theory and its qualitative transfer to neural models, we estimate influence with a counterfactual proxy for spurious reliance and use a generalized Gauss--Newton approximation to curvature for language models. Across log-linear policies, neural models, and large language models, our selection rules consistently achieve comparable reductions in spurious reliance with fewer ties than uniform selection.