Preference Optimization

Follow this topic Ask about this topic

Preference optimization uses comparisons, rankings, or desirability signals to steer models toward outputs people or task-specific evaluators prefer. Common approaches include reinforcement learning from human feedback, Direct Preference Optimization (DPO) and variants such as KTO, as well as listwise, group-wise, and token-level objectives; newer methods also use online feedback, synthetic preferences, and task-specific rewards. Work spans language-model alignment on instruction-following and reasoning benchmarks, as well as image, audio, recommendation, translation, and robotic systems. A central challenge is that preferences can be noisy, inconsistent, or specific to a user or context, while optimizing proxy rewards can cause unwanted behavior or reduce the model’s existing capabilities.

Latest papers

155 papers24 papers in the last four weeks, up 41% on the four weeks before. 0.1% of all new papers.

Doubly Robust Alignment for Large Language Models
Jun 1, 2025Erhan Xu, Kai Ye, Hongyi Zhou +3LLM Alignment RL from Human Feedback
Department of Statistics, LSE, London, UK · Department of Mathematics, Tsinghua University, Beijing, China · School of Design, LCC, UAL, London, UK +1
Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomy
May 28, 2025Saleh Afzoon, Ali Shahsavandi, Phuong Thao Huynh +4LLM Personalization Human-AI Interaction
School of Computing, Macquarie University, Sydney, Australia · Department of Computer Engineering and Information Technology, Shiraz University of Technology, Shiraz, Iran
Atomic Consistency Preference Optimization for Long-Form Question Answering
May 14, 2025Jingfeng Chen, Raghuveer Thirukovalluru, Junlin Wang +2LLM Hallucination Mitigation LLM Alignment
Dual-Difficulty Curriculum Learning for Direct Preference Optimization
Apr 10, 2025Mengyang Li, Haozhan Geng, Zhong Zhang +1Pairwise Preference Learning LLM Alignment
Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission Tianjin Normal University Tianjin, China
KTO: Model Alignment as Prospect Theoretic Optimization
Feb 2, 2024Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff +2LLM Alignment Human Preference Modeling
Stanford University · Contextual AI

Jun 1, 2025J/K move · Enter open · S save

Doubly Robust Alignment for Large Language Models

Erhan Xu, Kai Ye, Hongyi Zhou, Luhan Zhu+2

Department of Statistics, LSE, London, UK · Department of Mathematics, Tsinghua University, Beijing, China · School of Design, LCC, UAL, London, UK+1

LLM Alignment RL from Human Feedback Preference Optimization Preference-Based RL

Open paper AskPDF arXiv

This paper studies reinforcement learning from human feedback (RLHF) for aligning large language models with human preferences. While RLHF has demonstrated promising results, many algorithms are highly sensitive to misspecifications in the underlying preference model (e.g., the Bradley-Terry model), the reference policy, or the reward function, resulting in undesirable fine-tuning. To address model misspecification, we propose a doubly robust preference optimization algorithm that remains consistent when either the preference model or the reference policy is correctly specified (without requiring both). Our proposal demonstrates superior and more robust performance than state-of-the-art algorithms, both in theory and in practice. The code is available at https://github.com/DRPO4LLM/DRPO4LLM

Latest papers

Doubly Robust Alignment for Large Language Models

Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomy

Atomic Consistency Preference Optimization for Long-Form Question Answering

Dual-Difficulty Curriculum Learning for Direct Preference Optimization

KTO: Model Alignment as Prospect Theoretic Optimization