Preference-Based RL

RL: Reinforcement Learning

Preference-based reinforcement learning studies how agents can learn what to do from judgments that one behavior is better than another, which can be easier to provide than precise numerical rewards. Common approaches learn a reward model from pairwise or ranked trajectories and optimize a policy with reinforcement learning, while direct preference optimization, active query selection, and uncertainty-aware exploration can reduce reliance on reward models or feedback. The methods are used in robotics, language-model alignment, and multi-objective or safety-critical decision-making, drawing on human, expert, and AI-generated feedback. A central challenge is making learning reliable when preferences are noisy, inconsistent, or diverse, while preserving exploration and preventing reward overoptimization; theoretical guarantees and evaluation methods remain active concerns.

Latest papers

127 papers23 papers in the last four weeks, up 92% on the four weeks before. 0.1% of all new papers.

Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph
May 8, 2026Ning Liu, Chuanneng Sun, Kristina Klinkner +1Pairwise Preference Learning LLM Alignment
Amazon
Implicit Preference Alignment for Human Image Animation
May 8, 2026Yuanzhi Wang, Xuhua Ren, Jiaxiang Cheng +5Human Motion Generation Human Avatar Animation
Work done during the internship at Tencent Hunyuan. · School of Artificial Intelligence, Beijing Normal University · Tencent Hunyuan
Multi-Objective Constraint Inference using Inverse reinforcement learning
May 7, 2026Syed Ihtesham Hussain Shah, Floris den Hengst, Aneta Lisowska +1Reinforcement Learning Constrained RL
Faculty of Sciences Vrije Universiteit Amsterdam
A Unified Pair-GRPO Family: From Implicit to Explicit Preference Constraints for Stable and General RL Alignment
May 7, 2026Hao YuPairwise Preference Learning LLM Alignment
Department of Automation, Tsinghua University
Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization
May 6, 2026Xin Yu, Liuchen Liao, Yiwen Zhang +3On-Policy Self-Distillation Language Model Distillation
1The Pennsylvania State University · 2TikTok
Data-dependent Exploration for Online Reinforcement Learning from Human Feedback
May 6, 2026Zhen-Yu Zhang, Yuting Tang, Jiandong Zhang +2RL Exploration Sample-Efficient RL
Center for Advanced Intelligence Project, RIKEN · Graduate School of Frontier Sciences, The University of Tokyo · Northeastern University +1
PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning
May 1, 2026Ziqin Yuan, Ruiqi Wang, Dezhong Zhao +2Reward Modeling Mixture of Experts
1Purdue University, West Lafayette, IN, USA. · 2Beijing University of Chemical Technology, Beijing, China. · 3Indiana University Bloomington, Bloomington, IN, USA.
Enhancing Reinforcement Learning for Radiology Report Generation with Evidence-aware Rewards and Self-correcting Preference Learning
Apr 15, 2026Qin Zhou, Guoyan Liang, Qianyi Yang +4Radiology Report Generation RL for Generative Models
Department of Computer Science and Engineering, ECUST, China · Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, P. R. China · Zhejiang University, Hangzhou, China +1
Learning Preference-Based Objectives from Clinical Narratives for Dynamic Sepsis Treatment
Apr 12, 2026Daniel J. Tan, Jayne Hui Zhen Chan, Kai Wen Hwang +3Reward Design for RL Reinforcement Learning
Institute of Data Science, National University of Singapore, Singapore · National University Hospital, Singapore · Saw Swee Hock School of Public Health, National University of Singapore, Singapore
UniCreative: Unifying Long-form Logic and Short-form Sparkle via Reference-Free Reinforcement Learning
Apr 7, 2026Xiaolong Wei, Zerun Zhu, Simin Niu +9Text Generation RL for Language Models
Beihang University · Baidu Inc. · Renmin University of China
Optimizing Neurorobot Policy under Limited Demonstration Data through Preference Regret
Apr 4, 2026Viet Dung Nguyen, Yuhang Song, Anh Nguyen +3Learning from Demonstration Robot Imitation Learning
Rochester Institute of Technology · Advanced Micro Devices, Inc. · University of Liverpool
Offline Constrained RLHF with Multiple Preference Oracles
Mar 31, 2026Brenden Latham, Mehrdad MoharramiConstrained RL Reinforcement Learning
Computer Science Department, University of Iowa, Iowa City, IA, USA
Reducing Oracle Feedback with Vision-Language Embeddings for Preference-Based RL
Mar 30, 2026Udita Ghosh, Dripta S. Raychaudhuri, Jiachen Li +2Preference-Based RL Data-Efficient Robot Learning
Univeristy of California, Riverside · AWS AI Labs (Work done outside AWS)
Safe Reinforcement Learning with Preference-based Constraint Inference
Mar 24, 2026Chenglin Li, Grant Ruan, Hua GengConstrained RL Safe RL
Department of Automation, Tsinghua University, Beijing, China · Laboratory for Information & Decision Systems, Massachusetts Institute of Technology, Cambridge, MA, USA
Privacy-Preserving Reinforcement Learning from Human Feedback via Decoupled Reward Modeling
Mar 23, 2026Young Hyun Cho, Will Wei SunPrivacy-Preserving ML Privacy-Preserving Language Models
Purdue University
Regret Bounds for Reinforcement Learning from Multi-Source Imperfect Preferences
Mar 20, 2026Ming Shi, Yingbin Liang, Ness B. Shroff +1Preference-Based RL RL from Human Feedback
Department of Electrical Engineering, University at Buffalo, Buffalo, NY 14260, USA · Department of Electrical and Computer Engineering, The Ohio State University, Columbus, OH 43210, USA · Department of Electrical and Computer Engineering and Computer Science and Engineering, The Ohio State University, Columbus, OH 43210, USA +1
Via Negativa for AI Alignment: Why Negative Constraints Are Structurally Superior to Positive Preferences
Mar 17, 2026Quan ChengLLM Alignment RL from Human Feedback
Tsinghua University
CRED: Counterfactual Reasoning and Environment Design for Active Preference Learning
Mar 9, 2026Yi-Shiuan Tung, Gyanig Kumar, Wei Jiang +2Active Preference Learning Preference Learning
Department of Computer Science, University of Colorado Boulder
CompassDPO: Dynamics-Controlled Direct Preference Optimization for Robust Safety Alignment
Mar 7, 2026Jilong Liu, Yonghui Yang, Pengyang Shao +5Direct Preference Optimization LLM Safety Alignment
Hefei University of Technology · National University of Singapore
Causally Robust Reward Learning from Reason-Augmented Preference Feedback
Mar 5, 2026Minjune Hwang, Yigit Korkmaz, Daniel Seita +1Reward Modeling Preference-Based RL
Thomas Lord Department of Computer Science, University of Southern California
Design-MLLM: A Reinforcement Alignment Framework for Verifiable and Aesthetic Interior Design
Mar 4, 2026Yuxuan Yang, Xiaotong Mao, Jingyao WangLLM Alignment Indoor Scene Generation
Nanjing Forestry University, Nanjing, China · Université de Lorraine, Nancy, France · Laboratoire Réactions et Génie des Procédés, Nancy, France +2
PARWiS: Winner determination under shoestring budgets using active pairwise comparisons
Mar 1, 2026Shailendra BhandariDueling Bandits Pairwise Preference Learning
Department of Computer Science OsloMet– Oslo Metropolitan University Oslo, Norway
BeautyGRPO: Aesthetic Alignment for Face Retouching via Dynamic Path Guidance and Fine-Grained Preference Modeling
Mar 1, 2026Jiachen Yang, Xianhui Lin, Yi Dong +4Image Editing RL for Image Editing
School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, China · vivo BlueImage Lab, vivo Mobile Communication Co., Ltd., China · Guangdong Provincial Key Laboratory of Information Security Technology, China
Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferences
Feb 26, 2026Junghyun Lee, Minju Hong, Kwang-Sung Jun +2Regret Minimization in RL Preference-Based RL
KAIST AI · KAIST EE · POSTECH CSE/AI
Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI Feedback
Feb 24, 2026Chenyang Zhao, Vinny Cahill, Ivana DusparicMulti-Objective Reinforcement Learning Preference-Based RL
Trinity College Dublin, Dublin, Ireland
Oracle-Robust Online Alignment for Large Language Models
Feb 24, 2026Zimeng Li, Mudit Gaur, Vaneet AggarwalLLM Alignment Preference-Based RL
Purdue University
OPRIDE: Offline Preference-based Reinforcement Learning via In-Dataset Exploration
Feb 19, 2026Yiqin Yang, Hao Hu, Yihuan Mao +10Preference-Based RL Offline RL
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · Moonshot AI · Tsinghua University +3
Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment
Feb 17, 2026Jialu Wang, Heinrich Peters, Asad A. Butt +6LLM Alignment Group Relative Policy Optimization
1Apple Inc., California, United States.
GraphAllocBench: A Flexible Benchmark for Preference-Conditioned Multi-Objective Policy Learning
Jan 28, 2026Zhiheng Jiang, Yunzhe Wang, Ryan Marr +3Multi-Objective Reinforcement Learning RL Benchmarks
University of California, Los Angeles · USC Institute for Creative Technologies · U.S. Army DEVCOM Army Research Laboratory
Trust, Don't Trust, or Flip: Robust Preference-Based Reinforcement Learning with Multi-Expert Feedback
Jan 26, 2026Seyed Amir Hosseini, Maryam Abdolali, Amirhosein Tavakkoli +4Preference-Based RL Robust RL
K. N. Toosi University of Technology, Tehran, Iran · Amirkabir University of Technology, Tehran, Iran · The University of Tokyo, Tokyo, Japan

May 8, 2026J/K move · Enter open · S save

Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph

Ning Liu, Chuanneng Sun, Kristina Klinkner, Shervin Malmasi

Amazon

Pairwise Preference Learning LLM Alignment Direct Preference Optimization Preference Optimization Preference-Based RL

Open paper AskPDF arXiv

Direct Preference Optimization (DPO) aligns language models using pairwise preference comparisons, offering a simple and effective alternative to Reinforcement Learning (RL) from human feedback. However, in many practical settings, training data consists of multiple rollouts per prompt, inducing rich preference structure that pairwise DPO fails to exploit. Collapsing such data into independent pairs discards transitivity, introduces redundant or conflicting supervision, and can lead to unstable optimization. We propose Graph Direct Preference Optimization (GraphDPO), a structured generalization of DPO that operates over directed acyclic preference graphs induced by rollout rankings. GraphDPO encodes dominance relations as edges and optimizes a graph-structured Plackett--Luce-inspired objective that aggregates supervision over dominated neighborhoods, propagating the transitive structure of the input preferences while recovering standard DPO as a special case. GraphDPO naturally handles tied and incomparable responses through an equivalence-class construction: responses with identical preference signals form graph layers, incomparable responses share a layer without an edge, and absent edges contribute zero loss, preventing the spurious gradients that strict total-order objectives incur. Under this layered construction, GraphDPO maintains linear per-prompt complexity via efficient log-sum-exp aggregation. We further incorporate optional ground-truth anchoring by inserting verified solutions as dominant nodes and applying an annealed schedule that stabilizes early training while gradually relaxing oracle supervision. Experiments on reasoning and program synthesis tasks demonstrate superior performance, suggesting that graph-structured preference modeling is a scalable and robust alternative to pairwise and listwise alignment objectives.