Preference-Based RL

RL: Reinforcement Learning

Preference-based reinforcement learning studies how agents can learn what to do from judgments that one behavior is better than another, which can be easier to provide than precise numerical rewards. Common approaches learn a reward model from pairwise or ranked trajectories and optimize a policy with reinforcement learning, while direct preference optimization, active query selection, and uncertainty-aware exploration can reduce reliance on reward models or feedback. The methods are used in robotics, language-model alignment, and multi-objective or safety-critical decision-making, drawing on human, expert, and AI-generated feedback. A central challenge is making learning reliable when preferences are noisy, inconsistent, or diverse, while preserving exploration and preventing reward overoptimization; theoretical guarantees and evaluation methods remain active concerns.

Latest papers

127 papers23 papers in the last four weeks, up 92% on the four weeks before. 0.1% of all new papers.

PrefO2O: Zero-Shot Preference-Based Reinforcement Learning via Offline-to-Online Latent Preference Transfer
Oct 22, 2026Ruiqi Wang, Dezhong Zhao, Xuesu Xiao +3Preference-Based RL Zero-Shot RL
University of Southern California, Los Angeles, CA, USA. · Beijing University of Chemical Technology, Beijing, China. · George Mason University, Fairfax, VA, USA. +1
Reward-Weighted Preference Optimization for Selective Combinatorial Optimization Problems
Oct 22, 2026Haohui Zhang, Xinyu Hu, Wouter van Heeswijk +2RL for Combinatorial Optimization Combinatorial Optimization
University of Twente, Enschede, Netherlands · University of Bath, Bath, United Kingdom · Delft University of Technology, Delft, Netherlands
Differentially Private Sparse Reward Estimation with Preference Feedback
Oct 21, 2026Meng Ding, Mingxi Lei, Jie Zhang +2Reward Modeling RL from Human Feedback
Department of Computer Science, University of Massachusetts Boston, Boston, MA, USA · Department of Computer Science and Engineering, University at Buffalo, SUNY, Buffalo, NY, USA · ETH Zürich, Zürich, Switzerland +2
One Policy, Many Styles: Where to Aggregate Preferences for Steerable Humanoid Control?
Oct 20, 2026Ziqian Huang, Yuzhuo Chen, Shimin Zhang +3Preference-Based RL Humanoid Robot Control
College of Computing Georgia Institute of Technology
Teaching AI Agent to Play Poker Strategically Through Preference Optimization
Oct 19, 2026Phurinat Polasa, Thanapon Thanadunpremdet, Phatthanasak Kraiduang +1Game-Playing Agents Preference-Based RL
Department of Computer Engineering, KMUTT, Thailand · Department of Computer Engineering, Mahidol University, Thailand · R&D, SCB DataX, SCBX Group, Thailand
Mitigating Reward Hacking in Preference Reward Models with Evolving Penalties
Oct 19, 2026Afonso Baldo, António Farinhas, Marcos Treviso +2Reward Hacking RL for Language Models
Sword Health · Instituto Superior Técnico, Universidade de Lisboa · Gandara AI +2
Progress-Guided Preference Generation for Aligning Generative Control Policies
Oct 19, 2026Tung M. Luu, Tri Ton, Duc M. Nguyen +8Preference-Based RL RL for Generative Models
KAIST · Center for AI Research & CECS, VinUniversity · VinRobotics +4
Robust Direct Preference Optimization under Preference Label-Flip Attacks
Oct 18, 2026Chenye Yang, Weiyu Xu, Lifeng LaiDirect Preference Optimization Adversarial Robustness
Department of Electrical and Computer Engineering, University of California, Davis, Davis, CA 95616, USA · Department of Electrical and Computer Engineering, University of Iowa, Iowa City, IA 52242, USA
Steerable Pluralistic Policy Alignment from Anonymized Crowd Feedback
Oct 17, 2026Akansha Kalra, Ellen R. Novoseller, Vernon J. Lawhern +1Pluralistic Alignment User Preference Modeling
Kahlert School of Computing, University of Utah, Salt Lake City, USA · U.S. Army DEVCOM Army Research Laboratory, Aberdeen Proving Ground, MD, USA
Reason-guided Human Reward Modeling for Preference-based Reinforcement Learning
Oct 16, 2026Anshul Nayak, Shahil Shaik, Jonathan Smereka +1Preference-Based RL Reward Modeling
Clemson University, Clemson, SC 29631 USA · U.S. Army DEVCOM Ground Vehicle Systems Center, Warren, MI 48092 USA
Beyond Reward Scoring: Online Multi-Agent Preference Learning for Multimodal Response Generation
Oct 16, 2026Ziyun Zou, Jiwoo Hong, Yifan Lu +5Preference Alignment Preference-Based RL
Mohamed bin Zayed University of Artificial Intelligence · Nubank · Dartmouth College
SPEAR: Selective Preference Elicitation with Adaptive Rewards for VLM-Guided Robotic Manipulation
Oct 15, 2026Zewen Wu, Ruizhe Liu, Pei Zhou +2Active Preference Learning Preference Learning
HKU Musketeers Foundation Institute of Data Science, The University of Hong Kong, Hong Kong SAR, China. · Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong SAR, China.
CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF
Oct 15, 2026Licheng Pan, Haocheng Yang, Yuzhou Chen +7RL from Human Feedback Noisy-Label Learning
Zhejiang University · Xiaohongshu Inc · National University of Singapore +3
Can You Trust Your Preference Models? Stackelberg Games for Alignment
Oct 10, 2026Riccardo Formenti, Aditya Agrawal, Vikas GargLLM Alignment Preference-Based RL
Department of Mathematics ETH Zürich · Department of Computer Science Aalto University · YaiYai Ltd and Aalto University
Matched-Context Group Policy Optimization for LLM Agent Training
Oct 9, 2026Chenxu Zhang, Yaorui Shi, Wenlong Zhang +3RL for LLM Agents Preference-Based RL
Imperial College London · Meituan · University of Science and Technology of China +1
PB-GRPO: Learning Socially Adaptive LLM Agents from Persona-Driven Simulation with Preference-Batched GRPO
Oct 2, 2026Jingquan Wang, Jun Yin, Xu Han +3LLM Agent Training Group Relative Policy Optimization
Amazon
GAW-PO: Preference Optimization with Gradient-Aligned Token Weights
Oct 1, 2026Andreea Dutulescu, Stefan Ruseti, Mihai Masala +2LLM Alignment Direct Preference Optimization
National University of Science and Technology POLITEHNICA Bucharest · NVIDIA
PROMO: Preference-conditioned Multi-Objective Reinforcement Learning for Quadrupedal Robots
Oct 1, 2026Amr Mousa, Rifny Rachman, Neil Karavis +2Multi-Objective Reinforcement Learning Energy-Efficient Robot Locomotion
The University of Manchester, United Kingdom · BAE Systems, United Kingdom · University of Warwick, United Kingdom
PrefPI: Preference-Guided Steering into Out-of-Distribution Behaviors
Sep 30, 2026Seungeun Rho, Wontaek Kim, Danfei Xu +1Robot Policy Learning Robot Policy Adaptation
School of Interactive Computing Georgia Institute of Technology
Patient-Centered Treatment Planning for Chronic Multimorbidity: A Hierarchical Reinforcement Learning Framework for Preference Modeling
Sep 30, 2026Nafiseh Payani, Soham Das, G. Anthony Wilson +1Healthcare Hierarchical RL
Department of Industrial and Systems Engineering at the University of Tennessee, Knoxville, TN, USA · Department of Industrial Engineering at the University of Arkansas, Fayetteville, AR, USA
Robust Risk-Sensitive Reinforcement Learning from Corrupted Human Feedback
Sep 30, 2026Xinyi Ni, Lifeng LaiRisk-Sensitive RL Robust RL
Department of Electrical and Computer Engineering, University of California, Davis, CA 95616, USA
Revisiting scaling laws for reward optimization
Sep 29, 2026Ali Aouad, Aymane El Gadarri, Vivek F. FariasReward Modeling Reward Over-Optimization
MIT
Pointwise or Pairwise: When Do Pairwise Losses Help Reward Learning, Provably?
Sep 29, 2026Junghyun Lee, Minsoo Ha, Sanghwa Kim +4Offline-to-Online Contextual Bandits Contextual Bandits
Kim Jaechul Graduate School of AI, KAIST Seoul 02455, Republic of Korea · Graduate School of AI, POSTECH Pohang 37673, Republic of Korea · Samsung Research Seoul 06765, Republic of Korea +1
Improving LLM Collaboration via Multi-Agent Preference Learning
Sep 26, 2026Shuo Liu, Xinzichen Li, Tianle Chen +1Multi-Agent Collaboration RL for LLM Agents
Northeastern University, Boston
On Preference Coverage Collapse from Hindsight Relabeling in Multi-Objective Reinforcement Learning
Sep 22, 2026Baptiste Bonin, Caro Strickland, Audrey DurandMulti-Objective Reinforcement Learning Experience Replay
Mila (Quebec AI Institute) · Université Laval · CIFAR AI Chair
Simultaneous Forward and Inverse Human-in-the-Loop Optimization
Sep 18, 2026Kyeongwon Park, Steven H. CollinsHuman-in-the-Loop AI Human-Robot Interaction
Department of Mechanical Engineering, Stanford University, United States
Scaling Vision-Language Reward Learning for Robot Manipulation in Parallel Simulation
Sep 18, 2026Lobna Joualy, Eric Demeester, Nikolaos TsiogkasRobotic RL Preference-Based RL
Department of Computer Science and Department of Mechanical Engineering, KU Leuven, 3590 Diepenbeek, Belgium.
IncentRL: The Trade-Off Between Preference Guidance and Task Performance
Sep 18, 2026Xuening Wu, Yanlan Kang, Shenqin YinReinforcement Learning Preference-Based RL
Fudan University
ArenaFlow: From Trajectory Ranking to Hierarchical Credit Propagation for Open-Ended Agent RL
Sep 18, 2026Qiang Zhang, Ruixue Ding, Fanrui Zhang +9Reinforcement Learning RL for LLM Agents
Alibaba Token Hub, Alibaba Group · Amap, Alibaba Group
The Operable Pareto Front: Distilling Offline Search into Run-Time Control for Multi-Objective UAV Edge-Computing Scheduling
Sep 16, 2026Qiao Liao, Zhiyong Feng, Bin Wu +1Trajectory Optimization Edge Computing
School of Computer Science and Technology, Tianjin University, Tianjin, China · School of Computer Software, Tianjin University, Tianjin, China · School of Information Science and Engineering, Shandong Agriculture and Engineering University, Shandong, China

Oct 22, 2026J/K move · Enter open · S save

PrefO2O: Zero-Shot Preference-Based Reinforcement Learning via Offline-to-Online Latent Preference Transfer

Ruiqi Wang, Dezhong Zhao, Xuesu Xiao, Erdem Biyik+2

University of Southern California, Los Angeles, CA, USA. · Beijing University of Chemical Technology, Beijing, China. · George Mason University, Fairfax, VA, USA.+1

Preference-Based RL Zero-Shot RL Offline-To-Online RL

Open paper AskPDF arXiv

Preference-based reinforcement learning (PbRL) has emerged as a promising paradigm for learning robot behaviors aligned with human intent. However, its scalability is fundamentally limited by the need to collect expensive human preference labels for every new task. In this work, we present PrefO2O, an offline-to-online latent preference transfer framework that enables zero-shot PbRL on unseen target tasks without requiring any human feedback. The core idea is to learn a task-invariant latent preference space from a small amount of offline preference data on source tasks, where transferable preference structures are disentangled from task-specific variations. This learned manifold allows PrefO2O to transfer preference structures and reward priors to new tasks under distribution shift, while confidence-weighted optimal transport alignment generates reliable pseudo-preference supervision for online reward adaptation. Extensive experiments on robotic manipulation and locomotion benchmarks demonstrate that PrefO2O substantially outperforms existing preference-transfer baselines and matches or exceeds PbRL methods trained with real human feedback, while generalizing across multi-to-one and one-to-one transfer settings as well as across different robot embodiments.