Preference-Based RL

RL: Reinforcement Learning

Preference-based reinforcement learning studies how agents can learn what to do from judgments that one behavior is better than another, which can be easier to provide than precise numerical rewards. Common approaches learn a reward model from pairwise or ranked trajectories and optimize a policy with reinforcement learning, while direct preference optimization, active query selection, and uncertainty-aware exploration can reduce reliance on reward models or feedback. The methods are used in robotics, language-model alignment, and multi-objective or safety-critical decision-making, drawing on human, expert, and AI-generated feedback. A central challenge is making learning reliable when preferences are noisy, inconsistent, or diverse, while preserving exploration and preventing reward overoptimization; theoretical guarantees and evaluation methods remain active concerns.

Latest papers

127 papers23 papers in the last four weeks, up 92% on the four weeks before. 0.1% of all new papers.

Oracle-RLAIF: An Improved Fine-Tuning Framework for Multi-modal Video Models using Reinforcement Learning from Ranking Feedback
Oct 2, 2025Derek Shi, Ruben Glatt, Christine Klymko +5Video-Language Models RL Fine-Tuning
Stanford University, CA, USA · Lawrence Livermore National Laboratory, CA, USA · Microsoft, WA, USA
Learning Acrobatic Flight from Preferences
Aug 26, 2025Colin Merk, Ismail Geles, Jiaxu Xing +3RL for Robotics Aerial Robotics
Robotics and Perception Group, University of Zurich, Switzerland · Autonomous Learning and Predictive Intelligence Lab, University of Zurich, Switzerland
From Dueling Bandits to Episodic MDPs: Learning with Adversarial Preference Feedback
Jul 15, 2025Taira Tsuchiya, Shinji Ito, Haipeng LuoPreference-Based RL Dueling Bandits
The University of Tokyo and RIKEN · University of Southern California
Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning
Jun 14, 2025Sara Rajaram, R. James Cotton, Fabian H. SinzOffline RL Robust RL
Institute for Computer Science and Campus Institute for Data Science, University of Göttingen, Göttingen, Germany · University Hospital Tübingen, Tübingen, Germany · International Max Planck Research School for Intelligent Systems, Tübingen, Germany +2
Doubly Robust Alignment for Large Language Models
Jun 1, 2025Erhan Xu, Kai Ye, Hongyi Zhou +3LLM Alignment RL from Human Feedback
Department of Statistics, LSE, London, UK · Department of Mathematics, Tsinghua University, Beijing, China · School of Design, LCC, UAL, London, UK +1
Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
May 15, 2025Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang +12Reward Modeling LLM Alignment
NLP Group, Fudan University · Hong Kong Baptist University · LLM Team, Meituan Inc
Corruption Robust Offline Reinforcement Learning with Human Feedback
Feb 9, 2024Debmalya Mandal, Andi Nika, Parameswaran Kamalaruban +2Offline RL Corruption Robustness
Dept. of Computer Science University of Warwick, UK · Max-Planck Institute for Software Systems, Germany · Independent Researcher London, UK

Oct 2, 2025J/K move · Enter open · S save

Oracle-RLAIF: An Improved Fine-Tuning Framework for Multi-modal Video Models using Reinforcement Learning from Ranking Feedback

Derek Shi, Ruben Glatt, Christine Klymko, Shubham Mohole+4

Stanford University, CA, USA · Lawrence Livermore National Laboratory, CA, USA · Microsoft, WA, USA

Video-Language Models RL Fine-Tuning RL for VLMs Preference-Based RL

Open paper AskPDF arXiv

Recent advances in large video-language models (VLMs) rely on extensive fine-tuning techniques that strengthen alignment between textual and visual comprehension. Leading pipelines typically pair supervised fine-tuning (SFT) with reinforcement learning from preference data to enhance video comprehension. However, as VLMs scale in parameter size, so does the cost of gathering enough human feedback. To make fine-tuning more cost-effective, recent frameworks explore reinforcement learning with AI feedback (RLAIF), which replace human preference with AI as a judge. Current RLAIF frameworks rely on a specialized reward model trained with video narratives to create calibrated scalar rewards -- an expensive and restrictive pipeline. We propose Oracle-RLAIF, a novel framework that replaces the trained reward model with a more general Oracle ranker which acts as a drop-in model ranking candidate model responses rather than scoring them. Alongside Oracle-RLAIF, we introduce

GRPO_{rank}

, a novel rank-based loss function based on Group Relative Policy Optimization (GRPO) that directly optimizes ordinal feedback with rank-aware advantages. Empirically, we demonstrate that Oracle-RLAIF consistently outperforms leading VLMs using existing fine-tuning methods when evaluated across various video comprehension benchmarks. Oracle-RLAIF paves the path to creating flexible and data-efficient frameworks for aligning large multi-modal video models with reinforcement learning from rank rather than score.