RL from Human Feedback

RL: Reinforcement Learning

Ask about this topic

Overview

RLHF studies how to use human judgments and interventions to teach AI systems which responses or actions are desirable, especially when goals are hard to specify with a fixed reward. Common approaches learn reward models from pairwise comparisons and optimize policies with PPO or GRPO, while methods such as Direct Preference Optimization learn directly from preference pairs; active learning and online feedback aim to reduce the data required. Research spans language-model alignment and benchmarks such as AlpacaEval and MT-Bench, as well as robot learning from corrections and demonstrations. A central challenge is that feedback can be noisy, biased, or diverse, and optimizing an imperfect reward can cause reward hacking, so work increasingly studies uncertainty, robustness, fairness, and how to measure real-world improvement rather than proxy scores alone.

Latest papers

CardsList

128 papers10 papers in the last four weeks, up 67% on the four weeks before. 0.1% of all new papers.

  1. DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)

    Jul 10, 2025Wenxuan Zhou, Shujian Zhang, Brice Magdalou +4Direct Preference OptimizationHuman Preference Modeling

    Google DeepMind · CEE-M, Univ Montpellier, CNRS, INRAE, Institut Agro · Work done while at Google DeepMind +1

  2. A Technical Survey of Reinforcement Learning Techniques for Large Language Models

    Jul 5, 2025Saksham Sahai Srivastava, Vaneet AggarwalRL for Language ModelsLLM Alignment

    University of Georgia, Athens, GA, USA · Purdue University, West Lafayette, IN, USA

  3. Doubly Robust Alignment for Large Language Models

    Jun 1, 2025Erhan Xu, Kai Ye, Hongyi Zhou +3LLM AlignmentRL from Human Feedback

    Department of Statistics, LSE, London, UK · Department of Mathematics, Tsinghua University, Beijing, China · School of Design, LCC, UAL, London, UK +1

  4. MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning

    May 30, 2025Jingyan Shen, Jiarui Yao, Rui Yang +5Pairwise Preference LearningReward Modeling

    New York University · University of Illinois Urbana-Champaign · Rice University

  5. Bias Fitting to Mitigate Length Bias of Reward Model in RLHF

    May 19, 2025Kangwen Zhao, Jianfeng Cai, Jinhua Zhu +5Reward ModelingLLM Alignment

    University of Science and Technology of China

  6. Reinforcement Learning from Human Feedback

    Apr 16, 2025Nathan LambertLLM AlignmentDirect Preference Optimization

  7. Distributionally Robust Reinforcement Learning with Human Feedback

    Mar 1, 2025Debmalya Mandal, Paulius Sasnauskas, Goran RadanovicDirect Preference OptimizationLLM Fine-Tuning

    Dept. of Computer Science University of Warwick, UK · Department of Computing Science University of Alberta, Edmonton, Canada · Max-Planck Institute for Software Systems, Germany

  8. Corruption Robust Offline Reinforcement Learning with Human Feedback

    Feb 9, 2024Debmalya Mandal, Andi Nika, Parameswaran Kamalaruban +2Offline RLCorruption Robustness

    Dept. of Computer Science University of Warwick, UK · Max-Planck Institute for Software Systems, Germany · Independent Researcher London, UK