Reward Learning

Reward learning focuses on automatically inferring or shaping reward functions for reinforcement learning agents, enabling them to learn complex tasks without explicit programming. Current research emphasizes efficient reward function design using large language models, inverse reinforcement learning from human demonstrations (including suboptimal or preference-based feedback), and novel algorithms to improve sample efficiency and robustness. These advancements are crucial for deploying reinforcement learning in real-world applications, such as robotics, personalized education, and the responsible development of AI systems, by enabling more effective and human-aligned learning.

Papers

November 18, 2024

Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
Yannick Metz, David Lindner, Raphaël Baur, Mennatallah El-Assady
Reinforcement Learning Human Feedback Area MaPPing Deep Space Multi Agent Learning Reward Learning Feedback Type

November 6, 2024

Temporal-Difference Learning Using Distributed Error Signals
Jonas Guan, Shon Eduard Verch, Claas Voelcker, Ethan C. Jackson, Nicolas Papernot, William A. Cunningham
Error Detection Reward Learning Deep RL Temporal Difference Learning

October 21, 2024

On The Global Convergence Of Online RLHF With Neural Parametrization
Mudit Gaur, Amrit Singh Bedi, Raghu Pasupathy, Vaneet Aggarwal
Large Language Model Reinforcement Learning Neural Network Reinforcement Learning From Human Feedback Global Convergence Reward Learning AI Alignment

October 11, 2024

Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videos
Harsh Mahesheka, Zhixian Xie, Zhaoran Wang, Wanxin Jin
Vision Language Model Bilevel Optimization Task Specific Model Reward Learning Complex Behavior Reward Design Online Video

October 8, 2024

Trajectory Improvement and Reward Learning from Comparative Language Feedback
Zhaojing Yang, Miru Jun, Jeremy Tien, Stuart J. Russell, Anca Dragan, Erdem Bıyık
Latent Space Reward Learning Robot Trajectory Language Feedback Trajectory Design Preference Based Reward

October 5, 2024

Reward Learning From Preference With Ties
Jinsong Liu, Dongdong Ge, Ruihao Zhu
Preference Feedback Reward Learning Preference Rating Synthetic Preference Preference Modeling Tie KD Preference Based

September 12, 2024

Reinforcement Learning Discovers Efficient Decentralized Graph Path Search Strategies
Alexei Pisacane, Victor-Alexandru Darvariu, Mirco Musolesi
Reinforcement Learning Real World Reward Learning Graph Search Navigable Graph

September 4, 2024

Large Language Models as Efficient Reward Function Searchers for Custom-Environment Multi-Objective Reinforcement Learning
Guanwen Xie, Jingzehua Xu, Yiyuan Yang, Yimian Ding, Shuai Zhang
Reinforcement Learning Reward Function Multi Objective Reinforcement Learning Reward Learning Task Specific Reward

June 26, 2024

Encouraging Responsible Use of Generative AI in Education: A Reward-Based Learning Approach
Aditi Singh, Abul Ehtesham, Saket Kumar, Gaurav Kumar Gupta, Tala Talaei Khoei
Training Data Generative AI Education Domain Reward Learning Learning Structure

June 3, 2024

Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning
Andreas Schlaginhaufen, Maryam Kamgarpour
Optimal Policy Inverse Reinforcement Learning Task Transferability Reward Report Reward Learning Transition Dynamic Regularized Reinforcement Learning

May 28, 2024

Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment
Jiaxiang Li, Siliang Zeng, Hoi-To Wai, Chenliang Li, Alfredo Garcia, Mingyi Hong
Reinforcement Learning Preference Learning Reward Learning Human Demonstration LLM Alignment

May 23, 2024

Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input
Andi Peng, Yuying Sun, Tianmin Shu, David Abel
Reward Model Linear Bandit Reward Learning Natural Language Input Social Learning Preference Based Reward Feature Preference

May 20, 2024

A Unified Linear Programming Framework for Offline Reward Learning from Human Demonstrations and Feedback
Kihyun Kim, Jiawei Zhang, Asuman Ozdaglar, Pablo A. Parrilo
Reinforcement Learning Human Feedback Inverse Reinforcement Learning Linear Programming Reward Learning Human Demonstration

May 12, 2024

Learning Reward for Robot Skills Using Large Language Models via Self-Alignment
Yuwei Zeng, Yao Mu, Lin Shao
Reward Function Reward Learning Robot Skill Self Alignment

April 30, 2024

Numeric Reward Machines
Kristina Levina, Nikolaos Pappas, Athanasios Karapantelakis, Aneta Vulgarakis Feljan, Jendrik Seipp
Q Learning Reward Learning Reward Machine Feasible Reward

April 23, 2024

ID-Aligner: Enhancing Identity-Preserving Text-to-Image Generation with Reward Feedback Learning
Weifeng Chen, Jiacheng Zhang, Jie Wu, Hefeng Wu, Xuefeng Xiao, Liang Lin
Reward Learning Identity Related Identity Preservation AI Art Reward Fine Tuning

April 19, 2024

Breaching the Bottleneck: Evolutionary Transition from Reward-Driven Learning to Reward-Agnostic Domain-Adapted Learning in Neuromodulated Neural Nets
Solvi Arnold, Reiji Suzuki, Takaya Arita, Kimitoshi Yamazaki
Reinforcement Learning Domain Adaptive Reward Learning Reward Signal Major Challenge Bottleneck Biological Intelligence

April 12, 2024

Hindsight PRIORs for Reward Learning from Human Preferences
Mudit Verma, Katherine Metcalf
Reinforcement Learning Policy Learning Human Preference Reward Learning Hindsight Instruction Reward Engineering

April 10, 2024

Reward Learning from Suboptimal Demonstrations with Applications in Surgical Electrocautery
Zohre Karimi, Shing-Hei Ho, Bao Thach, Alan Kuntz, Daniel S. Brown
Reinforcement Learning Financial Application Reward Function Robotic Surgery Reward Learning Suboptimal Demonstration Robust Reward

April 4, 2024

Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, Tengyang Xie
Language Model Reinforcement Learning Nash Equilibrium Preference Feedback Reward Learning Reward Maximization

Reward Learning

Papers

Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework

Temporal-Difference Learning Using Distributed Error Signals

On The Global Convergence Of Online RLHF With Neural Parametrization

Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videos

Trajectory Improvement and Reward Learning from Comparative Language Feedback

Reward Learning From Preference With Ties

Reinforcement Learning Discovers Efficient Decentralized Graph Path Search Strategies

Large Language Models as Efficient Reward Function Searchers for Custom-Environment Multi-Objective Reinforcement Learning

Encouraging Responsible Use of Generative AI in Education: A Reward-Based Learning Approach

Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning

Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment

Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input

A Unified Linear Programming Framework for Offline Reward Learning from Human Demonstrations and Feedback

Learning Reward for Robot Skills Using Large Language Models via Self-Alignment

Numeric Reward Machines

ID-Aligner: Enhancing Identity-Preserving Text-to-Image Generation with Reward Feedback Learning

Breaching the Bottleneck: Evolutionary Transition from Reward-Driven Learning to Reward-Agnostic Domain-Adapted Learning in Neuromodulated Neural Nets

Hindsight PRIORs for Reward Learning from Human Preferences

Reward Learning from Suboptimal Demonstrations with Applications in Surgical Electrocautery

Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences