Reward Learning

Reward learning focuses on automatically inferring or shaping reward functions for reinforcement learning agents, enabling them to learn complex tasks without explicit programming. Current research emphasizes efficient reward function design using large language models, inverse reinforcement learning from human demonstrations (including suboptimal or preference-based feedback), and novel algorithms to improve sample efficiency and robustness. These advancements are crucial for deploying reinforcement learning in real-world applications, such as robotics, personalized education, and the responsible development of AI systems, by enabling more effective and human-aligned learning.

Papers

January 9, 2023

On The Fragility of Learned Reward Functions
Lev McKinney, Yawen Duan, David Krueger, Adam Gleave
Reward Function Reward Learning Preference Based Reward

January 3, 2023

Benchmarks and Algorithms for Offline Preference-Based Reward Learning
Daniel Shin, Anca D. Dragan, Daniel S. Brown
New Benchmark Practical Algorithm Reward Function Policy OpTimization Reward Learning Action Free Offline

December 9, 2022

On the Sensitivity of Reward Inference to Misspecified Human Models
Joey Hong, Kush Bhatia, Anca Dragan
Reward Function Reward Learning Human Model Network Sensitivity Reward Estimation Accuracy Reward Inference

November 30, 2022

Time-Efficient Reward Learning via Visually Assisted Cluster Ranking
David Zhang, Micah Carroll, Andreea Bobu, Anca Dragan
Human Feedback Reward Learning Human Label Visual Clustering

November 16, 2022

A Neural Active Inference Model of Perceptual-Motor Learning
Zhizhuo Yang, Gabriel J. Diaz, Brett R. Fajen, Reynold Bailey, Alexander Ororbia
Active Inference Reward Learning Visuo Motor Control Anticipatory Behavior

November 10, 2022

Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control
Xiang Fan, Yiwei Lyu, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency
Language Model Shot Learning Language Generation Pretrained Language Model Reward Learning Probability Distribution Nested Entity

October 28, 2022

Relative Behavioral Attributes: Filling the Gap between Symbolic Goal Specification and Reward Learning from Human Preferences
Lin Guan, Karthik Valmeekam, Subbarao Kambhampati
Human Preference Reward Learning Agent Behavior Goal Generation Behavioral Feature Conservative Reward

October 27, 2022

Learning on the Job: Self-Rewarding Offline-to-Online Finetuning for Industrial Insertion of Novel Connectors from Vision
Ashvin Nair, Brian Zhu, Gokul Narayanan, Eugen Solowjow, Sergey Levine
LeArning Abstract Strong Generalization Offline Reinforcement Learning Vision Paper Reward Function Reward Learning Insertion Task Connector Insertion

October 20, 2022

Robust Imitation via Mirror Descent Inverse Reinforcement Learning
Dong-Sig Han, Hyunseo Kim, Hyundo Lee, Je-Hwan Ryu, Byoung-Tak Zhang
Inverse Reinforcement Learning Reward Learning Mirror Descent Adversarial Imitation Learning Robust Imitation

October 17, 2022

Symbol Guided Hindsight Priors for Reward Learning from Human Preferences
Mudit Verma, Katherine Metcalf
Reward Function Preference Feedback Human Preference Reward Learning Hindsight Instruction

October 3, 2022

Reward Learning with Trees: Methods and Evaluation
Tom Bewley, Jonathan Lawry, Arthur Richards, Rachel Craddock, Ian Henderson
Global Evaluation NCD Method Reward Function Tree Specie Reward Learning Interpretable Tree Interpretable Reward Reward Tree

September 28, 2022

Argumentative Reward Learning: Reasoning About Human Preferences
Francis Rhys Ward, Francesco Belardinelli, Francesca Toni
Reinforcement Learning Human Preference Reward Learning

September 25, 2022

Reward Learning using Structural Motifs in Inverse Reinforcement Learning
Raeid Saqur
High Dimensional Reward Function Inverse Reinforcement Learning Reward Learning Sequential Task Novel Technique Sequence Motif

August 23, 2022

The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types
Gaurav R. Ghosal, Matthew Zurek, Daniel S. Brown, Anca D. Dragan
Mixed Effect Reward Learning Human Rationality Noisy Preference Multi Point Feedback Learning Reward

August 5, 2022

Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Jing Xu, Megan Ung, Mojtaba Komeili, Kushal Arora, Y-Lan Boureau, Jason Weston
Human Feedback Skill Learning Software Deployment Reward Learning Open Domain Dialogue Learning Framework Conversational Capability

July 20, 2022

Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations
Haoran Xu, Xianyuan Zhan, Honglei Yin, Huiling Qin
Offline Reinforcement Learning Reward Learning Suboptimal Demonstration Offline Imitation Learning Expert Policy

March 18, 2022

SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning
Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, Kimin Lee
Reinforcement Learning Data Augmentation Reward Function Reward Learning Preference Based Reinforcement Learning Semi Supervised Reward

March 17, 2022

GAC: A Deep Reinforcement Learning Model Toward User Incentivization in Unknown Social Networks
Shiqing Wu, Weihua Li, Quan Bai
Actor Critic Reward Learning Incentive Mechanism Social Influence Unknown Network Incentive Allocation

March 14, 2022

Invariance in Policy Optimisation and Partial Identifiability in Reward Learning
Joar Skalse, Matthew Farrugia-Roberts, Stuart Russell, Alessandro Abate, Adam Gleave
Reward Function Policy OpTimization Alpha Invariance Reward Learning Task Specific Reward Partial Identifiability

January 25, 2022

Dynamics-Aware Comparison of Learned Reward Functions
Blake Wulfe, Ashwin Balakrishna, Logan Ellis, Jean Mercat, Rowan McAllister, Adrien Gaidon
Reward Function Reward Learning