Reward Function

Reward functions, crucial for guiding reinforcement learning agents towards desired behaviors, are the focus of intense research. Current efforts center on automatically learning reward functions from diverse sources like human preferences, demonstrations (including imperfect ones), and natural language descriptions, often employing techniques like inverse reinforcement learning, large language models, and Bayesian optimization within various architectures including transformers and generative models. This research is vital for improving the efficiency and robustness of reinforcement learning, enabling its application to complex real-world problems where manually designing reward functions is impractical or impossible. The ultimate goal is to create more adaptable and human-aligned AI systems.

Papers

February 20, 2023

Multiagent Inverse Reinforcement Learning via Theory of Mind Reasoning
Haochen Wu, Pedro Sequeira, David V. Pynadath
Theoretical Understanding Reward Function Maximum Entropy Reinforcement Learning Unknown Teammate Mind Reasoning

February 19, 2023

AIIR-MIX: Multi-Agent Reinforcement Learning Meets Attention Individual Intrinsic Reward Mixing Network
Wei Li, Weiyan Liu, Shitong Shao, Shiyi Huang
Multi Agent Reinforcement Learning Reward Function Intrinsic Reward

February 9, 2023

CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learning
Sheng Yue, Guanbo Wang, Wei Shao, Zhaofeng Zhang, Sen Lin, Ju Ren, Junshan Zhang
Reward Function Inverse Reinforcement Learning Exploration Exploitation Reward Misspecification Conservative Reward

February 3, 2023

Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased
Chao Yu, Jiaxuan Gao, Weilin Liu, Botian Xu, Hao Tang, Jiaqi Yang, Yu Wang, Yi Wu
Zero Shot Multi Agent Reinforcement Learning Real Human Reward Function Self Play Adaptive Policy

January 25, 2023

DIFFER: Decomposing Individual Reward for Fair Experience Replay in Multi-Agent Reinforcement Learning
Xunhan Hu, Jian Zhao, Wengang Zhou, Ruili Feng, Houqiang Li
Multi Agent Reinforcement Learning Reward Function Reward Model Cooperative Multi Agent Reinforcement Learning Experience Replay Diverse Strategy Social Reward

January 23, 2023

A Data-Efficient Visual-Audio Representation with Intuitive Fine-tuning for Voice-Controlled Robots
Peixin Chang, Shuijing Liu, Tianchen Ji, Neeloy Chakraborty, Kaiwen Hong, Katherine Driggs-Campbell
Contrastive Learning Fine Tuning Reward Function Data Efficient Audio Visual Representation Robot Goal

January 21, 2023

The configurable tree graph (CT-graph): measurable problems in partially observable and distal reward environments for lifelong reinforcement learning
Andrea Soltoggio, Eseoghene Ben-Iwhiwhu, Christos Peridis, Pawel Ladosz, Jeffery Dick, Praveen K. Pilly, Soheil Kolouri
Reward Function Sparse Reward Sparse Reward Environment Lifelong Reinforcement Learning Memory Less Agent Solved Problem Hierarchical Reward

January 20, 2023

Modeling Moral Choices in Social Dilemmas with Multi-Agent Reinforcement Learning
Elizaveta Tennant, Stephen Hailes, Mirco Musolesi
Multi Agent Reinforcement Learning Reward Function Social Dilemma Moral Agent Moral Decision

January 18, 2023

DIRECT: Learning from Sparse and Shifting Rewards using Discriminative Reward Co-Training
Philipp Altmann, Thomy Phan, Fabian Ritz, Thomas Gabor, Claudia Linnhoff-Popien
Reward Function Many Sparse Policy OpTimization Reward Collapse

January 11, 2023

Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Models
Yi Liu, Gaurav Datta, Ellen Novoseller, Daniel S. Brown
Reinforcement Learning High Efficiency Reward Function Policy OpTimization Learned Dynamic

January 9, 2023

On The Fragility of Learned Reward Functions
Lev McKinney, Yawen Duan, David Krueger, Adam Gleave
Reward Function Reward Learning Preference Based Reward

January 5, 2023

Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping
Lina Mezghani, Sainbayar Sukhbaatar, Piotr Bojanowski, Alessandro Lazaric, Karteek Alahari
Reward Function Reward Shaping Long Horizon Task Goal Conditioned Policy Continuous Control Task Offline Goal Conditioned Reinforcement Learning

January 3, 2023

Benchmarks and Algorithms for Offline Preference-Based Reward Learning
Daniel Shin, Anca D. Dragan, Daniel S. Brown
New Benchmark Practical Algorithm Reward Function Policy OpTimization Reward Learning Action Free Offline

January 2, 2023

January 1, 2023

Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits
Ruibo Liu, Chenyan Jia, Ge Zhang, Ziyu Zhuang, Tony X Liu, Soroush Vosoughi
Language Model Reward Function LLM Fine Tuning Value Alignment Human Value Text Alignment Text Editing

December 31, 2022

Contextual Bandits and Optimistically Universal Learning
Moise Blanchard, Steve Hanneke, Patrick Jaillet
Multi Armed Bandit Reward Function Contextual Bandit Contextual Bandit Problem Universal Learning Universal Consistency

December 17, 2022

Training Robots to Evaluate Robots: Example-Based Interactive Reward Functions for Policy Learning
Kun Huang, Edward S. Hu, Dinesh Jayaraman
Non Humanoid Robot Reward Function Real Robot Interactive Behavior Training Robot Robotic Skill

December 9, 2022

On the Sensitivity of Reward Inference to Misspecified Human Models
Joey Hong, Kush Bhatia, Anca Dragan
Reward Function Reward Learning Human Model Network Sensitivity Reward Estimation Accuracy Reward Inference

December 7, 2022

Tiered Reward Functions: Specifying and Fast Learning of Desired Behavior
Zhiyuan Zhou, Shreyas Sundara Raman, Henry Sowerby, Michael L. Littman
Reward Function Reward Report Fast Learning Task Specific Reward Reward Function Design

Reward Function

Papers

Multiagent Inverse Reinforcement Learning via Theory of Mind Reasoning

AIIR-MIX: Multi-Agent Reinforcement Learning Meets Attention Individual Intrinsic Reward Mixing Network

CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learning

Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased

DIFFER: Decomposing Individual Reward for Fair Experience Replay in Multi-Agent Reinforcement Learning

A Data-Efficient Visual-Audio Representation with Intuitive Fine-tuning for Voice-Controlled Robots

The configurable tree graph (CT-graph): measurable problems in partially observable and distal reward environments for lifelong reinforcement learning

Modeling Moral Choices in Social Dilemmas with Multi-Agent Reinforcement Learning

DIRECT: Learning from Sparse and Shifting Rewards using Discriminative Reward Co-Training

Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Models

On The Fragility of Learned Reward Functions

Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping

Benchmarks and Algorithms for Offline Preference-Based Reward Learning

SIRL: Similarity-based Implicit Representation Learning

On the Challenges of using Reinforcement Learning in Precision Drug Dosing: Delay and Prolongedness of Action Effects

Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits

Contextual Bandits and Optimistically Universal Learning

Training Robots to Evaluate Robots: Example-Based Interactive Reward Functions for Policy Learning

On the Sensitivity of Reward Inference to Misspecified Human Models

Tiered Reward Functions: Specifying and Fast Learning of Desired Behavior