Sparse Reward

Sparse reward reinforcement learning tackles the challenge of training agents in environments where positive feedback is infrequent, hindering efficient learning. Current research focuses on improving exploration strategies through techniques like optimistic Thompson sampling and intrinsic reward shaping, often employing deep deterministic policy gradients (DDPG), transformers, and generative flow networks (GFlowNets) to address the problem. These advancements aim to enhance sample efficiency and improve the performance of reinforcement learning agents in complex, real-world scenarios characterized by sparse rewards, such as robotics and multi-agent systems. The resulting improvements in sample efficiency and robustness have significant implications for various applications, including robotics, personalized recommendations, and human-AI collaboration.

Papers

June 12, 2022

Dealing with Sparse Rewards in Continuous Control Robotics via Heavy-Tailed Policies
Souradip Chakraborty, Amrit Singh Bedi, Alec Koppel, Pratap Tokekar, Dinesh Manocha
Sparse Reward Sparse Reward Environment Continuous Control Task Heavy Tailed Policy

June 8, 2022

Deep Hierarchical Planning from Pixels
Danijar Hafner, Kuang-Huei Lee, Ian Fischer, Pieter Abbeel
World Model Sparse Reward Tetromino Pixel Hierarchical Reinforcement Learning Latent Intent

June 3, 2022

Challenges to Solving Combinatorially Hard Long-Horizon Deep RL Tasks
Andrew C. Li, Pashootan Vaezipoor, Rodrigo Toro Icarte, Sheila A. McIlraith
Deep Reinforcement Learning Technical Challenge Complex Reasoning Sparse Reward Long Horizon Task

May 31, 2022

Hierarchies of Reward Machines
Daniel Furelos-Blanco, Mark Law, Anders Jonsson, Krysia Broda, Alessandra Russo
Sparse Reward Part Whole Hierarchy Finite State Machine Reward Machine Subtask Assignment

May 28, 2022

Reinforcement Learning for Branch-and-Bound Optimisation using Retrospective Trajectories
Christopher W. F. Parsonson, Alexandre Laterre, Thomas D. Barrett
Reinforcement Learning Machine Learning Sparse Reward Handwritten Trajectory Branch and Bound Combinatorial Optimisation Problem Mixed Integer Linear Program

May 19, 2022

May 2, 2022

Exploration in Deep Reinforcement Learning: A Survey
Pawel Ladosz, Lilian Weng, Minwoo Kim, Hyondong Oh
Reinforcement Learning Timely Survey Deep Reinforcement Learning Environment Exploration Sparse Reward Exploration Method Exploration Strategy

April 16, 2022

Efficient Reinforcement Learning for Unsupervised Controlled Text Generation
Bhargav Upadhyay, Akhilesh Sudhakar, Arjun Maheswaran
Text Generation Sparse Reward Efficient Reinforcement Learning Unsupervised Text Style Transfer

April 5, 2022

Learning to Bid Long-Term: Multi-Agent Reinforcement Learning with Long-Term and Sparse Reward in Repeated Auction Games
Jing Tan, Ramin Khalili, Holger Karl
LeArning Abstract Multi Agent Reinforcement Learning Long Term Reinforcement Learning Algorithm Sparse Reward Reward Signal Short Term Sequential Game

March 25, 2022

Dealing with Sparse Rewards Using Graph Neural Networks
Matvey Gerasyov, Ilya Makarov
Graph Neural Network Deep Reinforcement Learning Sparse Reward Reward Shaping

March 21, 2022

Self-Imitation Learning from Demonstrations
Georgiy Pshikhachev, Dmitry Ivanov, Vladimir Egorov, Aleksei Shpilman
Reinforcement Learning Sparse Reward Noisy Demonstration Suboptimal Demonstration Learning From Demonstration

March 17, 2022

The Frost Hollow Experiments: Pavlovian Signalling as a Path to Coordination and Communication Between Agents
Patrick M. Pilarski, Andrew Butcher, Elnaz Davoodi, Michael Bradley Johanson, Dylan J. A. Brenneis, Adam S. R. Parker, Leslie Acker, Matthew M. Botvinick, Joseph Modayil, Adam White
Agent Smith Timely Communication Sparse Reward Reinforcement Learning Agent Prior Coordination Communication Learning

March 2, 2022

February 24, 2022

Collaborative Training of Heterogeneous Reinforcement Learning Agents in Environments with Sparse Rewards: What and When to Share?
Alain Andres, Esther Villar-Rodriguez, Javier Del Ser
Environment Feature Sparse Reward Intrinsic Reward Heterogeneous Agent Intrinsic Motivation Reinforcement Learning Environment Collaborative Training Exploration Strategy Heterogeneous Multi Agent Reinforcement Learning

February 19, 2022

Shaping Advice in Deep Reinforcement Learning
Baicen Xiao, Bhaskar Ramasubramanian, Radha Poovendran
Reinforcement Learning Deep Reinforcement Learning Multi Agent Reinforcement Learning Sparse Reward Single Agent Reinforcement Learning

February 15, 2022

Disentangling Successor Features for Coordination in Multi-agent Reinforcement Learning
Seung Hyun Kim, Neale Van Stralen, Girish Chowdhary, Huy T. Tran
Multi Agent Reinforcement Learning Sparse Reward Multi Agent Environment Prior Coordination Individual Agent Decentralized Agent Successor Feature

February 9, 2022

Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstration
Desik Rengarajan, Gargi Vaidya, Akshay Sarvesh, Dileep Kalathil, Srinivas Shakkottai
Reinforcement Learning Sparse Reward Efficient Reinforcement Learning Human Guidance Optimal Behavior Real World Reinforcement Learning Reward Feedback

February 5, 2022

ASHA: Assistive Teleoperation via Human-in-the-Loop Reinforcement Learning
Sean Chen, Jensen Gao, Siddharth Reddy, Glen Berseth, Anca D. Dragan, Sergey Levine
Robotic Manipulation Sparse Reward Human in the Loop Human in the Loop Reinforcement User Feedback Assistive Teleoperation

Sparse Reward

Papers

Dealing with Sparse Rewards in Continuous Control Robotics via Heavy-Tailed Policies

Deep Hierarchical Planning from Pixels

Challenges to Solving Combinatorially Hard Long-Horizon Deep RL Tasks

Hierarchies of Reward Machines

Reinforcement Learning for Branch-and-Bound Optimisation using Retrospective Trajectories

Dexterous Robotic Manipulation using Deep Reinforcement Learning and Knowledge Transfer for Complex Sparse Reward-based Tasks

Image Augmentation Based Momentum Memory Intrinsic Reward for Sparse Reward Visual Scenes

Exploration in Deep Reinforcement Learning: A Survey

Efficient Reinforcement Learning for Unsupervised Controlled Text Generation

Learning to Bid Long-Term: Multi-Agent Reinforcement Learning with Long-Term and Sparse Reward in Repeated Auction Games

Dealing with Sparse Rewards Using Graph Neural Networks

Self-Imitation Learning from Demonstrations

The Frost Hollow Experiments: Pavlovian Signalling as a Path to Coordination and Communication Between Agents

Learning in Sparse Rewards settings through Quality-Diversity algorithms

Follow your Nose: Using General Value Functions for Directed Exploration in Reinforcement Learning

Collaborative Training of Heterogeneous Reinforcement Learning Agents in Environments with Sparse Rewards: What and When to Share?

Shaping Advice in Deep Reinforcement Learning

Disentangling Successor Features for Coordination in Multi-agent Reinforcement Learning

Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstration

ASHA: Assistive Teleoperation via Human-in-the-Loop Reinforcement Learning