Dense Reward

Dense reward in reinforcement learning (RL) focuses on designing reward functions that provide frequent, informative feedback to the agent, contrasting with sparse rewards that only signal success or failure. Current research explores methods for automatically generating dense rewards from sparse data, leveraging techniques like large language models and intrinsic motivation, and adapting existing RL algorithms (e.g., PPO, TD3, REDQ) to effectively utilize this richer feedback. This shift towards dense rewards promises to significantly improve sample efficiency and the ability to train RL agents for complex tasks in robotics and other domains, reducing the need for extensive manual reward engineering.

Papers

October 23, 2024

Adaptive Dense Reward: Understanding the Gap Between Action and Reward Space in Alignment
Yanshi Li, Shaopan Xiong, Gengru Chen, Xiaoyang Li, Yijia Luo, Xingyao Zhang, Yanhui Huang, Xingyuan Bu, Yingshui Tan, Chun Yuan, Jiamang Wang, Wenbo Su, Bo Zheng
Reinforcement Learning Alignment Problem Deep Space Action Feature Reinforcement Learning From Human Feedback Reward Report Level Supervision Dense Reward

September 13, 2024

Quasimetric Value Functions with Dense Rewards
Khadichabonu Valieva, Bikramjit Banerjee
Sparse Reward Dense Reward Quasimetric Learning

July 31, 2024

Image-Based Deep Reinforcement Learning with Intrinsically Motivated Stimuli: On the Execution of Complex Robotic Tasks
David Valencia, Henry Williams, Yuning Xing, Trevor Gee, Minas Liarokapis, Bruce A. MacDonald
Reinforcement Learning Deep Reinforcement Learning Efficient Exploration Intrinsic Motivation Complex Robotic Task Test Stimulus Dense Reward

May 26, 2024

Rewarded Region Replay (R3) for Policy Learning with Discrete Action Space
Bangzheng Li, Ningshan Ma, Zifan Wang
Action Space Policy Reinforcement Learning Dense Reward Data Rich Region

April 25, 2024

DrS: Learning Reusable Dense Rewards for Multi-Stage Tasks
Tongzhou Mu, Minghua Liu, Hao Su
Sparse Reward Reward Engineering Dense Reward Multi Stage Task

March 11, 2024

Unveiling the Significance of Toddler-Inspired Reward Transition in Goal-Oriented Reinforcement Learning
Junseok Park, Yoonsung Kim, Hee Bin Yoo, Min Whoo Lee, Kibeom Kim, Won-Seok Choi, Minsu Lee, Byoung-Tak Zhang
Reinforcement Learning Importance Aware Dense Reward Goal Oriented Reinforcement Learning Reward Transition

February 13, 2024

A Dense Reward View on Aligning Text-to-Image Diffusion with Preference
Shentao Yang, Tianqi Chen, Mingyuan Zhou
Text to Image Diffusion Model Preference Feedback Preference Alignment Text to Image Diffusion Dense Reward

February 1, 2024

Dense Reward for Free in Reinforcement Learning from Human Feedback
Alex J. Chan, Hao Sun, Samuel Holt, Mihaela van der Schaar
Reinforcement Learning Human Feedback Reward Model Reward Shaping Traditional Reinforcement Learning Dense Reward

December 14, 2023

Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft
Hao Li, Xue Yang, Zhaokai Wang, Xizhou Zhu, Jie Zhou, Yu Qiao, Xiaogang Wang, Hongsheng Li, Lewei Lu, Jifeng Dai
Sparse Reward Reward Learning Reward Structure Dense Reward

December 10, 2023

Efficient Sparse-Reward Goal-Conditioned Reinforcement Learning with a High Replay Ratio and Regularization
Takuya Hiraoka
Reinforcement Learning Sparse Reward Environment Double Q Learning Dense Reward

October 31, 2023

Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning
Ruizhe Shi, Yuyao Liu, Yanjie Ze, Simon S. Du, Huazhe Xu
Pre Trained Language Model Offline Reinforcement Learning Real Power Dense Reward

October 4, 2023

Reinforcement Learning with Foundation Priors: Let the Embodied Agent Efficiently Learn on Its Own
Weirui Ye, Yunsheng Zhang, Haoyang Weng, Xianfan Gu, Shengjie Wang, Tong Zhang, Mengchen Wang, Pieter Abbeel, Yang Gao
Actor Critic Goal Conditioned Reinforcement Learning Generalist Model Dense Reward

September 13, 2023

Stable In-hand Manipulation with Finger Specific Multi-agent Shadow Reward
Lingfeng Tao, Jiucai Zhang, Xiaoli Zhang
Hand Manipulation Dexterous in Hand Manipulation Dense Reward Stable Manipulation

Dense Reward

Papers

Adaptive Dense Reward: Understanding the Gap Between Action and Reward Space in Alignment

Quasimetric Value Functions with Dense Rewards

Image-Based Deep Reinforcement Learning with Intrinsically Motivated Stimuli: On the Execution of Complex Robotic Tasks

Rewarded Region Replay (R3) for Policy Learning with Discrete Action Space

DrS: Learning Reusable Dense Rewards for Multi-Stage Tasks

Unveiling the Significance of Toddler-Inspired Reward Transition in Goal-Oriented Reinforcement Learning

A Dense Reward View on Aligning Text-to-Image Diffusion with Preference

Dense Reward for Free in Reinforcement Learning from Human Feedback

Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft

Efficient Sparse-Reward Goal-Conditioned Reinforcement Learning with a High Replay Ratio and Regularization

Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning

Reinforcement Learning with Foundation Priors: Let the Embodied Agent Efficiently Learn on Its Own

Stable In-hand Manipulation with Finger Specific Multi-agent Shadow Reward