Sparse-Reward RL

RL: Reinforcement Learning

Sparse-reward reinforcement learning asks how agents can discover useful behavior when environments provide little feedback, a common obstacle in long-horizon robotics, games, and interactive tasks. Exploration methods such as intrinsic motivation, goal relabeling, curricula, and hierarchical skills help agents reach informative states in settings like MiniGrid, Montezuma’s Revenge, and MuJoCo. Other approaches derive denser credit from successful or failed trajectories, learned progress rewards, value estimates, or language-model feedback, including for agents evaluated on ALFWorld and WebShop. A central tension is that extra guidance can improve learning efficiency but may be noisy, encourage reward hacking, or shift the objective, so reliable credit assignment and transfer beyond familiar benchmarks remain open challenges.

Latest papers

164 papers31 papers in the last four weeks, up 107% on the four weeks before. 0.2% of all new papers.

Soft Deterministic Policy Gradient with Gaussian Smoothing
May 7, 2026Hyunjun Na, Donghwan LeeRL Control Policy Gradient
School of Electrical Engineering, KAIST 291 Daehak-ro, Yuseong-gu, Daejeon 34141, Korea
QHyer: Q-conditioned Hybrid Attention-mamba Transformer for Offline Goal-conditioned RL
May 3, 2026Xing Lei, Jincheng Wang, Xuetao Zhang +1Hybrid Mamba-Transformer Architectures Offline RL
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · Computer Science, University College London · School of Engineering, Westlake University
Joint Energy Management and Coordinated AIGC Workload Scheduling for Distributed Data Centers: A Diffusion-Aided Reward Shaping Approach
May 3, 2026Yang Fu, Peng Qin, Liming Chen +3Reward Shaping Data Center Energy Management
Yang Fu, Peng Qin, Member, IEEE, Liming Chen, Zihao Zhang, Hao Yu, and Yifei Wang
$S^3$ -R1: Learning to Retrieve and Answer Step-by-Step with Synthetic Data
May 2, 2026Harsh Goel, Akhil Udathu, Susmija Jabbireddy +2Multi-Hop QA Synthetic Data Generation
The University of Texas at Austin · Work done during an internship at Google DeepMind · Google DeepMind
AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learning
May 1, 2026Haotian Zhao, Songlin Zhou, Yuxin Zhang +9Agentic RL Credit Assignment in RL
Baidu · Tsinghua University · Fudan University
Sample-efficient Neuro-symbolic Proximal Policy Optimization
Apr 28, 2026Simone Murari, Celeste Veronese, Daniele MeliReinforcement Learning Proximal Policy Optimization
Dept. of Computer Science, University of Verona, Italy
Zero Shot Coordination for Sparse Reward Tasks with Diverse Reward Shapings
Apr 28, 2026Keenan Powell, Peihong Yu, Pratap TokekarMulti-Agent Coordination Reward Shaping
Department of Computer Science University of Maryland, College Park
ReCast: Recasting Learning Signals for Reinforcement Learning in Generative Recommendation
Apr 24, 2026Peiyan Zhang, Hanmo Liu, Chengxuan Tong +3RL for Generative Models Sparse-Reward RL
1Huawei Technologies Co., Ltd.
Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learning
Apr 22, 2026Aravind Venugopal, Jiayu Chen, Xudong Wu +3Reward Shaping Temporal Credit Assignment
Carnegie Mellon University · The University of Hong Kong · INFIFORCE Intelligent Technology +1
Reducing Credit Assignment Variance via Counterfactual Reasoning Paths
Apr 20, 2026Fei Ding, Yongkang Zhang, Youwei Wang +1RL for Language Model Reasoning Credit Assignment in RL
Alibaba Group · Tsinghua University
Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization
Apr 15, 2026Junzhe Wang, Zhiheng Xi, Yajie Yang +4LLM Agents RL for LLM Agents
Fudan University · Shanghai Artificial Intelligence Laboratory · Shanghai Key Laboratory of Intelligent Information Processing
Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Sequence-Level Likelihood
Apr 14, 2026Xingyu Lin, Yilin Wen, Du Su +5RL for Language Model Reasoning Policy Optimization
College of Computer Science and Technology, Jilin University · Key Laboratory of Symbolic Computation and Knowledge Engineering of MOE, Jilin University · Baidu Inc. +2
Utilizing and Calibrating Hindsight Process Rewards via Reinforcement with Mutual Information Self-Evaluation
Apr 13, 2026Jiashu Yao, Heyan Huang, Zeming Liu +1Reinforcement Learning RL for LLM Agents
Beijing Institute of Technology · Beihang University
SEARL: Joint Optimization of Policy and Tool Graph Memory for Self-Evolving Agents
Apr 9, 2026Xinshun Feng, Xinhao Song, Lijun Li +2Self-Evolving Agents Memory-Augmented Agents
Shanghai Artificial Intelligence Laboratory · Shanghai Jiaotong University
Smart Commander: A Hierarchical Reinforcement Learning Framework for Fleet-Level PHM Decision Optimization
Apr 8, 2026Yong Si, Mingfei Lu, Jing Li +4Hierarchical RL Predictive Maintenance
School of Aerospace Engineering, Tsinghua University, China · College of Information and Control Engineering, Xi’an University of Architecture and Technology, China · Hangzhou International Innovation Institute, Beihang University, China +2
Reason in Chains, Learn in Trees: Self-Rectification and Grafting for Multi-turn Agent Policy Optimization
Apr 8, 2026Yu Li, Sizhe Tang, Tian LanRL for Language Model Reasoning RL for LLM Agents
Department of Electrical and Computer Engineering, George Washington University
Design Conditions for Intra-Group Learning of Sequence-Level Rewards: Token Gradient Cancellation
Apr 4, 2026Fei Ding, Yongkang Zhang, youwei wang +1RL for Language Model Reasoning Step-Level Credit Assignment in RL
Alibaba Group · Tsinghua University
MedLoc-R1: Performance-Aware Curriculum Reward Scheduling for GRPO-Based Medical Visual Grounding
Mar 30, 2026Guangjing Yang, Ziyuan Qin, Chaoran Zhang +6Medical Image Grounding Reward Shaping
Beijing University of Posts and Telecommunications · Emory University · Peking University +1
Match or Replay: Self Imitating Proximal Policy Optimization
Mar 29, 2026Gaurav Chaudhary, Laxmidhar Behera, Washim Uddin MondalRL Exploration Sample-Efficient RL
Department of Electrical Engineering, Indian Institute of Technology Kanpur
Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration
Mar 23, 2026Zakaria Mhammedi, James CohanUncertainty-Aware Exploration RL Exploration
Google Research, NYC
P^2O: Joint Policy and Prompt Optimization
Mar 23, 2026Xinyu Lu, Kaiqi Zhang, Jinglin Yang +6RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences +2
HISR: Hindsight Information Modulated Segmental Process Rewards For Multi-turn Agentic Reinforcement Learning
Mar 19, 2026Zhicong Lu, Zichuan Lin, Wei Jia +7RL for LLM Agents Agentic RL
Aerospace Information Research Institute, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Tencent Hunyuan +1
SQL-ASTRA: Alleviating Sparse Feedback in Agentic SQL via Column-Set Matching and Trajectory Aggregation
Mar 17, 2026Long Li, Zhijian Zhou, Jiangxuan Long +5Text-to-SQL Step-Level Credit Assignment in RL
Griffith University, Brisbane, Australia · Fudan University, Shanghai, China · Shanghai Innovation Institute, China +4
Hindsight-Anchored Policy Optimization: Learning Through Hindsight with Thompson Sampling-Inspired Adaptive Gating
Mar 11, 2026Yuning Wu, Ke Wang, Haoran Liu +3Reinforcement Learning RL for Language Model Reasoning
Amazon
TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMs
Mar 11, 2026Yutao Xie, Nathaniel Thomas, Nicklas Hansen +3RL for Language Model Reasoning Reward Shaping
UC San Diego · AWS AI
$V_{0.5}$ : Generalist Value Model as a Prior for Sparse RL Rollouts
Mar 11, 2026Yi-Kai Zhang, Yueqing Sun, Hongyan Hao +4Reinforcement Learning with Verifiable Rewards Value Function Estimation
School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University · Meituan, China
Reward-Zero: Language Embedding Driven Implicit Reward Mechanisms for Reinforcement Learning
Mar 10, 2026Heng Zhang, Haddy Alchaer, Arash Ajoudani +1Reward Design for RL Dense Reward RL
Human-Robot Interfaces and Interaction Lab, Istituto Italiano di Tecnologia, Genova, Italy · Ph.D. program of national interest in Robotics and Intelligent Machines (DRIM) and Universit ‘a di Genova, Genoa, Italy. · Edwardson School of Industrial Engineering, Purdue University, West Lafayette, IN, USA. +2
Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learning
Mar 4, 2026Lei Huang, Xiang Cheng, Chenxiao Zhao +6RL Exploration RL for Language Models
Harbin Institute of Technology · Xiaohongshu Inc.
Reinforcement Learning with Symbolic Reward Machines
Mar 3, 2026Thomas Krug, Daniel NeiderNon-Markovian RL Reinforcement Learning
TU Dortmund University · Research Center Trustworthy Data Science and Security
Learn Hard Problems During RL with Reference Guided Fine-tuning
Mar 1, 2026Yangzhen Wu, Shanda Li, Zixin Wen +5LLM Mathematical Reasoning RL for Language Model Reasoning
UC Berkeley · Carnegie Mellon University · ByteDance Seed

May 7, 2026J/K move · Enter open · S save

Soft Deterministic Policy Gradient with Gaussian Smoothing

Hyunjun Na, Donghwan Lee

School of Electrical Engineering, KAIST 291 Daehak-ro, Yuseong-gu, Daejeon 34141, Korea

RL Control Policy Gradient Policy Gradient Methods Sparse-Reward RL Deep Deterministic Policy Gradient

Open paper AskPDF arXiv

Deterministic policy gradient (DPG) is widely utilized for continuous control; however, it inherently relies on the differentiability of the critic with respect to the action during policy updates. This assumption is violated in practical control problems involving sparse or discrete rewards, leading to ill-defined policy gradients and unstable learning. To address these challenges, we propose a principled alternative based on a smoothed Bellman equation formulated via Gaussian smoothing. Specifically, we define a novel action-value function based on a smoothed Bellman equation and derive the soft deterministic policy gradient (Soft-DPG). Our formulation eliminates explicit dependence on critic action-gradients and ensures that the gradient remains well-defined even for non-smooth Q-functions. We instantiate this framework into a deep reinforcement learning algorithm, which we call soft deep deterministic policy gradient (Soft DDPG). Empirical evaluations on standard continuous control benchmarks and their discretized-reward variants show that Soft DDPG remains competitive in dense-reward settings and provides clear gains in most discretized-reward environments, where standard DDPG is more sensitive to irregular critic landscapes.