Sparse-Reward RL

RL: Reinforcement Learning

Sparse-reward reinforcement learning asks how agents can discover useful behavior when environments provide little feedback, a common obstacle in long-horizon robotics, games, and interactive tasks. Exploration methods such as intrinsic motivation, goal relabeling, curricula, and hierarchical skills help agents reach informative states in settings like MiniGrid, Montezuma’s Revenge, and MuJoCo. Other approaches derive denser credit from successful or failed trajectories, learned progress rewards, value estimates, or language-model feedback, including for agents evaluated on ALFWorld and WebShop. A central tension is that extra guidance can improve learning efficiency but may be noisy, encourage reward hacking, or shift the objective, so reliable credit assignment and transfer beyond familiar benchmarks remain open challenges.

Latest papers

164 papers31 papers in the last four weeks, up 107% on the four weeks before. 0.2% of all new papers.

Effectiveness of Curriculum Learning Depends on Reward Sparsity and Competing Optima
Oct 22, 2026John J. Vastola, Ann Huang, Satpreet H. Singh +2Curriculum RL Reinforcement Learning
Department of Neurobiology, Harvard Medical School · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University · Department of Psychology and Center for Brain Science, Harvard University
Progress Reward Model for Reinforcement Learning via Large Language Models
Oct 15, 2026Xiuhui Zhang, Ning Gao, Xingyu Jiang +4Reward Design for RL LLM-Guided RL
Beihang University, 37 Xueyuan Road, Haidian District, Beijing
Bellman-Consistent Trajectory RL for LLM Arithmetic Reasoning
Oct 15, 2026Lianfa LiRL for Language Model Reasoning Reinforcement Learning
Department of Computer Science, Stanford University · LREIS, Institute of Geographical Sciences and Natural Resources Research, Chinese Academy of Sciences
Long-Horizon RL Needs Muon
Oct 14, 2026Kai Ruan, Jinghao Lin, Tao Chen +3Long-Horizon RL Long-Horizon LLM Agents
Gaoling School of Artificial Intelligence, Renmin University of China · Duke University · Institute of Automation, Chinese Academy of Sciences +1
Reinforcement Learning via Advantage Guidance
Oct 14, 2026Zhiyuan Zhou, Sergey LevineReinforcement Learning Offline RL
UC Berkeley
Contextual Credit Policy Optimization
Oct 13, 2026Huiwen Han, Ziyao Zeng, Zihao Zhu +6RL for LLM Agents Step-Level Credit Assignment in RL
Stanford University · Texas A&M University · Yale University +3
Adaptive-Group Process Rewards for GUI Agents
Oct 13, 2026Zirui Shang, Yang Liu, Xin Shu +3RL for GUI Agents Process Reward Models
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology · State Key Laboratory of General Artificial Intelligence, BIGAI · Wuhan University +1
DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress Meter
Oct 13, 2026Xuyan Ye, Xiyan Fu, Shengda Fan +3RL for LLM Agents Reinforcement Learning
College of Computing and Data Science, Nanyang Technological University · Gaoling School of Artificial Intelligence, Renmin University of China
WeGR: Towards Unified Generative Retrieval via Behavior-aware Tokenization and Preference Alignment
Oct 12, 2026Jiajun Han, Guoxin Ma, Felix +9Generative Retrieval RL for IR
Tencent Penguin Island
LLM Value Functions for Credit Assignment in Agentic RL
Oct 11, 2026Kaijie Zhu, Weixiang Yan, Omar Attia +3RL for LLM Agents Agentic RL
Apple
Harnessing VLM in Embodied RL: From Reward Regressor to Success Detector via Exploration-Augmented Key-Observation Gating
Oct 11, 2026Xin Liu, Yutong Jiang, Jian Su +4Reward Design for RL Reinforcement Learning
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of the Chinese Academy of Sciences · AGIBOT
PTGPO: Prefix Tree Group Policy Optimization
Oct 10, 2026Ruolin Chen, Yang Li, Jihang Wang +4RL for LLM Agents Temporal Credit Assignment
Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · HyperAI Team, Xiaomi Corporation +1
AlphaNIST: Emergent Assembly Skills from Binary Task Rewards
Oct 9, 2026Octi Zhang, Iretiayo Akinola, Rosario Scalise +12Robotic Assembly Long-Horizon Robotic Manipulation
NVIDIA · University of Washington
Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresight
Oct 6, 2026Haoxiang Zhang, Qinglin Chen, Hiroaki Hayashi +9Agentic RL RL for Language Model Reasoning
Salesforce AI Research · UC San Diego · Texas A&M University +1
Task-Space Imitation Guidance for Efficient Reinforcement Learning
Oct 5, 2026Salar Asayesh, Hossein Darani, Todd Cao +2Reward Shaping Imitation Learning
Sanctuary AI, Canada
Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding
Oct 5, 2026Hoang Phan, Minh Pham, Chau Pham +3RL for Language Model Reasoning LLM-Guided RL
New York University · University at Buffalo · Monash University
Bidirectional Voronoi-biased Exploration Curriculum for Reinforcement Learning
Oct 2, 2026Juri Pfammatter, Kaixian Qu, Clemens Schwarke +2Curriculum RL RL Exploration
Robotic Systems Lab, ETH Zürich, Zürich, Switzerland · NVIDIA
EVOL: Simulator-Guided Evolutionary Expert Synthesis for Deployment-Free Learning Path Recommendation
Oct 2, 2026Geonwoo Bang, Dongho Kim, Moohong MinRecommender Systems Sparse-Reward RL
Department of Computer Education, Sungkyunkwan University Seoul, Republic of Korea · Department of Immersive Media Engineering/Social Innovation Convergence Program, Sungkyunkwan University Seoul, Republic of Korea · Department of Computer Education/Social Innovation Convergence Program, Sungkyunkwan University Seoul, Republic of Korea
AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning
Oct 2, 2026Xin Wang, Wenhao Wu, Menghao Zhang +3Step-Level Credit Assignment in RL RL for LLM Agents
Tsinghua University · Xiaomi Inc. · Nanjing University
Understanding Enrichment in Reinforcement Learning
Oct 2, 2026Jinwoo Kim, Shraddha BarkeImportance Sampling for RL Reinforcement Learning
University of California-San Diego La Jolla, CA, USA · Microsoft Research · Microsoft Research Redmond, WA, USA
Dependency-Aware Reward Shaping for Agentic Reinforcement Learning
Oct 1, 2026Ziyi Chen, Yan Zhang, Jianhui Wei +2RL for Language Models Agentic RL
University of Illinois Urbana-Champaign · National University of Singapore · Zhejiang University +1
Make Sparse Rewards Count: Density-Aware Reward Aggregation for Multi-Reward RL
Sep 30, 2026Tong Zheng, Skylar Zhai, Zhan Cheng +7Multi-Objective Reinforcement Learning Reinforcement Learning
University of Chinese Academy of Sciences · University of Minnesota Twin Cities · University of Wisconsin–Madison +5
T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
Sep 30, 2026Bo-Wen Zhang, Junwei He, Maoqi Liu +6Agentic RL Reward Shaping
State Key Laboratory of Novel Software Technology, Nanjing University · School of Intelligence Science and Technology, Nanjing University · ByteDance
From Imitation to Reward Discovery: On-Policy Warmup for Agentic RL
Sep 30, 2026Yitong Qiao, Tiantian He, Lei Liu +4Agentic RL Reinforcement Learning with Verifiable Rewards
Zhejiang University · Ant Healthcare, Ant Group
Guide, Then Let Go: Gap-Adaptive Teacher Scheduling for Sparse-Reward Agentic RL
Sep 29, 2026Youling Huang, Tiankuo Xu, Jiaji Liu +10Agentic RL LLM-Guided RL
DUT · Kuaishou · XJTU +5
State Trace Rationale As Auxiliary Task in Reinforcement Learning
Sep 29, 2026Muhammad U. Nasir, Alex Vogt, Steven D. James +1Reinforcement Learning Representation Learning for RL
University of the Witwatersrand Johannesburg, South Africa · New York University New York, USA
Going Beyond State-Reaching: Learning Abstractions for Intrinsically Motivated Option Discovery
Sep 29, 2026Akhil Bagaria, Anita De Mello Koch, George KonidarisHierarchical RL RL Exploration
Brown University
SpikeCredit: Temporal Credit Carrier for Reinforcement Learning with Sparse Rewards
Sep 28, 2026Yingchao Yu, Pengfei Sun, Wenxuan Pan +4Credit Assignment in RL Step-Level Credit Assignment in RL
School of Information and Intelligent Science, Donghua University, Shanghai, China · School of Engineering, Westlake University, Hangzhou, China · Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom +1
TIDE: Teacher-Student Transition via Informative Distillation and Exploration for Agentic RL
Sep 28, 2026Yibin Huang, Xinming Xu, Conghui ZhuAgentic RL On-Policy Distillation
Faculty of Computing, Harbin Institute of Technology · Tsinghua University
When Sparse Reward Meets Dense Distillation: Training Dynamics of On-Policy Distillation
Sep 28, 2026Xinke Jiang, Tao Feng, Zhibang Yang +4Gradient Interference Reinforcement Learning with Verifiable Rewards
National Engineering Research Center of Software Engineering, Peking University, Beijing, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China +1

Oct 22, 2026J/K move · Enter open · S save

Effectiveness of Curriculum Learning Depends on Reward Sparsity and Competing Optima

John J. Vastola, Ann Huang, Satpreet H. Singh, Samuel J. Gershman+1

Department of Neurobiology, Harvard Medical School · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University · Department of Psychology and Center for Brain Science, Harvard University

Curriculum RL Reinforcement Learning Sparse-Reward RL Policy Gradient Methods Reward Design for RL

Open paper AskPDF arXiv

While animals and people tend to learn a task more quickly and reliably when they first train on simpler versions of it, curriculum learning's effectiveness in artificial settings varies, and appears substantially greater in reinforcement learning than in supervised learning. In this work, we construct and analyze a minimal model of policy learning to better understand why. We consider an open-loop, episodic target interception task whose parameters---especially how close the agent must be to the target to receive an informative reward signal---can be chosen so that the agent either receives informative rewards frequently ('dense' rewards) or rarely ('sparse' rewards). We mathematically and numerically analyze the dynamics of REINFORCE-based policy learning in both cases. In the dense reward case, curricula have only marginal benefits; in the sparse case, curricula can be required for learning to occur at all. We find that this is especially true when the agent has the conflicting goals of minimizing effort and accumulating high reward, since this can produce distinct reward landscape optima which compete with one another. Our characterization of useful curricula in this setting indicates that they address two issues: they (i) make rewards less sparse to speed up learning, and (ii) shape the reward landscape to steer agents away from bad local optima.