Sparse-Reward RL

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

19 papers in the last four weeks, up 138% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 111

All topics

Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresight
Oct 6, 2026Haoxiang Zhang, Qinglin Chen, Hiroaki Hayashi +9Agentic RL RL for Language Model Reasoning
Salesforce AI Research · UC San Diego · Texas A&M University +1
Task-Space Imitation Guidance for Efficient Reinforcement Learning
Oct 5, 2026Salar Asayesh, Hossein Darani, Todd Cao +2Reward Shaping Imitation Learning
Sanctuary AI, Canada
Dependency-Aware Reward Shaping for Agentic Reinforcement Learning
Oct 1, 2026Ziyi Chen, Yan Zhang, Jianhui Wei +2RL for Language Models Agentic RL
University of Illinois Urbana-Champaign · National University of Singapore · Zhejiang University +1
Make Sparse Rewards Count: Density-Aware Reward Aggregation for Multi-Reward RL
Sep 30, 2026Tong Zheng, Skylar Zhai, Zhan Cheng +7Multi-Objective Reinforcement Learning Reinforcement Learning
University of Chinese Academy of Sciences · University of Minnesota Twin Cities · University of Wisconsin–Madison +5
T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
Sep 30, 2026Bo-Wen Zhang, Junwei He, Maoqi Liu +6Agentic RL Reward Shaping
State Key Laboratory of Novel Software Technology, Nanjing University · School of Intelligence Science and Technology, Nanjing University · ByteDance
From Imitation to Reward Discovery: On-Policy Warmup for Agentic RL
Sep 30, 2026Yitong Qiao, Tiantian He, Lei Liu +4Agentic RL Reinforcement Learning with Verifiable Rewards
Zhejiang University · Ant Healthcare, Ant Group
Guide, Then Let Go: Gap-Adaptive Teacher Scheduling for Sparse-Reward Agentic RL
Sep 29, 2026Youling Huang, Tiankuo Xu, Jiaji Liu +10Agentic RL LLM-Guided RL
DUT · Kuaishou · XJTU +5
State Trace Rationale As Auxiliary Task in Reinforcement Learning
Sep 29, 2026Muhammad U. Nasir, Alex Vogt, Steven D. James +1Reinforcement Learning Representation Learning for RL
University of the Witwatersrand Johannesburg, South Africa · New York University New York, USA
Going Beyond State-Reaching: Learning Abstractions for Intrinsically Motivated Option Discovery
Sep 29, 2026Akhil Bagaria, Anita De Mello Koch, George KonidarisHierarchical RL RL Exploration
Brown University
SpikeCredit: Temporal Credit Carrier for Reinforcement Learning with Sparse Rewards
Sep 28, 2026Yingchao Yu, Pengfei Sun, Wenxuan Pan +4Credit Assignment in RL Step-Level Credit Assignment in RL
School of Information and Intelligent Science, Donghua University, Shanghai, China · School of Engineering, Westlake University, Hangzhou, China · Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom +1
TIDE: Teacher-Student Transition via Informative Distillation and Exploration for Agentic RL
Sep 28, 2026Yibin Huang, Xinming Xu, Conghui ZhuAgentic RL On-Policy Distillation
Faculty of Computing, Harbin Institute of Technology · Tsinghua University
When Sparse Reward Meets Dense Distillation: Training Dynamics of On-Policy Distillation
Sep 28, 2026Xinke Jiang, Tao Feng, Zhibang Yang +4Gradient Interference Reinforcement Learning with Verifiable Rewards
National Engineering Research Center of Software Engineering, Peking University, Beijing, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China +1
UniOPSD: Unifying Outcome and Hindsight Feedback for Agentic Reinforcement Learning
Sep 28, 2026Zenghuang Fu, Zhaoyang Li, Qiuyuan Ai +8Agentic RL Credit Assignment in RL
University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Peking University +3
Demonstration-Free Success-Probability Reward Learning for Generalist Robot Policies
Sep 27, 2026Duo Wu, Haifeng Wang, Rongwei Lu +6Robotic RL Sparse-Reward RL
Tsinghua University · Yuanxing Robotics · Pengcheng Laboratory
SAGE: Mitigating Long-Horizon Reasoning Biases via Topological Guidance
Sep 24, 2026Xinyue Zeng, Jiawei Zhang, Yujun Yan +1Sparse-Reward RL
CS Department Virginia Tech · CS Department University of Wisconsin Madison · CS Department Dartmouth College
Reinforcement Learning with Verifiable Rewards for Small Search Agents
Sep 23, 2026Gaurisankar Jayadas, Aske Plaat, Álvaro Serra-Gómez +1Reinforcement Learning Reinforcement Learning with Verifiable Rewards
Leiden Institute of Advanced Computer Science, Leiden University, Leiden, The Netherlands
FLARE: A Full-Lifecycle Dense Supervision Paradigm for Long-Horizon Coding Agents via Generative Reward Model
Sep 20, 2026Jingxuan Xu, Gang Wu, Yanan Wu +13RL for Code Generation Reward Modeling
Independent Researcher · Nanjing University · Peking University +1
Improving Offline Goal-Conditioned Reinforcement Learning via Selective Reward Stimulation
Sep 16, 2026Jing ZhangReward Shaping Offline RL
The Hong Kong University of Science and Technology
AlphaRJM: Reward-Jump Memory for Stochastic Return-Guided Alpha Discovery
Sep 8, 2026Sayan Dhan, Selvaraju NatarajanQuantitative Finance Sparse-Reward RL
Department of Mathematics Indian Institute of Technology Guwahati
Environments as Scaffold: Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon Tasks
Sep 8, 2026Hongbang Yuan, Zhuoran Jin, Yixin CaoReinforcement Learning Sparse-Reward RL
1Fudan University · 2CASIA · 3Shanghai Innovation Institute
Phase-and-First-Arrival VLM Feedback for Sparse-Reward Reinforcement Learning in Surgical Manipulation
Sep 7, 2026Wanli Liuchen, Fangyuan Wang, Bin Li +4Reward Shaping Temporal Credit Assignment
The Hong Kong Polytechnic University · The Chinese University of Hong Kong · Mohamed bin Zayed University of Artificial Intelligence +1
Coverage, Not Targeting: A Structural Regime in Multi-Turn Agent Credit Assignment
Sep 2, 2026Chenyu Zhou, Qiliang Jiang, Shuning Wu +1Credit Assignment LLM Agents
School of Engineering, Institute of Science Tokyo, Japan · College of Control Science and Engineering, Zhejiang University, China · Department of Electrical and Computer Engineering, National University of Singapore, Singapore
Explore More, Drift Less: Outcome-Only Reinforcement Learning Can Suffice for Long-Horizon Interactive Agents
Sep 1, 2026Liming Pu, Xiaoxia Li, Yifu Liu +2Agentic RL RL Post-Training
Alibaba Research
Locally-Guided Actor-Critic: Training a Goal-conditioned Actor with a Subgoal-aware Critic
Aug 31, 2026Olivier Serris, Stéphane Doncieux, Olivier SigaudReinforcement Learning Sparse-Reward RL
ISIR · Sorbonne Université, CNRS
S2-HWM: Sparse Event-Structured Hierarchical World Model for Long-Horizon Surgical Robot Manipulation
Aug 13, 2026Shuzhe Zhang, Xin Zhu, Yinling Qian +1Long-Horizon Robotic Manipulation World Models for Robotics
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences
I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization
Aug 13, 2026Yubo Zhang, Xinhong Ma, Zezhong Tan +1On-Policy Self-Distillation Group Relative Policy Optimization
Qwen Large Model Application Team, Alibaba
Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents
Aug 13, 2026Haoze Wu, Chuqiao Kuang, Tianyi Zhuang +1Group Relative Policy Optimization Credit Assignment in RL
The Hong Kong University of Science and Technology · Huawei Technologies Ltd.
Learning Loco-Manipulation From SMPC Demonstrations With Sparse Offline-to-Online RL
Aug 12, 2026Martin Schuck, Maks Sorokin, Simone Manni +5Robot Skill Learning Sparse-Reward RL
1RAI Institute · 2Technical University of Munich · ETH Zurich
Exploration-Driven Personalized Federated Reinforcement Learning via Intrinsic Motivation
Aug 11, 2026Md Rafid Islam, Rafsan Jany, Zahid Hasan +1Federated RL Intrinsic Motivation
Department of Electrical and Computer Engineering North South University Dhaka, Bangladesh · Digital Health Research Division Korea Institute of Oriental Medicine Daejeon, South Korea · Department of Electrical and Computer Engineering The University of Alabama in Huntsville Huntsville, AL, USA
Learning from Environmental Feedback: Credit Assignment across Multiple Timescales for Agentic Reinforcement Learning
Aug 8, 2026Yifu Huo, Shunjie Xing, Chenglong Wang +8Agentic RL Credit Assignment in RL
School of Computer Science and Engineering, Northeastern University, Shenyang, China · School of Computer Science · 2NiuTrans Research, Shenyang, China +1
Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning
Aug 7, 2026Haoyu Zheng, Yun Zhu, Qing Wang +1Agentic RL Credit Assignment in RL
Zhejiang University · Tencent · Shanghai AI Laboratory
Gated-BEPO: Confidence-Gated Bellman Credit Assignment for Large Language Model Agents
Aug 7, 2026Hongxi Yan, Ziyue Huang, Shichao Fan +1Credit Assignment in RL Step-Level Credit Assignment in RL
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · Zhongguancun Laboratory, Beijing, China · Qualcomm, Beijing, China
Agentic Reinforcement Learning with Self-Distilled Reward Shaping
Aug 4, 2026Ranxu Zhang, Guinan Chen, Chenshaodong +5Reward Shaping Token-Level Credit Assignment
Alibaba Group · University of Science and Technology of China Hefei, China · Alibaba Group Hangzhou, China
Neurosymbolic Reasoning with Incremental Knowledge for Sample Efficient Hierarchical Reinforcement Learning
Aug 4, 2026Subrat Prasad Panda, Blaise Genest, Arvind EaswaranReinforcement Learning Symbolic Planning
CCDS, NTU Singapore · IPAL, CNRS, France
PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning
Aug 3, 2026Chunji Lv, Yangguang Wei, Junlin Liu +6Language Model Distillation On-Policy Distillation
Beijing Institute of Technology · Meituan · Institute of Automation, Chinese Academy of Sciences +1
BiCAA: Bidirectional Credit Assignment for Search-Augmented Agent
Aug 2, 2026Yibin Huang, Bin Xu, Hailong Cao +1Process Reward Models Credit Assignment in RL
Faculty of Computing, Harbin Institute of Technology
Explore Beyond the Boundary Using Entropic Information
Jul 31, 2026Bumgeun Park, Donghwan LeeReinforcement Learning RL Exploration
Korea Advanced Institute of Science and Technology, Daejeon 34051, Korea
ReDiPPO: Reference-Guided Value Calibration and Discrepancy-Aware Token Reweighting for Mathematical Reasoning
Jul 30, 2026Zhenrong Zhang, Fei Wu, Jun Du +2RL for Language Model Reasoning Credit Assignment in RL
1IFLYTEK Research · University of Science and Technology of China
SCOUT: Per-Context Reset Curricula for Sparse-Reward Reinforcement Learning
Jul 29, 2026Siddharth Aphale, Ayushman SinghCurriculum RL Curriculum Learning
Stanford University · Sesame AI
Hierarchical Soft Actor-Critic for Sparse-Reward Long-Horizon Reinforcement Learning
Jul 26, 2026Zahra Abdalla Elashaal, Afef Hfaiedh, Nahla Khraief +2Reinforcement Learning Hierarchical RL
Lab. of Robotics, Informatics and Complex Systems, National Engineering School of Tunis, University of Tunis El Manar, Tunis, Tunisia · Faculty of Information Technology, University of Tripoli, Tripoli, Libya · Lab. of Robotics, Informaticsand Complex Systems, National Engineering School of Tunis, University of Tunis El Manar, Tunis, Tunisia +3
LEACL: LLM-Enhanced Automatic Curriculum Learning for Reinforcement Learning in Long-Horizon Manipulation Tasks
Jul 26, 2026Faraz Heravi, James Ouyang, Zifan Xu +3LLM-Guided RL Robotic Manipulation
The University of Texas at Austin · Nanyang Technological University · Sony AI
The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and The Channel, Not the Content, Decides What Works
Jul 23, 2026Yu WangReward Hacking Reward Shaping
Independent researcher
Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Tasks
Jul 22, 2026Kaibing Yang, Guangfeng Cai, Shengtian Yang +6Group Relative Policy Optimization RL Exploration
Southeast University · Kuaishou Technology · Work done during an internship at Kuaishou Technology, supervised by Jun Xu. +1
S3: Stable Subgoal Selection by Constraining Uncertainty of Coarse Dynamics in Hierarchical Reinforcement Learning
Jul 21, 2026Kshitij Kumar Srivastava, Kshitij JerathHierarchical RL Intrinsic Motivation
University of Massachusetts, Lowell United States
Counterfactual Shapley Credit Assignment
Jul 18, 2026Mingxuan Li, Kai-Zhan Lee, Elias BareinboimCredit Assignment in RL Temporal Credit Assignment
Department of Computer Science, Columbia University, New York, NY, USA
Reachability-Aware Pretraining for Efficient Target-Oriented Path Exploration in Temporal Knowledge Graph Reasoning
Jul 16, 2026Chien-Liang Liu, Tsao-Lun ChenTemporal Link Prediction Sparse-Reward RL
Artificial Intelligence Research Center, Chang Gung University · 2Graduate Institute of Oral Biology, National Taiwan University · 3Electrical Engineering, National Taiwan University of Science and Technology
SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning
Jul 16, 2026Jinyang Wu, Shuo Yang, Zhengxi Lu +8Agentic RL On-Policy Distillation
1Tsinghua University · 2Zhejiang University · 3The Chinese University of Hong Kong +2
AlphaZero in Sparsely Rewarded Games: Limits and Auxiliary Supervision
Jul 9, 2026Brent Kong, Tejas Ram, Tony Yue YuGame-Playing Agents Self-Play RL
Department of Mathematics · California Institute of Technology · Department of Computer Science
STAPO: Selective Trajectory-Aware Policy Optimization for LLM Agent Training
Jul 6, 2026Qiuyi Qi, Tian Liang, Mutian Bao +8Uncertainty Quantification Sparse-Reward RL
Zhejiang University · Ant Group · City University of Hong Kong
RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents
Jul 6, 2026Qiang Liu, Taian Guo, Ruizhi Qiao +1Reinforcement Learning Sparse-Reward RL
Tencent YouTu Lab
ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy
Jul 3, 2026Zijun Xie, Yuyang You, Yongzhi Li +6Token-Level Credit Assignment RL for Language Model Reasoning
Peking University · Baidu Inc. · Kuaishou Inc.
Hierarchical Reinforcement Learning in StarCraft Micromanagement with Influence Maps and Cluster-based Scripts
Jun 29, 2026Chunhui Bai, Changhe Li, Dequan Li +2Hierarchical RL Sparse-Reward RL
School of Artificial Intelligence and Automation, China University of Geosciences, Wuhan 430074, China · State Key Laboratory of Digital Intelligent Technology for Unmanned Coal Mining, Anhui University of Science & Technology, Huainan 232001, China · School of Artificial Intelligence, Anhui University of Science & Technology, Hefei 231131, China +1
KbSD: Knowledge Boundary aware Self-Distillation for Behavioral Calibration in Agentic Search
Jun 29, 2026Tao Feng, Xinke Jiang, Chao WuLanguage Model Calibration Agentic RAG
Zhejiang University, Hangzhou, China · Peking University, Beijing, China
Automating Potential-based Reward Shaping with Vision Language Model Guidance
Jun 25, 2026Henrik Müller, Daniel KudenkoReward Shaping Sparse-Reward RL
L3S Research Center Leibniz University Hannover Hannover, 30167, Germany
Mesh-RL: Coupled subgrid reinforcement learning
Jun 24, 2026Behnam Gheshlaghi, Bahador Rashidi, Shahin AtakishiyevReinforcement Learning Credit Assignment in RL
Independent Researcher · University of Alberta
Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents
Jun 24, 2026Peng Xu, Sijia Chen, Junzhuo Li +1Reward Shaping Credit Assignment in RL
†The Hong Kong University of Science and Technology (Guangzhou) · ‡The Hong Kong University of Science and Technology
Learning with a Single Rollout via Monte Carlo Pass@k Critic
Jun 24, 2026Fengdi Che, Yang Liu, Lei Yu +4RL for Language Model Reasoning Credit Assignment in RL
University of Alberta · BIGAI · University of Toronto +4
Learning Process Rewards via Success Visitation Matching for Efficient RL
Jun 22, 2026Raymond Tsao, Andrew Wagenmaker, Sergey LevineProcess Reward Models Credit Assignment in RL
UC Berkeley
Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning
Jun 22, 2026Yunan Wang, Minghui Song, Zihan Zhang +6Agentic RL Reinforcement Learning
1Peking University · 2Microsoft Corporation
Hierarchical Reinforcement Learning for Sparse-Reward Search in Commutative Algebra
Jun 22, 2026Giorgi Butbaia, Paul Orland, Coco Huang +7Hierarchical RL Sparse-Reward RL
Department of Mathematics, California Institute of Technology, Pasadena, CA · Department of Mathematics, Temple University, Philadelphia, PA · Department of Mathematics, University of Kansas, Lawrence, KS +1

Oct 6, 2026, cs.AIJ/K move · Enter open · S save

Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresight

Haoxiang Zhang, Qinglin Chen, Hiroaki Hayashi, Zhuofeng Li+8

Salesforce AI Research · UC San Diego · Texas A&M University+1

Agentic RL RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards Self-Distillation Sparse-Reward RL

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) turns agent experience into learning signals primarily through scalar outcome rewards after interaction. For group-relative objectives, however, this signal vanishes when all rollouts receive the same reward, even though their trajectories may reveal useful information about what the task requires and how the agent fails. We ask a complementary question: can hindsight teach an agent what it could have anticipated before acting? We introduce prospective learning, which uses post-hoc experience to supervise foresight predictions from the pre-interaction view, and instantiate it with Self-Retrospection Distillation (SRD). Intuitively, a completed trajectory reveals knowledge that would have been useful and pitfalls that should be avoided; SRD distills this privileged hindsight into trajectory-blind foresight of the same policy. Foresight serves only as a training target and need not be explicitly generated at inference time. Across 10 tool-integrated reasoning and long-horizon agentic tasks, SRD complements RLVR and self-distillation baselines with gains of up to

24.2

pp. Its advantage is especially pronounced when reward contrast is scarce: when

37

98\%

of rollout groups are reward-uniform across model scales, yet SRD can still exploit learning signal from sampled trajectories. In the 2B setting, where

98\%

of groups are all-failure, the RLVR training ends up at

0.0\%

success, while adding SRD reaches

60.6\%

under the same rollout budget. Our results suggest that post-hoc agent experience is useful not only for evaluating or improving behavior, but also for shaping predictive representations before available interaction.

Momentum

Related topics

Latest papers 111