Sparse-Reward RL

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

19 papers in the last four weeks, up 138% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 111

All topics

Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning
Aug 7, 2026Haoyu Zheng, Yun Zhu, Qing Wang +1Agentic RL Credit Assignment in RL
Zhejiang University · Tencent · Shanghai AI Laboratory
Gated-BEPO: Confidence-Gated Bellman Credit Assignment for Large Language Model Agents
Aug 7, 2026Hongxi Yan, Ziyue Huang, Shichao Fan +1Credit Assignment in RL Step-Level Credit Assignment in RL
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · Zhongguancun Laboratory, Beijing, China · Qualcomm, Beijing, China
Agentic Reinforcement Learning with Self-Distilled Reward Shaping
Aug 4, 2026Ranxu Zhang, Guinan Chen, Chenshaodong +5Reward Shaping Token-Level Credit Assignment
Alibaba Group · University of Science and Technology of China Hefei, China · Alibaba Group Hangzhou, China
Neurosymbolic Reasoning with Incremental Knowledge for Sample Efficient Hierarchical Reinforcement Learning
Aug 4, 2026Subrat Prasad Panda, Blaise Genest, Arvind EaswaranReinforcement Learning Symbolic Planning
CCDS, NTU Singapore · IPAL, CNRS, France
PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning
Aug 3, 2026Chunji Lv, Yangguang Wei, Junlin Liu +6Language Model Distillation On-Policy Distillation
Beijing Institute of Technology · Meituan · Institute of Automation, Chinese Academy of Sciences +1
BiCAA: Bidirectional Credit Assignment for Search-Augmented Agent
Aug 2, 2026Yibin Huang, Bin Xu, Hailong Cao +1Process Reward Models Credit Assignment in RL
Faculty of Computing, Harbin Institute of Technology
Explore Beyond the Boundary Using Entropic Information
Jul 31, 2026Bumgeun Park, Donghwan LeeReinforcement Learning RL Exploration
Korea Advanced Institute of Science and Technology, Daejeon 34051, Korea
ReDiPPO: Reference-Guided Value Calibration and Discrepancy-Aware Token Reweighting for Mathematical Reasoning
Jul 30, 2026Zhenrong Zhang, Fei Wu, Jun Du +2RL for Language Model Reasoning Credit Assignment in RL
1IFLYTEK Research · University of Science and Technology of China
SCOUT: Per-Context Reset Curricula for Sparse-Reward Reinforcement Learning
Jul 29, 2026Siddharth Aphale, Ayushman SinghCurriculum RL Curriculum Learning
Stanford University · Sesame AI
Hierarchical Soft Actor-Critic for Sparse-Reward Long-Horizon Reinforcement Learning
Jul 26, 2026Zahra Abdalla Elashaal, Afef Hfaiedh, Nahla Khraief +2Reinforcement Learning Hierarchical RL
Lab. of Robotics, Informatics and Complex Systems, National Engineering School of Tunis, University of Tunis El Manar, Tunis, Tunisia · Faculty of Information Technology, University of Tripoli, Tripoli, Libya · Lab. of Robotics, Informaticsand Complex Systems, National Engineering School of Tunis, University of Tunis El Manar, Tunis, Tunisia +3
LEACL: LLM-Enhanced Automatic Curriculum Learning for Reinforcement Learning in Long-Horizon Manipulation Tasks
Jul 26, 2026Faraz Heravi, James Ouyang, Zifan Xu +3LLM-Guided RL Robotic Manipulation
The University of Texas at Austin · Nanyang Technological University · Sony AI
The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and The Channel, Not the Content, Decides What Works
Jul 23, 2026Yu WangReward Hacking Reward Shaping
Independent researcher
Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Tasks
Jul 22, 2026Kaibing Yang, Guangfeng Cai, Shengtian Yang +6Group Relative Policy Optimization RL Exploration
Southeast University · Kuaishou Technology · Work done during an internship at Kuaishou Technology, supervised by Jun Xu. +1
S3: Stable Subgoal Selection by Constraining Uncertainty of Coarse Dynamics in Hierarchical Reinforcement Learning
Jul 21, 2026Kshitij Kumar Srivastava, Kshitij JerathHierarchical RL Intrinsic Motivation
University of Massachusetts, Lowell United States
Counterfactual Shapley Credit Assignment
Jul 18, 2026Mingxuan Li, Kai-Zhan Lee, Elias BareinboimCredit Assignment in RL Temporal Credit Assignment
Department of Computer Science, Columbia University, New York, NY, USA
Reachability-Aware Pretraining for Efficient Target-Oriented Path Exploration in Temporal Knowledge Graph Reasoning
Jul 16, 2026Chien-Liang Liu, Tsao-Lun ChenTemporal Link Prediction Sparse-Reward RL
Artificial Intelligence Research Center, Chang Gung University · 2Graduate Institute of Oral Biology, National Taiwan University · 3Electrical Engineering, National Taiwan University of Science and Technology
SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning
Jul 16, 2026Jinyang Wu, Shuo Yang, Zhengxi Lu +8Agentic RL On-Policy Distillation
1Tsinghua University · 2Zhejiang University · 3The Chinese University of Hong Kong +2
AlphaZero in Sparsely Rewarded Games: Limits and Auxiliary Supervision
Jul 9, 2026Brent Kong, Tejas Ram, Tony Yue YuGame-Playing Agents Self-Play RL
Department of Mathematics · California Institute of Technology · Department of Computer Science
STAPO: Selective Trajectory-Aware Policy Optimization for LLM Agent Training
Jul 6, 2026Qiuyi Qi, Tian Liang, Mutian Bao +8Uncertainty Quantification Sparse-Reward RL
Zhejiang University · Ant Group · City University of Hong Kong
RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents
Jul 6, 2026Qiang Liu, Taian Guo, Ruizhi Qiao +1Reinforcement Learning Sparse-Reward RL
Tencent YouTu Lab
ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy
Jul 3, 2026Zijun Xie, Yuyang You, Yongzhi Li +6Token-Level Credit Assignment RL for Language Model Reasoning
Peking University · Baidu Inc. · Kuaishou Inc.
Hierarchical Reinforcement Learning in StarCraft Micromanagement with Influence Maps and Cluster-based Scripts
Jun 29, 2026Chunhui Bai, Changhe Li, Dequan Li +2Hierarchical RL Sparse-Reward RL
School of Artificial Intelligence and Automation, China University of Geosciences, Wuhan 430074, China · State Key Laboratory of Digital Intelligent Technology for Unmanned Coal Mining, Anhui University of Science & Technology, Huainan 232001, China · School of Artificial Intelligence, Anhui University of Science & Technology, Hefei 231131, China +1
KbSD: Knowledge Boundary aware Self-Distillation for Behavioral Calibration in Agentic Search
Jun 29, 2026Tao Feng, Xinke Jiang, Chao WuLanguage Model Calibration Agentic RAG
Zhejiang University, Hangzhou, China · Peking University, Beijing, China
Automating Potential-based Reward Shaping with Vision Language Model Guidance
Jun 25, 2026Henrik Müller, Daniel KudenkoReward Shaping Sparse-Reward RL
L3S Research Center Leibniz University Hannover Hannover, 30167, Germany
Mesh-RL: Coupled subgrid reinforcement learning
Jun 24, 2026Behnam Gheshlaghi, Bahador Rashidi, Shahin AtakishiyevReinforcement Learning Credit Assignment in RL
Independent Researcher · University of Alberta
Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents
Jun 24, 2026Peng Xu, Sijia Chen, Junzhuo Li +1Reward Shaping Credit Assignment in RL
†The Hong Kong University of Science and Technology (Guangzhou) · ‡The Hong Kong University of Science and Technology
Learning with a Single Rollout via Monte Carlo Pass@k Critic
Jun 24, 2026Fengdi Che, Yang Liu, Lei Yu +4RL for Language Model Reasoning Credit Assignment in RL
University of Alberta · BIGAI · University of Toronto +4
Learning Process Rewards via Success Visitation Matching for Efficient RL
Jun 22, 2026Raymond Tsao, Andrew Wagenmaker, Sergey LevineProcess Reward Models Credit Assignment in RL
UC Berkeley
Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning
Jun 22, 2026Yunan Wang, Minghui Song, Zihan Zhang +6Agentic RL Reinforcement Learning
1Peking University · 2Microsoft Corporation
Hierarchical Reinforcement Learning for Sparse-Reward Search in Commutative Algebra
Jun 22, 2026Giorgi Butbaia, Paul Orland, Coco Huang +7Hierarchical RL Sparse-Reward RL
Department of Mathematics, California Institute of Technology, Pasadena, CA · Department of Mathematics, Temple University, Philadelphia, PA · Department of Mathematics, University of Kansas, Lawrence, KS +1

Aug 7, 2026, cs.LGJ/K move · Enter open · S save

Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning

Haoyu Zheng, Yun Zhu, Qing Wang, Wenqiao Zhang

Zhejiang University · Tencent · Shanghai AI Laboratory

Agentic RL Credit Assignment in RL Sparse-Reward RL Policy Learning RL for LLM Agents

Open paper AskPDF arXiv

Recent agentic reinforcement learning methods use hindsight to complement sparse outcome rewards. However, a completed rollout can yield many such signals, leaving their appropriate allocation across turns unclear. We introduce TRIAL, a trajectory-relative hindsight distillation framework with a unified turn-aligned scoring protocol. For each decision turn, TRIAL extracts an outcome view of that decision's realized consequence and evaluates the same response under ordinary and hindsight-conditioned contexts. The signed log-probability gap determines the direction and local strength of token-level supervision, while turn-level magnitudes are normalized jointly over the realized trajectory. The resulting allocation multipliers have an eligible-token-weighted mean of one, redistributing dense supervision across turns while fixing its average multiplier. Experiments on WebShop and ALFWorld with different backbones show that TRIAL outperforms GRPO across all eight combinations of backbone, environment, and evaluation metric, while achieving the best or tied-best performance among six methods on six of them. On WebShop with Qwen3-1.7B, TRIAL improves the success rate from 56.4% to 75.2% and the task score from 78.7% to 85.7%. Controlled ablations further show that trajectory-relative turn allocation provides substantial gains beyond those of dense hindsight distillation alone.

Momentum

Related topics

Latest papers 111