Delayed RL

RL: Reinforcement Learning

Follow this topic Ask about this topic

Latest papers

13 papers1 paper in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Visual Causal-chain Bookmarking: Retrieval-Weighted Credit Assignment for Reinforcement Learning from Delayed Reward
Oct 16, 2026Prathap SelvakumarCredit Assignment in RL Step-Level Credit Assignment in RL
University of Manchester
Explore Beyond the Boundary Using Entropic Information
Jul 31, 2026Bumgeun Park, Donghwan LeeReinforcement Learning RL Exploration
Korea Advanced Institute of Science and Technology, Daejeon 34051, Korea
Reinforcement Learning on Cost-Constrained Quadrupedal Hardware
Jul 29, 2026Javier C. Weddington, Bence P. Ölveczky, Stephen A. BaccusDynamical Systems Reinforcement Learning
Neurosciences Interdepartmental Program, Stanford University, Stanford, CA 94305 USA · Department of Organismic and Evolutionary Biology, Harvard University, Cambridge, MA 02138 USA · Department of Neurobiology, Stanford University School of Medicine, Stanford, CA 94305 USA
Diffusion-Guided Uncertainty-Aware Delayed Policy Optimization
Jul 6, 2026Junqi Tu, Zejiao Liu, Fangfei Li +1Reinforcement Learning Robotic RL
The Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science and Technology, Shanghai, China · The School of Mathematics, East China University of Science and Technology, Shanghai, China
Retroactive Advantage Correction: Closed-Form V-Trace Bias Correction for Delay-Aware RLHF
Jun 25, 2026Arnav RajAsynchronous RL Reinforcement Learning
Department of Computer Science and Engineering, Indian Institute of Technology Delhi, New Delhi, India.
Multi-Agent Reinforcement Learning from Delayed Marketplace Feedback for Objective-Weight Adaptation in Three-Sided Dispatch
Jun 11, 2026Haochen Wu, Yi Hou, Shiguang XieMulti-Objective Reinforcement Learning Reinforcement Learning
DoorDash Inc.
Transferable Delay-Aware Reinforcement Learning via Implicit Causal Graph Modeling
May 12, 2026Chenran Zhao, Dianxi Shi, Yaowen Zhang +2Causal Representation Learning Reinforcement Learning
College of Computer Science and Technology, National University of Defense Technology, Changsha, China · Institute of Military Transportation, Tianjin, China · Intelligent Game and Decision Lab (IGDL), Beijing, China
Delay-Empowered Causal Hierarchical Reinforcement Learning
May 12, 2026Chenran Zhao, Dianxi Shi, Haotian Wang +4Reinforcement Learning Hierarchical RL
College of Computer Science and Technology, National University of Defense Technology, Changsha, China · Intelligent Game and Decision Lab (IGDL), Beijing, China · School of Artificial Intelligence, Hebei University of Technology, Tianjin, China +1
FutureWorld: A Live Reinforcement Learning Environment for Predictive Agents with Real-World Outcome Rewards
Apr 29, 2026Zhixin Han, Yanzhi Zhang, Chuyang Wei +11Agentic RL Reinforcement Learning
College of Software, Nankai University · Zhongguancun Academy, Beijing, China · Academy of Mathematics and Systems Science, Chinese Academy of Sciences +3
Delayed homomorphic reinforcement learning for environments with delayed feedback
Apr 4, 2026Jongsoo Lee, Jangwon Kim, Soohee HanDelayed RL Reinforcement Learning
Department of Convergence IT Engineering, POSTECH, Pohang, 37673, South Korea · Department of Electrical Engineering, POSTECH, Pohang, 37673, South Korea
Minimax Optimal Strategy for Delayed Observations in Online Reinforcement Learning
Mar 3, 2026Harin Lee, Kevin JamiesonDelayed RL Reinforcement Learning
Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, Washington.
State-Action Inpainting Diffuser for Continuous Control with Delay
Mar 2, 2026Dongqi Han, Wei Wang, Enze Zhang +1Delayed RL Reinforcement Learning
Microsoft Research Asia · University of Pennsylvania · Shanghai Jiaotong University
Adaptive Reinforcement Learning for Unobservable Random Delays
Jun 17, 2025John Wikman, Alexandre Proutiere, David BromanReinforcement Learning Robotic RL
EECS and Digital Futures, KTH Royal Institute of Technology, Stockholm, Sweden.

Oct 16, 2026J/K move · Enter open · S save

Visual Causal-chain Bookmarking: Retrieval-Weighted Credit Assignment for Reinforcement Learning from Delayed Reward

Prathap Selvakumar

University of Manchester

Credit Assignment in RL Step-Level Credit Assignment in RL Delayed RL Causal RL RL for LLM Agents

Open paper AskPDF arXiv

RL agents trained on a single trajectory-level return face a credit-assignment problem: when a reward arrives many steps after the decision that caused it, every action gets the same signal regardless of causal responsibility. This paper studies Visual Causal-chain Bookmarking (VCBM), which concentrates credit on the decision points that determine an episode's return, via two instantiations. The first targets a fully observed, low-dimensional MDP (a delayed-reward watch-repair task), where VCBM statistically discovers which state components the agent causally controls and estimates per-decision-point action values directly, evaluated against ground truth and RUDDER, an LSTM-based baseline. VCBM converges to 95.07% correct decisions in a mean of 1,532.3 episodes across 20 seeds, versus RUDDER's slower, more variable convergence (mean 3,327.1 episodes; p<0.001); VCBM's discovered causal structure exactly matches ground truth in all seeds. The second targets a partially observed, language-based setting: an LLM agent trained with Leave-One-Out PPO (LOOP) in AppWorld, where causal discovery is intractable and a structural heuristic (state-mutating tool calls) substitutes for it, with credit distributed via retrieval over episodic memory and blended with LOOP's advantage. Over 200 iterations on identical Qwen2.5-7B-Instruct/LoRA/FSDP2 configurations, VCBM-blended LOOP improves task success over plain LOOP (0.354 vs. 0.458), reduces final-step max importance weight, and improves token efficiency, with a small KL exception favouring LOOP. The watch-repair result supports the claim that weighting causally relevant steps is more sample-efficient than an undifferentiated baseline or a learned return-redistribution network; the AppWorld result, limited to one seed per arm by a 2-GPU budget, is a compute-limited demonstration at a harder, natural-language scale, not a statistical confirmation.