Delayed homomorphic reinforcement learning for environments with delayed feedback

Jongsoo Lee; Jangwon Kim; Soohee Han

cs.LGApr 4, 2026

Delayed homomorphic reinforcement learning for environments with delayed feedback

Authors: Jongsoo Lee, Jangwon Kim, Soohee Han

Organizations: Department of Convergence IT Engineering, POSTECH, Pohang, 37673, South Korea · Department of Electrical Engineering, POSTECH, Pohang, 37673, South Korea

Abstract

Reinforcement learning in real-world systems often involves delayed feedback, which breaks the Markov assumption and impedes both learning and control. Canonical augmentation-based approaches cause state-space explosion, which imposes a severe sample-complexity burden. Despite recent progress, state-of-the-art augmentation-based baselines either mainly alleviate the burden on the critic or rely on non-unified treatments for the actor and critic. In this study, we propose delayed homomorphic reinforcement learning (DHRL), a framework grounded in MDP homomorphisms that defines a belief-equivalence relation over the augmented state space to collapse control-redundant augmented states. In principle, this yields exact abstraction under deterministic dynamics and approximate abstraction under stochastic dynamics, enabling both the actor and critic to benefit from a structured abstraction mechanism. In finite domains, exact abstraction preserves optimality and recovers the delay-free sample-complexity order, whereas approximate abstraction admits a value-loss bound on the resulting policy. For continuous domains, we introduce deep delayed homomorphic policy gradient (D $^2$ HPG), a deep actor-critic instantiation of the DHRL framework. Experiments on continuous-control tasks in MuJoCo show that D $^2$ HPG outperforms strong augmentation-based baselines.

Figures & tables

Figure 1: Schematic overview of the DHRL framework.

Figure 2: Schematic overview of the practical instantiation of D 2 HPG.

Environment		Ant-v3	HalfCheetah-v3	Hopper-v3	Walker2d-v3	Humanoid-v3	InvertedPendulum-v2
$\Delta$	Algorithm	Ant-v3	HalfCheetah-v3	Hopper-v3	Walker2d-v3	Humanoid-v3	InvertedPendulum-v2
–	Delay-free SAC	$3279.2_{\pm 180}$	$8608.4_{\pm 57}$	$2435.2_{\pm 23}$	$3305.5_{\pm 234}$	$3228.1_{\pm 410}$	$964.3_{\pm 29}$
5	Naive SAC	$-74.9_{\pm 4}$	$-276.3_{\pm 5}$	$88.8_{\pm 10}$	$44.5_{\pm 20}$	$398.3_{\pm 6}$	$32.1_{\pm 2}$
	Augmented SAC	$881.9_{\pm 103}$	$2130.9_{\pm 344}$	$2230.8_{\pm 178}$	$1265.4_{\pm 303}$	$629.2_{\pm 56}$	$935.7_{\pm 38}$
	Delayed SAC	$1093.6_{\pm 132}$	$1753.1_{\pm 198}$	$1536.6_{\pm 248}$	$858.8_{\pm 207}$	$505.9_{\pm 68}$	$925.5_{\pm 18}$
	VDPO	$4373.3_{\pm 181}$	$4819.5_{\pm 36}$	$1917.6_{\pm 105}$	$3402.9_{\pm 263}$	$2843.2_{\pm 482}$	${764.5}_{\pm 136}$

Table 1: Results on the continuous-control MuJoCo tasks under fixed delays with

\Delta\in\{5,10,20\}

. Each algorithm was evaluated for one million time steps with five different seeds, where the standard deviations of average returns are denoted by

\pm

. The best performance is highlighted.

Figure 3: Normalized performance of D 2 HPG under actuator-level stochastic perturbations in multiple MuJoCo tasks with delay

\Delta=10

, normalized with respect to the noise-free model. The perturbations are sampled from a normal distribution with standard deviation

\sigma\in\{0.01,0.03,0.05,0.07\}

. All results are reported over one million time steps with five different seeds.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameters	Values
Actor network	$256,256$
Critic network	$256,256$
Learning rate (actor)	$3\times 10^{-4}$
Learning rate (beta-critic)	$3\times 10^{-4}$
Temperature ( $\alpha$ )	$0.2$
Target smoothing coefficient ( $\beta$ )	$0.005$

Appendix

Table 2: Hyperparameters for the baseline algorithms.

Environment	State dimension	Action dimension	Action range	Time step (s)
Ant-v3	$27$	$8$	$[-1.0,1.0]$	$0.05$
HalfCheetah-v3	$17$	$6$	$[-1.0,1.0]$	$0.05$
Walker2d-v3	$17$	$6$	$[-1.0,1.0]$	$0.008$
Hopper-v3	$11$	$3$	$[-1.0,1.0]$	$0.008$
Humanoid-v3	$376$	$17$	$[-0.4,0.4]$	$0.015$
InvertedPendulum-v2	$4$	$1$	$[-3.0,3.0]$	$0.04$

Appendix

Table 3: Environment details of the MuJoCo benchmark.

Panel of Figure 4 — Figure 4: Visual illustration of continuous control tasks in MuJoCo: (a) Ant-v3 (b) HalfCheetah-v3, (c) Walker2d-v3, (d) Hopper-v3, (e) Humanoid-v3, and (f) InvertedPendulum-v2.

Panel of Figure 5 — Figure 5: Normalized performance of D 2 HPG variants in MuJoCo, where D 2 HPG-BPQL is used as the baseline. Each algorithm was evaluated for one million time steps with five different seeds.

Figure 6: Normalized performance of D 2 HPG under fixed delays with

\Delta=10

(fixed) and random delays uniformly sampled from

\{1,2,\dots,\Delta_{\text{max}}\}

with

\Delta_{\text{max}}=10

(random), evaluated on multiple MuJoCo tasks over one million time steps with five random seeds.

Figure 7: Wall-clock runtime comparison across baselines over one million time steps.

Figure 8: Performance curves of each algorithm on the MuJoCo benchmarks with

\Delta=5

Figure 9: Performance curves of each algorithm on the MuJoCo benchmarks with

\Delta=10

Figure 10: Performance curves of each algorithm on the MuJoCo benchmarks with

\Delta=20

Delayed RL

See all 13 papers

Diffusion-Guided Uncertainty-Aware Delayed Policy Optimization
Jul 6, 2026Junqi Tu, Zejiao Liu, Fangfei Li +1Reinforcement Learning Robotic RL
The Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science and Technology, Shanghai, China · The School of Mathematics, East China University of Science and Technology, Shanghai, China
Delay-Empowered Causal Hierarchical Reinforcement Learning
May 12, 2026Chenran Zhao, Dianxi Shi, Haotian Wang +4Reinforcement Learning Hierarchical RL
College of Computer Science and Technology, National University of Defense Technology, Changsha, China · Intelligent Game and Decision Lab (IGDL), Beijing, China · School of Artificial Intelligence, Hebei University of Technology, Tianjin, China +1
Minimax Optimal Strategy for Delayed Observations in Online Reinforcement Learning
Mar 3, 2026Harin Lee, Kevin JamiesonDelayed RL Reinforcement Learning
Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, Washington.

Jul 6, 2026J/K move · Enter open · S save

Diffusion-Guided Uncertainty-Aware Delayed Policy Optimization

Junqi Tu, Zejiao Liu, Fangfei Li, Yang Tang

The Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science and Technology, Shanghai, China · The School of Mathematics, East China University of Science and Technology, Shanghai, China

Reinforcement Learning Robotic RL Delayed RL

Open paper AskPDF arXiv

Reinforcement learning in real world environments often suffers from severe performance degradation due to delayed feedback. Existing approaches typically mitigate performance degradation caused by observation delays by constructing augmented states or predicting the true states. However, these methods often overlook the inherent discrepancy between delayed state and true states induced by stochastic MDP. We theoretically prove the existence of such a discrepancy and show that it leads to the degradation of the optimal policy. To address this challenge, we propose Diffusion Guided Uncertainty Aware Delayed Policy Optimization (DUPO). Our method explicitly models the relationship between delayed state message and the current state using a diffusion model, and leverages the resulting discrepancy estimates to weight delayed policies. Extensive experiments on continuous robotic control tasks with multiple stochastic delays demonstrate that DUPO consistently outperforms existing methods and remains effective even under long and random delay scenarios.

Abstract

Delayed RL

Explore similar work

Diffusion-Guided Uncertainty-Aware Delayed Policy Optimization

Delay-Empowered Causal Hierarchical Reinforcement Learning

Minimax Optimal Strategy for Delayed Observations in Online Reinforcement Learning