Delayed homomorphic reinforcement learning for environments with delayed feedback

Jongsoo Lee; Jangwon Kim; Soohee Han

cs.LGApr 4, 2026

Delayed homomorphic reinforcement learning for environments with delayed feedback

Authors: Jongsoo Lee, Jangwon Kim, Soohee Han

Organizations: Department of Convergence IT Engineering, POSTECH, Pohang, 37673, South Korea · Department of Electrical Engineering, POSTECH, Pohang, 37673, South Korea

Abstract

Reinforcement learning in real-world systems often involves delayed feedback, which breaks the Markov assumption and impedes both learning and control. Canonical augmentation-based approaches cause state-space explosion, which imposes a severe sample-complexity burden. Despite recent progress, state-of-the-art augmentation-based baselines either mainly alleviate the burden on the critic or rely on non-unified treatments for the actor and critic. In this study, we propose delayed homomorphic reinforcement learning (DHRL), a framework grounded in MDP homomorphisms that defines a belief-equivalence relation over the augmented state space to collapse control-redundant augmented states. In principle, this yields exact abstraction under deterministic dynamics and approximate abstraction under stochastic dynamics, enabling both the actor and critic to benefit from a structured abstraction mechanism. In finite domains, exact abstraction preserves optimality and recovers the delay-free sample-complexity order, whereas approximate abstraction admits a value-loss bound on the resulting policy. For continuous domains, we introduce deep delayed homomorphic policy gradient (D $^2$ HPG), a deep actor-critic instantiation of the DHRL framework. Experiments on continuous-control tasks in MuJoCo show that D $^2$ HPG outperforms strong augmentation-based baselines.

Figures & tables

Figure 1: Schematic overview of the DHRL framework.

Figure 2: Schematic overview of the practical instantiation of D 2 HPG.

Environment		Ant-v3	HalfCheetah-v3	Hopper-v3	Walker2d-v3	Humanoid-v3	InvertedPendulum-v2
$\Delta$	Algorithm	Ant-v3	HalfCheetah-v3	Hopper-v3	Walker2d-v3	Humanoid-v3	InvertedPendulum-v2
–	Delay-free SAC	$3279.2_{\pm 180}$	$8608.4_{\pm 57}$	$2435.2_{\pm 23}$	$3305.5_{\pm 234}$	$3228.1_{\pm 410}$	$964.3_{\pm 29}$
5	Naive SAC	$-74.9_{\pm 4}$	$-276.3_{\pm 5}$	$88.8_{\pm 10}$	$44.5_{\pm 20}$	$398.3_{\pm 6}$	$32.1_{\pm 2}$
	Augmented SAC	$881.9_{\pm 103}$	$2130.9_{\pm 344}$	$2230.8_{\pm 178}$	$1265.4_{\pm 303}$	$629.2_{\pm 56}$	$935.7_{\pm 38}$
	Delayed SAC	$1093.6_{\pm 132}$	$1753.1_{\pm 198}$	$1536.6_{\pm 248}$	$858.8_{\pm 207}$	$505.9_{\pm 68}$	$925.5_{\pm 18}$
	VDPO	$4373.3_{\pm 181}$	$4819.5_{\pm 36}$	$1917.6_{\pm 105}$	$3402.9_{\pm 263}$	$2843.2_{\pm 482}$	${764.5}_{\pm 136}$

Table 1: Results on the continuous-control MuJoCo tasks under fixed delays with

\Delta\in\{5,10,20\}

. Each algorithm was evaluated for one million time steps with five different seeds, where the standard deviations of average returns are denoted by

\pm

. The best performance is highlighted.

Figure 3: Normalized performance of D 2 HPG under actuator-level stochastic perturbations in multiple MuJoCo tasks with delay

\Delta=10

, normalized with respect to the noise-free model. The perturbations are sampled from a normal distribution with standard deviation

\sigma\in\{0.01,0.03,0.05,0.07\}

. All results are reported over one million time steps with five different seeds.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameters	Values
Actor network	$256,256$
Critic network	$256,256$
Learning rate (actor)	$3\times 10^{-4}$
Learning rate (beta-critic)	$3\times 10^{-4}$
Temperature ( $\alpha$ )	$0.2$
Target smoothing coefficient ( $\beta$ )	$0.005$

Appendix

Table 2: Hyperparameters for the baseline algorithms.

Environment	State dimension	Action dimension	Action range	Time step (s)
Ant-v3	$27$	$8$	$[-1.0,1.0]$	$0.05$
HalfCheetah-v3	$17$	$6$	$[-1.0,1.0]$	$0.05$
Walker2d-v3	$17$	$6$	$[-1.0,1.0]$	$0.008$
Hopper-v3	$11$	$3$	$[-1.0,1.0]$	$0.008$
Humanoid-v3	$376$	$17$	$[-0.4,0.4]$	$0.015$
InvertedPendulum-v2	$4$	$1$	$[-3.0,3.0]$	$0.04$

Appendix

Table 3: Environment details of the MuJoCo benchmark.

Panel of Figure 4 — Figure 4: Visual illustration of continuous control tasks in MuJoCo: (a) Ant-v3 (b) HalfCheetah-v3, (c) Walker2d-v3, (d) Hopper-v3, (e) Humanoid-v3, and (f) InvertedPendulum-v2.

Panel of Figure 5 — Figure 5: Normalized performance of D 2 HPG variants in MuJoCo, where D 2 HPG-BPQL is used as the baseline. Each algorithm was evaluated for one million time steps with five different seeds.

Figure 6: Normalized performance of D 2 HPG under fixed delays with

\Delta=10

(fixed) and random delays uniformly sampled from

\{1,2,\dots,\Delta_{\text{max}}\}

with

\Delta_{\text{max}}=10

(random), evaluated on multiple MuJoCo tasks over one million time steps with five random seeds.

Figure 7: Wall-clock runtime comparison across baselines over one million time steps.

Figure 8: Performance curves of each algorithm on the MuJoCo benchmarks with

\Delta=5

Figure 9: Performance curves of each algorithm on the MuJoCo benchmarks with

\Delta=10

Figure 10: Performance curves of each algorithm on the MuJoCo benchmarks with

\Delta=20

Delayed RL

See all 13 papers

Jul 6, 2026

Diffusion-Guided Uncertainty-Aware Delayed Policy Optimization

Reinforcement learning in real world environments often suffers from severe performance degradation due to delayed feedback. Existing approaches typically mitigate performance degradation caused by observation delays by constructing augmented states or predicting the true states. However, these methods often overlook the inherent discrepancy between delayed state and true states induced by stochastic MDP. We theoretically prove the existence of such a discrepancy and show that it leads to the degradation of the optimal policy. To address this challenge, we propose Diffusion Guided Uncertainty Aware Delayed Policy Optimization (DUPO). Our method explicitly models the relationship between delayed state message and the current state using a diffusion model, and leverages the resulting discrepancy estimates to weight delayed policies. Extensive experiments on continuous robotic control tasks with multiple stochastic delays demonstrate that DUPO consistently outperforms existing methods and remains effective even under long and random delay scenarios.

Junqi Tu, Zejiao Liu, Fangfei Li +1

The Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science and Technology, Shanghai, China · The School of Mathematics, East China University of Science and Technology, Shanghai, China

Reinforcement Learning Robotic RL Delayed RL

Ask PDF

May 12, 2026

Delay-Empowered Causal Hierarchical Reinforcement Learning

Many real-world tasks involve delayed effects, where the outcomes of actions emerge after varying time lags. Existing delay-aware reinforcement learning methods often rely on state augmentation, prior knowledge of delay distributions, or access to non-delayed data, limiting their generalization. Hierarchical reinforcement learning, by contrast, inherently offers advantages in handling delays due to its hierarchical structure, yet existing methods are restricted to fixed delays. To address these limitations, we propose Delay-Empowered Causal Hierarchical Reinforcement Learning (DECHRL). DECHRL explicitly models both the causal structure of state transitions and their associated stochastic delay distributions. These are then incorporated into a delay-aware empowerment objective that drives proactive exploration toward highly controllable states, thereby improving performance under temporal uncertainty. We evaluate DECHRL in modified 2D-Minecraft and MiniGrid environments featuring stochastic delays. Experimental results show that DECHRL effectively models temporal delays and significantly outperforms baselines in decision-making under temporal uncertainty.

Chenran Zhao, Dianxi Shi, Haotian Wang +4

College of Computer Science and Technology, National University of Defense Technology, Changsha, China · Intelligent Game and Decision Lab (IGDL), Beijing, China · School of Artificial Intelligence, Hebei University of Technology, Tianjin, China +1

Reinforcement Learning Hierarchical RL RL Exploration Causal RL Delayed RL

Ask PDF

Mar 3, 2026

Minimax Optimal Strategy for Delayed Observations in Online Reinforcement Learning

We study reinforcement learning with delayed state observation, where the agent observes the current state after some random number of time steps. We propose an algorithm that combines the augmentation method and the upper confidence bound approach. For tabular Markov decision processes (MDPs), we derive a regret bound of

\tilde{\mathcal{O}}(H \sqrt{D_{\max} SAK})

, where

S

and

A

are the cardinalities of the state and action spaces,

H

is the time horizon,

K

is the number of episodes, and

D_{\max}

is the maximum length of the delay. We also provide a matching lower bound up to logarithmic factors, showing the optimality of our approach. Our analytical framework formulates this problem as a special case of a broader class of MDPs, where their transition dynamics decompose into a known component and an unknown but structured component. We establish general results for this abstract setting, which may be of independent interest.

Harin Lee, Kevin Jamieson

Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, Washington.

Delayed RL Reinforcement Learning Regret Minimization in RL

Ask PDF

Abstract

Delayed RL

Explore similar work

Diffusion-Guided Uncertainty-Aware Delayed Policy Optimization

Delay-Empowered Causal Hierarchical Reinforcement Learning

Minimax Optimal Strategy for Delayed Observations in Online Reinforcement Learning