Organizations: School of Computer Science and Engineering, UNSW Sydney, Australia · Frankfurt Institute for Advanced Studies, Germany · Escuela de Ingeniería, Universidad Central de Chile, Chile
Sensory delays create temporal misalignments between the state of the environment and the observations available to an agent, turning Markovian control problems into partially observable ones. In this work, we inquire whether such misalignments can be mitigated during training by using hindsight to recover aligned state-action transitions. We introduce Hindsight Critics, an asymmetric extension of the Soft Actor-Critic for reinforcement learning under partial observability. The critics are trained on hindsight-aligned transitions whereas the actor learns from the resulting privileged value estimates despite only having access to delayed observations. Comparisons against delay-aware and unaware baselines on the DelayBench suite show that Hindsight Critics can partially compensate for observation delays without requiring modifications to the agent's observations during deployment.
Figures & tables
Figure 1: Return-delay curves for PPO, SAC and Hindsight Critics (HC) over three illustrative environments. Scattered dots indicate the return over a single evaluation episode. Sigmoid curves are fitted to those returns and used to compute the AUC scores. SAC achieves high zero-delay performances but is particularly sensitive to delays. Hindsight Critics can partially mitigate this degradation.
Environment
PPO
A2C
DDPG
TD3
SAC
HC
Pendulum-v1
-1068.7
-1245.3
-139.3
-140.9
-156.7
-154.8
InvertedPendulum-v5
1000.0
856.8
761.8
954.4
1000.0
1000.0
InvertedDoublePendulum-v5
8589.9
501.8
7496.3
8616.9
9319.8
9319.8
Reacher-v5
-4.7
-26.8
-4.5
-4.7
-5.6
-5.7
HalfCheetah-v5
1942.3
813.8
4235.7
–
5282.5
4571.6
Hopper-v5
1964.0
148.0
873.6
89.6
3247.0
2630.4
Table 1: Zero-delay average returns of Hindsight Critics (HC) compared with other common RL algorithms across multiple environments. The highest score for each environment is highlighted in bold .
Environment
PPO
A2C
DDPG
TD3
SAC
HC
Pendulum-v1
.112
.000
.192
.205
.182
.634
InvertedPendulum-v5
.215
.028
.013
.019
.023
.024
InvertedDoublePendulum-v5
.035
.002
.038
.034
.037
.043
Reacher-v5
.701
.374
.725
.597
.509
.490
HalfCheetah-v5
.144
.010
.041
–
.034
.237
Hopper-v5
.095
.007
.027
.010
.034
.030
Table 2: AUC scores of Hindsight Critics (HC) compared with other common RL algorithms across multiple environments. The highest score for each environment is highlighted in bold .
history stacking
forward model
Environment
PPO
SAC
HC
PPO
SAC
HC
Pendulum-v1
.243
.813
.859
.130
.740
.817
InvertedPendulum-v5
.219
.023
.024
.228
.023
.024
InvertedDoublePendulum-v5
.036
.046
.049
.047
.046
.048
Reacher-v5
.757
.862
.875
.714
.859
.857
HalfCheetah-v5
.128
.187
.406
.164
.349
.539
Table 3: AUC scores for the delay-aware versions (history stacking and forward model) of Hindsight Critics (HC), PPO, and SAC, across multiple environments. The highest score for each delay-aware version and environment is highlighted in bold .
We formally establish the equivalence between Observation Delay (OD) and Action Delay (AD) in cooperative partially observable multi-agent systems using observation-action histories. We show that both systems generate identical admissible joint-policy sets, and their induced state-action-observation trajectories are identical in distribution, leading to identical optimal solutions in Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs). This formally generalizes existing infinite-horizon single-agent results to any-horizon partially observable cooperative multi-agent problems with decentralized policy execution, and allows any mixed-delay configuration to be reduced to a pure OD system. We further prove that in Transition-Independent MDPs (TI-MDPs), the observation-action history reduces to a tractable minimal local augmented state. However, we show through numerical experiments that although the optimal solution spaces are structurally isomorphic, the practical learning dynamics are fundamentally different. First, using the minimal local augmented state, the equivalence no longer holds when transitions are not independent. Second, operational constraints and causal credit-assignment errors in Temporal Difference (TD) algorithms induce different learning behaviors across regimes. Finally, leveraging this structural equivalence to bypass these learning challenges, we demonstrate successful multi-agent zero-shot policy transfer from OD to AD, paving the way for unified, efficient solution methods in complex delayed systems.
Jules Sintes, Ana Bušić, Jiamin Zhu
Inria · DIENS, École Normale Supérieure, PSL University · Paris, France +2
We study reinforcement learning with delayed state observation, where the agent observes the current state after some random number of time steps. We propose an algorithm that combines the augmentation method and the upper confidence bound approach. For tabular Markov decision processes (MDPs), we derive a regret bound of O~(HDmaxSAK), where S and A are the cardinalities of the state and action spaces, H is the time horizon, K is the number of episodes, and Dmax is the maximum length of the delay. We also provide a matching lower bound up to logarithmic factors, showing the optimality of our approach. Our analytical framework formulates this problem as a special case of a broader class of MDPs, where their transition dynamics decompose into a known component and an unknown but structured component. We establish general results for this abstract setting, which may be of independent interest.
Harin Lee, Kevin Jamieson
Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, Washington.
When a reinforcement learning agent cannot observe the full state, we usually blame its policies: it cannot see enough to represent a good one. We show that in a solvable case the bigger problem lies elsewhere. Even when a good policy is available and the agent's value function is expressive enough to describe it exactly, learning still ends up somewhere far worse. We study a partially observed linear-quadratic problem in which a standard actor-critic learner can be solved in closed form. At our default setting the best policy the agent can represent is already close to optimal, costing 10.4% more than the ideal controller that observes everything. Learning does not find it. The algorithm instead comes to rest at a policy that is 35% worse than the best one available to it, and we can say exactly where and why. The cause is a bias in what the critic learns rather than a limit on what the actor can express. Because the agent cannot attribute what it sees to the part of the state it cannot observe, the critic misreads that unexplained variation as sharp curvature in its own value estimates, and the actor follows that error away from the optimum. We derive closed-form expressions for the resulting policy, for its cost, and for the one design choice that removes the problem, which is how far the learner looks ahead before trusting its own value estimates. Deep reinforcement learning experiments follow these predictions closely. Notably, giving the agent memory of past observations does not help, while changing how far it looks ahead does.
Idil Gözel
University College London · Department of Physics and Astronomy University College London, London, UK