cs.LGNov 9, 2026

Can Hindsight Critics Guide Delayed Actors?

Authors: Francisco M. López, Francisco Cruz

Organizations: School of Computer Science and Engineering, UNSW Sydney, Australia · Frankfurt Institute for Advanced Studies, Germany · Escuela de Ingeniería, Universidad Central de Chile, Chile

Abstract

Sensory delays create temporal misalignments between the state of the environment and the observations available to an agent, turning Markovian control problems into partially observable ones. In this work, we inquire whether such misalignments can be mitigated during training by using hindsight to recover aligned state-action transitions. We introduce Hindsight Critics, an asymmetric extension of the Soft Actor-Critic for reinforcement learning under partial observability. The critics are trained on hindsight-aligned transitions whereas the actor learns from the resulting privileged value estimates despite only having access to delayed observations. Comparisons against delay-aware and unaware baselines on the DelayBench suite show that Hindsight Critics can partially compensate for observation delays without requiring modifications to the agent's observations during deployment.

Figures & tables

Explore similar work

CardsList
  1. Structural Equivalence and Learning Dynamics in Delayed MARL

    May 5, 2026Jules Sintes, Ana Bušić, Jiamin ZhuPartially Observable Markov Decision ProcessesZero-Shot Policy Transfer

  2. Minimax Optimal Strategy for Delayed Observations in Online Reinforcement Learning

    Mar 3, 2026Harin Lee, Kevin JamiesonDelayed RLReinforcement Learning

  3. Learning Suffers More Than the Policy Class Under Partial Observability: A Closed-Form Analysis

    Aug 7, 2026Idil GözelRL ControlReinforcement Learning