Memory-State Critic for Asymmetric Actor-Critic with Application to Vision-Based Pursuit-Evasion

Arthur Louette; Alejandro Sánchez Roncero; Gaspard Lambrechts; Pascal Leroy; Julien Hansen; Petter Ögren; Damien Ernst

cs.LGOct 2, 2026

Memory-State Critic for Asymmetric Actor-Critic with Application to Vision-Based Pursuit-Evasion

Authors: Arthur Louette, Alejandro Sánchez Roncero, Gaspard Lambrechts, Pascal Leroy, Julien Hansen, Petter Ögren, Damien Ernst

Organizations: University of Liège, Belgium · KTH Royal Institute of Technology, Sweden · McGill University & Mila Québec AI Institute, Canada · Belerion, Belgium

Abstract

In partially observable Markov decision processes, the optimal policy generally depends on the history of observations and past actions. Asymmetric actor-critic methods have become popular to learn such policies when additional information, such as the true state of the environment, is available during training. The critic, which is not needed at execution, is given access to the state. A critic conditioned on the state alone is generally ill-defined and yields biased policy gradients. Conditioning on the state and the history, the history-state critic restores both. In this paper, we show that conditioning the critic on the state and the policy's own memory, i.e., the internal representation of the history through which the policy selects its actions, is already well-defined and gives unbiased policy gradients, removing the need for a second recurrent approximator of the history. We call it the memory state critic. It follows that a critic based on the policy's memory need not backpropagate its loss into that memory, even though the memory is a lossy encoding of the history. We evaluate the memory-state critic in a vision-based pursuit-evasion environment between two quadrotors across two arena types. The pursuer is the learning agent, and the evader is sampled per episode from a fixed pool of heuristic behaviours. The results show that the memory-state critic outperforms the history-state critic and converges faster. In addition to being unbiased compared to the state-only critic, it maintains a slight edge in the wall arena, where the actor's history carries information that the privileged state alone does not.

Figures & tables

Figure 1: Asymmetric actor-critic architectures with a recurrent policy. Left: the history-state critic maintains a second recurrent encoder

f_{\psi}

on the critic side, in parallel with the policy’s encoder

f_{\theta}

. Right: the memory-state critic reuses the policy’s hidden state

z^{a}

with a stop-gradient, removing the second recurrent module. Both critics consume the state

s

in their MLP head.

Figure 2: Vision-based pursuit-evasion environment. Left: IsaacLab simulates many arenas in parallel. Each contains a pursuer (red), an evader (blue), and an optional wall with a gap that can hide the evader. Middle: a representative depth channel from the pursuer’s camera. Right: the binary opponent segmentation mask. Both channels are $64\times 64$ .

Critic	Symbol	Description
Observation-state	$V_{\psi}(s,o,a)$	State, current observation, previous action.
State-only	$V_{\psi}(s)$	State only. Biased with a policy conditioned on the history.
History-state	$V_{\psi}(s,z^{c})$	State and a critic-side recurrent encoding of the history.
Memory-state (ours)	$V_{\psi}(s,z^{a})$	State and the policy’s memory with stop-gradient.

Table 1: Critic variants compared in the experiments.

Figure 3: Pursuer episode return during training. Lines are interquartile means over 20 seeds ( Agarwal et al., 2022 ) . The shaded intervals are $95\%$ bootstrap confidence intervals. The memory-state critic matches or exceeds the history-state critic in both arenas and converges faster. It exceeds the state-only critic in the wall arena and matches it in the open arena.

Figure 4: Terminal-condition breakdown per critic and arena. Each cell shows the fraction of evaluation episodes ending in the given outcome as the mean over 20 seeds, $\pm$ one standard deviation. The wall arena adds the pursuer wall collision outcome.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Discount factor $\gamma$	0.99 (open) / 0.999 (wall)
GAE $\lambda$	0.95
Learning rate	$3\times 10^{-4}$
LR scheduler	KL-adaptive ( $\tau_{\text{KL}}=0.008$ )
Rollout length	32 steps
Mini-batches	4

Appendix

Table 2: PPO hyperparameters, shared across all critic variants.

Asynchronous Advantage Actor-Critic

See all 9 papers

Oct 3, 2026

Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regression

Actor-critic methods reuse past experience to improve sample efficiency. However, historical data are typically regarded as off-policy samples for the current policy-improvement update. This work introduces Regularized Dual Averaging Actor Critic (RDA2C), which assigns a distinct role to replay. In regularized dual averaging, the subsequent policy is determined by accumulated policy-improvement feedback, so historical advantage estimates contribute directly to the actor objective rather than solely to the most recent update. RDA2C stores state-action samples with critic-estimated advantage labels, fits a dual score model

Z_θ

to the aggregated dataset, and derives the current policy from the accumulated score model using the entropy mirror map. In this way, replay defines an empirical dual objective from which the policy is computed. To analyze RDA2C, we establish a finite-time value-gap decomposition, separating the regularized dual-averaging term from errors due to stale-replay supervised fitting, critic bias, finite-buffer variance, and replay coverage, and stating the assumptions under which each error is bounded. RDA2C accepts advantage labels from any critic. With GAE labels, RDA2C outperforms PPO on six of eight MuJoCo tasks and eight of twelve Atari games. RDA2C also outperforms AAPDA, the closest dual-averaging baseline, on six of eight MuJoCo tasks. With twin-

Q

labels, RDA2C matches SAC at matched batch size and update frequency.

Nianli Peng, Geoffrey J. Gordon, Kianté Brantley

Harvard University · Carnegie Mellon University

Actor-Critic Methods Experience Replay Off-Policy RL

Ask PDF

May 23, 2026

Refined Analysis of Entropy-Regularized Actor-Critic

In this paper, we study the role of the critic in actor--critic for entropy-regularized, finite, discounted environments. We establish that, when the critic is exact, using the latter as a baseline is a variance-reduction method in a strong sense. In this case, actor--critic with stochastic gradients matches the sample complexity of deterministic policy gradient, reaching an

ε

-optimal regularized value with

\tilde{O}(\log(1/ε))

samples. In practice, the critic is learned alongside the actor: the variance of the actor update is then influenced by the critic's variance and bias. Specifically, when the critic has a sufficiently small error, the variance reduction and rapid convergence are preserved. This suggests to learn the critic first, keeping it up to date after each actor update, underscoring the crucial role of accurate critic estimation in actor--critic methods.

Safwan Labbi, Paul Mangold, Daniil Tiapkin +1

CMAP, CNRS, École Polytechnique, Institut Polytechnique de Paris, 91120 Palaiseau, France · Université Paris-Saclay, CNRS, LMO, 91405, Orsay, France · Google DeepMind +2

Reinforcement Learning Maximum Entropy RL Policy Gradient Methods Actor-Critic Methods Sample-Efficient RL

Ask PDF

Sep 27, 2026

When Does Backpropagating Through Policy Memory Matter? Separating Representation Credit from Closed-Loop Credit

When a policy with memory is trained through a differentiable simulator, credit can reach what it stores along two routes: representation credit teaches the network that writes the memory what to summarize, while closed-loop credit travels back through the physical states that earlier actions produced and that the memory later stored. Stopping gradients at stored history, as truncated backpropagation and Transformer-XL do, removes both routes at once. Keeping every forward computation identical and removing one route at a time, we trained Transformer policies for quadrotor tracking to determine when each route matters. Credit through memory improved learning only when velocity observations were noisy or hidden. When velocity was hidden, mildly decaying the gradient between consecutive physical states (

γ= 0.8

) eliminated the truncation cost of removing closed-loop credit while leaving representation credit essential, a separation that held in 14 of 16 new seeds. Two further systems bound this result: in a fixed-wing aircraft, representation credit could replace closed-loop credit on its own, leaving no cost to separate, and in a cart-pole, training with full state-edge credit was too unstable to compare the routes. In both the quadrotor and a fully observed vessel model, static gradient norms failed to predict which cut cost more, so memory truncation choices are best guided by short training runs.

Xingjian Li, Jianhua Z. Huang, Junli Duan

School of Data Science, The Chinese University of Hong Kong, Shenzhen · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen · COSCO Shipping Technology Co., Ltd.

Policy Learning Memory-Augmented Agents Neural Network Training Dynamics Recurrent Neural Networks Robotic Control

Ask PDF

Abstract

Asynchronous Advantage Actor-Critic

Explore similar work

Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regression

Refined Analysis of Entropy-Regularized Actor-Critic

When Does Backpropagating Through Policy Memory Matter? Separating Representation Credit from Closed-Loop Credit