When Does Backpropagating Through Policy Memory Matter? Separating Representation Credit from Closed-Loop Credit

Xingjian Li; Jianhua Z. Huang; Junli Duan

cs.LGSep 27, 2026

When Does Backpropagating Through Policy Memory Matter? Separating Representation Credit from Closed-Loop Credit

Authors: Xingjian Li, Jianhua Z. Huang, Junli Duan

Organizations: School of Data Science, The Chinese University of Hong Kong, Shenzhen · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen · COSCO Shipping Technology Co., Ltd.

Abstract

When a policy with memory is trained through a differentiable simulator, credit can reach what it stores along two routes: representation credit teaches the network that writes the memory what to summarize, while closed-loop credit travels back through the physical states that earlier actions produced and that the memory later stored. Stopping gradients at stored history, as truncated backpropagation and Transformer-XL do, removes both routes at once. Keeping every forward computation identical and removing one route at a time, we trained Transformer policies for quadrotor tracking to determine when each route matters. Credit through memory improved learning only when velocity observations were noisy or hidden. When velocity was hidden, mildly decaying the gradient between consecutive physical states ( $γ= 0.8$ ) eliminated the truncation cost of removing closed-loop credit while leaving representation credit essential, a separation that held in 14 of 16 new seeds. Two further systems bound this result: in a fixed-wing aircraft, representation credit could replace closed-loop credit on its own, leaving no cost to separate, and in a cart-pole, training with full state-edge credit was too unstable to compare the routes. In both the quadrotor and a fully observed vessel model, static gradient norms failed to predict which cut cost more, so memory truncation choices are best guided by short training runs.

Figures & tables

Panel of Figure 1 — Figure 1: Two routes for credit through memory. (a) A step $j$ of a rollout whose stored representation $n_{j}$ is read at a later step $t$ . Credit that reaches $n_{j}$ can end at the parameters $\theta$ inside its computation (representation credit, blue) or leave through the observation $o_{j}$ into the state $s_{j}$ and continue to earlier actions (closed-loop credit, vermilion). The state-edge decay $\gamma$ scales every state-to-state edge. (b) Each ablation arm keeps (filled) or removes (open) each route. All arms produce the same rollout and differ only in where the learning signal goes.

Arm	Credit removed	Forward computation	Used in
full policy	none	reference	all sections
no representation credit	$P_{\mathrm{rep}}$ and $P_{\mathrm{mix}}$	identical	5.1 – 5.6
no closed-loop credit	$P_{\mathrm{loop}}$ and $P_{\mathrm{mix}}$	identical	5.1 – 5.6
no credit through memory	all paths through memory	identical	5.1 , 5.5 , 5.6
no observation gradient	every path through an observation	identical	5.1
short memory ( $w=1,2$ )	none (memory restricted)	attention window $w$	5.1

Table 1: Ablation arms. Each removes one class of backward paths, and only the short-memory controls change the forward computation.

Figure 2: Credit through memory matters only when velocity is unreliable. Quadrotor,

\gamma=1

. Truncation cost is the increase in final error relative to the full policy trained with the same seed. (a) Memory restricted to the current step. (b) Removing representation credit, closed-loop credit, or both. (c) Stopping gradients at every observation. Small points are seeds (16 per condition), and large points and bars are means with

95\%

intervals. Vertical scales differ between panels.

Figure 3: Mild state-edge decay removes the contribution of closed-loop credit. Quadrotor. (a) Error of the full policy during training at

\gamma=1

0.8

, and

0

, velocity hidden (16 seeds). (b, c) Truncation cost of each route, the increase in final error when it is removed, at

\gamma=1

and

0.8

in the 16 new seeds, with velocity hidden (b) or noisy (c) . Thin lines are seeds and thick lines means. (d) How much more truncation cost fell for closed-loop credit than for representation credit from

\gamma=1

0.8

, per seed; positive values follow the prediction. Black points and bars are means with

97.5\%

intervals, widened because two predictions share these runs (Appendix A ), and horizontal ticks are medians. With velocity hidden, closed-loop credit lost its value at

\gamma=0.8

while representation credit kept it.

Figure 4: Further differences between the two routes. Quadrotor, noise

0.20\text{ m/s}

unless stated. (a, b) Truncation cost when credit is kept for the

s

most recent stored steps and removed for older ones, with noisy (a) or hidden (b) velocity, where

s=0

is the full cut. (c) Truncation cost after 1000 and 2000 updates in 8 seeds, with thin lines for seeds. (d) Truncation cost when the cut starts at a later update, with the optimizer state carried over. Bands and bars are

95\%

intervals over 16 seeds unless stated.

		Credit removed
System	Velocity	both routes	closed-loop only	representation only
Quadrotor, 16 seeds	hidden	28% (16/16)	13% (14/16)	23% (16/16)
Fixed-wing, 16 seeds	hidden	5.8% (11/16)	0.3% (10/16)	4.9% (11/16)
Vessel, 16 streams	observed	$-0.5\%$ (6/16)	$-0.4\%$ (6/16)	1.0% (12/16)

Table 2: Truncation cost of each route in three systems,

\gamma=1

: medians over seeds of the increase in final error when the route is removed, with the number of seeds in which the error increased. We use medians because a few fixed-wing seeds had large costs. Means with intervals are in Section 5.1 and Appendices C , E , and F , and the cart-pole, whose training was too unstable to compare the routes, is reported in Appendix E .

Figure 5: Gradient magnitude does not predict truncation cost. (a) Norm of one route’s contribution, relative to the full gradient, against the truncation cost of removing that route, for the quadrotor at

\gamma=1

(filled) and

\gamma=0

(open) under four velocity conditions, and the vessel model (stars). (b) Norm of the gradient without any credit through memory, relative to the full gradient, against the cost of removing all credit through memory. Gradient norms are medians over 64 probe batches from four separately trained full policies at update 1000, and costs are means over the 16 seeds of Section 5.1 . For the vessel, norms are medians over 46 windows at the final checkpoint, and costs are means over its 16 streams.

Situation	What we observed	Suggested practice	Basis
State observed accurately	Cutting memory credit changed error by $<0.3\%$ (quadrotor), $<1.5\%$ (vessel)	Truncation is safe	Quadrotor, vessel
Velocity noisy or hidden, full physical credit	Cutting memory credit cost $13\text{--}34\%$	Keep credit through memory	Quadrotor, 16 seeds
Velocity hidden, mild state-edge decay	Closed-loop truncation cost vanished; representation credit kept its value	If cutting, keep representation credit	Quadrotor, 16 new seeds
Closed-loop credit replaceable	Removing it alone cost almost nothing under every measure; removing both cost $5.8\%$ (median)	Cutting closed-loop credit may be harmless; check by training	Fixed-wing, 16 seeds
Unstable plant with full physical credit	Runs relapsed, and costs depended on when the error was measured	Stabilize training before comparing cuts	Cart-pole, 16 seeds
Detaching all observations	Error rose $51\text{--}107\%$ , even with exact velocity	Avoid with memory policies at $\gamma=1$	Quadrotor, 16 seeds

Table 3: Suggestions for practice and the observations they rest on.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

	Quadrotor	Cart-pole	Fixed-wing	Vessel
State, action dimensions	12, 4	4, 1	12, 4	4, 2
Control step, horizon	$0.05\text{ s}$ , 32	$0.05\text{ s}$ , 32	$0.05\text{ s}$ , 32	$5\text{ min}$ , 288
Policy	2 layers, width 64, 4 heads			$2\times 2$ layers, 256, 8 heads
Parameters	94,324	93,129	93,940	3,677,698
Initialization	random	random	random	pretrained
Learning rate	$10^{-3}$	$10^{-3}$	$5\times 10^{-4}$	$3\times 10^{-5}$

Appendix

Table A1: Training settings. All policies are trained from the stated initialization with AdamW and no learning-rate schedule.

Prediction	Where	Outcome
Going from exact to noisy velocity raises the cost of removing representation credit more than that of removing closed-loop credit (quadrotor)	5.2	Not supported: $-3.0$ $[-9.8,3.0]$
With velocity hidden, removing representation credit costs more than removing closed-loop credit	5.2	Supported in the original seeds: $11.2$ $[0.1,23.3]$ , 13 of 16; close in the new seeds ( $20\%$ and $19\%$ )
Removing state-edge credit ( $\gamma=0$ , noise $0.20$ ) lowers the closed-loop cost more than the representation cost	5.3	Not supported: $3.4$ $[-2.7,10.3]$ ; training stalled
Mild decay ( $\gamma=0.8$ ), new seeds, velocity hidden: lowers the closed-loop cost more than the representation cost	5.3	Supported: $39$ $[10,81]$ , 14 of 16
Mild decay ( $\gamma=0.8$ ), new seeds, noise $0.20$ : lowers the closed-loop cost more than the representation cost	5.3	Not established: $3.8$ $[-1.1,9.1]$
Keeping two recent steps recovers more of the representation cost than of the closed-loop cost	5.4	Reversed at noise $0.20$ (ratio difference $-0.55$ $[-0.82,-0.34]$ ); not resolved with velocity hidden

Appendix

Table B1: Predictions stated before the runs that test them. Differences are in percentage points with the widened intervals of Appendix A .

Arm	exact	noise 0.05	noise 0.20	hidden
no representation credit	-0.2 [-0.4, 0.1]	-0.9 [-2.1, 0.2]	12.6 [10.1, 15.2]	26.1 [19.4, 33.9]
no closed-loop credit	0.2 [-0.0, 0.4]	6.3 [4.1, 8.4]	16.0 [12.7, 19.9]	14.9 [9.0, 22.2]
no credit through memory	-0.2 [-0.4, 0.0]	3.3 [1.8, 4.8]	26.9 [24.1, 29.8]	33.9 [24.9, 43.5]
interaction	-0.2 [-0.5, 0.0]	-2.1 [-4.4, 0.3]	-1.8 [-5.5, 1.8]	-7.1 [-17.6, 3.7]
no observation gradient	75.0 [44.6, 121.4]	51.5 [41.2, 62.4]	74.1 [66.1, 82.3]	106.5 [90.4, 123.1]
memory of 1 step	-0.3 [-0.5, 0.0]	2.7 [1.6, 3.6]	38.4 [35.3, 41.4]	106.8 [99.3, 113.8]

Appendix

Table C1: Quadrotor truncation costs at

\gamma=1

(

\%

), means with

95\%

intervals over 16 seeds. The last row gives the full policy’s median final error.

Velocity	$\gamma$	Arm	Same- $\gamma$ cost	Common-scale cost
noise 0.20	0	no rep.	0.0 [-0.2, 0.3]	0.5 [-3.8, 4.5]
		no loop	0.1 [-0.0, 0.2]	1.4 [-0.6, 3.1]
		no memory	0.1 [-0.2, 0.4]	1.6 [-2.9, 6.4]
	0.8	no rep.	7.6 [4.7, 10.6]	13.0 [8.1, 18.2]
		no loop	5.4 [3.0, 7.9]	9.1 [5.1, 13.4]
		no memory	13.1 [10.1, 16.4]	22.7 [17.5, 28.4]

Appendix

Table C2: Quadrotor truncation costs (

\%

) under state-edge decay, original 16 seeds. Same-

\gamma

costs are relative to the full policy at the same

\gamma

, and common-scale costs measure the excess error in units of the full policy’s error at

\gamma=1

. The full policy’s error was

16

and

18

times its

\gamma=1

value at

\gamma=0

(noise

0.20

and hidden velocity), and

1.7

times at

\gamma=0.8

Velocity	$\gamma$	Arm	Same- $\gamma$ cost	Common-scale cost
hidden	1	no rep.	20.2 [13.4, 27.5]	–
		no loop	18.5 [6.1, 39.7]	–
		no memory	37.3 [29.3, 46.1]	–
	0.8	no rep.	40.8 [24.0, 69.0]	65.8 [39.4, 110.1]
		no loop	-0.4 [-2.1, 1.5]	-0.6 [-3.5, 2.5]
		no memory	24.8 [18.6, 30.9]	40.1 [30.4, 49.3]

Appendix

Table C3: Quadrotor truncation costs (

\%

) in the 16 new seeds.

{}^{\dagger}97.5\%

interval, widened as described in Appendix A .

Velocity	Cost vs. full policy at $\gamma=0$	Common-scale cost	Full policy, $\gamma=0$ / $\gamma=1$
exact	0.0 [-0.1, 0.1]	0.3 [-1.7, 2.3]	20.3
noise 0.05	-0.0 [-0.1, 0.1]	-0.1 [-1.8, 1.5]	18.7
noise 0.20	0.1 [-0.0, 0.2]	1.7 [-0.4, 3.6]	16.1
hidden	-0.0 [-0.1, 0.1]	-0.4 [-2.3, 1.6]	18.5

Appendix

Table C4: One-step physical credit combined with no observation gradient, the configuration closest to the controllers of Nachkov et al. (2025) . Costs (

\%

) are relative to the full policy at

\gamma=0

; the last column gives how much worse the full policy is at

\gamma=0

than at

\gamma=1

Original 16 seeds, $\gamma=1$
		at	mean	median	mean	median	lowest
Velocity	Arm	1000	800–1000	800–1000	600–1000	600–1000	500–1000
hidden	no rep.	26.1 (16)	25.3 (16)	25.3 (16)	25.0 (16)	25.4 (16)	20.7 (16)
	no loop	14.9 (14)	17.4 (16)	13.9 (16)	15.3 (16)	10.9 (16)	8.0 (16)
	no memory	33.9 (16)	38.8 (16)	40.4 (16)	39.1 (16)	37.6 (16)	28.2 (16)
noise 0.20	no rep.	12.6 (16)	11.3 (16)	11.0 (15)	9.0 (16)	10.2 (16)	12.1 (16)

Appendix

Table C5: Quadrotor truncation costs (

\%

) under six measures of the final error (Appendix A ). Means over 16 seeds, with the number of seeds in which the error increased. For the difference of drops from

\gamma=1

\gamma=0.8

, the mean and the median, with the number of seeds in which the difference was positive. The first measure is the one used in the main text.

Steps kept	No rep.	No loop	No memory
0 (full cut)	12.6 [10.1, 15.2]	16.0 [12.6, 20.0]	26.9 [24.1, 29.9]
1	11.9 [9.8, 14.0]	8.7 [6.5, 11.2]	21.8 [18.3, 25.7]
2	11.5 [9.3, 13.8]	5.9 [4.0, 7.8]	17.3 [14.5, 20.4]
4	4.3 [2.5, 5.8]	2.1 [0.8, 3.5]	8.1 [6.5, 9.7]
8	0.5 [-0.8, 1.8]	0.5 [-0.4, 1.4]	1.3 [0.4, 2.3]
16	-0.3 [-0.8, 0.1]	-0.1 [-0.4, 0.2]	-0.3 [-0.8, 0.1]

Appendix

Table D1: Truncation cost (

\%

) when credit through memory is kept for the

s

most recent stored steps, for noise

0.20\text{ m/s}

(top) and hidden velocity (bottom). Means with

95\%

intervals over 16 seeds.

Arm	1000 updates	2000 updates	Change	Seeds with increase
no rep.	13.9 [10.0, 18.0]	22.0 [15.6, 28.8]	8.1 [-1.4, 17.0]	5/8
no loop	14.8 [10.4, 19.9]	5.8 [3.5, 8.0]	-8.9 [-13.4, -4.4]	2/8
no memory	27.8 [25.6, 30.3]	21.3 [16.4, 25.8]	-6.5 [-11.9, -0.9]	3/8

Appendix

Table D2: Truncation cost (

\%

) after 1000 and 2000 updates of the same runs, 8 seeds.

Switch at	No rep., carried	No rep., reset	No loop, carried	No loop, reset	Reset only
250	13.8 [11.7, 16.7]	7.6 [4.0, 11.9]	14.7 [11.6, 18.2]	12.6 [9.4, 17.0]	-7.2 [-8.7, -5.8]
500	9.8 [8.6, 11.2]	8.1 [6.2, 10.3]	9.6 [7.4, 11.8]	8.2 [5.7, 11.0]	-5.5 [-6.9, -4.3]
750	4.8 [2.9, 6.8]	5.4 [3.7, 7.0]	5.5 [3.5, 7.7]	5.7 [4.9, 6.6]	0.3 [-1.1, 1.6]

Appendix

Table D3: Truncation cost (

\%

) when the cut starts at update

\tau

, with the optimizer state carried over or reset. The last column resets the optimizer without any cut.

Switch at update 500	Optimizer state carried	Optimizer state reset
no rep. $\to$ full	-1.1 [-2.9, 0.6]	-4.9 [-6.7, -3.2]
no loop $\to$ full	6.3 [4.6, 7.9]	-4.0 [-5.5, -2.7]

Appendix

Table D4: Truncation cost (

\%

) when a cut that ran from the start is lifted at update 500.

$\gamma$	Arm	Mean [95% interval]	Median (seeds with increase)	Runs that relapsed
1	full policy	–	–	3/16
	no rep.	161.5 [-13.6, 494.3]	2.9 (9/16)	3/16
	no loop	860.4 [8.9, 2515.6]	9.1 (9/16)	9/16
	no memory	45.1 [-0.7, 100.7]	9.3 (12/16)	8/16
0.9	full policy	–	–	2/16
	no rep.	37.0 [6.2, 83.5]	9.8 (13/16)	4/16

Appendix

Table E1: Cart-pole truncation costs (

\%

), velocity hidden, all 16 seeds. Means with

95\%

intervals, and medians with the number of seeds in which the error increased. The last column counts the runs in each arm that relapsed (Appendix A ).

$\gamma$	Arm	Mean [95% interval]	Median (seeds with increase)
1	no rep.	7.2 [1.5, 13.3]	4.9 (11/16)
	no loop	3.7 [0.3, 8.0]	0.3 (10/16)
	no memory	8.7 [2.8, 15.3]	5.8 (11/16)
	interaction	-2.2 [-5.8, 0.6]	0.0 (8/16)
0.9	no rep.	18.1 [0.8, 39.0]	12.3 (10/16)
	no loop	21.2 [-1.0, 52.8]	6.1 (10/16)

Appendix

Table E2: Fixed-wing truncation costs (

\%

), velocity hidden, 16 seeds. Means with

95\%

intervals, and medians with the number of seeds in which the error increased.

		at	mean	median	mean	median	lowest
System	Arm	1000	800–1000	800–1000	600–1000	600–1000	500–1000
fixed-wing	no rep.	4.9 (11)	1.4 (9)	0.9 (10)	1.1 (11)	1.6 (12)	2.0 (10)
	no loop	0.3 (10)	0.1 (11)	-0.2 (6)	0.2 (10)	0.1 (9)	0.2 (11)
	no memory	5.8 (11)	2.5 (10)	2.2 (12)	1.3 (12)	3.0 (13)	4.1 (12)
	difference of drops	-9.2 (7)	2.2 (8)	2.2 (9)	-0.2 (8)	5.2 (10)	-7.1 (6)
cart-pole	no rep.	2.9 (9)	9.1 (10)	8.4 (10)	-0.6 (8)	0.4 (8)	3.8 (11)

Appendix

Table E3: Fixed-wing and cart-pole truncation costs (

\%

) at

\gamma=1

under six measures of the final error (Appendix A ), velocity hidden. Medians over 16 seeds, with the number of seeds in which the error increased. The last row of each system gives the difference of drops from

\gamma=1

\gamma=0.9

, with the number of seeds in which it was positive. The first measure is the one used in the main text.

Arm	Cost [95% interval]	Median	Increase	Clipped ( $\%$ )	Events, first six	Events, further ten
full policy	–	–	–	11.4 (10.4–12.9)	6	0
no rep.	0.7 [0.0, 1.2]	1.0	12/16	10.2 (9.1–11.2)	2	5
no loop	-0.3 [-1.2, 0.7]	-0.4	6/16	12.5 (11.1–13.5)	22	78
no memory	0.4 [-0.6, 1.5]	-0.5	6/16	11.0 (9.9–12.7)	79	64
no obs. gradient	-0.4 [-1.3, 0.4]	-0.3	7/16	13.9 (12.5–15.2)	0	0

Appendix

Table F1: Vessel model, 16 sampling streams. Truncation costs (

\%

) average the final errors at updates 2000, 2200, and 2400. Clipping is the mean share of clipped updates per run, with its range. Events are numerical events in the six streams where we first noticed the pattern and in the ten further streams.

$\gamma$	Velocity	$\\|P_{\mathrm{rep}}\\|/\\|g\\|$	$\\|P_{\mathrm{loop}}\\|/\\|g\\|$	$\\|P_{\mathrm{mix}}\\|/\\|g\\|$	$\\|g_{\text{no-mem}}\\|/\\|g\\|$
1	exact	0.21	0.11	0.02	0.94
	noise 0.05	1.74	0.81	1.11	1.27
	noise 0.20	0.66	0.66	0.45	1.09
	hidden	0.68	1.52	1.19	0.93
0	exact	0.21	0.04	0.01	1.02
	noise 0.05	0.21	0.05	0.01	1.04

Appendix

Table G1: Quadrotor gradient norms relative to the full gradient, medians over 64 probe batches at update 1000.

Checkpoint	$\\|P_{\mathrm{rep}}\\|/\\|g\\|$	$\\|P_{\mathrm{loop}}\\|/\\|g\\|$	$D$ , median [95% interval]	Windows
residual model, 200 updates	0.022	0.012	0.018 [0.014, 0.025]	46
residual model, 500 updates	0.069	0.039	0.031 [0.017, 0.040]	46
residual model, 2400 updates (main)	0.376	0.120	0.025 [-0.000, 0.045]	46
direct-output model, pretrained	0.003	0.902	-0.146 [-0.260, -0.046]	46
direct-output model, fine-tuned A	0.006	0.812	-0.372 [-0.406, -0.290]	46
direct-output model, fine-tuned B	0.006	0.840	-0.337 [-0.396, -0.252]	46

Appendix

Table G2: Vessel gradient norms with full physical credit (medians over 46 windows) and the statistic

D

with a

95\%

interval.

Policy Learning

Policy learning develops decision rules that choose actions or interventions to maximize outcomes from data, feedback, or interaction.

See all 479 papers

Aug 24, 2026

A Physical Response-and-Memory Model for Muon Optimization

Training large language models is costly. How low a loss the same compute can ultimately reach depends on how each step's gradient is converted into a weight update; the rule that performs this conversion is the optimizer. From SGD and AdamW to the recent Muon, effective update rules have mostly been shaped by engineering intuition and then selected on benchmarks. Muon semi-orthogonalizes the momentum matrix before applying the update and has kept breaking records on public training benchmarks; yet why the semi-orthogonalized direction works, and over how long a history the momentum should average, are two questions at present answered mainly by experience. Here we treat the weight matrix during training as a responsive medium with memory and build a physical model for it, in which both questions find answers: the semi-orthogonalized direction is the maximally dissipative response under an output-side safety budget, which explains why it works; momentum is the internal stress accumulated by the medium; how long it should average is set by the relaxation of this stress, and a real medium relaxes on more than one timescale, the simplest form being one fast and one slow. On this basis we propose the Bi-Maxwell optimizer. The framework further yields a testable consequence: gradient directions change fast early in training and more slowly later, so the optimal memory length should grow with training stage; step-by-step measurements of a proxy for it by a read-only probe across 8 independent training trajectories are consistent with this consequence. Replacing the memory kernel alone, from a single timescale to two, brings training to the target loss in noticeably fewer steps on a public large-language-model optimizer benchmark.

Yinze Hu, Hongjun Xiang, Xingao Gong +1

Key Laboratory of Computational Physical Sciences (Ministry of Education), Institute of Computational Physical Sciences, State Key Laboratory of Surface Physics, and Department of Physics, Fudan University, Shanghai 200433, China · School of Physics Science and Engineering, Tongji University, Shanghai 200092, China

Neural Network Optimization Muon Optimizer Neural Network Training Dynamics LLM Training

Ask PDF

Aug 25, 2026

Delayed Optimizer-State Transport Shapes Short-Horizon Training Decisions

Adaptive optimizers retain gradient history in moment variables, allowing a local change in loss weighting to alter later updates. We examine whether this delayed transport is large enough to change prospective short-horizon decisions. On committed future-minibatch sequences, we differentiate eight-step AdamW trajectories through the complete model--optimizer state and select exposure-matched Math--Code loss schedules before independent evaluation. Across 12 unused 0.3M Transformer histories, full transport lowers token-disjoint loss relative to an optimizer-aware immediate derivative in 10/12 histories (mean benefit

4.71\times10^{-4}

; exact one-sided sign test,

p=0.0193

). The two controllers act equally often but select different schedules in 60/96 windows. Crossed checkpoint--future-path tests attribute this reordering to the interaction between optimizer state and near-future data, while an independent Ising--CNN experiment shows that deleting moment-state transport destroys accurate response prediction. Full-transport scores also concentrate exact-rollout winners in larger candidate libraries, focusing finite-amplitude evaluation on a shortlist. On these committed short paths, optimizer memory and near-future data order are therefore actionable components of the training state, providing a mechanism-based criterion for when finite-horizon rather than one-step intervention is required.

Jinhui Guo

School of Physics, Beihang University, Beijing 100191, China

Neural Network Optimization Neural Network Training Dynamics Learning to Optimize Adaptive Loss Weighting

Ask PDF

Oct 2, 2026

Memory-State Critic for Asymmetric Actor-Critic with Application to Vision-Based Pursuit-Evasion

In partially observable Markov decision processes, the optimal policy generally depends on the history of observations and past actions. Asymmetric actor-critic methods have become popular to learn such policies when additional information, such as the true state of the environment, is available during training. The critic, which is not needed at execution, is given access to the state. A critic conditioned on the state alone is generally ill-defined and yields biased policy gradients. Conditioning on the state and the history, the history-state critic restores both. In this paper, we show that conditioning the critic on the state and the policy's own memory, i.e., the internal representation of the history through which the policy selects its actions, is already well-defined and gives unbiased policy gradients, removing the need for a second recurrent approximator of the history. We call it the memory state critic. It follows that a critic based on the policy's memory need not backpropagate its loss into that memory, even though the memory is a lossy encoding of the history. We evaluate the memory-state critic in a vision-based pursuit-evasion environment between two quadrotors across two arena types. The pursuer is the learning agent, and the evader is sampled per episode from a fixed pool of heuristic behaviours. The results show that the memory-state critic outperforms the history-state critic and converges faster. In addition to being unbiased compared to the state-only critic, it maintains a slight edge in the wall arena, where the actor's history carries information that the privileged state alone does not.

Arthur Louette, Alejandro Sánchez Roncero, Gaspard Lambrechts +4

University of Liège, Belgium · KTH Royal Institute of Technology, Sweden · McGill University & Mila Québec AI Institute, Canada +1

Asynchronous Advantage Actor-Critic Actor-Critic Methods Partially Observable RL Visuomotor Policy Learning

Ask PDF

Abstract

Policy Learning

Explore similar work

A Physical Response-and-Memory Model for Muon Optimization

Delayed Optimizer-State Transport Shapes Short-Horizon Training Decisions

Memory-State Critic for Asymmetric Actor-Critic with Application to Vision-Based Pursuit-Evasion