Memory or a Clock? Auditing a Memory Advantage in XLand-MiniGrid with Seeds, Maps and Evaluation Noise Apart

Rodrigo Reveron-Fonseca

cs.LGOct 24, 2026

Memory or a Clock? Auditing a Memory Advantage in XLand-MiniGrid with Seeds, Maps and Evaluation Noise Apart

Authors: Rodrigo Reveron-Fonseca

Organizations: Tecnológico de Monterrey

Abstract

On a partially observable XLand-MiniGrid task, an LSTM agent beats a memory-free MLP by 25 percentage points. The gap looks like a clear case of memory. We ask two questions about it. First, how much of a measured gap is noise? Separating training seed, map and evaluation randomness in 60 runs, each re-evaluated three times, we find that about ten paired seeds detect a 10-point gap in final success. The binary "task acquired" verdict, a common summary of learning curves, is 76% evaluation noise and would need about a hundred seeds for the same gap. Second, is the gap memory? On a sealed panel of new maps, opened once, a memory-free MLP of the LSTM's size comes within 3.3 points of the LSTM when it is given the time step, and gains no measurable benefit from its width without it. With the time step, an MLP a quarter of the LSTM's size, and the LSTM with its state erased at every step, also come within about 5 points. A linear probe reads the time step from the trained LSTM's state. On this task, most of the apparent memory advantage is an advantage of knowing the time, which matters because episodes are cut at a fixed limit and the reward shrinks with time. We recommend giving memory-free baselines the time step, and we release the protocol, the event-driven tooling and every checkpoint.

Figures & tables

	LSTM	MLP	LSTM $-$ MLP (paired)
Trainable parameters	11,367	2,919
Final test success (mean)	84.0 [80.8, 86.9]	58.9 [55.9, 61.7]	$+25.1$ [20.8, 29.5]
Final test success (IQM)	84.9	59.2	$+24.3$
Final test success, re-evaluated	83.9 [81.0, 86.7]	58.7 [56.5, 61.0]
Validation AUC	44.6 [43.0, 46.1]	38.8 [37.9, 39.7]	$+5.8$ [4.0, 7.6]
Seeds acquiring B	26 / 30	0 / 30

Table 1: Part I summary over 30 paired seeds. Final test success is one evaluation on 64 test maps; “re-evaluated” averages three further evaluations with new sampling seeds. Brackets are 95% bootstrap intervals over seeds; IQM is the interquartile mean ( Agarwal et al., 2021 ) .

Component	LSTM	MLP	LSTM $-$ MLP
$\sigma^{2}_{\text{seed}}$	0.0059 [0.0030, 0.0087]	0.0032 [0.0012, 0.0053]	0.0096 [0.0043, 0.0151]
$\sigma^{2}_{\text{map}}$	0.0346 [0.0275, 0.0469]	0.0911 [0.0820, 0.1056]	0.0368 [0.0332, 0.0486]
$\sigma^{2}_{\text{seed}\times\text{map}}$	0.0174 [0.0104, 0.0210]	0.0184 [0.0090, 0.0231]	0.0362 [0.0191, 0.0461]
$\sigma^{2}_{\text{noise}}$	0.0778 [0.0661, 0.0906]	0.1313 [0.1210, 0.1413]	0.2111 [0.1918, 0.2332]
Noise share of one evaluation	16%	37%	25%

Table 2: Variance components of final test success per episode ( Equation 1 ;

S=30

M=64

R=3

), with 95% seed-bootstrap intervals.

Figure 3: Composition of the between-seed variance of a run’s score from one evaluation, for each metric. Grey is the part due only to the randomness of evaluation. The MLP never acquires task B, so its acquisition verdict has no variance and is omitted.

Figure 4: Seeds needed for a paired two-sided test (

\alpha=0.05

, power 0.8) to detect a true gap between LSTM and MLP, with one evaluation per run, by metric. Dotted lines mark common seed counts. For the acquisition verdict the gap is a difference in the fraction of seeds that acquire the task.

	$\delta=5$ pp		$\delta=10$ pp		$\delta=20$ pp		$\pm 5$ pp interval
Metric	$n_{r}=1$	$n_{r}=3$	$n_{r}=1$	$n_{r}=3$	$n_{r}=1$	$n_{r}=3$	$n_{r}=1$	$n_{r}=3$
Final test success	43	36	11	9	3	3	21	18
Validation AUC	9	8	3	2	2	2	5	4
Acquired (yes/no)	369	183	93	46	24	12	181	90

Table 3: Seeds needed for the paired LSTM

-

MLP comparison on the fixed test panel, with

n_{r}

evaluations per run. Left: to detect a true gap

\delta

(

\alpha=0.05

, power 0.8). Right: for a 95% interval of half-width 5 pp around the mean gap.

Arm	Params	Clock	lr	Seeds	Sealed (%)	Acquired	AUC
LSTM	11,367	no	$10^{-3}$	20	95.0 [93.7, 96.3]	20/20 (175k)	0.60
MLP-wide + clock	11,399	yes	$10^{-3}$	20	91.7 [89.7, 93.5]	20/20 (180k)	0.62
stack4 + clock	11,381	yes	$3\cdot 10^{-4}$	10	74.2 [69.3, 78.5]	5/10 (270k)	0.38
LSTM-reset	11,367	no	$10^{-3}$	20	67.0 [65.1, 68.8]	2/20 (240k)	0.45
MLP-wide, no clock †	11,401	no	$10^{-3}$	10	65.1 [62.6, 67.6]	0/10	0.44
MLP (small)	2,919	no	$10^{-3}$	10	61.9 [59.1, 64.6]	0/10	0.46

Table 4: Part II arms. Sealed: success of the validation-selected checkpoint on 64 new maps

\times

4 sampling seeds, mean over seeds with a 95% bootstrap interval. Acquired: seeds meeting the acquisition rule within 300,000 interactions, with the median step among them. AUC: normalised area under the validation curve.

	Contrast	Mean	BCa 95%	$p$	Holm	Ahead	Verdict
P1	LSTM $-$ MLP-wide + clock	$+3.3$	[0.9, 6.0]	0.021	0.041	12/20	not supported ∗
P2	LSTM $-$ LSTM-reset	$+28.0$	[25.3, 29.9]	$2\cdot 10^{-6}$	$6\cdot 10^{-6}$	20/20	supported
H3	LSTM $+$ MLP-wide $-$ 2 stack4	$+39.2$	[32.5, 47.9]	0.002	0.010	10/10	supported
E1	clock: MLP-wide $\pm$ clock	$+26.6$	[21.5, 30.5]	0.002		10/10
E2	width: MLP-wide, no clock $-$ MLP	$+3.2$	[ $-0.4$ , 5.8]	0.092		9/10
E3	LSTM $-$ MLP (small)	$+34.1$	[31.6, 38.2]	0.002		10/10

Table 5: Contrasts on the sealed panel (percentage points).

p

: exact two-sided sign-flip test; Holm: adjusted within the family (

m=3

principal,

m=5

secondary). Ahead: seeds on which the contrast is positive. The exploratory family has no correction and no verdict.

Figure 7: From the small MLP to the LSTM, one ingredient at a time, on the 10 seeds that every arm shares (sealed panel). Width alone adds little; telling a memory-free network the time closes most of the gap.

	Contrast	Mean	BCa 95%	$p$	Ahead
M1	LSTM-reset + clock $-$ LSTM-reset	$+24.1$	[21.6, 27.0]	0.002	10/10
M2	LSTM $-$ LSTM-reset + clock	$+4.5$	[2.5, 6.9]	0.002	10/10
M3	MLP + clock $-$ MLP	$+28.8$	[24.0, 31.6]	0.002	10/10
M4	MLP-wide + clock $-$ MLP + clock	$+1.0$	[ $-2.7$ , 5.1]	0.66	6/10
M5	LSTM $-$ MLP + clock	$+5.3$	[2.6, 8.6]	0.012	8/10

Table 6: Mechanism contrasts on the sealed panel (percentage points; exploratory, 10 paired seeds, exact sign-flip

p

, no correction).

Features	Trained state	Initial-weights state	Observation	Shuffled labels
LSTM	0.84	0.52	$-0.19$	$-0.10$
LSTM-reset	0.04	$-0.07$	$-0.06$	$-0.03$

Table 7: Linear probe for the episode step

t

(out-of-fold

R^{2}

, folds split by map; mean over 20 seeds per arm). Episodes last 9.5 steps on average because successful ones end early.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents
Jul 15, 2026Eric Hanchen Jiang, Zhi Zhang, Yuchen Wu +11Persistent Memory for Language Models Memory Management for LLM Agents
University of California Los Angeles · University of Washington · Northwestern University
DolphinBench: Mapping the Pareto Frontier of Agent Memory
Sep 21, 2026Soumil Rathi, Deshraj Yadav, Taranjeet SinghAgent Memory Persistent Memory for Language Models
Mem0
MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks
Feb 18, 2026Zexue He, Yu Wang, Churan Zhi +11Agent Memory LLM Agent Memory
Stanford University · UCSD · UIUC +3

Jul 15, 2026J/K move · Enter open · S save

Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents

Eric Hanchen Jiang, Zhi Zhang, Yuchen Wu, Levina Li+10

University of California Los Angeles · University of Washington · Northwestern University

Persistent Memory for Language Models Memory Management for LLM Agents Agent Memory Management Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Large Language Model (LLM) agents increasingly rely on external memory systems to accumulate experience across tasks. Yet nearly all existing approaches, from graph-structured memories to reflective insight stores, access memory through fixed, hand-designed heuristics. We argue that this static view of memory is a core bottleneck for agentic learning because optimal memory behavior is fundamentally context-dependent. The early stages of the tasks, benefit from minimal retrieval because memory is sparse; recurring goal types benefit from plan reuse rather than generic nearest-neighbor lookup; stuck agents benefit from re-retrieval with alternative queries; and across long task streams, the memory store itself must be consolidated and pruned to remain useful. We present Memory as a Controlled Process (MemCon), a framework that models memory operations as a Markov Decision Process and learns an online policy that adaptively decides when, what, and how much to retrieve, when to inject a distilled plan, and when to consolidate or forget. MemCon is backend-agnostic: it wraps any existing memory implementation, learns from task-by-task binary feedback with no pretraining and no additional LLM calls, and uses a lightweight tabular contextual bandit with UCB exploration that converges within tens of tasks. Across 6 benchmarks, 3 agent frameworks, and 3 LLM backbones, MemCon consistently outperforms multiple memory baselines by up to 15.2 points in task success while reducing token consumption by 5--20%.

Abstract

Reinforcement Learning

Explore similar work

Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents

DolphinBench: Mapping the Pareto Frontier of Agent Memory

MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks