When Does Experience Memory Help Closed-Loop LLM Driving Agents? A Controlled Evaluation with Resampled Warmup Pools, Token-Matched Fillers, and Audited Baselines

Yingxin Jia; Xuanyi Li

cs.AIOct 20, 2026

When Does Experience Memory Help Closed-Loop LLM Driving Agents? A Controlled Evaluation with Resampled Warmup Pools, Token-Matched Fillers, and Audited Baselines

Authors: Yingxin Jia, Xuanyi Li

Organizations: Independent Researchers

Abstract

When does experience memory help a closed-loop LLM driving agent? Published gains cannot say, because the two factors we find to matter most, which warmup experiences the pool contains and what any injected tokens do to the prompt, are controlled in none of them. We answer with a controlled evaluation on one audited HighwayEnv testbed (1Hz, gpt-4o-mini for the decisive contrasts). We treat the warmup pool as a resampled random factor (ten disjoint pools) and add token-matched irrelevant-filler controls, oracle-certified diagnostics, caution-pricing metrics, and quantified audits of the baseline ports. The probe suite spans flat-text retrieval (a DiLu port), a single-frame grid port (RESPOND-inspired), our episodic temporal-graph memory, a retrieval-free consolidated rulebook, and a flat K-step observation-history baseline. Four findings follow, each a property of this testbed. (1) Headline memory gains belong to the warmup draw. The reported pool is the best of ten for both retrieval memories, which beat No-Memory in only one pool of ten (sign test p=0.011), and even the rulebook's apparent pool-immunity collapsed on five fresh draws. (2) No retrieval design tested exceeds token-matched filler. The collision-only merge ceiling is reached by a single 15-token caution instruction, and metrics that price caution erase the rulebook's lead. (3) Representational sophistication is nowhere robustly operative. (4) Baseline port quality silently inverts conclusions. On this testbed, a single-pool comparison without filler controls would have reported memory gains that these controls reverse. We release the testbed, audited ports, seeds, and caches.

Figures & tables

Figure 1: Overview. (a) Memory designs from our probe suite and from the literature drive one frozen planner in closed loop, and only the memory system varies. (b) The controls we introduce: the warmup pool resampled as a random factor, token-matched irrelevant filler, and pre-registered diagnostics with scripted oracle certificates. (c) Swapping one

8

-episode warmup pool flips the flat memory from aggressive (all hazard episodes crash) to timid (all time out) on the history-necessity diagnostic. (d) The early- SLOWER share tracks merge safety. The retrieval-free rulebook turns this dial hardest, yet five fresh pools swing it from

0.03

0.98

Finding	Confirmatory evidence	Test; location
M1	Ten disjoint $8$ -episode pools, $n\!=\!100$ each: DiLu port and TG-Mem above No-Memory in $1/10$ pools, below in $5/10$ ; pool-averaged effect $-0.20$ [ $-0.29,-0.10$ ] / [ $-0.30,-0.09$ ]. Rulebook $0.97$ – $0.98$ on A–E, $0.03$ – $0.98$ on pre-registered F–J.	Exact sign test ( $p\!=\!0.011$ ); per-pool tests, Holm; bootstrap over pools; Fig. 3 , Tab. S45, S51, S52
M2	Padding filler $0.800$ vs. TG-Mem $0.660$ / DiLu port $0.670$ ( $n\!=\!100$ paired); caution line $0.980$ above every memory and filler; always- SLOWER certificate $0.980$ (no LLM); rulebook SaS $0.00$ .	McNemar ( $p\!=\!0.029$ , $0.024$ ; $\leq\!4\times 10^{-9}$ ); Tab. 2 , S47, S50
M3	DiLu port vs. TG-Mem indistinguishable ( $n\!=\!100$ paired); every arm $0/20$ hazard successes on the oracle-solvable telegraph-v0 ( $n\!=\!40$ , pre-registered); $-$ temporal edges $0.55$ vs. $0.66$ ( $n\!=\!40$ , pool A).	McNemar ( $p\!=\!1.0$ ; $0.07$ ); Tab. 2 , S30
M4	Crippled DiLu port $0.375$ vs. audited port $0.670$ ; grid port fails two of three reproduction audits ( $51\%$ decisions, $49\%$ cells reproduced).	Port audits; Tab. S11

Table 1: Evidence map: each finding, the confirmatory result behind it, and the test used (

n\!=\!100

paired seeds on merge,

n\!=\!40

on diagnostics;

n\!\leq\!20

cells are exploratory).

Figure 2: TG-Mem in one view. (a) An episode as a typed temporal graph, with spatial edges within a frame, temporal

t\!\to\!t{+}1

edges across frames, and the action and outcome stored as metadata (model-inferred causal edges are disclosed as such). (b) The perceive

\to

encode

\to

retrieve

\to

plan

\to

reflect loop over the graph store.

Method	Coll.-free $\uparrow$	SaS $\uparrow$	Speed	Steps	Tok./dec. $\downarrow$
No-Memory	0.510	0.40	24.2	12.7	210
DiLu port	0.670	0.67	25.6	12.1	274
Grid port (RESPOND-inspired)	0.140	0.14	29.0	0 7.4	251
TG-Mem (ours)	0.660	0.45	23.2	13.6	1,038
Flat $K$ -step history (ours)	0.750	0.31	22.5	14.7	527
Irrelevant-padding control (ours)	0.800	0.29	22.3	15.4	1,037

Table 2: Closed-loop merge-v0 (

n\!=\!100

/method, one shared seed set, gpt-4o-mini , audited v2 ports, warmup pool A). Paired McNemar tests, Holm-adjusted over the four warmup arms: grid port below everything (

p_{\text{holm}}\!\leq\!3\times 10^{-9}

), TG-Mem vs. No-Memory

p_{\text{holm}}\!=\!0.041

, DiLu port vs. No-Memory

0.073

, DiLu port vs. TG-Mem

1.0

. Pool A is the best of ten draws, and under B–J the memory rows collapse to

0.08

–

0.48

(Figure 3 ). SaS is the pre-registered safe-and-swift composite (§ 4 ), and speed is in m/s from certified replays. History and padding rows are warmup-free controls outside the Holm family (raw paired

p

: padding above TG-Mem

0.029

, above DiLu port

0.024

). Lower block: TG-Mem ablation on the same pool (

n\!=\!40

shared seeds, full TG-Mem

0.700

, paired McNemar vs. full and unpaired Fisher,

{}^{\ast}p\!<\!0.05

, pool-conditional, § 5.5 ). Tokens count every LLM call.

Figure 3: The warmup-pool lottery on merge-v0 over all ten disjoint

8

-episode pools (

n\!=\!100

per cell, Wilson

95\%

intervals) against the pool-free No-Memory arm (dashed), the band spanned by the four token-matched fillers, and the always- SLOWER and caution-line ceiling (dotted). Both retrieval memories exceed No-Memory only under pool A. Open markers re-warm pools A and B on five times the budget. The retrieval-free rulebook (stars) looked pool-immune on the five original pools and fell to

0.03

–

0.78

on three of the five pre-registered fresh pools.

Memory-Augmented Language Model Agents

Memory-augmented language model agents use persistent facts, interaction histories, and skills to support personalization and long-horizon tasks.

See all 444 papers

From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory
Jun 7, 2026Yishuo Cai, Xingyu Guo, Xuancheng Huang +8Continual Learning for LLM Agents LLM Agent Memory
Peking University, Beijing, China · Central South University, Changsha, China · Zhipu AI, Beijing, China +3
Towards Autonomous Memory Agents
Feb 25, 2026Xinle Wu, Rui Zhang, Mustafa Anis Hussain +1Memory-Augmented Language Model Agents Memory Management for LLM Agents
National University of Singapore, Singapore
Useful Memories Become Faulty When Continuously Updated by LLMs
May 13, 2026Dylan Zhang, Yanshan Lin, Zhengkun Wu +4AI Agent Reliability Persistent Memory for Language Models
University of Illinois Urbana-Champaign · IIIS, Tsinghua University

Jun 7, 2026J/K move · Enter open · S save

From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory

Yishuo Cai, Xingyu Guo, Xuancheng Huang, Jinhua Du+7

Peking University, Beijing, China · Central South University, Changsha, China · Zhipu AI, Beijing, China+3

Continual Learning for LLM Agents LLM Agent Memory Test-Time RL Test-Time Training RL for LLM Agents

Open paper AskPDF arXiv

Large language model (LLM) agents are increasingly deployed in long-running settings where improving through experience at test time becomes important. A common approach is to update an explicit memory after each interaction to guide future decisions. However, most existing methods rely on hand-designed prompting rules, making it difficult to align memory updates with downstream objectives over multi-step horizons consistently. We propose MemoPilot, a plug-in memory copilot that explicitly trains the memory update process to improve a frozen LLM's performance across sequential interactions. We formulate memory updating as a multi-turn decision problem and optimize it end-to-end with multi-turn GRPO. Our training recipe introduces (i) a turn-wise reward signal and (ii) a context-independent, turn-level advantage estimation across rollouts, enabling finer-grained credit assignment and more stable training in multi-turn settings. We evaluate MemoPilot on two testbeds: multi-round Rock-Paper-Scissors (RPS) and Limit Texas Hold'em (LHE). Across both environments, MemoPilot substantially improves test-time learning of a frozen player over strong baselines, ranking first in Elo ratings on both games (1762 on LHE and 1590 on RPS) and outperforming all baseline memory methods and proprietary models, including DeepSeek-V3.2.

Abstract

Memory-Augmented Language Model Agents

Explore similar work

From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory

Towards Autonomous Memory Agents

Useful Memories Become Faulty When Continuously Updated by LLMs