When Does Experience Memory Help Closed-Loop LLM Driving Agents? A Controlled Evaluation with Resampled Warmup Pools, Token-Matched Fillers, and Audited Baselines
When does experience memory help a closed-loop LLM driving agent? Published gains cannot say, because the two factors we find to matter most, which warmup experiences the pool contains and what any injected tokens do to the prompt, are controlled in none of them. We answer with a controlled evaluation on one audited HighwayEnv testbed (1Hz, gpt-4o-mini for the decisive contrasts). We treat the warmup pool as a resampled random factor (ten disjoint pools) and add token-matched irrelevant-filler controls, oracle-certified diagnostics, caution-pricing metrics, and quantified audits of the baseline ports. The probe suite spans flat-text retrieval (a DiLu port), a single-frame grid port (RESPOND-inspired), our episodic temporal-graph memory, a retrieval-free consolidated rulebook, and a flat K-step observation-history baseline. Four findings follow, each a property of this testbed. (1) Headline memory gains belong to the warmup draw. The reported pool is the best of ten for both retrieval memories, which beat No-Memory in only one pool of ten (sign test p=0.011), and even the rulebook's apparent pool-immunity collapsed on five fresh draws. (2) No retrieval design tested exceeds token-matched filler. The collision-only merge ceiling is reached by a single 15-token caution instruction, and metrics that price caution erase the rulebook's lead. (3) Representational sophistication is nowhere robustly operative. (4) Baseline port quality silently inverts conclusions. On this testbed, a single-pool comparison without filler controls would have reported memory gains that these controls reverse. We release the testbed, audited ports, seeds, and caches.
Figures & tables
Figure 1: Overview. (a) Memory designs from our probe suite and from the literature drive one frozen planner in closed loop, and only the memory system varies. (b) The controls we introduce: the warmup pool resampled as a random factor, token-matched irrelevant filler, and pre-registered diagnostics with scripted oracle certificates. (c) Swapping one 8 -episode warmup pool flips the flat memory from aggressive (all hazard episodes crash) to timid (all time out) on the history-necessity diagnostic. (d) The early- SLOWER share tracks merge safety. The retrieval-free rulebook turns this dial hardest, yet five fresh pools swing it from 0.03 to 0.98 .
Finding
Confirmatory evidence
Test; location
M1
Ten disjoint 8 -episode pools, n=100 each: DiLu port and TG-Mem above No-Memory in 1/10 pools, below in 5/10 ; pool-averaged effect −0.20 [ −0.29,−0.10 ] / [ −0.30,−0.09 ]. Rulebook 0.97 – 0.98 on A–E, 0.03 – 0.98 on pre-registered F–J.
Exact sign test ( p=0.011 ); per-pool tests, Holm; bootstrap over pools; Fig. 3 , Tab. S45, S51, S52
M2
Padding filler 0.800 vs. TG-Mem 0.660 / DiLu port 0.670 ( n=100 paired); caution line 0.980 above every memory and filler; always- SLOWER certificate 0.980 (no LLM); rulebook SaS 0.00 .
DiLu port vs. TG-Mem indistinguishable ( n=100 paired); every arm 0/20 hazard successes on the oracle-solvable telegraph-v0 ( n=40 , pre-registered); − temporal edges 0.55 vs. 0.66 ( n=40 , pool A).
McNemar ( p=1.0 ; 0.07 ); Tab. 2 , S30
M4
Crippled DiLu port 0.375 vs. audited port 0.670 ; grid port fails two of three reproduction audits ( 51% decisions, 49% cells reproduced).
Port audits; Tab. S11
Table 1: Evidence map: each finding, the confirmatory result behind it, and the test used ( n=100 paired seeds on merge, n=40 on diagnostics; n≤20 cells are exploratory).
Figure 2: TG-Mem in one view. (a) An episode as a typed temporal graph, with spatial edges within a frame, temporal t→t+1 edges across frames, and the action and outcome stored as metadata (model-inferred causal edges are disclosed as such). (b) The perceive → encode → retrieve → plan → reflect loop over the graph store.
Method
Coll.-free ↑
SaS ↑
Speed
Steps
Tok./dec. ↓
No-Memory
0.510
0.40
24.2
12.7
210
DiLu port
0.670
0.67
25.6
12.1
274
Grid port (RESPOND-inspired)
0.140
0.14
29.0
0 7.4
251
TG-Mem (ours)
0.660
0.45
23.2
13.6
1,038
Flat K -step history (ours)
0.750
0.31
22.5
14.7
527
Irrelevant-padding control (ours)
0.800
0.29
22.3
15.4
1,037
Table 2: Closed-loop merge-v0 ( n=100 /method, one shared seed set, gpt-4o-mini , audited v2 ports, warmup pool A). Paired McNemar tests, Holm-adjusted over the four warmup arms: grid port below everything ( pholm≤3×10−9 ), TG-Mem vs. No-Memory pholm=0.041 , DiLu port vs. No-Memory 0.073 , DiLu port vs. TG-Mem 1.0 . Pool A is the best of ten draws, and under B–J the memory rows collapse to 0.08 – 0.48 (Figure 3 ). SaS is the pre-registered safe-and-swift composite (§ 4 ), and speed is in m/s from certified replays. History and padding rows are warmup-free controls outside the Holm family (raw paired p : padding above TG-Mem 0.029 , above DiLu port 0.024 ). Lower block: TG-Mem ablation on the same pool ( n=40 shared seeds, full TG-Mem 0.700 , paired McNemar vs. full and unpaired Fisher, ∗p<0.05 , pool-conditional, § 5.5 ). Tokens count every LLM call.
Figure 3: The warmup-pool lottery on merge-v0 over all ten disjoint 8 -episode pools ( n=100 per cell, Wilson 95% intervals) against the pool-free No-Memory arm (dashed), the band spanned by the four token-matched fillers, and the always- SLOWER and caution-line ceiling (dotted). Both retrieval memories exceed No-Memory only under pool A. Open markers re-warm pools A and B on five times the budget. The retrieval-free rulebook (stars) looked pool-immune on the five original pools and fell to 0.03 – 0.78 on three of the five pre-registered fresh pools.
Large language model (LLM) agents are increasingly deployed in long-running settings where improving through experience at test time becomes important. A common approach is to update an explicit memory after each interaction to guide future decisions. However, most existing methods rely on hand-designed prompting rules, making it difficult to align memory updates with downstream objectives over multi-step horizons consistently. We propose MemoPilot, a plug-in memory copilot that explicitly trains the memory update process to improve a frozen LLM's performance across sequential interactions. We formulate memory updating as a multi-turn decision problem and optimize it end-to-end with multi-turn GRPO. Our training recipe introduces (i) a turn-wise reward signal and (ii) a context-independent, turn-level advantage estimation across rollouts, enabling finer-grained credit assignment and more stable training in multi-turn settings. We evaluate MemoPilot on two testbeds: multi-round Rock-Paper-Scissors (RPS) and Limit Texas Hold'em (LHE). Across both environments, MemoPilot substantially improves test-time learning of a frozen player over strong baselines, ranking first in Elo ratings on both games (1762 on LHE and 1590 on RPS) and outperforming all baseline memory methods and proprietary models, including DeepSeek-V3.2.
Yishuo Cai, Xingyu Guo, Xuancheng Huang +8
Peking University, Beijing, China · Central South University, Changsha, China · Zhipu AI, Beijing, China +3
Recent memory agents improve LLMs by extracting experiences and conversation history into an external storage. This enables low-overhead context assembly and online memory update without expensive LLM training. However, existing solutions remain passive and reactive; memory growth is bounded by information that happens to be available, while memory agents seldom seek external inputs in uncertainties. We propose autonomous memory agents that actively acquire, validate, and curate knowledge at a minimum cost. U-Mem materializes this idea via (i) a cost-aware knowledge-extraction cascade that escalates from cheap self/teacher signals to tool-verified research and, only when needed, expert feedback, and (ii) semantic-aware Thompson sampling to balance exploration and exploitation over memories and mitigate cold-start bias. On both verifiable and non-verifiable benchmarks, U-Mem consistently beats prior memory baselines and can surpass RL-based optimization, improving HotpotQA (Qwen2.5-7B) by 14.6 points and AIME25 (Gemini-2.5-flash) by 7.33 points.
Learning from past experience benefits from two complementary forms of memory: episodic traces -- raw trajectories of what happened -- and consolidated abstractions distilled across many episodes into reusable, schema-like lessons. Recent agentic-memory systems pursue the consolidated form: an LLM rewrites past trajectories into a textual memory bank that it continuously updates with new interactions, promising self-improving agents without parameter updates. Yet we find that such consolidated memories produced by today's LLMs are often faulty even when derived from useful experiences. As consolidation proceeds, memory utility first rises, then degrades, and can fall below the no-memory baseline. More surprisingly, even when consolidating from ground-truth solutions, GPT-5.4 fails on 54% of a set of ARC-AGI problems it had previously solved without memory. We trace the regression to the consolidation step rather than the underlying experience: the same trajectories yield qualitatively different memories under different update schedules, and an episodic-only control that simply retains those trajectories remains competitive with the consolidators we test. In a controlled ARC-AGI Stream environment that exposes Retain, Delete, and Consolidate actions, agents preserve raw episodes by default and double the accuracy of their forced-consolidation counterparts; disabling consolidation entirely (episodic management only) matches this auto regime. Practically, robust agent memory should treat raw episodes as first-class evidence and gate consolidation explicitly rather than firing it after every interaction. Looking forward, reliable agentic memory will require LLMs that can consolidate without overwriting the evidence they depend on.
Dylan Zhang, Yanshan Lin, Zhengkun Wu +4
University of Illinois Urbana-Champaign · IIIS, Tsinghua University