When Correct Memory Goes Wrong: Fuzzing Persistent Memory Use in LLM Agents

Yuqiao Meng; Luoxi Tang; Yingxue Zhang; Yuchen Yang; Zhaohan Xi

cs.DCSep 29, 2026

When Correct Memory Goes Wrong: Fuzzing Persistent Memory Use in LLM Agents

Authors: Yuqiao Meng, Luoxi Tang, Yingxue Zhang, Yuchen Yang, Zhaohan Xi

Organizations: Binghamton University, State University of New York, Binghamton, NY, USA · The Pennsylvania State University, University Park, PA, USA

Abstract

Persistent memory helps LLM agents carry information across long interactions, but correct memory can still be used incorrectly when queries change or memory states evolve. Existing work mainly studies memory content errors or evaluates fixed test cases, leaving memory-use failures hard to discover systematically. We formulate this issue as a fuzzing problem and categorize such failures into query-related and memory-state failures. We then develop U-Fuzz, which starts from memory checkpoints as test seeds, mutates queries or memory states under explicit mutation obligations, validates each mutant, and uses observed memory behavior to guide iterative testing while keeping failure labels outside the search. We evaluate U-Fuzz across several memory systems against diverse fuzzing baselines, and further test an output-only setting with API-based LLMs where memory retrieval is hidden. Across these settings, U-Fuzz consistently uncovers more confirmed memory-use failures, showing that its search remains effective across different memory architectures and even when only final responses are observable.

Figures & tables

Figure 1: Illustrations of two memory-use failures: (1) paraphrasing a query changes the evidence retrieved from the same memory state; (2) after a memory update, stale evidence is ranked above the current value.

Figure 2: Overview of U-Fuzz . U-Fuzz builds seeds and references from memory checkpoints, generates validated query and memory-state mutations, and executes each parent-mutant pair to detect memory-use failures. Retrieval coverage and divergence guide later fuzzing rounds, while failure labels remain outside the fuzzing search.

	Method	Mem0		A-Mem		Graphiti		MemOS
		UF@ $B$	Cov@ $B$	UF@ $B$	Cov@ $B$	UF@ $B$	Cov@ $B$	UF@ $B$	Cov@ $B$
LoCoMo	Random Mutation	12.7 $\pm 1.5$	0.40 $\pm.02$	21.7 $\pm 5.1$	0.24 $\pm.08$	18.7 $\pm 3.2$	0.27 $\pm.12$	15.0 $\pm 4.6$	0.34 $\pm.10$
	Unguided LLM	22.0 $\pm 2.0$	0.47 $\pm.03$	29.7 $\pm 3.8$	0.39 $\pm.08$	27.3 $\pm 4.0$	0.35 $\pm.10$	26.0 $\pm 3.0$	0.45 $\pm.12$
	LLM-as-Judge	23.3 $\pm 2.5$	0.41 $\pm.01$	35.7 $\pm 2.3$	0.38 $\pm.04$	23.3 $\pm 1.5$	0.34 $\pm.02$	25.3 $\pm 0.6$	0.45 $\pm.02$
	Coverage-Guided	27.7 $\pm 4.0$	0.59 $\pm.03$	31.3 $\pm 2.5$	0.54 $\pm.05$	30.3 $\pm 3.5$	0.47 $\pm.04$	30.0 $\pm 1.0$	0.65 $\pm.02$
	U-Fuzz-Q	33.0 $\pm 2.6$	0.57 $\pm.07$	57.0 $\pm 1.0$	0.51 $\pm.03$	39.7 $\pm 2.9$	0.42 $\pm.05$	41.3 $\pm 2.5$	0.57 $\pm.03$

Table 1: Performance under a fixed execution budget

B

(

B{=}8000

for LoCoMo and

B{=}4000

for LongMemEval-S). Results report mean and standard deviation over three runs. Appendix C also provides some case studies of found memory-use failures.

Figure 3: Fuzzing progress as the execution budget (x-axis) increases on LoCoMo. Each column corresponds to one memory system. Shaded regions indicate one standard deviation over three runs.

Figure 4: Component ablation on LoCoMo. Each bar reports the change in UF@

B

or Cov@

B

after removing one component from the full U-Fuzz . Mutation ablations remove one query- or memory-state operator, while Feedback ablations remove one search-feedback indicator. Error bars show one standard deviation over three runs.

Figure 5: Failure discovery under output-only observability on LoCoMo. Each bar reports UF@

B

B=2000

, wherein we prompt LLMs to output reasoning and responses. Error bars indicate one standard deviation over three runs.

Explore similar work

MemFail: Stress-Testing Failure Modes of LLM Memory Systems
May 26, 2026Ishir Garg, Neel Kolhe, Dawn Song +1Large Language Model Memory Large Language Models Fail
University of California, Berkeley
From Attack Success to Attack Severity: Counterfactual Memory Attacks on LLM Agents
Sep 28, 2026Mingxi Zou, Langzhang Liang, Zhuo Wang +3Large Language Model Agents Harms
Shanghai Academy of AI for Science (SAIS) · Fudan University · Monash University
Memory Provenance Laundering in LLM Agents: A Non-Amplification Firewall for Persistent Memory
Jul 31, 2026Jinghan Xu, Yiyong Xiao, Wanru Shao +2Semantic Memory Large Language Model Agents

May 26, 2026, cs.AIJ/K move · Enter open · S save

MemFail: Stress-Testing Failure Modes of LLM Memory Systems

Ishir Garg, Neel Kolhe, Dawn Song, Xuandong Zhao

University of California, Berkeley

Large Language Model Memory Large Language Models Fail Large Language Model Benchmarks Latent Failure Patterns Diagnostic Benchmark

Open paper AskPDF arXiv

Large language model (LLM) agents increasingly rely on external memory systems to remain consistent across long-horizon interactions, but little empirical work has been done to understand the specific failure modes and design choices that these systems present. Existing benchmarks report aggregate question-answering accuracy and treat memory systems as black boxes, making it impossible to attribute an incorrect answer to a particular failure mode of the system. We introduce MemFail, a diagnostic benchmark that isolates the failure modes of modern LLM memory systems. We begin by formalizing memory systems as the composition of three canonical operations -- summarization, storage, and retrieval -- and identify the potential failure modes induced by each. Based on these hypothesized failure modes, we construct five datasets spanning four tasks, each adversarially designed to test a specific operation of a memory system. Using these datasets, we evaluate four state-of-the-art memory systems on MemFail and demonstrate how MemFail can be used to empirically understand the tradeoffs induced by differences in memory system architectures.

Abstract

Explore similar work

MemFail: Stress-Testing Failure Modes of LLM Memory Systems

From Attack Success to Attack Severity: Counterfactual Memory Attacks on LLM Agents

Memory Provenance Laundering in LLM Agents: A Non-Amplification Firewall for Persistent Memory