The Price of Safety: Benign-Case Utility and Token Overhead of Memory-Poisoning Defenses in LLM Agents

Pritom Bhowmik

cs.CRSep 19, 2026

The Price of Safety: Benign-Case Utility and Token Overhead of Memory-Poisoning Defenses in LLM Agents

Authors: Pritom Bhowmik

Abstract

Memory-poisoning defenses for LLM agents are typically evaluated by how well they stop attacks, even though almost all traffic they encounter is benign. We measure their cost when no attack is present. Our harness holds the memory backend, retrieval, answer prompt, and judge constant, changing only the defense. We test four defenses on half of LoCoMo: five long conversations with 750 turns and 255 questions per run, repeated three times each. Replication proved necessary: one run showed a reranker cost on adversarial questions that the other two did not. The three write-time defenses (sanitization, provenance checking, and LLM-based anomaly detection) never fired on benign traffic. We detect no accuracy cost from them at a resolution of about $\pm$ 4 points; anomaly detection adds 1.2% token overhead. The read-time reranker, with 2.7% token overhead, fired on 9.9% of its memory decisions and withheld memory from 45% of questions. It lowered core accuracy by 4.4 points (95% CI [-9.0, -0.05]; McNemar p = 0.064). The effect is borderline in magnitude, but its direction held across all six runs that used the reranker. A blind audit found content-based justification for only 2 of 50 sampled drops. Stacking all four defenses raised token overhead to 4.1% but did not compound the accuracy cost. In our runs, cost tracked what each defense judged and how much context it used, rather than whether it used an LLM. Attack success rate reveals none of this, so benign cost should be reported alongside it. We release the harness, labels, and per-run outputs.

Figures & tables

Panel of Figure 1 — Figure 1: Measurement pipeline. The write path runs once per conversation turn; the read path runs once per question, so the reranker judges the same memory repeatedly and without its source turn. Each condition enables one defense; the stacked condition enables all four. Everything else is held fixed.

Replicate 1, used to write the detector
		Agrees with hand label
Answers labeled	$\boldsymbol{n}$	Regex	Detector
Regex said abstained (random sample)	38	38	38
Regex said failed (all)	20	12	20
Replicates 2 and 3, unseen by the detector
Regex said failed (all), first labels	31	0	23

Table 1: Adversarial answers labeled by hand, blind to condition, and how many each scorer agrees with. Rows group answers by the original regex’s call. Final scores use the hand label wherever one exists.

Test	Unit	Limit at our scale
Bootstrap CI	question	reflects question sampling only
Exact McNemar	question	14 to 24 discordant pairs
Exact sign-flip	conversation	$p$ cannot fall below 0.0625

Table 2: The three tests, their unit of analysis, and their limit at our scale.

Role	Model	Provider
Agent	gemini-3.1-flash-lite	Google
LLM defenses	gemini-3.1-flash-lite	Google
Embedding	text-embedding-3-small	OpenAI
Judge	claude-haiku-4-5-20251001	Anthropic

Table 3: Models used in the experiment. The agent performs memory extraction and answering; the LLM defenses are anomaly detection and the reranker. The judge’s provider differs from the backbone’s.

Defense	Stage	$\boldsymbol{\Delta_{d}}$	95% CI		$\boldsymbol{p_{\mathrm{McN}}}$	$\boldsymbol{p_{\mathrm{flip}}}$	Per replicate	$\boldsymbol{\tau_{d}}$	FQR
Sanitize	write	$+1.8$	$[-1.7$ ,	$+5.6]$	0.791	0.625	$+2.2,\ -4.0,\ +1.9$	0.0%	0.0%
Provenance	write	$+1.9$	$[-2.2$ ,	$+6.0]$	0.481	0.500	$-0.1,\ -3.2,\ +2.7$	0.0%	0.0%
Anomaly	write	$+3.1$	$[-0.9$ ,	$+7.1]$	0.359	0.250	$0.0,\ +0.4,\ +3.1$	1.2%	0.0%
Rerank	read	$-4.4$	$[-9.0$ ,	$-0.05]$	0.064	0.375	$-7.9,\ -10.7,\ -3.7$	2.7%	9.9%
Stacked	both	$-3.1$	$[-7.4$ ,	$+1.0]$	0.383	0.500	$-5.1,\ -6.7,\ -2.1$	4.1%	8.9%

Table 4: Benign-case results relative to the undefended baseline.

\Delta_{d}

is the paired change in core accuracy, in points (majority vote over three replicates, conversations weighted equally), with its stratified bootstrap 95% CI and exact McNemar and sign-flip

p

-values. Per replicate gives

\Delta_{d}

within each replicate.

\tau_{d}

is defense tokens over baseline tokens. FQR is quarantines per decision.

Figure 2: Accuracy change against token overhead. Bars are 95% bootstrap intervals. Sanitize and provenance both cost 0%; they are offset for visibility.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Conversation	Turns	Questions	Evidence kept	Used (core / adv.)
conv-26	419	199	86	75 (66 / 9)
conv-30	369	105	51	51 (38 / 13)
conv-41	663	193	32	32 (25 / 7)
conv-42	629	260	42	42 (33 / 9)
conv-43	680	242	55	55 (38 / 17)
Total	2,760	999	266	255 (200 / 55)

Appendix

Table 5: Question retention at 150-turn truncation. Turns and Questions are for the full conversation; Evidence kept counts questions whose evidence turns all fall within the first 150 turns.

Condition	Conv.	Rep.	Mem.	$\boldsymbol{n_{q}}$	Core	Adv.	FQR
None	conv-26	1	309	75	0.652	1.000	–
None	conv-26	2	292	75	0.697	1.000	–
None	conv-26	3	316	75	0.621	1.000	–
None	conv-30	1	282	51	0.684	0.923	–
None	conv-30	2	248	51	0.737	1.000	–
None	conv-30	3	253	51	0.711	1.000	–

Appendix

Table 6: Per-cell results, all 90 runs. Core is core accuracy within the run; Adv. is adversarial abstention with hand labels applied. Quar. counts quarantines, and FQR divides them by the decisions made in that cell (memories judged, for the reranker stage).

LLM Agent Safety

Large language model agent safety studies how to keep tool-using, persistent, and multi-agent systems aligned with user intent and authorized limits.

See all 322 papers

Jul 16, 2026

MemPoison: Uncovering Persistent Memory Threats and Structural Blind Spots in LLM Agents

Persistent external memory enhances agent continuity but introduces persistent security vulnerabilities: adversarial content can be injected via standard interaction channels, retained across turns, and later distort downstream behavior. To address this challenge, we propose MemPoison, a comprehensive benchmark and analysis framework featuring 1227 hand-validated cases across four attack types, three injection channels, and three representative memory substrates, evaluated on seven open-weight and three closed-weight model families. We introduce a three-tier taxonomy: (L1) direct single-record corruption, (L2) compositional multi-record corruption and (L3) context-triggered dormant corruption. Our evaluations reveal a distinct defense frontier: while baseline write-time defenses, such as consistency checks, substantially suppress direct L1 attacks, they fail to reliably suppress L2 and L3 attacks. Through mechanistic influence decomposition (MID), we demonstrate structural blind spots in write-time defenses, which admit seemingly benign records that later become harmful through joint retrieval composition or trigger-conditioned activation. Our findings advocate for shifting from static filtering to adaptive, context-sensitive memory defense strategies.

Jifeng Gao, Kang Xia, Yi Zhang +5

State Key Laboratory for Novel Software Technology, Nanjing University, China · NARI Group Corporation/State Grid Electric Power Research Institute, China

LLM Agent Memory LLM Agent Security AI Agent Security Benchmarks Agent Memory Poisoning

Ask PDF

Oct 9, 2026

Forgetting is Not Always Bad: A Neuro-Inspired Memory Repair Mechanism for Poisoned LLM Agents

Large language model (LLM) agents are highly vulnerable to memory-poisoning attacks. Existing defenses primarily rely on external modules for either static memory isolation or continuous online auditing, resulting in low memory efficiency and high computational overhead. To address these limitations, inspired by neuroscientific mechanisms that weak reactivation can destabilize memories and induce decay in the absence of reinforcement, we propose an agent-intrinsic training-free memory repair framework, DREAM (Dynamic Reactivation for Engram Attenuation in Memory), that enables selective functional forgetting of poisoned memories. Specifically, DREAM implements a three-stage pipeline: perturbation-induced implicit reactivation to reveal the structural fragility of malicious memories, adaptive anomaly diagnosis based on multi-dimensional activation patterns to detect topological anomalies, and a dynamic memory repair module that selectively suppresses harmful memories while reinforcing benign ones. Extensive experiments across diverse attackers and real-world tasks demonstrate that DREAM reduces attack success rates by over 95% against backdoor poisoning while preserving strong benign utility. DREAM also achieves competitive robustness against injection attacks. In terms of runtime and token consumption, DREAM achieves up to a 2.13

\times

speedup and a 37.7% reduction in token consumption compared with A-MemGuard. Furthermore, DREAM also achieves a task success rate of 92.96% on poisoned multi-agent systems.

Lei Liu, Yunji Liang, Xiaowen Zhang +4

Northwestern Polytechnical University · National Univerisity of Singapore

Agent Memory Poisoning Backdoor Defense in LLMs Memory-Augmented Language Model Agents Prompt Injection Attacks on AI Agents

Ask PDF

Jun 3, 2026

From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agents

Memory is a core component of AI agents, enabling them to accumulate knowledge across interactions and improve performance. However, persistent memory introduces the risk of memory poisoning, where a single adversarial memory write can exert long-term influence over agent behavior. We present a systematic study of memory poisoning in LLM-based agents. We identify four memory write channels and nine structural vulnerabilities in model capabilities, system prompt design, and agent system architecture that make these channels exploitable. Based on these vulnerabilities, we develop a taxonomy of six classes of memory poisoning attacks. Furthermore, we design MPBench -- a benchmark for evaluating memory poisoning attacks, and show that agents designed to write and retrieve memory more aggressively are more exploitable. We also show that existing prompt injection defenses fail to cover memory poisoning attacks. Our findings provide a foundation for understanding and mitigating memory poisoning attacks against AI agents.

Pritam Dash, Tongyu Ge, Aditi Jain +2

Huawei Canada · University of Waterloo, Canada · Work done during an internship at Huawei.

AI Agent Security Benchmarks Agent Memory Poisoning Memory-Augmented Language Model Agents

Ask PDF

Abstract

LLM Agent Safety

Explore similar work

MemPoison: Uncovering Persistent Memory Threats and Structural Blind Spots in LLM Agents

Forgetting is Not Always Bad: A Neuro-Inspired Memory Repair Mechanism for Poisoned LLM Agents

From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agents