The Price of Safety: Benign-Case Utility and Token Overhead of Memory-Poisoning Defenses in LLM Agents

Pritom Bhowmik

cs.CRSep 19, 2026

The Price of Safety: Benign-Case Utility and Token Overhead of Memory-Poisoning Defenses in LLM Agents

Authors: Pritom Bhowmik

Abstract

Memory-poisoning defenses for LLM agents are typically evaluated by how well they stop attacks, even though almost all traffic they encounter is benign. We measure their cost when no attack is present. Our harness holds the memory backend, retrieval, answer prompt, and judge constant, changing only the defense. We test four defenses on half of LoCoMo: five long conversations with 750 turns and 255 questions per run, repeated three times each. Replication proved necessary: one run showed a reranker cost on adversarial questions that the other two did not. The three write-time defenses (sanitization, provenance checking, and LLM-based anomaly detection) never fired on benign traffic. We detect no accuracy cost from them at a resolution of about $\pm$ 4 points; anomaly detection adds 1.2% token overhead. The read-time reranker, with 2.7% token overhead, fired on 9.9% of its memory decisions and withheld memory from 45% of questions. It lowered core accuracy by 4.4 points (95% CI [-9.0, -0.05]; McNemar p = 0.064). The effect is borderline in magnitude, but its direction held across all six runs that used the reranker. A blind audit found content-based justification for only 2 of 50 sampled drops. Stacking all four defenses raised token overhead to 4.1% but did not compound the accuracy cost. In our runs, cost tracked what each defense judged and how much context it used, rather than whether it used an LLM. Attack success rate reveals none of this, so benign cost should be reported alongside it. We release the harness, labels, and per-run outputs.

Figures & tables

Panel of Figure 1 — Figure 1: Measurement pipeline. The write path runs once per conversation turn; the read path runs once per question, so the reranker judges the same memory repeatedly and without its source turn. Each condition enables one defense; the stacked condition enables all four. Everything else is held fixed.

Replicate 1, used to write the detector
		Agrees with hand label
Answers labeled	$\boldsymbol{n}$	Regex	Detector
Regex said abstained (random sample)	38	38	38
Regex said failed (all)	20	12	20
Replicates 2 and 3, unseen by the detector
Regex said failed (all), first labels	31	0	23

Table 1: Adversarial answers labeled by hand, blind to condition, and how many each scorer agrees with. Rows group answers by the original regex’s call. Final scores use the hand label wherever one exists.

Test	Unit	Limit at our scale
Bootstrap CI	question	reflects question sampling only
Exact McNemar	question	14 to 24 discordant pairs
Exact sign-flip	conversation	$p$ cannot fall below 0.0625

Table 2: The three tests, their unit of analysis, and their limit at our scale.

Role	Model	Provider
Agent	gemini-3.1-flash-lite	Google
LLM defenses	gemini-3.1-flash-lite	Google
Embedding	text-embedding-3-small	OpenAI
Judge	claude-haiku-4-5-20251001	Anthropic

Table 3: Models used in the experiment. The agent performs memory extraction and answering; the LLM defenses are anomaly detection and the reranker. The judge’s provider differs from the backbone’s.

Defense	Stage	$\boldsymbol{\Delta_{d}}$	95% CI		$\boldsymbol{p_{\mathrm{McN}}}$	$\boldsymbol{p_{\mathrm{flip}}}$	Per replicate	$\boldsymbol{\tau_{d}}$	FQR
Sanitize	write	$+1.8$	$[-1.7$ ,	$+5.6]$	0.791	0.625	$+2.2,\ -4.0,\ +1.9$	0.0%	0.0%
Provenance	write	$+1.9$	$[-2.2$ ,	$+6.0]$	0.481	0.500	$-0.1,\ -3.2,\ +2.7$	0.0%	0.0%
Anomaly	write	$+3.1$	$[-0.9$ ,	$+7.1]$	0.359	0.250	$0.0,\ +0.4,\ +3.1$	1.2%	0.0%
Rerank	read	$-4.4$	$[-9.0$ ,	$-0.05]$	0.064	0.375	$-7.9,\ -10.7,\ -3.7$	2.7%	9.9%
Stacked	both	$-3.1$	$[-7.4$ ,	$+1.0]$	0.383	0.500	$-5.1,\ -6.7,\ -2.1$	4.1%	8.9%

Table 4: Benign-case results relative to the undefended baseline.

\Delta_{d}

is the paired change in core accuracy, in points (majority vote over three replicates, conversations weighted equally), with its stratified bootstrap 95% CI and exact McNemar and sign-flip

p

-values. Per replicate gives

\Delta_{d}

within each replicate.

\tau_{d}

is defense tokens over baseline tokens. FQR is quarantines per decision.

Figure 2: Accuracy change against token overhead. Bars are 95% bootstrap intervals. Sanitize and provenance both cost 0%; they are offset for visibility.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Conversation	Turns	Questions	Evidence kept	Used (core / adv.)
conv-26	419	199	86	75 (66 / 9)
conv-30	369	105	51	51 (38 / 13)
conv-41	663	193	32	32 (25 / 7)
conv-42	629	260	42	42 (33 / 9)
conv-43	680	242	55	55 (38 / 17)
Total	2,760	999	266	255 (200 / 55)

Appendix

Table 5: Question retention at 150-turn truncation. Turns and Questions are for the full conversation; Evidence kept counts questions whose evidence turns all fall within the first 150 turns.

Condition	Conv.	Rep.	Mem.	$\boldsymbol{n_{q}}$	Core	Adv.	FQR
None	conv-26	1	309	75	0.652	1.000	–
None	conv-26	2	292	75	0.697	1.000	–
None	conv-26	3	316	75	0.621	1.000	–
None	conv-30	1	282	51	0.684	0.923	–
None	conv-30	2	248	51	0.737	1.000	–
None	conv-30	3	253	51	0.711	1.000	–

Appendix

Table 6: Per-cell results, all 90 runs. Core is core accuracy within the run; Adv. is adversarial abstention with hand labels applied. Quar. counts quarantines, and FQR divides them by the decisions made in that cell (memories judged, for the reranker stage).

LLM Agent Safety

Large language model agent safety studies how to keep tool-using, persistent, and multi-agent systems aligned with user intent and authorized limits.

See all 322 papers

MemPoison: Uncovering Persistent Memory Threats and Structural Blind Spots in LLM Agents
Jul 16, 2026Jifeng Gao, Kang Xia, Yi Zhang +5LLM Agent Memory LLM Agent Security
State Key Laboratory for Novel Software Technology, Nanjing University, China · NARI Group Corporation/State Grid Electric Power Research Institute, China
Forgetting is Not Always Bad: A Neuro-Inspired Memory Repair Mechanism for Poisoned LLM Agents
Oct 9, 2026Lei Liu, Yunji Liang, Xiaowen Zhang +4Agent Memory Poisoning Backdoor Defense in LLMs
Northwestern Polytechnical University · National Univerisity of Singapore
From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agents
Jun 3, 2026Pritam Dash, Tongyu Ge, Aditi Jain +2AI Agent Security Benchmarks Agent Memory Poisoning
Huawei Canada · University of Waterloo, Canada · Work done during an internship at Huawei.

Jul 16, 2026J/K move · Enter open · S save

MemPoison: Uncovering Persistent Memory Threats and Structural Blind Spots in LLM Agents

Jifeng Gao, Kang Xia, Yi Zhang, Xiaobin Hong+4

State Key Laboratory for Novel Software Technology, Nanjing University, China · NARI Group Corporation/State Grid Electric Power Research Institute, China

LLM Agent Memory LLM Agent Security AI Agent Security Benchmarks Agent Memory Poisoning

Open paper AskPDF arXiv

Persistent external memory enhances agent continuity but introduces persistent security vulnerabilities: adversarial content can be injected via standard interaction channels, retained across turns, and later distort downstream behavior. To address this challenge, we propose MemPoison, a comprehensive benchmark and analysis framework featuring 1227 hand-validated cases across four attack types, three injection channels, and three representative memory substrates, evaluated on seven open-weight and three closed-weight model families. We introduce a three-tier taxonomy: (L1) direct single-record corruption, (L2) compositional multi-record corruption and (L3) context-triggered dormant corruption. Our evaluations reveal a distinct defense frontier: while baseline write-time defenses, such as consistency checks, substantially suppress direct L1 attacks, they fail to reliably suppress L2 and L3 attacks. Through mechanistic influence decomposition (MID), we demonstrate structural blind spots in write-time defenses, which admit seemingly benign records that later become harmful through joint retrieval composition or trigger-conditioned activation. Our findings advocate for shifting from static filtering to adaptive, context-sensitive memory defense strategies.

Abstract

LLM Agent Safety

Explore similar work

MemPoison: Uncovering Persistent Memory Threats and Structural Blind Spots in LLM Agents

Forgetting is Not Always Bad: A Neuro-Inspired Memory Repair Mechanism for Poisoned LLM Agents

From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agents