From Attack Success to Attack Severity: Counterfactual Memory Attacks on LLM Agents

Mingxi Zou; Langzhang Liang; Zhuo Wang; Yiyang Zhao; Lizhen Qu; Zenglin Xu

cs.AISep 28, 2026

From Attack Success to Attack Severity: Counterfactual Memory Attacks on LLM Agents

Authors: Mingxi Zou, Langzhang Liang, Zhuo Wang, Yiyang Zhao, Lizhen Qu, Zenglin Xu

Organizations: Shanghai Academy of AI for Science (SAIS) · Fudan University · Monash University

Abstract

As LLM agents increasingly rely on persistent memory for long-horizon and personalized behavior, they can retain and reuse information across interactions, but this also creates a lasting channel through which malicious memory writes can influence future behavior. Persistent-memory attacks are typically evaluated by whether they succeed, yet successful attacks can leave persistent states with substantially different downstream consequences. We study this severity as a distinct attack-design objective and formalize it with counterfactual memory regret (CMR), the paired increase in expected downstream loss relative to clean memory. We introduce MemHarm, which predeclares a finite class of sparse, grounded semantic edits, evaluates candidates through the normal agent memory interface using offline paired-loss feedback, and certifies resolved selections within that class. Compared with attack-success optimization, CMR-guided selection produces substantially larger downstream loss while retaining most of the success-rate gain. Across two agent benchmarks and diverse memory designs, MemHarm attains the highest CMR point estimates among the evaluated general attacks on identical support. Factor-removal interventions link this harm to the selected semantic factor, and native-agent deployments verify the write-to-fresh-process attack path.

Figures & tables

Figure 1: Attack success does not determine severity. Both attacks achieve the same binary success outcome, yet they induce different downstream task losses.

Figure 2: Native-agent MPBench evaluation. (a) MemHarm’s write–persist–retrieve attack path in OpenClaw and Hermes. (b) Full-case ASR, conditional RSR, and E2E versus MPBench Avg and per-metric Best; Appendix H.1 gives the protocol.

Figure 3: MemHarm overview. MemHarm freezes plausible semantic memory edits and uses offline memory-only paired rollouts to select a high-CMR attack.

GPT-4o	General attack baselines
Backbone	Attack	AgentDojo				$\tau^{3}$ -bench
Backbone	Attack	CMR $\uparrow$	ASR (%) $\uparrow$	BUD (pp) $\downarrow$	$\mathrm{BMAR}_{\geq 4}$ (%) $\uparrow$	CMR $\uparrow$	ASR (%) $\uparrow$	BUD (pp) $\downarrow$	$\mathrm{BMAR}_{\geq 4}$ (%) $\uparrow$
	AgentPoison	$-0.069\pm 0.033$	$2.5\pm 1.8$	$4.2\pm 2.5$	$9.2\pm 5.0$	$0.204\pm 0.087$	$60.0\pm 14.5$	$3.3\pm 2.3$	$20.4\pm 12.2$
	MINJA	$-0.097\pm 0.104$	$13.6\pm 4.8$	$8.8\pm 3.4$	$21.7\pm 7.5$	$0.184\pm 0.119$	$60.0\pm 14.2$	$7.2\pm 3.0$	$12.0\pm 8.3$
	InjecMEM	$-0.072\pm 0.093$	$12.5\pm 4.3$	$6.1\pm 2.9$	$2.5\pm 2.5$	$0.194\pm 0.095$	$63.3\pm 14.9$	$5.0\pm 2.7$	$6.1\pm 8.2$
	MemPoison	$0.008\pm 0.022$	$6.7\pm 1.6$	$5.4\pm 2.7$	$2.2\pm 2.4$	$0.224\pm 0.109$	$66.7\pm 13.7$	$4.3\pm 2.5$	$2.0\pm 4.1$

Table 1: Primary held-out attack comparison. CMR, ASR, and

\mathrm{BMAR}_{\geq 4}

use applicable support; BUD uses shared non-target probes. Entries are point estimates with 95% intervals; blue bold and purple shading mark the best and second-best estimates.

	GPT-4o		Llama-3.1-70B
Attack / comparison	AgentDojo	$\tau^{3}$ -bench	AgentDojo	$\tau^{3}$ -bench
(a) General attacks: common-support CMR $\uparrow$ identical six-contract support
AgentPoison	$-0.052\pm 0.034$	$0.195\pm 0.086$	$-0.035\pm 0.041$	$0.171\pm 0.091$
MINJA	$-0.081\pm 0.072$	$0.176\pm 0.103$	$-0.052\pm 0.078$	$0.158\pm 0.108$
InjecMEM	$-0.059\pm 0.067$	$0.185\pm 0.091$	$-0.041\pm 0.071$	$0.169\pm 0.097$
MemPoison	$0.011\pm 0.026$	$0.216\pm 0.088$	$0.012\pm 0.029$	$0.195\pm 0.094$

Table 2: Fair-support severity comparisons. (a) Common-support CMR for general attacks. (b) Contract-matched CMR gains of MemHarm over specialized baselines. Entries show 95% intervals where applicable.

Figure 4: Severity structure and generalization. Panels (a–d) summarize objective robustness, cross-memory transfer, semantic-factor severity, and attack-quality diagnostics. Whiskers denote 95% CIs where available; transfer labels report macro retention.

Figure 5: Identification and reliability. Panels (a–d) summarize memory-path identification, class-conditional certification, certificate difficulty, and construction validity. Whiskers denote 95% CIs where available.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Attack variable	Reported insertion / retention path	Role of retrieval in the reported mechanism	Reported detectability / plausibility framing	Evaluation target
PoisonedRAG	Malicious passages	Retrieval-corpus or index content; agent-memory state is not the primary target	Central to the attack	Perplexity-based and duplicate-filter defenses evaluated	Attacker-chosen answer for an attacker-chosen target question
AgentPoison	Trigger-conditioned poisoned demonstrations	Poisoned long-term-memory or knowledge-base entries	Central in the reported retrieval-equipped configurations	Trigger coherence; benign-task behavior also measured	Triggered target action and end-to-end adversarial impact
MINJA	Query-induced malicious records with bridging steps	Records induced through query-only interaction and retained by the agent	Central to later activation; query-only describes the insertion route	Query-only interaction constraint	Targeted malicious reasoning on later victim queries
InjecMEM	Topic anchor paired with an instruction payload	A record induced in one interaction and retained through the memory path	Retriever-agnostic anchor designed for topic-conditioned later retrieval	Topic-conditioned retrieval	Pre-specified output for related queries
MemoryGraft	Poisoned successful experiences or procedure templates	Ingestion artifacts that induce persistent poisoned experience records	Experience retrieval is central	Successful-looking traces	Persistent behavioral drift on semantically similar tasks
MemPoison	Dialogue-delivered trigger–payload relation	A triggerable backdoor retained through selective memory processing	Depends on the memory extraction/retrieval pipeline	Selective-memory survival	Trigger-conditioned later responses

Appendix

Table 3: Qualitative comparison with prior memory attacks using each source paper’s terminology and evaluation protocol.

Figure 6: Expanded fixed-confidence audit for direct signed CMR over 128 independent task–memory instances, with 32 instances in each benchmark–backbone panel. (a) Mutually exclusive terminal outcomes are shown together with the realized frozen-class size, mean paired pulls, and final unresolved gap. No minimum-budget-only outcome occurred. (b) Certified budgets are summarized among the 80 dual certificates. (c) Adaptive and uniform allocation use the same confidence sequences,

(\eta,\delta)

, and stopping predicates at a matched 91.4% at-least-optimality rate. (d) Resolution is stratified by the completed-audit empirical top-two arm gap.

Figure 7: Sensitivity to violations of the memory-only assumption, averaged over AgentDojo and

\tau^{3}

-bench. (a) Enabling each residual channel increases observed paired CMR above the isolated value of 0.446; channels are ordered by this inflation. (b) The corresponding

\widehat{\kappa}_{X}

and

\widehat{\kappa}_{x_{a}}

diagnostics quantify residual coupling for the clean and edited branches.

Memory	Write unit	Read interface	Factor representation	Defense-relevant property
Vector episodic	Raw episode or document chunk	Similarity retrieval	Implicit span-level fact	Preserves source spans, but over-retrieves plausible edits
Rolling summary	Session or task summary	Injected summary context	Compressed natural-language rule	Loses fine-grained provenance during compression
User profile	Structured or semi-structured profile field	Profile lookup / injection	Explicit preference, threshold, or binding	Rejects unsupported fields under strict schemas
Reflection store	Generated lesson or rule	Retrieved reflection	Reusable behavioral prior	Converts local facts into broad rules
Graph memory	Entity–relation edge	Graph neighborhood lookup	Explicit typed edge	Benefits from schema validation and signed edges
Hybrid memory	Retrieval plus summary/profile state	Mixed retrieval and injection	Implicit and explicit	Inherits provenance loss and structured validation hooks

Appendix

Table 7: Memory contracts used to instantiate the same semantic attack variable across heterogeneous backends.

Figure 9: Defense-stress results for GPT-4o, equal-weight macro-averaged over AgentDojo and $\tau^{3}$ -bench across five independent seeds. (a) Each cell reports mean CMR $\pm$ standard error; lower is better for the defender. The orange outline marks MemHarm and the green outline marks the combined provenance, confirmation, and causal-audit defense. (b) The aligned benign-utility cost is reported as mean drop $\pm$ standard error.

Figure 10: Cross-memory transfer after source-only selection is frozen. (a) Macro-averages over two benchmarks and two backbones; each block contains 48 held-out source tasks and five independent seeds. Whiskers are task-cluster bootstrap 95% confidence intervals and direct labels include the retention ratio $R_{\mathrm{transfer}}$ . (b) Per-backbone CMR point estimates underlying the macro-average; orange outlines mark source-memory MemHarm rows.

Figure 11: Held-out stealth, sparsity, recovery, and retention audits. (a) MemoryGraft and MemHarm are compared on six metrics using each metric’s native scale; whiskers denote task-cluster bootstrap 95% confidence intervals and

\Delta_{\mathcal{H}}

is a simultaneous UCB. (b) The retention audit uses 240 MemHarm writes and 240 matched-clean writes, at least three reviewers per item, and task-level bootstrap intervals. (c) The attack–clean appearance gap. (d) Bootstrap interval for ordinal agreement measured by Krippendorff’s

\alpha

Selection policy	CMR	ASR (%)
Random	0.6830	57.55
ASR-guided	0.6958	75.30
CMR-guided	0.7830	71.28

Appendix

Table 8: Fixed-space objective comparison on held-out rollouts. All three policies select from the same admissible native-agent edit class; CMR is reported on its natural scale and ASR as a percentage.

Figure 13: Front-end construction and validation quality on the two primary benchmarks. (a) Automatic factor-extractor precision, recall, Cohen’s

\kappa

, and schema coverage. (b) Oracle CMR gap, where lower is better. (c) Benign-manifold acceptance, false-accept, and false-reject rates; the domain-specific

\theta_{\mathrm{lm}}

and frozen detector-threshold sets are shown below the panel.

Explore similar work

May 22, 2026cs.AI

MemAudit: Post-hoc Auditing of Poisoned Agent Memory via Causal Attribution and Structural Anomaly Detection

Large language model agents increasingly rely on persistent memory to store past interactions, retrieve relevant demonstrations, and improve long-horizon task execution. However, this memory mechanism also creates a practical security vulnerability: an adversarial user may inject malicious records into the agent's memory through ordinary interaction, and these records can later be retrieved to steer the agent's reasoning and actions. Existing defenses primarily focus on online intervention, such as prompt filtering or output blocking, but they do not address the post-hoc question of which stored memories are responsible after harmful behavior has already been observed. We propose \textbf{MemAudit}, a post-hoc causal memory auditing framework for memory-augmented LLM agents. The framework combines two complementary signals: (1) a counterfactual memory influence score that measures each memory's causal contribution to harmful outputs, and (2) a memory consistency graph that identifies structurally anomalous memories within the broader memory store. We evaluate MemAudit against MINJA, a query-only memory injection attack in which malicious records are generated and stored through normal agent interactions rather than direct memory-bank modification. Across both QA and reasoning-agent settings, MemAudit substantially reduces attack success rates under realistic post-hoc auditing scenarios. The results show that QA attack success is reduced from

70\%

0\%

, while RAP attack success drops from

83.3\%

0\%

Zhewen Tan, Yilun Yao, Huiyan Jin +9

Poisoning Attacker Large Language Model Model Auditing Counterfactual Learning

Ask PDF

Jul 6, 2026cs.CR

Your Agent's Memories Are Not Its Own: Forged Reasoning Attacks on LLM Agent Memory and Defenses

Persistent memory has enabled large language model (LLM) agents to store factual knowledge, prior decisions, reasoning histories, tool usage information, and context. While this has improved the agent's functionality and continuity across tasks, it has also introduced a new attack surface: the agent's own reasoning history. In this paper, we introduce the Forged Amplifying Rationale Memory Attack (FARMA), which poisons an agent's remembered reasoning rather than its factual knowledge. It inserts forged reasoning traces using evasive language that bypasses keyword-based defenses, then amplifies them through self-referential reinforcement that defeats consensus-based defenses. To address FARMA, we introduce SENTINEL, a layered defense pipeline to detect forged reasoning entries. Its central component is the Reasoning Guard that structurally analyzes candidate entries for forgery using five weighted signals. We evaluate FARMA and SENTINEL across multiple agents and different LLM models with 50 trials and show that FARMA achieves an attack success rate of up to 100% under baseline conditions and is capable of defeating defense mechanisms like keyword filter and A-MemGuard. Our evaluation also shows that SENTINEL reduces FARMA's attack success rate to as low as 0% with no false positives observed across 326 benign agent traces. Our work demonstrates the need to protect not only an agent's retrieved content but also the integrity of its reasoning history.

Neeraj Karamchandani, Piyush Nagasubramaniam, Sencun Zhu +1

The Pennsylvania State University, University Park, PA, USA

LLM Reasoning Strategies LLM Defense Mechanisms Benign

Ask PDF

Jul 31, 2026cs.CR

Memory Provenance Laundering in LLM Agents: A Non-Amplification Firewall for Persistent Memory

Long-term memory lets large language model(LLM) agents reuse prior preferences and work flows, but it also turns untrusted observations into persistent action context. We identify memory provenance laundering: during LLM-based memory consolidation, an external observation may be rewritten as apparent user history or workflow support, preserving an action trigger while erasing the low-trust source that should limit its authority. Existing prompt filters, content sanitizers, and tool guards do not enforce source-authority non-amplification after lossy memory consolidation. We formalize this boundary and instantiate it as Provenance-Preserving Memory Fire wall (PPMF), a lightweight memory middleware that preserves platform-maintained provenance and authorizes tool calls by matching action risk to the authority of action-relevant memories. In our schema-grounded evaluation with fixed risk policies, vulnerable consolidated memories reach up to 1.000 attack success rate(ASR); with intact platform-maintained provenance, confirmation, and risk labels, no evaluated unauthorized high-risk action passes the PPMF gate while confirmed benign actions and targeted low-risk memory use remain executable.

Jinghan Xu, Yiyong Xiao, Wanru Shao +2

Semantic Memory Large Language Model Agents Persistence Provenance Semantic Firewall

Ask PDF