Eternal Sunshine of the Spotless Mind: Systematically Erasing LLM's Memories

Olga Ohrimenko

cs.CLSep 29, 2026

Eternal Sunshine of the Spotless Mind: Systematically Erasing LLM's Memories

Authors: Olga Ohrimenko

Organizations: School of Computing and Information Systems The University of Melbourne

Abstract

We consider persistent LLMs that accumulate memories of their interactions with a user over time. Such LLMs maintain memories using external storage, which they can query to overcome the limitations of a fixed context window. Such systems have numerous practical applications, as they can draw on all past interactions when responding to user queries. In this paper, we ask whether LLMs can forget information shared with them upon a user's request. We find that current LLMs fail to delete such information---even when they claim to have forgotten it and even when operating with a limited context. To this end, we consider a new direction of study: Deletion of LLM Memories. We show that naively removing messages that match a user's deletion request is insufficient, since conversations naturally introduce message dependencies that cause information to persist. To correctly handle deletion requests, we propose the DeLLM framework. It dynamically constructs relevant context for each LLM query and maintains a provenance graph of messages to determine which ones must be removed during deletion. Our experiments show that DeLLM achieves a high deletion rate while maintaining utility.

Figures & tables

Fig. 1 : Example of persistence of personal information in ChatGPT. Here, a user shares a fact and asks ChatGPT to forget it. However, the fact reappears in the next interaction.

Fig. 2 : User asks Copilot to remove sensitive information. Though the information is removed from “Saved memories” and Copilot says it will not retain it, the fact is referenced in the last interaction since it persists in the chat history.

Fig. 3 : Example conversation with an LLM ( gpt-4.1-nano ) where information about user’s location can be inferred from messages 4&8, even if 1&2 are removed. Message identifiers appear on the right in brackets.

Fig. 4 : Sliding window context does not guarantee deletion. Here, an LLM ( gpt-4.1-nano ) is given as context only the last five messages exchanged between it and the user. Rectangular boundaries over messages indicate which messages are included in the contexts of LLM’s messages 4, 6 and 8. For example, to answer the last user message 7, the LLM is given messages 3-7. Here, even though message 1 is not part of the context of message 8, information about it persists in message 8 since message 4 depends on 1 and message 4 is part of 8’s context.

Fig. 5 : Processing a user query with $\mathsf{DeLLM}$ (Section V-B ): 1) User sends a query. 2) $\mathsf{DeLLM}$ constructs context by retrieving from $\mathbb{D}$ messages relevant for answering the query and 3) sends the query and the context to LLM. 4) LLM replies. 5) $\mathsf{DeLLM}$ saves messages in $\mathbb{D}$ , adds dependency between LLM’s response and messages in its context to $\mathbb{G}$ , and sends the response to the user.

Fig. 6 : Processing a deletion request with $\mathsf{DeLLM}$ (Section V-C ): 1) User sends a deletion request. 2) $\mathsf{DeLLM}$ queries $\mathbb{D}$ to retrieve a message set $M$ matching the deletion criteria. 3) $\mathsf{DeLLM}$ traverses $\mathbb{G}$ to find messages tainted by messages in $M$ . 4) $\mathsf{DeLLM}$ removes $M$ and all the tainted messages from $\mathbb{D}$ and $\mathbb{G}$ .

Fig. 7 : A provenance graph, where a node corresponds to a message identifier and a directed edge from

i

j

indicates that

i

was used as part of a context for message

j

. If message 1 needs to be removed, then all messages dependent on 1 need to be removed. They can be found by traversing the graph to find all the nodes reachable from 1. The traversal will find that messages 2, 4 and 8 need to be removed as well.

LLM	System	Datasets
LLM	System	DMR	TOFU
gpt-4.1-nano	$\mathsf{LLM~w/~memory}$	84%	68%
gpt-4.1-nano	$\boldsymbol{\mathsf{DeLLM}}$	79%	89%
gpt-5-nano	$\mathsf{LLM~w/~memory}$	76%	82%
gpt-5-nano	$\boldsymbol{\mathsf{DeLLM}}$	69%	84%

TABLE I : Utility across datasets, LLM models and systems. Accuracy (%) based on LLM judge. Higher utility is preferred.

LLM	System	Datasets
		TOFU		PF
		Before	After	Before	After
gpt-4.1-nano	$\mathsf{LLM~w/o~memory}$	32.5	32.5	1	1
	$\mathsf{LLM~w/~memory}$	75	70	100	96
	$\boldsymbol{\mathsf{DeLLM}}$	95	27.5	100	2.5
gpt-5-nano	$\mathsf{LLM~w/o~memory}$	35	35	1.7	1.7

TABLE II : Deletion rate across datasets, LLM models and systems. Accuracy (%) based on LLM judge. A lower After score is preferred, as it indicates that the utility on the forget data has decreased following a deletion request for that data.

LLM	System	Datasets
		TOFU		PF
		Before	After	Before	After
gpt-4.1-nano	$\mathsf{LLM~w/~memory}$	62	62.3	100	99.7
gpt-4.1-nano	$\boldsymbol{\mathsf{DeLLM}}$	83.5	89.3	100	100
gpt-5-nano	$\mathsf{LLM~w/~memory}$	74.1	74.2	100	100
gpt-5-nano	$\boldsymbol{\mathsf{DeLLM}}$	76.3	85.3	99.7	100

TABLE III : Impact of deletion of forget data on utility over retain dataset. Accuracy (%) based on LLM judge. Higher After score is preferred, as it indicates that the utility on the retain data has not been impacted significantly after the deletion request on forget data.

Fig. 8 : Utility on forget and retain data after a deletion request. Lower utility on the forget data and higher utility on the retain data indicate better performance. We compare $\mathsf{DeLLM}$ with semantic deletion, evaluated over five similarity thresholds, by removing messages that semantically match the forget data. To match $\mathsf{DeLLM}$ ’s deletion rate, the semantic deletion method must use a sufficiently aggressive threshold, which results in unrelated messages being deleted and therefore reduces utility on the retain data.

LLM	System	Datasets
LLM	System	DMR	TOFU
gpt-4.1-nano	$\mathsf{LLM~w/~memory}$	66.6%	12.0%
gpt-4.1-nano	$\mathsf{DeLLM}$	64.2%	34.3%
gpt-5-nano	$\mathsf{LLM~w/~memory}$	60.5%	14.8%
gpt-5-nano	$\mathsf{DeLLM}$	57.6%	24.4%

TABLE IV : Utility across datasets, LLM models and systems. Accuracy based on ROUGE-L recall score.

LLM	System	Datasets
		TOFU		PF
		Before	After	Before	After
gpt-4.1-nano	$\mathsf{LLM~w/o~memory}$	18.8	18.8	1.5	1.5
	$\mathsf{LLM~w/~memory}$	13.0	13.0	75.8	65.3
	$\boldsymbol{\mathsf{DeLLM}}$	37.7	24.0	82.2	20.4
gpt-5-nano	$\mathsf{LLM~w/o~memory}$	17.5	17.5	10.3	10.3

TABLE V : Deletion rate across datasets, LLM models and systems. ROUGE-L recall scores reported. A lower After score is preferred, as it indicates that the utility on the forget data has decreased following a deletion request for that data.

LLM	System	Datasets
		TOFU		PF
		Before	After	Before	After
gpt-4.1-nano	$\mathsf{LLM~w/~memory}$	11.1	11.1	70.3	66.9
gpt-4.1-nano	$\boldsymbol{\mathsf{DeLLM}}$	30.9	29.3	83.0	80.8
gpt-5-nano	$\mathsf{LLM~w/~memory}$	14.1	13.5	79.1	77.2
gpt-5-nano	$\boldsymbol{\mathsf{DeLLM}}$	22.6	23.8	87.4	86.3

TABLE VI : Impact of deletion of forget data on utility over retain dataset. ROUGE-L recall scores reported. Higher After score is preferred, as it indicates that the utility on the retain data has not been impacted significantly after the deletion request on forget data.

Panel of Figure 15 — Fig. 9 : (Left) Sample interaction with Gemini where the user shares their location and asks several questions that depend on the location. (Right) The user deletes messages 3–6 that mention the location “Paris” explicitly in the conversation in the left figure. The user then continues the conversation by asking question (11). Gemini’s response still uses the location “Paris” that is inferred from the remaining messages. Note: This interaction happened outside of France and the authors are not based in France.

LLM Unlearning

Large language model unlearning studies how to remove specified knowledge or behaviors while preserving a model’s unrelated capabilities.

See all 191 papers

Oct 15, 2026

MemCut: Every Secret Leaves a Trail in the Long-Term Memory of LLM Agents

Long-term memory has become a crucial component of large language model (LLM) agents, with most research focusing on retrieving relevant information accurately and efficiently. However, a dependable memory system must also forget sensitive user data upon request. To evaluate this capability, we introduce MemCut, a systematic benchmark for user-requested forgetting. Its 456 hand-designed secrets in long-horizon conversations have fully annotated evidence trails to test recovery after deletion, including from messages that never state the secrets directly. We propose two recovery metrics. Native recovery measures reader success through the system's retrieval, whereas residual recovery measures the same reader's success when directly given surviving objects traced to annotated disclosures or evidence. We establish three reference deletion baselines to test how far existing retrievers and write models support forgetting across five memory designs and four LLMs. Disclosure removal reduces native recovery far more than residual recovery. LLM-guided deletion of ten additional turns from twenty retrieval candidates leaves 39-76% residual recovery. Subset sweeps (144 facts, three systems, two models) show that widening the pool at a fixed budget reduces residual recovery more than deleting all original candidates. These sweeps identify candidate coverage as a bottleneck in the reference baselines.

Cheng Zhang, Xuanning Cao, Zhicheng Zheng +1

The Hong Kong University of Science and Technology (Guangzhou) · University of Science and Technology of China · The Hong Kong University of Science and Technology

LLM Agent Memory Privacy Leakage in Language Models Long-Term Conversational Memory LLM Unlearning

Ask PDF

Oct 13, 2026

Can Agents Forget When Asked? A Benchmark for Memory Unlearning in LLM Agents

Long-term memory has become a core component of large language model (LLM) agents. As agents accumulate personal information over long-term use, users increasingly ask them to delete sensitive or outdated facts. A memory system should honor such requests. It should remove exactly what was requested and keep the rest of its memory intact. However, existing memory benchmarks evaluate only memory writing, retrieval, and organization, so whether agents can forget at the user's explicit request remains under-evaluated. To address this gap, we propose Bemula, the first benchmark dedicated to memory unlearning. It comprises 50 multi-session dialogue instances, 135 forget requests of six types, and 1,742 test questions. Each dialogue has a control version without the requests, so that only systems that have stored the target can earn credit for forgetting. Across the paired versions, five test suites measure storage, removal of the requested information, preservation of unrelated memories, relearning of similar facts, and resistance to extraction attacks. On Bemula, we evaluate six mainstream memory systems under three backbone LLMs, together with the Full Context LLM on two of them. The results are negative. Across all evaluated systems and backbones, no system passes more than 1.5% of forget requests end to end, and most pass none at all. These systems either leave the requested information extractable under simple attacks or delete far more than requested. Memory unlearning therefore remains an open challenge for agent memory design. We release our data and code at https://github.com/Nboxff/bemula-benchmark.

Bofan Xie, Chaokun Wang, Guanchen Ge +4

Tsinghua University · Sina Weibo

LLM Agent Memory Memory Agent Benchmarks LLM Unlearning Machine Unlearning Machine Unlearning Evaluation

Ask PDF

Oct 22, 2026

AfterDeleteBench: Auditing Post-Deletion Information Persistence in LLM Agents

Many LLM agents maintain external persistent memory and derive additional state from past interactions, such as summaries. Deleting an original memory record therefore need not remove information carried by its descendants. We introduce AfterDeleteBench, a controlled behavioral audit for this failure mode in fixed-weight LLM agents. For each synthetic secret, we compare an exposed-then-deleted agent with an otherwise matched agent that was never exposed to the secret, allowing post-deletion recovery to be attributed to the original exposure. In our primary setting, a single query recovers the deleted secret from Qwen2.5-7B-Instruct in roughly two thirds of cases, while matched never-exposed controls have zero recovery. Crucially, recovery does not require a surviving verbatim copy: when the complete secret string is absent and only separately stored components remain under a public composition rule, exact reconstruction succeeds in 62-99% of cases across Qwen, Mistral, and Phi-3. We also show that the response interface is part of the measurement system: target-bearing replies can be scored as failures when the requested format and parser are misaligned. Finally, we separate reconstruction from write-back and fresh-session recall. Provenance-aware closure eliminates observed recovery in the tested positive-baseline settings, whereas an exact-literal write filter blocks reinsertion but cannot erase information already present. Together, these results show that source deletion should be evaluated behaviorally at the level of the agent and its memory system, not only as a storage operation.

Ehsan Saleh, Roozbeh Nahavandi, Sama Salarian +2

The Ohio State University

LLM Agent Evaluation Memory Management for LLM Agents AI Agent Auditing

Ask PDF

Abstract

LLM Unlearning

Explore similar work

MemCut: Every Secret Leaves a Trail in the Long-Term Memory of LLM Agents

Can Agents Forget When Asked? A Benchmark for Memory Unlearning in LLM Agents

AfterDeleteBench: Auditing Post-Deletion Information Persistence in LLM Agents