Eternal Sunshine of the Spotless Mind: Systematically Erasing LLM's Memories

Olga Ohrimenko

cs.CLSep 29, 2026

Eternal Sunshine of the Spotless Mind: Systematically Erasing LLM's Memories

Authors: Olga Ohrimenko

Organizations: School of Computing and Information Systems The University of Melbourne

Abstract

We consider persistent LLMs that accumulate memories of their interactions with a user over time. Such LLMs maintain memories using external storage, which they can query to overcome the limitations of a fixed context window. Such systems have numerous practical applications, as they can draw on all past interactions when responding to user queries. In this paper, we ask whether LLMs can forget information shared with them upon a user's request. We find that current LLMs fail to delete such information---even when they claim to have forgotten it and even when operating with a limited context. To this end, we consider a new direction of study: Deletion of LLM Memories. We show that naively removing messages that match a user's deletion request is insufficient, since conversations naturally introduce message dependencies that cause information to persist. To correctly handle deletion requests, we propose the DeLLM framework. It dynamically constructs relevant context for each LLM query and maintains a provenance graph of messages to determine which ones must be removed during deletion. Our experiments show that DeLLM achieves a high deletion rate while maintaining utility.

Figures & tables

Fig. 1 : Example of persistence of personal information in ChatGPT. Here, a user shares a fact and asks ChatGPT to forget it. However, the fact reappears in the next interaction.

Fig. 2 : User asks Copilot to remove sensitive information. Though the information is removed from “Saved memories” and Copilot says it will not retain it, the fact is referenced in the last interaction since it persists in the chat history.

Fig. 3 : Example conversation with an LLM ( gpt-4.1-nano ) where information about user’s location can be inferred from messages 4&8, even if 1&2 are removed. Message identifiers appear on the right in brackets.

Fig. 4 : Sliding window context does not guarantee deletion. Here, an LLM ( gpt-4.1-nano ) is given as context only the last five messages exchanged between it and the user. Rectangular boundaries over messages indicate which messages are included in the contexts of LLM’s messages 4, 6 and 8. For example, to answer the last user message 7, the LLM is given messages 3-7. Here, even though message 1 is not part of the context of message 8, information about it persists in message 8 since message 4 depends on 1 and message 4 is part of 8’s context.

Fig. 5 : Processing a user query with $\mathsf{DeLLM}$ (Section V-B ): 1) User sends a query. 2) $\mathsf{DeLLM}$ constructs context by retrieving from $\mathbb{D}$ messages relevant for answering the query and 3) sends the query and the context to LLM. 4) LLM replies. 5) $\mathsf{DeLLM}$ saves messages in $\mathbb{D}$ , adds dependency between LLM’s response and messages in its context to $\mathbb{G}$ , and sends the response to the user.

Fig. 6 : Processing a deletion request with $\mathsf{DeLLM}$ (Section V-C ): 1) User sends a deletion request. 2) $\mathsf{DeLLM}$ queries $\mathbb{D}$ to retrieve a message set $M$ matching the deletion criteria. 3) $\mathsf{DeLLM}$ traverses $\mathbb{G}$ to find messages tainted by messages in $M$ . 4) $\mathsf{DeLLM}$ removes $M$ and all the tainted messages from $\mathbb{D}$ and $\mathbb{G}$ .

Fig. 7 : A provenance graph, where a node corresponds to a message identifier and a directed edge from

i

j

indicates that

i

was used as part of a context for message

j

. If message 1 needs to be removed, then all messages dependent on 1 need to be removed. They can be found by traversing the graph to find all the nodes reachable from 1. The traversal will find that messages 2, 4 and 8 need to be removed as well.

LLM	System	Datasets
LLM	System	DMR	TOFU
gpt-4.1-nano	$\mathsf{LLM~w/~memory}$	84%	68%
gpt-4.1-nano	$\boldsymbol{\mathsf{DeLLM}}$	79%	89%
gpt-5-nano	$\mathsf{LLM~w/~memory}$	76%	82%
gpt-5-nano	$\boldsymbol{\mathsf{DeLLM}}$	69%	84%

TABLE I : Utility across datasets, LLM models and systems. Accuracy (%) based on LLM judge. Higher utility is preferred.

LLM	System	Datasets
		TOFU		PF
		Before	After	Before	After
gpt-4.1-nano	$\mathsf{LLM~w/o~memory}$	32.5	32.5	1	1
	$\mathsf{LLM~w/~memory}$	75	70	100	96
	$\boldsymbol{\mathsf{DeLLM}}$	95	27.5	100	2.5
gpt-5-nano	$\mathsf{LLM~w/o~memory}$	35	35	1.7	1.7

TABLE II : Deletion rate across datasets, LLM models and systems. Accuracy (%) based on LLM judge. A lower After score is preferred, as it indicates that the utility on the forget data has decreased following a deletion request for that data.

LLM	System	Datasets
		TOFU		PF
		Before	After	Before	After
gpt-4.1-nano	$\mathsf{LLM~w/~memory}$	62	62.3	100	99.7
gpt-4.1-nano	$\boldsymbol{\mathsf{DeLLM}}$	83.5	89.3	100	100
gpt-5-nano	$\mathsf{LLM~w/~memory}$	74.1	74.2	100	100
gpt-5-nano	$\boldsymbol{\mathsf{DeLLM}}$	76.3	85.3	99.7	100

TABLE III : Impact of deletion of forget data on utility over retain dataset. Accuracy (%) based on LLM judge. Higher After score is preferred, as it indicates that the utility on the retain data has not been impacted significantly after the deletion request on forget data.

Fig. 8 : Utility on forget and retain data after a deletion request. Lower utility on the forget data and higher utility on the retain data indicate better performance. We compare $\mathsf{DeLLM}$ with semantic deletion, evaluated over five similarity thresholds, by removing messages that semantically match the forget data. To match $\mathsf{DeLLM}$ ’s deletion rate, the semantic deletion method must use a sufficiently aggressive threshold, which results in unrelated messages being deleted and therefore reduces utility on the retain data.

LLM	System	Datasets
LLM	System	DMR	TOFU
gpt-4.1-nano	$\mathsf{LLM~w/~memory}$	66.6%	12.0%
gpt-4.1-nano	$\mathsf{DeLLM}$	64.2%	34.3%
gpt-5-nano	$\mathsf{LLM~w/~memory}$	60.5%	14.8%
gpt-5-nano	$\mathsf{DeLLM}$	57.6%	24.4%

TABLE IV : Utility across datasets, LLM models and systems. Accuracy based on ROUGE-L recall score.

LLM	System	Datasets
		TOFU		PF
		Before	After	Before	After
gpt-4.1-nano	$\mathsf{LLM~w/o~memory}$	18.8	18.8	1.5	1.5
	$\mathsf{LLM~w/~memory}$	13.0	13.0	75.8	65.3
	$\boldsymbol{\mathsf{DeLLM}}$	37.7	24.0	82.2	20.4
gpt-5-nano	$\mathsf{LLM~w/o~memory}$	17.5	17.5	10.3	10.3

TABLE V : Deletion rate across datasets, LLM models and systems. ROUGE-L recall scores reported. A lower After score is preferred, as it indicates that the utility on the forget data has decreased following a deletion request for that data.

LLM	System	Datasets
		TOFU		PF
		Before	After	Before	After
gpt-4.1-nano	$\mathsf{LLM~w/~memory}$	11.1	11.1	70.3	66.9
gpt-4.1-nano	$\boldsymbol{\mathsf{DeLLM}}$	30.9	29.3	83.0	80.8
gpt-5-nano	$\mathsf{LLM~w/~memory}$	14.1	13.5	79.1	77.2
gpt-5-nano	$\boldsymbol{\mathsf{DeLLM}}$	22.6	23.8	87.4	86.3

TABLE VI : Impact of deletion of forget data on utility over retain dataset. ROUGE-L recall scores reported. Higher After score is preferred, as it indicates that the utility on the retain data has not been impacted significantly after the deletion request on forget data.

Panel of Figure 15 — Fig. 9 : (Left) Sample interaction with Gemini where the user shares their location and asks several questions that depend on the location. (Right) The user deletes messages 3–6 that mention the location “Paris” explicitly in the conversation in the left figure. The user then continues the conversation by asking question (11). Gemini’s response still uses the location “Paris” that is inferred from the remaining messages. Note: This interaction happened outside of France and the authors are not based in France.

LLM Unlearning

Large language model unlearning studies how to remove specified knowledge or behaviors while preserving a model’s unrelated capabilities.

See all 191 papers

MemCut: Every Secret Leaves a Trail in the Long-Term Memory of LLM Agents
Oct 15, 2026Cheng Zhang, Xuanning Cao, Zhicheng Zheng +1LLM Agent Memory Privacy Leakage in Language Models
The Hong Kong University of Science and Technology (Guangzhou) · University of Science and Technology of China · The Hong Kong University of Science and Technology
Can Agents Forget When Asked? A Benchmark for Memory Unlearning in LLM Agents
Oct 13, 2026Bofan Xie, Chaokun Wang, Guanchen Ge +4LLM Agent Memory Memory Agent Benchmarks
Tsinghua University · Sina Weibo
AfterDeleteBench: Auditing Post-Deletion Information Persistence in LLM Agents
Oct 22, 2026Ehsan Saleh, Roozbeh Nahavandi, Sama Salarian +2LLM Agent Evaluation Memory Management for LLM Agents
The Ohio State University

Oct 15, 2026J/K move · Enter open · S save

MemCut: Every Secret Leaves a Trail in the Long-Term Memory of LLM Agents

Cheng Zhang, Xuanning Cao, Zhicheng Zheng, Hui Xiong

The Hong Kong University of Science and Technology (Guangzhou) · University of Science and Technology of China · The Hong Kong University of Science and Technology

LLM Agent Memory Privacy Leakage in Language Models Long-Term Conversational Memory LLM Unlearning

Open paper AskPDF arXiv

Long-term memory has become a crucial component of large language model (LLM) agents, with most research focusing on retrieving relevant information accurately and efficiently. However, a dependable memory system must also forget sensitive user data upon request. To evaluate this capability, we introduce MemCut, a systematic benchmark for user-requested forgetting. Its 456 hand-designed secrets in long-horizon conversations have fully annotated evidence trails to test recovery after deletion, including from messages that never state the secrets directly. We propose two recovery metrics. Native recovery measures reader success through the system's retrieval, whereas residual recovery measures the same reader's success when directly given surviving objects traced to annotated disclosures or evidence. We establish three reference deletion baselines to test how far existing retrievers and write models support forgetting across five memory designs and four LLMs. Disclosure removal reduces native recovery far more than residual recovery. LLM-guided deletion of ten additional turns from twenty retrieval candidates leaves 39-76% residual recovery. Subset sweeps (144 facts, three systems, two models) show that widening the pool at a fixed budget reduces residual recovery more than deleting all original candidates. These sweeps identify candidate coverage as a bottleneck in the reference baselines.

Abstract

LLM Unlearning

Explore similar work

MemCut: Every Secret Leaves a Trail in the Long-Term Memory of LLM Agents

Can Agents Forget When Asked? A Benchmark for Memory Unlearning in LLM Agents

AfterDeleteBench: Auditing Post-Deletion Information Persistence in LLM Agents