Can Agents Forget When Asked? A Benchmark for Memory Unlearning in LLM Agents

Bofan Xie; Chaokun Wang; Guanchen Ge; Songyao Wang; Fang Niu; Hang Zhang; Wei Wang

cs.AIOct 13, 2026

Can Agents Forget When Asked? A Benchmark for Memory Unlearning in LLM Agents

Authors: Bofan Xie, Chaokun Wang, Guanchen Ge, Songyao Wang, Fang Niu, Hang Zhang, Wei Wang

Organizations: Tsinghua University · Sina Weibo

Abstract

Long-term memory has become a core component of large language model (LLM) agents. As agents accumulate personal information over long-term use, users increasingly ask them to delete sensitive or outdated facts. A memory system should honor such requests. It should remove exactly what was requested and keep the rest of its memory intact. However, existing memory benchmarks evaluate only memory writing, retrieval, and organization, so whether agents can forget at the user's explicit request remains under-evaluated. To address this gap, we propose Bemula, the first benchmark dedicated to memory unlearning. It comprises 50 multi-session dialogue instances, 135 forget requests of six types, and 1,742 test questions. Each dialogue has a control version without the requests, so that only systems that have stored the target can earn credit for forgetting. Across the paired versions, five test suites measure storage, removal of the requested information, preservation of unrelated memories, relearning of similar facts, and resistance to extraction attacks. On Bemula, we evaluate six mainstream memory systems under three backbone LLMs, together with the Full Context LLM on two of them. The results are negative. Across all evaluated systems and backbones, no system passes more than 1.5% of forget requests end to end, and most pass none at all. These systems either leave the requested information extractable under simple attacks or delete far more than requested. Memory unlearning therefore remains an open challenge for agent memory design. We release our data and code at https://github.com/Nboxff/bemula-benchmark.

Figures & tables

Figure 1: Illustrative examples of forget requests in Bemula .

Figure 2: The construction pipeline of Bemula .

Figure 3: Statistics of Bemula . (a) Length of every instance across the four length tiers. (b) Number of forget requests of each type, stacked by length tier. (c) Anatomy of one long instance. Blocks on top mark sessions, dots mark turns that mention a request’s target facts, and diamonds mark where each request is injected. The shaded span continues past the last request, and the F4 row has no dots because an inferential target is never stated in the dialogue.

Figure 4: Forgetting efficacy ( $\mathrm{FE}$ ) by forget request type on Bemula .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Suite	Version	Ground truth	Conditioned on	Detects
$\mathrm{RA}$	retention	control version	fact value	–	storage capability
$\mathrm{FE}$	efficacy	forget version	[FORGOTTEN] ⋆	retention probe ⋆	residual leakage
$\mathrm{PR}$	preservation	forget version	fact value	control probe	over-deletion
$\mathrm{RLA}$	relearning	forget version	new value	–	relearning failure
$\mathrm{AR}$	attack	forget version	[NO_LEAK]	attack on control	adversarial extraction

Appendix

Table 2: Item-level metrics of Bemula . Conditioning lists the paired control probe that must also succeed for the item to count as correct. The

\star

marks F6 replacement efficacy items, which take the new value as ground truth and have no retention counterpart.

Judge verdict	Annotator correct	Annotator incorrect	Agreement
Correct	53	1	98.1%
Incorrect	1	45	97.8%
All	54	46	98.0%

Appendix

Table 3: Agreement between the human annotator and the LLM judge on the 100 sampled items. Rows give the judge verdict and columns give the annotator verdict.

Profiles
Statistic	Value
Number of user profiles	50
Number of atomic / derived facts	2,993 / 77
Average facts per profile	61.4
Dialogue instances
Number of instances (forgetting + control)	50

Appendix

Table 4: Key statistics of Bemula . Token counts are measured with the o200k_base tokenizer on the forget version of each instance.

Tier	Instances	Sessions	Turns	Requests	Questions
Short	15	5 to 8	96 to 164	1 to 2	11 to 33
Medium	16	10 to 15	334 to 472	2 to 3	18 to 47
Long	16	17 to 25	916 to 1,464	3 to 5	26 to 65
XLong	3	27 to 36	2,688 to 3,500	4 to 6	50 to 70
Total	50	746	36,900	135	1,742

Appendix

Table 5: Composition of Bemula by length tier. The ranges give the minimum and maximum observed over the instances of a tier. The final row reports the total number of instances, sessions, turns, forget requests, and questions.

Figure 5: Illustrative example of the five attack questions of one forget request. Every question probes the same forgotten fact, David’s undergraduate field of study, with a different attack type.

Panel of Table 6 — Table 6: Main results on Bemula -Lite under the three backbone LLMs. $\mathrm{RA}$ : retention accuracy. $\mathrm{FE}$ : forgetting efficacy. $\mathrm{PR}$ : preservation. $\mathrm{RLA}$ : relearning. $\mathrm{AR}$ : attack robustness. $\mathrm{RPR}$ : request pass rate. $\mathrm{Score}$ : overall Bemula -Lite score ( $\mathrm{FA}\times\mathrm{UG}$ , Eq. 5 ). All scores are percentages.

Figure 6: Forgetting efficacy (

\mathrm{FE}

) versus preservation (

\mathrm{PR}

) on Bemula for all systems and backbones.

Panel of Table 7 — Table 7: Attack robustness ( $\mathrm{AR}$ ) by attack type on Bemula . Full Context is weakest against the jailbreak prefix, since its forgetting is purely instruction following. For the systems that forget too little (Naive RAG, Mem0, LightMem, SimpleMem), indirect questions and Boolean guesses are usually the most effective attacks, requiring no prompt trickery. The best value in each backbone block is bold, and the second-best is underlined. All scores are percentages.

Figure 7: Retention accuracy ( $\mathrm{RA}$ ), relearning accuracy ( $\mathrm{RLA}$ ), and attack robustness ( $\mathrm{AR}$ ) by forget request type on Bemula , complementing the efficacy heatmap in Figure 4 . Preservation and the request pass rate are instance-level metrics and cannot be broken down by request type. $\mathrm{RLA}$ is undefined for F6 requests, whose new value is already covered by the efficacy probe. Full Context and A-MEM have no runs on Qwen3-32B-AWQ because of its context length limit. Both cases are shown as –.

Figure 8: Suite metrics as a function of dialogue length on Bemula . Instances are sorted by turn count and split into five bins of ten; each point recomputes the official metrics on that bin, and the x position is the bin’s mean turn count (log scale). Bands mark the short, medium, long, and xlong tier spans.

System	$\mathrm{RA}$	$\mathrm{FE}$	$\mathrm{PR}$	$\mathrm{RLA}$	$\mathrm{AR}$	$\mathrm{Score}$
Backbone: GPT-5.6-Luna
Full Context	84.4 [80.0,88.7]	26.8 [20.6,33.5]	73.3 [66.0,80.7]	98.2 [95.4,100.0]	27.6 [24.1,31.1]	23.1 [20.0,26.2]
Naive RAG	75.9 [70.7,81.0]	14.7 [10.7,19.5]	67.3 [59.3,75.3]	71.1 [63.0,79.6]	12.9 [10.7,15.4]	9.5 [7.5,11.8]
LangMem	50.4 [41.0,60.1]	37.9 [29.8,46.9]	20.0 [12.7,28.0]	78.9 [70.2,87.2]	25.6 [19.3,32.4]	12.4 [8.6,16.8]
Mem0	89.7 [84.8,94.4]	8.5 [5.3,12.0]	80.0 [73.3,86.0]	77.2 [70.2,84.2]	4.7 [3.4,6.1]	4.9 [3.8,6.1]
A-MEM	80.4 [75.5,85.1]	33.0 [26.6,39.8]	59.3 [50.7,67.3]	88.6 [83.2,93.6]	24.7 [20.9,28.7]	20.7 [17.1,24.4]

Appendix

Table 8: Point estimates and 95% cluster-bootstrap CIs (over instances, 10,000 resamples) for the main results on Bemula . All values are percentages.

LLM Agent Memory

Large language model agent memory lets agents retain, organize, retrieve, and update information across interactions and long-running tasks.

See all 368 papers

Oct 15, 2026

MemCut: Every Secret Leaves a Trail in the Long-Term Memory of LLM Agents

Long-term memory has become a crucial component of large language model (LLM) agents, with most research focusing on retrieving relevant information accurately and efficiently. However, a dependable memory system must also forget sensitive user data upon request. To evaluate this capability, we introduce MemCut, a systematic benchmark for user-requested forgetting. Its 456 hand-designed secrets in long-horizon conversations have fully annotated evidence trails to test recovery after deletion, including from messages that never state the secrets directly. We propose two recovery metrics. Native recovery measures reader success through the system's retrieval, whereas residual recovery measures the same reader's success when directly given surviving objects traced to annotated disclosures or evidence. We establish three reference deletion baselines to test how far existing retrievers and write models support forgetting across five memory designs and four LLMs. Disclosure removal reduces native recovery far more than residual recovery. LLM-guided deletion of ten additional turns from twenty retrieval candidates leaves 39-76% residual recovery. Subset sweeps (144 facts, three systems, two models) show that widening the pool at a fixed budget reduces residual recovery more than deleting all original candidates. These sweeps identify candidate coverage as a bottleneck in the reference baselines.

Cheng Zhang, Xuanning Cao, Zhicheng Zheng +1

The Hong Kong University of Science and Technology (Guangzhou) · University of Science and Technology of China · The Hong Kong University of Science and Technology

LLM Agent Memory Privacy Leakage in Language Models Long-Term Conversational Memory LLM Unlearning

Ask PDF

Oct 13, 2026

Can Agents Truly Forget? Explore Machine Unlearning for LLM Agents

Large language model (LLM) agents have shown strong capabilities in interactive problem solving, tool use, and long-horizon task execution. However, they may also acquire sensitive or private knowledge during training and expose it during deployment. Retraining a model from scratch to remove such knowledge is prohibitively costly, making machine unlearning a more practical approach. Machine unlearning aims to remove targeted undesirable knowledge from trained models without retraining from scratch while preserving general utility. When language models are deployed as agents, parameter-level unlearning becomes insufficient: forgotten information may still persist in memory artifacts or be reacquired via tool calls. In this paper, we formulate the problem of agent unlearning as removing private information from an LLM-based agent across three information channels: parametric knowledge, persistent memory, and tool-call acquisition. To study this problem, we introduce we introduce Agent Unlearning Benchmark (AUBench), a controlled benchmark that extends the fictitious-author setting of TOFU into an agent environment with persistent memory, an offline web snapshot, tool-accessible documents, and provenance-tracked interaction trajectories. By evaluating existing parameter-level and memory-level unlearning methods in this setting, AUBench reveals the gap between model-level forgetting and system-level agent unlearning, and provides a foundation for developing unlearning methods that operate across the full lifecycle of LLM agents.

Junkai Lin, Ge Xu, Zheng Xu +2

Institute of Automation, Chinese Academy of Sciences · The Chinese University of Hong Kong · Nanyang Technological University +2

Machine Unlearning LLM Unlearning Machine Unlearning Evaluation LLM Agent Memory

Ask PDF

Sep 4, 2026

Forgetting Without Restarting: Execution-State Unlearning for Stateful LLM Agents

Long-running LLM agents are stateful: beyond the transcript they accrete compressed summaries, plaintext memory, pending tool plans, and, under every serving API, a KV cache. Yet today's "forget" operations delete a plaintext memory record and stop, leaving every artifact derived from the revoked information intact. We formalize execution-state unlearning: after a forget request, the agent must behave as if it had never observed the target. Modeling the runtime as a deterministic transition system, we prove that the pre-target trajectory prefix is shared with this counterfactual world for free, that the post-target suffix is irreducibly tainted without token-level attribution, and that exact unlearning requires at least

T-τ+1

recomputed transitions, where

τ

is the target's injection step. Provenance-Guided Selective Replay attains this bound as a cross-layer contract spanning prompt, compressed memory, and cache: a provenance graph locates the injection point, checkpoint restoration reduces to cropping the KV cache, and sanitized replay regenerates the counterfactual suffix. Audited with elicitation, stochastic, and string-free behavioral tests across three agent suites, nine baselines, and three model families, memory deletion leaves leakage unchanged, instruction-based forgetting collapses under elicitation (Leak@probes = 1.00), and source redaction still acts on a revoked preference in 80% of episodes, while selective replay is indistinguishable from a full reset at up to 9x fewer recomputed tokens.

Chao Yao, Yangbo Wei, Zhen Huang +5

Arizona State University, USA · Eastern Institute of Technology, Ningbo, China

LLM Unlearning LLM Agent Memory Machine Unlearning

Ask PDF

Abstract

LLM Agent Memory

Explore similar work

MemCut: Every Secret Leaves a Trail in the Long-Term Memory of LLM Agents

Can Agents Truly Forget? Explore Machine Unlearning for LLM Agents

Forgetting Without Restarting: Execution-State Unlearning for Stateful LLM Agents