Can Agents Forget When Asked? A Benchmark for Memory Unlearning in LLM Agents

Bofan Xie; Chaokun Wang; Guanchen Ge; Songyao Wang; Fang Niu; Hang Zhang; Wei Wang

cs.AIOct 13, 2026

Can Agents Forget When Asked? A Benchmark for Memory Unlearning in LLM Agents

Authors: Bofan Xie, Chaokun Wang, Guanchen Ge, Songyao Wang, Fang Niu, Hang Zhang, Wei Wang

Organizations: Tsinghua University · Sina Weibo

Abstract

Long-term memory has become a core component of large language model (LLM) agents. As agents accumulate personal information over long-term use, users increasingly ask them to delete sensitive or outdated facts. A memory system should honor such requests. It should remove exactly what was requested and keep the rest of its memory intact. However, existing memory benchmarks evaluate only memory writing, retrieval, and organization, so whether agents can forget at the user's explicit request remains under-evaluated. To address this gap, we propose Bemula, the first benchmark dedicated to memory unlearning. It comprises 50 multi-session dialogue instances, 135 forget requests of six types, and 1,742 test questions. Each dialogue has a control version without the requests, so that only systems that have stored the target can earn credit for forgetting. Across the paired versions, five test suites measure storage, removal of the requested information, preservation of unrelated memories, relearning of similar facts, and resistance to extraction attacks. On Bemula, we evaluate six mainstream memory systems under three backbone LLMs, together with the Full Context LLM on two of them. The results are negative. Across all evaluated systems and backbones, no system passes more than 1.5% of forget requests end to end, and most pass none at all. These systems either leave the requested information extractable under simple attacks or delete far more than requested. Memory unlearning therefore remains an open challenge for agent memory design. We release our data and code at https://github.com/Nboxff/bemula-benchmark.

Figures & tables

Figure 1: Illustrative examples of forget requests in Bemula .

Figure 2: The construction pipeline of Bemula .

Figure 3: Statistics of Bemula . (a) Length of every instance across the four length tiers. (b) Number of forget requests of each type, stacked by length tier. (c) Anatomy of one long instance. Blocks on top mark sessions, dots mark turns that mention a request’s target facts, and diamonds mark where each request is injected. The shaded span continues past the last request, and the F4 row has no dots because an inferential target is never stated in the dialogue.

Figure 4: Forgetting efficacy ( $\mathrm{FE}$ ) by forget request type on Bemula .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Suite	Version	Ground truth	Conditioned on	Detects
$\mathrm{RA}$	retention	control version	fact value	–	storage capability
$\mathrm{FE}$	efficacy	forget version	[FORGOTTEN] ⋆	retention probe ⋆	residual leakage
$\mathrm{PR}$	preservation	forget version	fact value	control probe	over-deletion
$\mathrm{RLA}$	relearning	forget version	new value	–	relearning failure
$\mathrm{AR}$	attack	forget version	[NO_LEAK]	attack on control	adversarial extraction

Appendix

Table 2: Item-level metrics of Bemula . Conditioning lists the paired control probe that must also succeed for the item to count as correct. The

\star

marks F6 replacement efficacy items, which take the new value as ground truth and have no retention counterpart.

Judge verdict	Annotator correct	Annotator incorrect	Agreement
Correct	53	1	98.1%
Incorrect	1	45	97.8%
All	54	46	98.0%

Appendix

Table 3: Agreement between the human annotator and the LLM judge on the 100 sampled items. Rows give the judge verdict and columns give the annotator verdict.

Profiles
Statistic	Value
Number of user profiles	50
Number of atomic / derived facts	2,993 / 77
Average facts per profile	61.4
Dialogue instances
Number of instances (forgetting + control)	50

Appendix

Table 4: Key statistics of Bemula . Token counts are measured with the o200k_base tokenizer on the forget version of each instance.

Tier	Instances	Sessions	Turns	Requests	Questions
Short	15	5 to 8	96 to 164	1 to 2	11 to 33
Medium	16	10 to 15	334 to 472	2 to 3	18 to 47
Long	16	17 to 25	916 to 1,464	3 to 5	26 to 65
XLong	3	27 to 36	2,688 to 3,500	4 to 6	50 to 70
Total	50	746	36,900	135	1,742

Appendix

Table 5: Composition of Bemula by length tier. The ranges give the minimum and maximum observed over the instances of a tier. The final row reports the total number of instances, sessions, turns, forget requests, and questions.

Figure 5: Illustrative example of the five attack questions of one forget request. Every question probes the same forgotten fact, David’s undergraduate field of study, with a different attack type.

Panel of Table 6 — Table 6: Main results on Bemula -Lite under the three backbone LLMs. $\mathrm{RA}$ : retention accuracy. $\mathrm{FE}$ : forgetting efficacy. $\mathrm{PR}$ : preservation. $\mathrm{RLA}$ : relearning. $\mathrm{AR}$ : attack robustness. $\mathrm{RPR}$ : request pass rate. $\mathrm{Score}$ : overall Bemula -Lite score ( $\mathrm{FA}\times\mathrm{UG}$ , Eq. 5 ). All scores are percentages.

Figure 6: Forgetting efficacy (

\mathrm{FE}

) versus preservation (

\mathrm{PR}

) on Bemula for all systems and backbones.

Panel of Table 7 — Table 7: Attack robustness ( $\mathrm{AR}$ ) by attack type on Bemula . Full Context is weakest against the jailbreak prefix, since its forgetting is purely instruction following. For the systems that forget too little (Naive RAG, Mem0, LightMem, SimpleMem), indirect questions and Boolean guesses are usually the most effective attacks, requiring no prompt trickery. The best value in each backbone block is bold, and the second-best is underlined. All scores are percentages.

Figure 7: Retention accuracy ( $\mathrm{RA}$ ), relearning accuracy ( $\mathrm{RLA}$ ), and attack robustness ( $\mathrm{AR}$ ) by forget request type on Bemula , complementing the efficacy heatmap in Figure 4 . Preservation and the request pass rate are instance-level metrics and cannot be broken down by request type. $\mathrm{RLA}$ is undefined for F6 requests, whose new value is already covered by the efficacy probe. Full Context and A-MEM have no runs on Qwen3-32B-AWQ because of its context length limit. Both cases are shown as –.

Figure 8: Suite metrics as a function of dialogue length on Bemula . Instances are sorted by turn count and split into five bins of ten; each point recomputes the official metrics on that bin, and the x position is the bin’s mean turn count (log scale). Bands mark the short, medium, long, and xlong tier spans.

System	$\mathrm{RA}$	$\mathrm{FE}$	$\mathrm{PR}$	$\mathrm{RLA}$	$\mathrm{AR}$	$\mathrm{Score}$
Backbone: GPT-5.6-Luna
Full Context	84.4 [80.0,88.7]	26.8 [20.6,33.5]	73.3 [66.0,80.7]	98.2 [95.4,100.0]	27.6 [24.1,31.1]	23.1 [20.0,26.2]
Naive RAG	75.9 [70.7,81.0]	14.7 [10.7,19.5]	67.3 [59.3,75.3]	71.1 [63.0,79.6]	12.9 [10.7,15.4]	9.5 [7.5,11.8]
LangMem	50.4 [41.0,60.1]	37.9 [29.8,46.9]	20.0 [12.7,28.0]	78.9 [70.2,87.2]	25.6 [19.3,32.4]	12.4 [8.6,16.8]
Mem0	89.7 [84.8,94.4]	8.5 [5.3,12.0]	80.0 [73.3,86.0]	77.2 [70.2,84.2]	4.7 [3.4,6.1]	4.9 [3.8,6.1]
A-MEM	80.4 [75.5,85.1]	33.0 [26.6,39.8]	59.3 [50.7,67.3]	88.6 [83.2,93.6]	24.7 [20.9,28.7]	20.7 [17.1,24.4]

Appendix

Table 8: Point estimates and 95% cluster-bootstrap CIs (over instances, 10,000 resamples) for the main results on Bemula . All values are percentages.

LLM Agent Memory

Large language model agent memory lets agents retain, organize, retrieve, and update information across interactions and long-running tasks.

See all 368 papers

MemCut: Every Secret Leaves a Trail in the Long-Term Memory of LLM Agents
Oct 15, 2026Cheng Zhang, Xuanning Cao, Zhicheng Zheng +1LLM Agent Memory Privacy Leakage in Language Models
The Hong Kong University of Science and Technology (Guangzhou) · University of Science and Technology of China · The Hong Kong University of Science and Technology
Can Agents Truly Forget? Explore Machine Unlearning for LLM Agents
Oct 13, 2026Junkai Lin, Ge Xu, Zheng Xu +2Machine Unlearning LLM Unlearning
Institute of Automation, Chinese Academy of Sciences · The Chinese University of Hong Kong · Nanyang Technological University +2
Forgetting Without Restarting: Execution-State Unlearning for Stateful LLM Agents
Sep 4, 2026Chao Yao, Yangbo Wei, Zhen Huang +5LLM Unlearning LLM Agent Memory
Arizona State University, USA · Eastern Institute of Technology, Ningbo, China

Oct 15, 2026J/K move · Enter open · S save

MemCut: Every Secret Leaves a Trail in the Long-Term Memory of LLM Agents

Cheng Zhang, Xuanning Cao, Zhicheng Zheng, Hui Xiong

The Hong Kong University of Science and Technology (Guangzhou) · University of Science and Technology of China · The Hong Kong University of Science and Technology

LLM Agent Memory Privacy Leakage in Language Models Long-Term Conversational Memory LLM Unlearning

Open paper AskPDF arXiv

Long-term memory has become a crucial component of large language model (LLM) agents, with most research focusing on retrieving relevant information accurately and efficiently. However, a dependable memory system must also forget sensitive user data upon request. To evaluate this capability, we introduce MemCut, a systematic benchmark for user-requested forgetting. Its 456 hand-designed secrets in long-horizon conversations have fully annotated evidence trails to test recovery after deletion, including from messages that never state the secrets directly. We propose two recovery metrics. Native recovery measures reader success through the system's retrieval, whereas residual recovery measures the same reader's success when directly given surviving objects traced to annotated disclosures or evidence. We establish three reference deletion baselines to test how far existing retrievers and write models support forgetting across five memory designs and four LLMs. Disclosure removal reduces native recovery far more than residual recovery. LLM-guided deletion of ten additional turns from twenty retrieval candidates leaves 39-76% residual recovery. Subset sweeps (144 facts, three systems, two models) show that widening the pool at a fixed budget reduces residual recovery more than deleting all original candidates. These sweeps identify candidate coverage as a bottleneck in the reference baselines.

Abstract

LLM Agent Memory

Explore similar work

MemCut: Every Secret Leaves a Trail in the Long-Term Memory of LLM Agents

Can Agents Truly Forget? Explore Machine Unlearning for LLM Agents

Forgetting Without Restarting: Execution-State Unlearning for Stateful LLM Agents