MemCut: Every Secret Leaves a Trail in the Long-Term Memory of LLM Agents

Cheng Zhang; Xuanning Cao; Zhicheng Zheng; Hui Xiong

cs.AIOct 15, 2026

MemCut: Every Secret Leaves a Trail in the Long-Term Memory of LLM Agents

Authors: Cheng Zhang, Xuanning Cao, Zhicheng Zheng, Hui Xiong

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · University of Science and Technology of China · The Hong Kong University of Science and Technology

Abstract

Long-term memory has become a crucial component of large language model (LLM) agents, with most research focusing on retrieving relevant information accurately and efficiently. However, a dependable memory system must also forget sensitive user data upon request. To evaluate this capability, we introduce MemCut, a systematic benchmark for user-requested forgetting. Its 456 hand-designed secrets in long-horizon conversations have fully annotated evidence trails to test recovery after deletion, including from messages that never state the secrets directly. We propose two recovery metrics. Native recovery measures reader success through the system's retrieval, whereas residual recovery measures the same reader's success when directly given surviving objects traced to annotated disclosures or evidence. We establish three reference deletion baselines to test how far existing retrievers and write models support forgetting across five memory designs and four LLMs. Disclosure removal reduces native recovery far more than residual recovery. LLM-guided deletion of ten additional turns from twenty retrieval candidates leaves 39-76% residual recovery. Subset sweeps (144 facts, three systems, two models) show that widening the pool at a fixed budget reduces residual recovery more than deleting all original candidates. These sweeps identify candidate coverage as a bottleneck in the reference baselines.

Figures & tables

Figure 1: The message that directly discloses the secret is deleted, but the secret remains recoverable from the surviving trail. The assistant can no longer answer a colleague, while ordinary messages that never state the secret still reveal it when combined.

Benchmark	Probe	Human design	LLM- free	All checked
LoCoMo	–	✗	✗	✓
LongMemEval-S	latest value	$\LEFTcircle$	✗	✓
MemoryAgentBench	latest value	✗	$\LEFTcircle$	✗
Memora	obsolete reuse	$\LEFTcircle$	✗	$\LEFTcircle$
ForgetEval	direct recall	$\LEFTcircle$	$\LEFTcircle$	$\LEFTcircle$
MemLeak	correlated cues	✗	✗	✗

Table 1: Data construction in benchmarks for long-term memory, forgetting, and unlearning. Probe : test after forgetting. Human design : facts designed by people. LLM-free : no fact invented by an LLM, which may only phrase text. All checked : every item checked by people.

\LEFTcircle

partly.

Figure 2: One MemCut sample, abridged. Top: weeks of ordinary conversation hold one disclosure (D), evidence that never states the fact (E1–E3), and look-alike records (L1–L4). Bottom: deletion removes the disclosure and the look-alikes that resemble it. Retrieval misses an evidence turn, so the system cannot answer. However, the stored evidence (E1–E3) together still gives the answer.

What each recovery metric needs	MemCut
Evidence is dispersed and redundant
sessions spanned by a fact’s evidence	8.0
evidence turns in the disclosure’s session	1.8%
minimal sufficient sets per fact	15.7
Evidence is harmless alone and complete
recovered from the best single evidence turn	0%

Table 2: Measurement guarantees of MemCut : conditions that make the recovery metrics interpretable, and the values that establish them (Qwen3.8-Flash as the reader).

			Overall			Fact assembly		Relation chain		Aggregation		Temporal state
	Model		Nat.	Res.	Hid.	Nat.	Res.	Nat.	Res.	Nat.	Res.	Nat.	Res.
Turn RAG	DeepSeek	S0	59.9	98.2	39.1	68.7	100.0	18.4	99.1	75.4	94.7	77.0	99.1
		S1	22.6	60.1	62.4	30.4	74.8	15.8	79.8	14.9	31.6	29.2	54.0
		S2	10.3	44.3	76.7	20.0	55.7	11.4	72.8	2.6	16.7	7.1	31.9
	Qwen	S0	56.6	97.8	42.2	60.9	100.0	24.6	100.0	45.6	91.2	95.6	100.0
		S1	19.1	58.6	67.4	22.6	46.1	7.9	74.6	22.8	38.6	23.0	75.2

Table 3: Recovery (%) after deletion in the two retrieval stores under their respective retrieval settings; all 456 facts. Nat. : through the system’s own retrieval. Res. (shaded by value): from the surviving evidence-derived memory. Hid. : hidden-risk share,

(\mathrm{Res}-\mathrm{Nat})/\mathrm{Res}

. S0 deletes the disclosure; S1 and S2 delete ten more turns.

			Overall			Fact assembly		Relation chain		Aggregation		Temporal state
	Model		Nat.	Res.	Hid.	Nat.	Res.	Nat.	Res.	Nat.	Res.	Nat.	Res.
MemoryBank	DeepSeek	S0	40.1	93.0	56.8	53.0	100.0	7.0	95.6	45.6	79.8	54.9	96.5
		S1	23.0	80.9	71.5	32.2	92.2	5.3	88.6	20.2	58.8	34.5	84.1
		S2	15.8	71.7	78.0	23.5	84.3	6.1	86.8	7.0	37.7	26.5	77.9
	Qwen	S0	41.9	95.8	56.3	53.9	97.4	7.9	96.5	50.9	89.5	54.9	100.0
		S1	21.7	80.5	73.0	28.7	88.7	5.3	92.1	17.5	57.9	35.4	83.2

Table 4: Recovery (%) after deletion in the three memory systems, as in Table 3 ; all 456 facts.

Figure 3: Why cleanup fails. (a, b) Residual recovery as the candidate pool widens (budget ten) and as the budget grows (pool twenty); dashed vertical lines mark the main setting. Solid lines: LLM-guided cleanup (S2); dashed: retrieval-ranked cleanup (S1); each line is the mean of DeepSeek and Qwen, with a band for their range, on the 144 facts of EP01–05 (Appendix B.3 ). (c) Native recovery after S0 against residual recovery after cleanup, per type and system; dark marks are means over the four write models, light marks are single models. (d) Hidden-risk share after cleanup by type, systems ordered by derivation strength.

Figure 4: A relation-chain fact under LLM-guided cleanup (Turn RAG, S2). (a) Recovery needs one evidence turn per link, so emptying any link makes it underivable. (b) Similarity ranks of evidence and disclosure (#7, deleted by S0), and what S2 deletes (struck through) from pools of 20, 30, and 50; E2 enters at 30 but is kept (circled).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

		Derivation type
Per fact	Overall	Fact assembly	Relation chain	Aggregation	Temporal state
Protected facts	456	115	114	114	113
Evidence
Evidence turns per fact	8.1	7.9	8.3	8.1	8.2
Sessions they span	8.0	7.8	8.2	8.1	8.0
Span in days (median)	29	31	31	28	25

Appendix

Table 5: Per-fact statistics of MemCut , overall and by derivation type (means unless noted). Share in disclosure’s session : facts with at least one evidence turn in the disclosure’s session. Sufficient sets and minimal cuts are author-identified and cross-checked by two independent algorithms. Every fact can be cut with at most five deletions, half the ten-turn budget; relation chains (highlighted) need the fewest. No evidence turn is shared across facts, and 62.6% of turns carry no annotation.

EP	Turns	Tokens	Sess.	Tok./ turn	Turns/ sess.	Days	Facts
EP01	2,136	103,805	26	48.6	82.2	30	28
EP02	2,152	77,846	28	36.2	76.9	27	28
EP03	2,200	85,815	28	39.0	78.6	27	28
EP04	2,200	70,715	29	32.1	75.9	28	27
EP05	2,738	99,776	33	36.4	83.0	32	33
EP06	2,798	100,768	35	36.0	79.9	34	32

Appendix

Table 6: Per-episode statistics: turns, tokens ( o200k_base ), sessions, tokens per turn, turns per session, days spanned, and protected facts.

	DeepSeek		Qwen		GLM		MiMo
System	Nat.	Res.	Nat.	Res.	Nat.	Res.	Nat.	Res.
Retrieval stores
Turn RAG	95.0	100.0	90.8	100.0	92.1	100.0	92.8	100.0
Session RAG	62.5	100.0	48.2	100.0	52.9	100.0	69.7	100.0
Memory systems
MemoryBank	89.9	99.8	89.0	99.8	90.6	99.8	89.5	99.3

Appendix

Table 7: Recovery (%) before deletion. Every store supports nearly every fact ( Res. ), and retrieval reveals most of them ( Nat. ) in every system but Session RAG.

	DeepSeek				Qwen				GLM				MiMo
System	Full	S0	S1	S2	Full	S0	S1	S2	Full	S0	S1	S2	Full	S0	S1	S2
Retrieval stores
Turn RAG	5.0	39.1	62.4	76.7	9.2	42.2	67.4	82.0	7.9	39.5	65.6	78.5	7.2	38.7	63.7	76.4
Session RAG	37.5	75.3	77.6	82.6	51.8	79.7	78.1	83.3	47.1	82.4	84.2	92.3	30.3	71.0	86.1	85.4
Memory systems
MemoryBank	9.9	56.8	71.5	78.0	10.8	56.3	73.0	81.6	9.2	55.6	73.8	81.9	9.9	54.8	71.3	78.1

Appendix

Table 8: Hidden-risk share (%), computed as

(\mathrm{Res}-\mathrm{Nat})/\mathrm{Res}

at every deletion level. It is a lower bound on the share of residual successes that native recovery misses (Appendix B.1 ). Shading is proportional to the value. Full is the store before any deletion. The share rises from Full to S2 in all 20 configurations.

	Turn RAG				MemoryBank				LightMem
	DeepSeek		Qwen		DeepSeek		Qwen		DeepSeek		Qwen
	S1	S2	S1	S2	S1	S2	S1	S2	S1	S2	S1	S2
Widening the candidate pool (budget $=10$ ; first column: pool size)
10	66.7	66.7	63.2	62.9	80.6	80.6	80.6	80.6	63.2	63.4	57.6	56.1
20 ⋆	66.7	52.1	63.2	47.9	80.6	72.2	80.6	72.2	62.5	49.3	52.1	47.2
30	66.7	47.2	63.2	44.1	80.6	68.1	80.6	58.7	62.5	44.9	52.1	39.3

Appendix

Table 9: Residual recovery (%) behind Figure 3 a–b: retrieval-ranked (S1) and LLM-guided (S2) cleanup as the candidate pool widens with the budget fixed at ten turns (top), and as the budget grows with the pool fixed at twenty (bottom). The sweeps run on the 144 facts of EP01–05 (Appendix B.3 ), so values at the main-table setting ( ⋆ ) differ from Tables 3 and 4 , which cover all 456. S2 is shaded in proportion to its value. Percentages are over completed runs: 98.9% of those planned, and at least 114 of the 144 facts in every cell.

System		Assem.	Chain	Aggr.	Temp.	All
MemoryBank	S1	2.2	1.2	2.9	2.3	2.2
	S2	3.0	1.7	3.7	3.2	2.9
Nemori	S1	2.9	1.8	3.5	2.9	2.8
	S2	3.6	2.4	3.9	3.5	3.4
LightMem	S1	3.9	2.4	5.2	4.2	3.9
	S2	4.8	2.9	6.0	5.1	4.7

Appendix

Table 10: Annotated evidence turns among the ten turns that cleanup deletes per fact, by system and derivation type ( Assem. : fact assembly; Chain : relation chain; Aggr. : aggregation; Temp. : temporal state), over all 456 facts and the DeepSeek and Qwen write models.

Pool	Selected	Off-target	Res. (%)
20	10	5.54	52.08
20	20	15.49	50.69
60	10	4.50	39.58

Appendix

Table 11: Deletion cost and residual recovery for DeepSeek–Turn RAG, S2, on the same 144 facts, all completed. Pool, selected, and off-target counts are in source turns. Selected and off-target counts exclude disclosure removal and are means per fact.

	Derivation type
	Fact	Rel.	Aggr.	Temp.
Recovery, mean %
Native before deletion	91.9	81.1	90.4	96.1
Native after S0 (visible)	73.8	18.2	60.9	76.6
Residual lost at S0	$-$ 0.1	2.3	14.7	0.9
Residual recovery after S2	60.2	71.8	24.3	58.4

Appendix

Table 12: Leakage by derivation type: means over the 16 configurations outside Session RAG (four write models, four systems), with the hidden-risk share computed from the mean recovery values. LLM judgment : residual recovery lowered by S2 relative to S1. Annotated structure from Table 5 . Fact: fact assembly; Rel.: relation chain; Aggr.: aggregation; Temp.: temporal state. Bold marks the type that stands out in each row.

		Full		S0		S1		S2
Type	System	Nat.	Res.	Nat.	Res.	Nat.	Res.	Nat.	Res.
DeepSeek
Fact assembly	MemoryBank	85.2	100.0	53.0	100.0	32.2	92.2	23.5	84.3
	Nemori	94.8	98.3	81.7	97.4	65.2	84.3	50.4	63.5
	LightMem	97.4	100.0	93.0	99.1	53.9	62.6	40.0	51.3
Relation chain	MemoryBank	84.2	100.0	7.0	95.6	5.3	88.6	6.1	86.8

Appendix

Table 13: Recovery (%) by derivation type in the three memory systems with DeepSeek and Qwen, adding the pre-deletion baseline (Full) that Table 4 omits (113–115 facts per type). Res. is shaded in proportion to its value, as in Table 3 .

LLM Agent Memory

Large language model agent memory lets agents retain, organize, retrieve, and update information across interactions and long-running tasks.

See all 373 papers

Can Agents Forget When Asked? A Benchmark for Memory Unlearning in LLM Agents
Oct 13, 2026Bofan Xie, Chaokun Wang, Guanchen Ge +4LLM Agent Memory Memory Agent Benchmarks
Tsinghua University · Sina Weibo
Forgetting Without Restarting: Execution-State Unlearning for Stateful LLM Agents
Sep 4, 2026Chao Yao, Yangbo Wei, Zhen Huang +5LLM Unlearning LLM Agent Memory
Arizona State University, USA · Eastern Institute of Technology, Ningbo, China
Eternal Sunshine of the Spotless Mind: Systematically Erasing LLM's Memories
Sep 29, 2026Olga OhrimenkoLLM Unlearning Persistent Memory for Language Models
School of Computing and Information Systems The University of Melbourne

Oct 13, 2026J/K move · Enter open · S save

Can Agents Forget When Asked? A Benchmark for Memory Unlearning in LLM Agents

Bofan Xie, Chaokun Wang, Guanchen Ge, Songyao Wang+3

Tsinghua University · Sina Weibo

LLM Agent Memory Memory Agent Benchmarks LLM Unlearning Machine Unlearning Machine Unlearning Evaluation

Open paper AskPDF arXiv

Long-term memory has become a core component of large language model (LLM) agents. As agents accumulate personal information over long-term use, users increasingly ask them to delete sensitive or outdated facts. A memory system should honor such requests. It should remove exactly what was requested and keep the rest of its memory intact. However, existing memory benchmarks evaluate only memory writing, retrieval, and organization, so whether agents can forget at the user's explicit request remains under-evaluated. To address this gap, we propose Bemula, the first benchmark dedicated to memory unlearning. It comprises 50 multi-session dialogue instances, 135 forget requests of six types, and 1,742 test questions. Each dialogue has a control version without the requests, so that only systems that have stored the target can earn credit for forgetting. Across the paired versions, five test suites measure storage, removal of the requested information, preservation of unrelated memories, relearning of similar facts, and resistance to extraction attacks. On Bemula, we evaluate six mainstream memory systems under three backbone LLMs, together with the Full Context LLM on two of them. The results are negative. Across all evaluated systems and backbones, no system passes more than 1.5% of forget requests end to end, and most pass none at all. These systems either leave the requested information extractable under simple attacks or delete far more than requested. Memory unlearning therefore remains an open challenge for agent memory design. We release our data and code at https://github.com/Nboxff/bemula-benchmark.

Abstract

LLM Agent Memory

Explore similar work

Can Agents Forget When Asked? A Benchmark for Memory Unlearning in LLM Agents

Forgetting Without Restarting: Execution-State Unlearning for Stateful LLM Agents

Eternal Sunshine of the Spotless Mind: Systematically Erasing LLM's Memories