From Anchors to Supervision: Memory-Graph Guided Corpus-Free Unlearning for Large Language Models

Wenxuan Li; Zhenfei Zhang; Mi Zhang; Geng Hong; Mi Wen; Xiaoyu You; Min Yang

cs.CLApr 15, 2026

From Anchors to Supervision: Memory-Graph Guided Corpus-Free Unlearning for Large Language Models

Authors: Wenxuan Li, Zhenfei Zhang, Mi Zhang, Geng Hong, Mi Wen, Xiaoyu You, Min Yang

Organizations: College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China · Shanghai University of Electric Power, Shanghai, China · School of Information Science and Engineering, East China University of Science and Technology, Shanghai, China

Abstract

Large language models (LLMs) may memorize sensitive or copyrighted content, raising significant privacy and legal concerns. While machine unlearning has emerged as a potential remedy, prevailing paradigms rely on user-provided forget sets, making unlearning requests difficult to audit and exposing systems to secondary leakage and malicious abuse. We propose MAGE, a Memory-grAph Guided Erasure framework for user-minimized, corpus-free unlearning. Given only a lightweight user anchor that identifies a target entity, MAGE probes the target LLM to recover target-related memorization, organizes it into a weighted local memory graph, and synthesizes scoped supervision for unlearning. MAGE is model-agnostic, can be plugged into standard unlearning methods, and requires no access to the original training corpus. Experiments on two benchmarks, TOFU and RWKU, demonstrate that MAGE's self-generated supervision achieves effective unlearning performance comparable to supervision generated with external reference, while preserving overall utility. These results support a practical and auditable unlearning workflow driven by minimal anchors rather than user-supplied forget corpora.

Figures & tables

Figure 1: Risk surface of externally supplied forget sets in existing LLM unlearning paradigms. In contrast, a user-minimized paradigm mitigates these risks by improving information protection and auditability.

Figure 2: The framework of MAGE. Internal Memory Mining : Given an unlearning request, MAGEmines target memorization to build a strength-weighted local memory graph via iterative expansion. Scoped Supervision Construction : It then performs strength-weighted path sampling and edge-to-event synthesis to generate compact forget and neighbor supervision for downstream unlearning.

Prior	RWKU	ELUDe	DirectQA	Ours
External Data (Wiki)	✓	✓	✗	✗
Extra LLM / Human	✓	✓	✗	✗
Correctness Check	✓	✗	✓	✗

Table 1: Priors and capabilities of forget-set generation methods. ✓ denotes used; ✗ denotes not used.

Finetune	Method	Forget Set $\downarrow$				Neighbor Set $\uparrow$			MIA Set		Utility Set $\uparrow$
		FB	QA	AA	All	FB	QA	All	FM $\uparrow$	RM $\downarrow$	MMLU	BBH	TruthfulQA	TriviaQA	AlpacaEval
Before		0.6788	0.6721	0.6829	0.6779	0.6306	0.7484	0.6895	1.9104	2.0433	0.4240	0.2654	0.2875	0.4056	6.2993
GA	RWKU	0.5817	0.5511	0.6536	0.5955	0.5600	0.7153	0.6376	3.1075	2.4128	0.3969	0.2346	0.3025	0.3675	6.3109
	ELUDe	0.6440	0.4646	0.6026	0.5704	0.6135	0.6944	0.6540	1.9933	2.1057	0.4167	0.2438	0.3175	0.3232	6.3341
	DirectQA	0.6531	0.6675	0.7030	0.6745	0.6353	0.6979	0.6666	1.9286	2.0572	0.4203	0.1975	0.2900	0.4640	6.2686
	MAGE	0.6373	0.4288	0.5438	0.5366	0.6263	0.6501	0.6382	1.9885	2.1016	0.4130	0.2515	0.3050	0.2955	6.3446

Table 2: Unlearning performance with different fine-tuning strategies and forget-set generation methods on Llama-2-7b-chat. Bold denotes the best result and underlined denotes the second best. Arrows indicate the direction of preference (

\downarrow

lower is better;

\uparrow

higher is better).

Baseline	Generation Method	Forget				Utility
		Prob $\downarrow$	ROUGE $\downarrow$	TruthRatio $\uparrow$	Forget Q. $\uparrow$	RS Score $\uparrow$	RAS Score $\uparrow$	WFS Score $\uparrow$	Model Utility $\uparrow$
Before		0.9951	0.9493	0.5306	0.0013	0.7133	0.1667	0.0508	0.6277
GA	TOFU	0.0353	0.0714	0.5903	0.1650	0.1906	0.1922	0.0725	0.3309
	DirectQA	0.4260	0.4521	0.5029	0.0286	0.5378	0.0830	0.0212	0.5783
	MAGE	0.1492	0.4500	0.5285	0.0541	0.4449	0.0046	0.0024	0.5155
NPO	TOFU	0.0359	0.0338	0.6272	0.1650	0.1638	0.2402	0.0948	0.2951

Table 3: Unlearning performance on TOFU-finetuned Llama-2-7b-chat. (↓ lower is better; ↑ higher is better)

Figure 3: Average entity overlap (Top-50 Jaccard) and frequency-distribution similarity across RWKU.

Attributes	Freq	DirectQA		MAGE
Attributes	Freq	Covered	F1	Covered	F1
BirthPlace	2	✓	1.00	✓	0.80
Gender	1		0		0
BirthYear	1		0	✓	0.67
Genre	3	✓	0	✓	0.67
Awards	1		0	✓	1.00

Table 4: Comparison of recovered attributes against TOFU ground-truth profiles.

Figure 4: Impact of the proportion of correct statements in the forget set on unlearning performance.

		Forget-Avg	Neighbor-Avg
GA+GD	Wiki	0.4834	0.5644
GA+GD	Neighbor	0.4861	0.5835
GA+KL	Wiki	0.5353	0.6030
GA+KL	Neighbor	0.5407	0.6157

Table 5: Impact of the Neighbor Set on Unlearning Performance

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Finetune	Method	Forget Set $\downarrow$				Neighbor Set $\uparrow$			MIA Set		Utility Set $\uparrow$
		FB	QA	AA	All	FB	QA	All	FM $\uparrow$	RM $\downarrow$	MMLU	BBH	TruthfulQA	TriviaQA	AlpacaEval
Before		0.6729	0.7290	0.6839	0.6953	0.6251	0.5819	0.6035	1.8448	1.9326	0.6864	0.4090	0.3775	0.3985	6.0196
GA	RWKU	0.6597	0.7404	0.6804	0.6935	0.6227	0.5692	0.5959	1.8645	1.9411	0.6879	0.3133	0.3800	0.4077	6.0383
	ELUDe	0.6694	0.6799	0.6625	0.6706	0.6113	0.5793	0.5953	1.8567	1.9383	0.6857	0.3148	0.3925	0.2670	6.1704
	DirectQA	0.6675	0.6541	0.6296	0.6504	0.6377	0.5534	0.5956	2.0591	2.0101	0.6784	0.4336	0.3775	0.3506	6.0679
	MAGE	0.6236	0.6724	0.6355	0.6438	0.6083	0.5590	0.5837	1.8663	1.9439	0.6842	0.4383	0.3975	0.2460	6.1535

Appendix

Table 6: Unlearning performance with different fine-tuning strategies and forget-set generation methods on Phi-3-mini-4k-instruct. Bold denotes the best result and underlined denotes the second best. Arrows indicate the direction of preference (

\downarrow

lower is better;

\uparrow

higher is better).

Baseline	Generation Method	Forget				Utility
		Prob $\downarrow$	ROUGE $\downarrow$	TruthRatio $\uparrow$	Forget Q. $\uparrow$	RS Score $\uparrow$	RAS Score $\uparrow$	WFS Score $\uparrow$	Model Utility $\uparrow$
Before		0.9951	0.9493	0.5306	0.0013	0.7133	0.1667	0.0508	0.6277
GA+GD	TOFU	0.1031	0.1631	0.7365	0.2657	0.1834	0.5251	0.3904	0.3497
	DirectQA	0.9843	0.9198	0.5395	0.0068	0.7015	0.1852	0.0616	0.6333
	MAGE	0.1608	0.4416	0.5283	0.1650	0.4748	0.0143	0.0048	0.5436
GA+KL	TOFU	0.4709	0.4409	0.5644	0.0541	0.5923	0.1617	0.0516	0.5943

Appendix

Table 7: Unlearning performance(GA+GD, GA+KL) on TOFU-finetuned Llama-2-7b-chat. (↓ lower is better; ↑ higher is better)

Baseline	Generation Method	Forget				Utility
		Prob $\downarrow$	ROUGE $\downarrow$	TruthRatio $\uparrow$	Forget Q. $\uparrow$	RS Score $\uparrow$	RAS Score $\uparrow$	WFS Score $\uparrow$	Model Utility $\uparrow$
Before		0.7742	0.6469	0.5077	0.0068	0.5664	0.2048	0.1236	0.5735
GA	TOFU	0.2995	0.3632	0.6180	0.2657	0.4173	0.5867	0.5741	0.5222
	DirectQA	0.5684	0.4337	0.5530	0.0143	0.5033	0.2418	0.1948	0.5586
	MAGE	0.2799	0.5457	0.4635	0.0286	0.4531	0.0059	0.0028	0.4948
NPO	TOFU	0.2197	0.3577	0.6328	0.0971	0.3290	0.1771	0.0832	0.4371

Appendix

Table 8: Unlearning performance with different forget set generation methods on TOFU using a TOFU-finetuned Llama-3.2-1B-Instruct model. (↓ lower is better; ↑ higher is better)

Attributes	Freq	Gold	DirectQA			MAGE
	Freq		Pred	F1	Covered	Pred	F1	Covered
BirthPlace	2	kuwait city, kuwait	kuwait city, kuwait;	1.00	✓	kuwait city;	0.80	✓
Gender	1	male		0			0
BirthYear	1	1956		0		1931; 1956;	0.67	✓
Genre	3	french literature	Arabic culture and Middle Eastern landscapes;	0	✓	french literature; literary fiction;	0.67	✓
Awards	1	prix goncourt		0		prix goncourt;	1.00	✓

Appendix

Table 9: Qualitative comparison of recovered attributes against TOFU ground-truth profiles.

Figure 5: Memory strength variation of 1-hop neighbors across models.

Figure 6: Impact of Memory Graph Coverage on Unlearning Performance.

LLM Unlearning

Large language model unlearning studies how to remove specified knowledge or behaviors while preserving a model’s unrelated capabilities.

See all 191 papers

Less is More: Geometric Unlearning for LLMs with Minimal Data Disclosure
May 3, 2026Chenchen Tan, Xinghao Li, Shujie Cui +3Privacy-Preserving Language Models Machine Unlearning
Faculty of Information Technology, Monash University, Clayton, Victoria, Australia. · Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center, Qilu University of Technology (Shandong Academy of Sciences), Jinan, China. · Shandong Provincial Key Laboratory of Computing Power Internet and Service Computing, Shandong Fundamental Research Center for Computer Science, Jinan, China.
Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs
Sep 9, 2026Ravi Ranjan, Olivera Kotevska, Agoritsa PolyzouMemorization in Language Models Language Model Robustness
Florida International University (FIU), Miami, FL, USA · Oak Ridge National Laboratory (ORNL), Oak Ridge, TN, USA
ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models
May 16, 2026Yujie Lin, Chengyi Yang, Zhishang Xiang +2Privacy Leakage in Language Models Model Editing
School of Informatics, Xiamen University, China · Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China · School of Film, Xiamen University, China +2

May 3, 2026J/K move · Enter open · S save

Less is More: Geometric Unlearning for LLMs with Minimal Data Disclosure

Chenchen Tan, Xinghao Li, Shujie Cui, Youyang Qu+2

Faculty of Information Technology, Monash University, Clayton, Victoria, Australia. · Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center, Qilu University of Technology (Shandong Academy of Sciences), Jinan, China. · Shandong Provincial Key Laboratory of Computing Power Internet and Service Computing, Shandong Fundamental Research Center for Computer Science, Jinan, China.

Privacy-Preserving Language Models Machine Unlearning LLM Unlearning

Open paper AskPDF arXiv

As large language models (LLMs) are increasingly deployed in real-world systems, they must support post-hoc removal of specific content to meet privacy and governance requirements. This motivates selective unlearning, which suppresses information about a particular entity or topic while preserving the LLM's general utility. However, most existing LLM unlearning methods require access to the original training corpus and rely on output-level refusal tuning or broad gradient updates, creating a tension among unlearning strength, non-target preservation, and data availability. We propose Geometric Unlearning (GU), an approach that operates directly on the model's prompt-conditioned hidden states without access to the original training corpus. Specifically, GU distills a compact, low-rank safe-behavior subspace from a small set of safe reference prompts and uses lightweight anchor-in-context synthetic prompts to trigger localized, projection-based alignment of hidden representations to this safe subspace. A teacher-distillation regularizer on synthetic non-target anchors further reduces collateral drift. Across privacy-oriented unlearning benchmarks (ToFU and UnlearnPII), GU achieves strong target suppression with minimal impact on non-target performance, demonstrating that effective unlearning can be achieved with minimal synthetic data.

Abstract

LLM Unlearning

Explore similar work

Less is More: Geometric Unlearning for LLMs with Minimal Data Disclosure

Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs

ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models