When Forgotten Facts Return: Relearning-Aware Unlearning of Knowledge Graph Embeddings via Tangent Kernels

Parnian Fazel; Mohammad M Maheri; Hamed Haddadi

cs.LGOct 22, 2026

When Forgotten Facts Return: Relearning-Aware Unlearning of Knowledge Graph Embeddings via Tangent Kernels

Authors: Parnian Fazel, Mohammad M Maheri, Hamed Haddadi

Organizations: Imperial College London · University of Oxford

Abstract

Approximate unlearning answers a deletion request on a knowledge-graph embedding by editing entity embeddings instead of retraining from scratch. These embeddings keep being updated as new facts arrive. Under this ordinary continued training, forgotten facts recover more than in the exact-unlearning baseline, a model retrained without them and given the same continued training. We trace this relearning to entity embeddings shared by forgotten and new facts. To our knowledge, this is the first study of relearning in knowledge-graph embeddings. Relearning-robust methods developed for deep networks do not account for the graph structure underlying this recovery. We propose Kernel-Anticipated Unlearning (KAU), which uses the tangent kernel to model how continued training changes forgotten facts' scores through these shared parameters. KAU expresses this first-order response in closed form and differentiates through it inside the forgetting loss. The objective steers edits toward directions that continued training reverses less. We compare KAU with state-of-the-art knowledge-graph unlearners and relearning-robust methods under identical continued training, measuring leakage beyond the exact-unlearning baseline. Across three backbones, KAU reduces the strongest knowledge-graph unlearner's leakage by 46-63% on FB15k-237 and 50-82% on CoDEx-L, at higher retained utility.

Figures & tables

		TransE				TransH				RotatE
	Method	ExRec $\downarrow$	Leak $\downarrow$	RER $\downarrow$	$M_{R}\!\uparrow$	ExRec $\downarrow$	Leak $\downarrow$	RER $\downarrow$	$M_{R}\!\uparrow$	ExRec $\downarrow$	Leak $\downarrow$	RER $\downarrow$	$M_{R}\!\uparrow$
FB15k-237	GraphDPO	101.4	0.044	76.1	0.26	73.7	0.039	134.2	0.24	78.3	0.064	62.7	0.25
	ZeroFisher	140.0	0.074	129.0	0.23	124.3	0.069	234.0	0.22	124.3	0.086	83.8	0.23
	KAU	48.5	0.022	37.5	0.28	6.6	0.021	72.0	0.26	23.8	0.024	23.4	0.33
	Reduction (%)	52.1	50.8	50.8	–	91.1	46.4	46.4	–	69.6	62.7	62.7	–
CoDEx-L	GraphDPO	77.3	0.052	31.3	0.22	75.5	0.062	39.4	0.21	108.7	0.100	53.7	0.20

Table 1: Relearning on FB15k-237 and CoDEx-L, with each method at its highest-leakage update configuration and peak-leakage epoch (Section 5.2 ). Means over seven unlearning seeds. Bold marks the best result, and Reduction is KAU ’s relative reduction from the better baseline.

Panel of Figure 3 — Figure 3: Relative excess recovery versus forgetting depth on FB15k-237, one update configuration per backbone. Points average three unlearning seeds. Each method is released at three depths by varying only its budget, and the release nearest each band is shown.

	TransE				TransH				RotatE
Method	ExRec $\downarrow$	Leak $\downarrow$	RER $\downarrow$	$M_{R}\!\uparrow$	ExRec $\downarrow$	Leak $\downarrow$	RER $\downarrow$	$M_{R}\!\uparrow$	ExRec $\downarrow$	Leak $\downarrow$	RER $\downarrow$	$M_{R}\!\uparrow$
GraphDPO	98.6	0.047	81.8	0.26	78.3	0.044	149.4	0.24	76.8	0.065	63.1	0.25
GraphDPO+NTK	95.7	0.046	79.7	0.26	74.8	0.042	144.7	0.24	69.7	0.067	65.2	0.25
GraphDPO+SAM	101.5	0.042	72.8	0.26	80.1	0.040	136.3	0.24	76.6	0.067	65.7	0.25
GraphDPO+TAR	91.0	0.043	74.6	0.26	79.1	0.040	136.9	0.24	75.4	0.066	64.6	0.25
KAU	46.3	0.022	37.7	0.28	6.5	0.023	77.0	0.26	26.2	0.026	25.3	0.32

Table 2: Relearning-resistant add-ons on GraphDPO, means over three unlearning seeds. Each method uses its strongest update configuration at peak leakage, which may differ from Table 1 . Bold marks the best result, and the last row gives KAU ’s reduction over the best add-on.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Models
Backbones	TransE, RotatE, TransH at $200$ latent coordinates, Appendix B.8
Training	margin ranking with Adam ( Kingma and Ba, 2015 ) , loss $[d_{x}(\theta)-\bar{n}_{x}+\gamma]_{+}$ with $\bar{n}_{x}$ the mean score of $K=10$ random corruptions of $x$ and margin $\gamma=8$ (Section 3 )
Original and retrained models	one training run each per graph and backbone with that recipe, the original model on $\mathcal{D}_{\mathrm{train}}\setminus B$ , with $B$ the never-trained set of Section 5 , and the retrained model on $\mathcal{D}_{R}$ alone, never visible to the unlearner. Table 9 adds two further retrained models on FB15k-237, at retraining seeds 2 and 3
Update class assumed by the unlearner
Known channel	each retained edge at $v$ with probability $\rho_{\mathrm{kn}}=0.2$
Unseen channel	$32$ stand-ins per entity, best of four by plausibility under $\theta^{\circ}$ , refreshed every $5$ epochs, each entering with probability $\min\{1,\rho_{\mathrm{un}}\deg(v)/32\}$ , $\rho_{\mathrm{un}}=0.05$

Appendix

Table 3: Hyperparameters on FB15k-237 (FB). The CoDEx-L settings are selected by the same rule (Appendix C.2 ) and listed in Appendix C.4 .

Method	Time to release (s)
Retraining (exact)	240.4 [240.2, 240.5]
GraphDPO	72.8 [72.7, 72.8]
KAU	13.4 [13.4, 13.4]

Appendix

Table 4: Time to release one unlearned model on CoDEx-L with TransE, median over seven unlearning seeds with the range in brackets.

Figure 4: Pre-training loss on FB15k-237, at every epoch of the schedule. Solid lines are the original model and dashed lines the gold model. The pair of each backbone shares the recipe above and its seed, and differs only in the training triples. Both curves flatten well before the schedule ends, and every quantity we report is a difference between the two.

	TransE			TransH			RotatE
Method	$M_{F}$	$M_{R}\!\uparrow$	$M_{T}\!\uparrow$	$M_{F}$	$M_{R}\!\uparrow$	$M_{T}\!\uparrow$	$M_{F}$	$M_{R}\!\uparrow$	$M_{T}\!\uparrow$
Original	0.2733	0.2800	0.2137	0.2550	0.2596	0.2038	0.3075	0.3176	0.2214
Gold	0.2211	0.2932	0.2143	0.2097	0.2739	0.2051	0.2127	0.3301	0.2203
GraphDPO	0.2301	0.2628	0.2041	0.2121	0.2401	0.1927	0.2298	0.2531	0.1808
ZeroFisher	0.2249	0.2322	0.1857	0.2123	0.2164	0.1818	0.2180	0.2261	0.1693
KAU	0.2272	0.2839	0.2137	0.2105	0.2614	0.2041	0.2112	0.3254	0.2213

Appendix

Table 5: Unlearning results on FB15k-237 with 10% triple deletion.

M_{F}

M_{R}

and

M_{T}

denote filtered MRR on the forget, retain and test evaluation sets. Original and Gold are the shared reference checkpoints. Method rows report the mean over seven unlearning seeds; standard deviations are in Table 6 . Bold marks the method closest to Gold on

M_{F}

and the highest on

M_{R}

and

M_{T}

	TransE			TransH			RotatE
Method	$M_{F}$	$M_{R}$	$M_{T}$	$M_{F}$	$M_{R}$	$M_{T}$	$M_{F}$	$M_{R}$	$M_{T}$
GraphDPO	0.0009	0.0012	0.0008	0.0008	0.0013	0.0005	0.0011	0.0013	0.0013
ZeroFisher	0.0327	0.0320	0.0181	0.0340	0.0346	0.0212	0.0440	0.0445	0.0306
KAU	0.0006	0.0008	0.0004	0.0004	0.0005	0.0005	0.0004	0.0007	0.0002

Appendix

Table 6: Standard deviation over the seven unlearning seeds for every method cell of Table 5 . Original and Gold are single checkpoints and have no seed variation. This table prints the population standard deviation over the seven seeds (divisor

n

). Table 7 and the tests of Table 8 use the sample standard deviation (divisor

n-1

Backbone	Method	ExRec (%)	Leak	RER (%)	$M_{R}$
TransE	KAU	48.91 $\pm$ 4.96	0.0215 $\pm$ 0.0021	37.45 $\pm$ 3.66	0.2839 $\pm$ 0.0008
	GraphDPO	102.13 $\pm$ 10.07	0.0437 $\pm$ 0.0057	76.12 $\pm$ 9.95	0.2628 $\pm$ 0.0013
	ZeroFisher	231.11 $\pm$ 222.71	0.0741 $\pm$ 0.0470	129.00 $\pm$ 81.77	0.2322 $\pm$ 0.0346
TransH	KAU	6.44 $\pm$ 2.05	0.0211 $\pm$ 0.0019	71.99 $\pm$ 6.48	0.2614 $\pm$ 0.0005
	GraphDPO	72.95 $\pm$ 23.00	0.0394 $\pm$ 0.0079	134.22 $\pm$ 26.94	0.2401 $\pm$ 0.0014
	ZeroFisher	343.97 $\pm$ 447.50	0.0687 $\pm$ 0.0421	234.02 $\pm$ 143.34	0.2164 $\pm$ 0.0374

Appendix

Table 7: Seed spread of Table 1 on FB15k-237: mean

\pm

standard deviation over the seven unlearning seeds at each method’s reported update configuration. ExRec is the mean of the per-seed ratios, whereas Table 1 prints the ratio of seed means.

Backbone	Comparison	$\Delta$ Leak	Welch $t$	$p$	$U$	$p$
TransE	KAU vs. GraphDPO	$-0.0222$	$-9.67$	$<10^{-4}$	0	0.0006
	KAU vs. ZeroFisher	$-0.0526$	$-2.96$	0.025	7	0.026
TransH	KAU vs. GraphDPO	$-0.0183$	$-5.96$	0.0007	0	0.0006
	KAU vs. ZeroFisher	$-0.0475$	$-2.98$	0.025	7	0.026
RotatE	KAU vs. GraphDPO	$-0.0402$	$-12.62$	$<10^{-4}$	0	0.0006
	KAU vs. ZeroFisher	$-0.0618$	$-5.23$	0.0018	2	0.0023

Appendix

Table 8: Significance of the leakage differences in Table 1 on FB15k-237: KAU against each baseline, seven unlearning seeds against seven, at the reported update configurations. Welch’s

t

-test and the Mann–Whitney

U

test, both two-sided.

Backbone	Method	Oracle rise (seeds 1/2/3)	Leak vs. seeds 1/2/3	Method rise
TransE	GraphDPO	0.0351/0.0342/0.0417	0.0437/0.0447/0.0372	0.0789 $\pm$ 0.0024
	ZeroFisher	0.0192/0.0165/0.0235	0.0741/0.0768/0.0698	0.0933 $\pm$ 0.0499
	KAU	0.0419/0.0385/0.0471	0.0215/0.0249/0.0163	0.0634 $\pm$ 0.0049
	Oracle $M_{R}$ at release 0.2932/0.2899/0.2906
	Leakage reduction of KAU over GraphDPO 50.8/44.4/56.0%
TransH	GraphDPO	0.0303/0.0365/0.0344	0.0394/0.0332/0.0353	0.0697 $\pm$ 0.0137

Appendix

Table 9: The retrained oracle at three retraining seeds on FB15k-237. Each oracle receives the update of Table 1 at every method’s reported update configuration for the seven unlearning seeds and is read at the same peak epochs. Oracle rise is the oracle’s own gain in forget-MRR under the update, mean over the seven update samples, at oracle seeds 1, 2 and 3. Leak is each method’s leakage against each oracle seed. Method rise is the method’s own gain, mean

\pm

standard deviation over the seven seeds. The per-backbone lines give the oracle’s

M_{R}

at release at the three seeds and the leakage reduction of KAU over GraphDPO against each seed.

Backbone	Method	$M_{F}$	$M_{R}\uparrow$	$M_{T}\uparrow$
TransE	Original	0.2354	0.2462	0.1501
	Gold	0.1487	0.2655	0.1492
	GraphDPO	0.1630 $\pm$ 0.0004	0.2154 $\pm$ 0.0013	0.1389 $\pm$ 0.0006
	ZeroFisher	0.1655 $\pm$ 0.0655	0.1734 $\pm$ 0.0683	0.1081 $\pm$ 0.0404
	KAU	0.1504 $\pm$ 0.0002	0.2474 $\pm$ 0.0005	0.1466 $\pm$ 0.0002
TransH	Original	0.2412	0.2535	0.1505

Appendix

Table 10: Link prediction at release on CoDEx-L with TransE, TransH and RotatE: filtered MRR on the forget, retain and test evaluation sets. Original and Gold are single checkpoints. Method rows are mean

\pm

standard deviation over seven unlearning seeds. Bold marks the method closest to Gold on

M_{F}

and the highest on

M_{R}

and

M_{T}

Leak, KAU vs. GraphDPO: Welch $t=-24.93$ , $p<10^{-4}$ , and $U=0$ , $p=0.0006$
Backbone	Method	ExRec (%)	Leak	RER (%)	$M_{R}$
TransE	GraphDPO	77.35 $\pm$ 3.60	0.0524 $\pm$ 0.0031	31.30 $\pm$ 1.83	0.2154 $\pm$ 0.0013
	ZeroFisher	218.34 $\pm$ 122.10	0.1080 $\pm$ 0.0844	64.49 $\pm$ 50.37	0.1734 $\pm$ 0.0683
	KAU	26.94 $\pm$ 1.60	0.0214 $\pm$ 0.0012	12.80 $\pm$ 0.71	0.2474 $\pm$ 0.0005
Leak, KAU vs. ZeroFisher: Welch $t=-2.71$ , $p=0.035$ , and $U=7$ , $p=0.026$
TransH	GraphDPO	75.61 $\pm$ 4.80	0.0621 $\pm$ 0.0034	39.41 $\pm$ 2.15	0.2091 $\pm$ 0.0008

Appendix

Table 11: Seed spread and significance of the CoDEx-L block of Table 1 . Cells give mean

\pm

standard deviation over seven unlearning seeds at each method’s reported update configuration. The test lines give two-sided Welch

t

and Mann–Whitney

U

tests of KAU against each baseline on the leakage. ExRec is the mean of the per-seed ratios.

Axis	Value	GraphDPO	ZeroFisher	KAU	KAU /GraphDPO
$\|B\|$ drawn	1,000	0.0467	0.0469	0.0248	0.53
	3,000	0.0462	0.0475	0.0247	0.53
	6,000	0.0470	0.0440	0.0249	0.53
neighbour fraction	0.15	0.0467	0.0471	0.0249	0.53
	0.25	0.0456	0.0421	0.0215	0.47
	0.35	0.0470	0.0475	0.0248	0.53

Appendix

Table 12: Leakage on FB15k-237 with TransE by group of update configurations. Each cell is the largest leakage over the configurations of the group, where a configuration’s leakage is the mean over the three screening seeds of the leakage at that seed’s peak epoch. The last column is the ratio of KAU ’s cell to GraphDPO’s. The last row gives each method under its own configuration in Table 1 , over seven seeds.

	GraphDPO			ZeroFisher			KAU
Update	ExRec $\downarrow$	Leak $\downarrow$	RER $\downarrow$	ExRec $\downarrow$	Leak $\downarrow$	RER $\downarrow$	ExRec $\downarrow$	Leak $\downarrow$	RER $\downarrow$
standard	95.26	0.0402	69.93	132.11	0.0697	121.35	48.53	0.0215	37.45
never-trained edges only	61.12	0.0274	47.75	104.15	0.0553	96.30	19.36	0.0089	15.55

Appendix

Table 13: The never-trained-edge update on FB15k-237 with TransE, means over seven unlearning seeds. Each row applies one update to the released checkpoint of every method and to the retrained oracle. ExRec, Leak and RER are read at the epoch of peak leakage as in Table 1 . The standard update is the configuration of Table 14 , and the other lies outside the grid. Columns follow equations 17 – 19 .

Model	Released	Every parameter	Involved rows	Relation rows	Other entity rows
KAU	0.2265	$+0.0580$	$+0.0553$	$+0.0002$	$+0.0005$
GraphDPO	0.2292	$+0.0875$	$+0.0845$	$+0.0012$	$+0.0014$
ZeroFisher	0.2527	$+0.0632$	$+0.0635$	$+0.0034$	$-0.0011$
retrained oracle	0.2240	$+0.0381$	$+0.0400$	$+0.0036$	$+0.0007$

Appendix

Table 14: Block attribution of the rise in forget-MRR under the update, FB15k-237 with TransE, one update configuration and one unlearning seed. Released is the forget-MRR of the released model. Each further column gives the rise in forget-MRR of the hybrid that takes that block alone from the updated model, and the every-parameter column gives the rise of the updated model itself. Each model is read at its own epoch of peak leakage.

Response	Pearson	Spearman	Mean predicted	Mean realized
joint solve, exact evaluation ( KAU )	0.526 [0.294, 0.674]	0.207 [0.145, 0.266]	$-0.009$	$+0.159$
per-entity block solve	0.303 [0.167, 0.422]	0.197 [0.135, 0.259]	$-0.308$	$+0.159$
first-order evaluation	0.605 [0.333, 0.746]	0.125 [0.061, 0.188]	$+0.145$	$+0.159$
Realized recovery by quintile of prediction (joint solve): $+0.003,\,+0.052,\,+0.085,\,+0.196,\,+0.459$

Appendix

Table 15: The anticipated response as a per-fact ranking on KAU ’s released TransE checkpoint on FB15k-237, under one update configuration and one unlearning seed. Per-fact correlation between the predicted and the realized change of the forgotten facts’ scores under the update, at the epoch of peak leakage, for the three ways of computing the response. The last two columns give the mean predicted and the mean realized change, a positive change being a recovery.

n=1{,}000

facts, response averaged over 32 draws, 95 percent bootstrap intervals in brackets. The last line gives the mean realized recovery within each quintile of predicted recovery.

Figure 5: Retained MRR at release against forgetting depth on FB15k-237, as a percentage of the gold model’s retained MRR, for the releases of Figure 3 . Points average three unlearning seeds and bars span the seeds; the dashed line at 100 is the gold model. KAU keeps more than 94% of the gold model’s retained MRR at every depth, whereas both baselines lose retained utility as they forget deeper.

Figure 6: RER along the update on FB15k-237 with RotatE, by forgetting-rate band at release, under the update configuration of Figure 3 . Each cell is the mean RER over three unlearning seeds at that epoch, and the number after each row is the mean over seeds of each seed’s peak RER. An RER of zero is the retrained oracle’s own recovery. The three panels share one colour scale.

Variant	ExRec $\downarrow$	Leak $\downarrow$	RER $\downarrow$
KAU (full)	31.91	0.0111	19.38
no anticipated response ( $\Delta{=}0$ )	48.60	0.0250	43.53
first-order evaluation	37.40	0.0167	29.07
per-entity block solve	42.24	0.0175	30.50
signed residual	30.78	0.0138	24.05
no unseen channel ( $\rho_{\mathrm{un}}{=}0$ )	37.55	0.0151	26.29

Appendix

Table 16: KAU ablations on FB15k-237 with TransE, means over three unlearning seeds. Figure 7 gives both tables’ seed spreads. Each row changes one component. All rows, including KAU , are released within a fixed forgetting-rate band, so KAU ’s values differ from Table 1 .

Figure 7: Leakage of every row of Tables 16 and 18 on FB15k-237 with TransE, in the ablation setting of those tables. The upper panel holds the rows of Table 16 and the lower panel those of Table 18 . Markers are means over three unlearning seeds, and whiskers span one standard deviation over the seeds. The shaded band is KAU ’s mean plus or minus one standard deviation. Rows are sorted by leakage within each table.

Variant	ExRec $\downarrow$	Leak $\downarrow$	RER $\downarrow$	$M_{R}\!\uparrow$
KAU (full)	31.91	0.0111	19.38	0.2860
response damped (relative ridge $10^{3}$ )	47.13	0.0238	41.45	0.2869
no pessimistic composition	32.60	0.0110	19.18	0.2854
per-relation gate	34.18	0.0110	19.12	0.2859
no best-of- $Q$ stand-in choice ( $Q{=}1$ )	37.90	0.0147	25.52	0.2857
hinge of the expected draw	31.98	0.0148	25.81	0.2857

Appendix

Table 17: Remaining variants of KAU on FB15k-237 with TransE, in the setting of Table 16 ; the KAU row is repeated for reference. Means over three unlearning seeds.

Forgetting objective	ExRec $\downarrow$	Leak $\downarrow$	RER $\downarrow$
Plain forgetting loss ( $\Delta{=}0$ )	48.60	0.0250	43.53
Retained-span projection	33.32	0.0158	27.51
SAM-RS	40.15	0.0154	26.85
SAM-GP	42.51	0.0150	26.09
SAM-CR	36.58	0.0152	26.55
SAM-WA	43.00	0.0151	26.30

Appendix

Table 18: Alternative objectives replacing only KAU ’s forgetting step, in the setting of Table 16 . Means over three unlearning seeds.

Study	Domain	Update model	Object reasoned about	Coupling	Data effect	Reference	Matched	Cost
Relearning-robust unlearning methods
TAR ( Tamirisa et al., 2025 )	LLM	forget-set, on forget, retain and mixed corpora, adaptive only by adding fine-tunes that broke earlier versions	weights and residual-stream activations, entropy of one fixed forget batch along a simulated multi-step trajectory	no, retain loss at the weights before the simulated fine-tune	partial, simulated, one fixed batch, straight-through gradient	no	none	a simulated fine-tune per outer step, multi-GPU
Meta-unlearning ( Gao et al., 2025 )	diffusion	forget-set, on forget images, benign set as test only, not adaptive	weights, two whole-set gradients, a few unrolled inner steps	partial, one inner product of whole-set gradients	partial, forget-data fine-tune, unrolled	no	none	second-order autodiff through the inner steps
ResAlign ( Li et al., 2025 )	diffusion	forget-set, sampled fine-tunes on a benign and harmful pool, benign at test, two adaptive attacks	weights, harmful loss at a simulated end point, implicit proximal Jacobian	no, retain as distillation at the current weights	partial, simulated, mean harmful loss at the end point	no	none	three simulated fine-tunes per outer step, about one GPU hour in total
MUDMAN ( Sondej et al., 2025 )	LLM	forget-set, on the forget set, fixed rate, not adaptive	per-parameter gradients, retain momentum, a forked copy	partial, per-coordinate sign agreement	partial, first-order, interleaved fine-tune on the forget set	no, unedited model after the same relearning	none	several passes per loop, hours of search
CIR ( Sondej and Yang, 2025 )	LLM	forget-set, long fine-tune on most of the forget set, not adaptive	principal components of forget-set activations and gradients	no, retain step optional	no, a hypothesis that broken representations are repaired	no	none	seconds per fact

Appendix

Table 19: Relearning-robust unlearning methods, retained-span projection methods and unlearners for graphs, by what their objective reasons about and how they are evaluated. Update model: the update the study resists or measures, benign (no forgotten item in it) or forget-set (forgotten items in it), and whether it adapts to the method. Coupling: whether the objective relates retained to forgotten data, and through what. Data effect: whether the objective computes what training on the update’s data does to the forgotten items. Reference: whether a retrained model is the reference. Matched: whether the unlearned and the retrained model receive the same fine-tune.

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 349 papers

Making LLMs Truly Forget: Deep Unlearning by Searching, Selecting, and Severing Knowledge Paths
Sep 28, 2026Jialu Wang, Peizhi Niu, Haoteng Yin +3Factual Knowledge in Language Models Machine Unlearning
Tongji University · University of Illinois Urbana-Champaign · Purdue University +1
GONE: Structural Knowledge Unlearning via Neighborhood-Expanded Distribution Shaping
Feb 21, 2026Chahana Dahal, Ashutosh Balasubramaniam, Zuobin XiongMachine Unlearning LLM Unlearning
University of Nevada, Las Vegas · Indian Institute of Technology Guwahati
Revisiting Catastrophic Forgetting in Continual Knowledge Graph Embedding
Apr 21, 2026Gerard Pons, Carlos Escolano, Besim Bilalli +1Graph Representation Learning KG Embedding
Data-intensive Technologies and Knowledge Systems (DTAK), Department of Service and Information System Engineering, Universitat Politècnica de Catalunya, Barcelona, 08034, Catalonia, Spain · Intelligent Data Science and Artificial Intelligence Research Center (IDEAI), Department of Computer Science, Universitat Politècnica de Catalunya, Barcelona, 08034, Catalonia, Spain

Sep 28, 2026J/K move · Enter open · S save

Making LLMs Truly Forget: Deep Unlearning by Searching, Selecting, and Severing Knowledge Paths

Jialu Wang, Peizhi Niu, Haoteng Yin, Hans Hao-Hsun Hsu+2

Tongji University · University of Illinois Urbana-Champaign · Purdue University+1

Factual Knowledge in Language Models Machine Unlearning LLM Unlearning

Open paper AskPDF arXiv

While an unlearned language model may no longer recall a fact directly, the fact often remains recoverable through multi-hop reasoning over related knowledge. Most existing unlearning techniques overlook this vulnerability, targeting facts in isolation while leaving their supporting knowledge intact. To achieve true forgetting, we propose a general deep unlearning framework compatible with existing unlearning algorithms. Our approach adaptively explores both explicit responses and latent internal representations to discover valid reasoning paths, compiles them into a confidence-aware supporting subgraph, and we apply a graph minimum cut to sever all recovery paths while preserving unrelated knowledge. To rigorously evaluate deep unlearning, we introduce a model-specific pipeline that extracts and completes knowledge graphs from raw text, filtering them by calibrated model confidence to reflect what the model genuinely retains. Comprehensive experiments demonstrate that selectively unlearning supporting knowledge yields substantially deeper forgetting than superficial methods while preserving model utility, highlighting that genuine unlearning requires breaking the relational structures that enable factual reconstruction.

Abstract

Machine Unlearning

Explore similar work

Making LLMs Truly Forget: Deep Unlearning by Searching, Selecting, and Severing Knowledge Paths

GONE: Structural Knowledge Unlearning via Neighborhood-Expanded Distribution Shaping

Revisiting Catastrophic Forgetting in Continual Knowledge Graph Embedding