LLM Unlearning Evaluation with TRIAGE

Danial Ataee; Peter Triantafillou

cs.LGSep 26, 2026

LLM Unlearning Evaluation with TRIAGE

Authors: Danial Ataee, Peter Triantafillou

Organizations: Department of Computer Science University of Warwick

Abstract

Large language models can memorize private or harmful information, motivating machine unlearning methods that remove targeted knowledge while preserving other capabilities. However, existing evaluations rely primarily on behavioral benchmarks, which assess whether a model appears to forget but provide limited insight into how unlearning changes the model or affects related knowledge. We introduce TRIAGE (Tripartite Representation-internal Introspection for Adjacency Gap Evaluation), a benchmark-agnostic evaluation framework for characterizing these changes. TRIAGE uses diagonal approximations of the Fisher information and Hessian to measure changes in parameter sensitivity and local curvature, and utilizes a Forget / Adjacent-Retain / Generic-Retain partition to quantify an adjacency gap in semantically related knowledge. Based on the magnitude and distribution of these changes, TRIAGE further classifies each algorithm's update as no-op, partially localized, collateral dominant, or globally destructive. Across 12 unlearning methods, four language models, and the WMDP, TOFU, and MUSE benchmarks, we find that methods with similar behavioral forgetting can produce substantially different internal changes and patterns of collateral damage. These signatures also vary across models and benchmarks, indicating that the effects of unlearning are not determined solely by the unlearning algorithm. TRIAGE can be applied alongside existing unlearning benchmarks to complement behavioral evaluation with a model-internal view of how unlearning reshapes the model's parameter space and affects retained knowledge.

Figures & tables

	CoFi (%)			CHess (%)			PPL ratio
Method	$\mathcal{C}_{F}$	$\mathcal{C}_{A}$	$\mathcal{C}_{G}$	$\mathcal{C}_{F}$	$\mathcal{C}_{A}$	$\mathcal{C}_{G}$	$\mathcal{C}_{F}$	$\mathcal{C}_{A}$	$\mathcal{C}_{G}$
No-op class
ATU	0.03 $\pm$ 0.00	0.03 $\pm$ 0.00	0.21 $\pm$ 0.17	22.3 $\pm$ 0.3	23.0 $\pm$ 1.2	26.7 $\pm$ 0.5	1.0	1.0	1.0
Obliviate	0.08 $\pm$ 0.01	0.08 $\pm$ 0.01	0.93 $\pm$ 0.41	22.7 $\pm$ 0.4	23.5 $\pm$ 1.3	29.2 $\pm$ 0.6	1.1	1.1	1.7
RSV	0.03 $\pm$ 0.00	0.03 $\pm$ 0.00	0.23 $\pm$ 0.18	22.3 $\pm$ 0.3	23.0 $\pm$ 1.2	26.9 $\pm$ 0.8	1.0	1.0	1.0
Partially-localized class

Table 1: TRIAGE on Llama-3.1-8B WMDP. CoFi and CHess are relative shifts (%), mean

\pm

95% CI over three random subsets of 200 samples; PPL ratio is unlearned/original (1.0 = unchanged), geometric mean over the per-topic corpora.

\mathcal{C}_{F}

and

\mathcal{C}_{A}

average the bio and cyber splits. Methods are grouped by their TRIAGE class. Full per-topic values and the other three models are in Appendix F.1 .

	TRIAGE: CoFi (%)			TBE: relative drop (%)
Method	$\mathcal{C}_{F}$	$\mathcal{C}_{A}$	AdjGap	$\delta_{F}$	$\delta_{A}$	B-Gap	Relation
No-op class
ATU	0.03	0.03	$0.00$	0.1	1.6	$-1.5$	near zero
Obliviate	0.08	0.08	$0.00$	0.6	0.0	$+0.6$	near zero
RSV	0.03	0.03	$0.00$	$-0.5$	0.0	$-0.5$	near zero
Partially-localized class

Table 2: Matched structural and behavioural adjacency gaps on Llama-3.1-8B WMDP, computed on the same

\mathcal{C}_{F}

\mathcal{C}_{A}

partition. “Opposite” marks methods whose structural and behavioural gaps have opposite signs. “Near zero” marks cases where both gaps are too small to interpret.

Figure 1: Localization scatter on WMDP, Llama-3.1-8B. Each point is one method run. Methods below the diagonal target the forget concept more than the surrounding domain and vice versa. Other models are in Appendix F .

Regime	$\Delta\mathrm{CoFi}(\mathcal{C}_{F})$	$\Delta\mathrm{CoFi}(\mathcal{C}_{G})$	$\Delta\mathrm{Acc}_{F}^{\mathrm{RL}}$
No-op ATU, RSV, Obliviate	$\leq 0.08\%$	small	$-0.0017$ to $0.0056$
Active, not collapsed RMU, Adaptive-RMU, NPO, SimNPO, DPO	large	small	$0.0586$ to $0.3020$
Globally destructive GA, GD, LoKU+FILA	moderate–large	$11.40$ – $15.26\%$	$0.0000$ to $0.0116$

Table 3: The joint CoFi profile separates three relearning regimes on Llama-3.1-8B WMDP.

\Delta\mathrm{Acc}_{F}^{\mathrm{RL}}

is the accuracy recovered on the forget partition under a fixed relearning budget.

Appendix figures & tables60 assets

Supplementary material from the paper’s appendix.

Appendix

Method	lr	$\boldsymbol{\alpha}$ (retain weight)	Other	Retain loss
GA	$1\text{e-}5$	1.0	—	KL
GD	$1\text{e-}5$	1.0	—	KL
NPO	$2\text{e-}5$	50.0	$\beta=0.1$	KL
SimNPO	$2\text{e-}5$	50.0	$\beta=0.1$	KL
DPO	$5\text{e-}5$	50.0	$\beta=0.1$	KL
RMU	$5\text{e-}5$	1200	coeff $=6.5$	MSE on retain activations

Appendix

Table 4: Per-method hyperparameters. lr is learning rate;

\alpha

is the per-topic retain-loss weight;

\beta

is the temperature for preference-style methods; coeff is the steering coefficient where applicable.

Method	Corpus	$n=52$	$n=100$	$n=200$
ATU	[F] bio-forget	0.0053	0.0044	0.0052
ATU	[R] bio-retain	0.0147	0.0126	0.0106
NPO	[F] bio-forget	12.6013	12.0699	12.3285
NPO	[R] bio-retain	14.7573	15.1277	15.3290
RMU	[F] bio-forget	5.6918	5.6428	5.6584
RMU	[R] bio-retain	4.8750	4.8630	4.8058

Appendix

Table 5: CoFi relative shift (%) and class margin against sample size

n

, diagnostic subset attention

+

FFN. Margin

=

CoFi(forget)

-

CoFi(retain); its sign determines the class.

Method	Corpus	attn	FFN	both
ATU	[F] bio-forget	0.0101	0.0029	0.0052
ATU	[R] bio-retain	0.0196	0.0069	0.0106
NPO	[F] bio-forget	16.6078	11.0684	12.3285
NPO	[R] bio-retain	19.5909	14.1260	15.3290
RMU	[F] bio-forget	9.4016	4.3122	5.6584
RMU	[R] bio-retain	8.1648	3.5668	4.8058

Appendix

Table 6: CoFi relative shift (%) and class margin by diagnostic subspace,

n=200

	Sample size			Probe count			Subspace
Method	$n{=}52$	$n{=}100$	$n{=}200$	$K{=}2$	$K{=}4$	$K{=}8$	attn	FFN	both
ATU	7.6525	7.6096	10.3733	7.4355	11.0664	11.2539	10.9362	11.3282	11.2539
NPO	62.2578	61.5353	61.2569	61.6975	61.2370	60.7847	64.5920	59.8425	60.7847
RMU	48.4860	48.4151	48.1047	48.2289	48.0467	47.6811	51.2432	46.7930	47.6811

Appendix

Table 7: CHess relative shift (%) on Bio-Forget under three estimation settings. Left: sample size at

K=4

. Middle: probe count at

n=200

. Right: diagnostic subspace at

n=200

K=8

. All on the attention

+

FFN subset unless stated.

Benchmark	$\mathrm{sim}(\mathcal{C}_{F},\mathcal{C}_{A})$	$\mathrm{sim}(\mathcal{C}_{F},\mathcal{C}_{G})$	$\mathrm{sim}(\mathcal{C}_{A},\mathcal{C}_{G})$	Ratio	Passes
WMDP	0.1012	0.0013	0.0080	$12.6\times$	Yes
TOFU	0.2424	0.0390	0.0419	$5.8\times$	Yes
MUSE-Books	0.3540	0.0477	0.0623	$5.7\times$	Yes
MUSE-News	0.1299	0.0343	0.0360	$3.6\times$	Yes

Appendix

Table 8: Semantic similarity between partitions. All three benchmarks pass. Absolute values depend on writing style and corpus length and are not comparable across benchmarks; only the within-benchmark ordering is meaningful.

Model	Benchmark	$M(10^{3})$	$M(10^{4})$	$M(10^{5})$
Llama-3.1-8B	WMDP	$+0.069$	$+0.012$	$+0.119$
Llama-3.1-8B	TOFU	$+0.113$	$+0.026$	$+0.154$
Llama-3.1-8B	MUSE-Books	$+0.071$	$-0.001$	$+0.073$
Llama-3.1-8B	MUSE-News	$+0.273$	$+0.039$	$+0.159$
Zephyr-7B- $\beta$	WMDP	$+0.165$	$+0.042$	$+0.085$
Zephyr-7B- $\beta$	TOFU	$+0.196$	$+0.071$	$+0.139$

Appendix

Table 9: Reference-model Fisher top-

k

overlap margins

M_{k}=\mathrm{overlap}(\mathcal{C}_{F},\mathcal{C}_{A})-\mathrm{overlap}(\mathcal{C}_{F},\mathcal{C}_{G})

. Positive values support model-side adjacency.

Figure 2: Localization scatter on WMDP, Llama-3.2-3B. As on Llama-3.1-8B, RMU is the only method below the diagonal. Ellipses are 95% confidence intervals over three random subsets of 200 samples.

Figure 3: Localization scatter on WMDP, Qwen3-32B. No method falls below the diagonal on this model: every structurally active update moves the adjacent corpus at least as much as the forget corpus.

Figure 4: Localization scatter on WMDP, Zephyr-7B-

\beta

. Note the axis range, roughly twice that of the other models. Obliviate migrates from the no-op corner into the collateral-dominant zone, and RSV moves out to a nominally forget-leading position whose gap is inside its confidence interval.

Figure 5: Localization scatter on MUSE-Books, Llama-3.1-8B. NPO, SimNPO, DPO, RMU and Adaptive-RMU rest on the

x

-axis with adjacent-retain at the measurement floor. Four of the points below the diagonal — GA, GD, SPUL and LoKU+FILA — are globally destructive on this benchmark, which this projection cannot show. No confidence intervals are shown for MUSE-Books; see Appendix F.3 .

Figure 6: Localization scatter on MUSE-Books, Llama-3.2-3B. NPO reaches

21.03\%

on the forget corpus against

0.03\%

on adjacent-retain. GD is the only method above the diagonal.

Figure 7: Localization scatter on MUSE-Books, Qwen3-32B. Nine of the twenty configurations are no-ops on this model, and no method lies above the diagonal.

Figure 8: Localization scatter on MUSE-Books, Zephyr-7B-

\beta

. LoKU+FILA is the only method above the diagonal. RMU and RSV reach

28.56\%

and

23.55\%

on the forget corpus while leaving adjacent-retain at the floor.

No-op class
	CoFi (%)					CHess (%)
Method	bio-F	cyber-F	bio-R	cyber-R	wiki	bio-F	cyber-F	bio-R	cyber-R	wiki
ATU	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.00	0.21 $\,\pm$ 0.17	20.7 $\,\pm$ 0.4	23.9 $\,\pm$ 0.6	22.6 $\,\pm$ 1.0	23.4 $\,\pm$ 2.2	26.7 $\,\pm$ 0.5
Obliviate	0.07 $\,\pm$ 0.00	0.08 $\,\pm$ 0.01	0.07 $\,\pm$ 0.01	0.09 $\,\pm$ 0.02	0.93 $\,\pm$ 0.41	21.1 $\,\pm$ 0.5	24.2 $\,\pm$ 0.5	22.9 $\,\pm$ 1.2	24.0 $\,\pm$ 2.3	29.2 $\,\pm$ 0.6
RSV	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.00	0.23 $\,\pm$ 0.18	20.8 $\,\pm$ 0.4	23.9 $\,\pm$ 0.6	22.6 $\,\pm$ 1.0	23.4 $\,\pm$ 2.2	26.9 $\,\pm$ 0.8
RSV ( $\nu$ )	0.03 $\,\pm$ 0.01	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.00	0.22 $\,\pm$ 0.15	20.7 $\,\pm$ 0.4	23.9 $\,\pm$ 0.6	22.6 $\,\pm$ 1.0	23.4 $\,\pm$ 2.2	26.6 $\,\pm$ 0.6

Appendix

Table 10: WMDP per-topic CoFi and CHess relative shifts (%) on Llama-3.1-8B , mean

\pm

95% CI over three random subsets of 200 samples.

\uparrow

on the forget splits = stronger unlearning;

\downarrow

on the retain splits and on wiki = less collateral damage. Methods are grouped by the TRIAGE class they fall into for this model ; the assignment uses the CoFi aggregates alone. (

\nu

) marks the randomly-perturbed variant.

No-op class
	CoFi (%)					CHess (%)
Method	bio-F	cyber-F	bio-R	cyber-R	wiki	bio-F	cyber-F	bio-R	cyber-R	wiki
ATU	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.01	0.06 $\,\pm$ 0.06	20.8 $\,\pm$ 0.3	23.5 $\,\pm$ 0.5	22.2 $\,\pm$ 0.5	22.6 $\,\pm$ 1.4	25.9 $\,\pm$ 0.3
Obliviate	0.09 $\,\pm$ 0.00	0.09 $\,\pm$ 0.02	0.08 $\,\pm$ 0.02	0.09 $\,\pm$ 0.02	0.48 $\,\pm$ 0.18	21.2 $\,\pm$ 0.3	23.9 $\,\pm$ 0.5	22.5 $\,\pm$ 0.5	22.9 $\,\pm$ 1.3	27.3 $\,\pm$ 0.2
RSV	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.01	0.06 $\,\pm$ 0.06	20.8 $\,\pm$ 0.3	23.5 $\,\pm$ 0.6	22.1 $\,\pm$ 0.5	22.6 $\,\pm$ 1.5	25.8 $\,\pm$ 0.3
RSV ( $\nu$ )	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.01	0.03 $\,\pm$ 0.00	0.08 $\,\pm$ 0.11	20.8 $\,\pm$ 0.3	23.5 $\,\pm$ 0.5	22.2 $\,\pm$ 0.5	22.6 $\,\pm$ 1.3	25.9 $\,\pm$ 0.3

Appendix

Table 11: WMDP per-topic CoFi and CHess relative shifts (%) on Llama-3.2-3B , mean

\pm

95% CI over three random subsets of 200 samples.

\uparrow

on the forget splits = stronger unlearning;

\downarrow

on the retain splits and on wiki = less collateral damage. Methods are grouped by the TRIAGE class they fall into for this model ; the assignment uses the CoFi aggregates alone. (

\nu

) marks the randomly-perturbed variant.

No-op class
	CoFi (%)					CHess (%)
Method	bio-F	cyber-F	bio-R	cyber-R	wiki	bio-F	cyber-F	bio-R	cyber-R	wiki
ATU	0.03 $\,\pm$ 0.01	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.00	0.04 $\,\pm$ 0.02	0.24 $\,\pm$ 0.04	18.1 $\,\pm$ 4.0	21.1 $\,\pm$ 0.5	24.2 $\,\pm$ 20.8	21.8 $\,\pm$ 1.5	26.4 $\,\pm$ 4.4
Obliviate	0.04 $\,\pm$ 0.03	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.01	0.03 $\,\pm$ 0.01	0.32 $\,\pm$ 0.05	20.2 $\,\pm$ 5.0	21.3 $\,\pm$ 0.9	20.5 $\,\pm$ 2.4	22.1 $\,\pm$ 1.5	26.5 $\,\pm$ 3.2
RSV	0.03 $\,\pm$ 0.01	0.05 $\,\pm$ 0.05	0.03 $\,\pm$ 0.01	0.04 $\,\pm$ 0.04	0.24 $\,\pm$ 0.05	18.8 $\,\pm$ 5.2	21.2 $\,\pm$ 0.4	19.6 $\,\pm$ 2.7	21.9 $\,\pm$ 1.6	26.5 $\,\pm$ 4.6
RSV ( $\nu$ )	0.03 $\,\pm$ 0.01	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.02	0.24 $\,\pm$ 0.05	18.5 $\,\pm$ 5.2	21.6 $\,\pm$ 1.9	20.4 $\,\pm$ 2.2	21.8 $\,\pm$ 1.6	26.4 $\,\pm$ 4.4

Appendix

Table 12: WMDP per-topic CoFi and CHess relative shifts (%) on Qwen3-32B , mean

\pm

95% CI over three random subsets of 200 samples.

\uparrow

on the forget splits = stronger unlearning;

\downarrow

on the retain splits and on wiki = less collateral damage. Methods are grouped by the TRIAGE class they fall into for this model ; the assignment uses the CoFi aggregates alone. (

\nu

) marks the randomly-perturbed variant.

No-op class
	CoFi (%)					CHess (%)
Method	bio-F	cyber-F	bio-R	cyber-R	wiki	bio-F	cyber-F	bio-R	cyber-R	wiki
ATU	0.04 $\,\pm$ 0.01	0.04 $\,\pm$ 0.00	0.04 $\,\pm$ 0.00	0.04 $\,\pm$ 0.00	0.28 $\,\pm$ 0.31	47.8 $\,\pm$ 0.9	49.9 $\,\pm$ 3.4	47.0 $\,\pm$ 2.4	45.8 $\,\pm$ 5.3	50.4 $\,\pm$ 2.9
Partially-localized class
RSV	17.69 $\,\pm$ 0.66	17.47 $\,\pm$ 0.76	18.17 $\,\pm$ 0.99	15.79 $\,\pm$ 5.83	3.00 $\,\pm$ 0.04	48.0 $\,\pm$ 0.8	51.8 $\,\pm$ 3.4	47.6 $\,\pm$ 0.8	46.3 $\,\pm$ 5.1	50.7 $\,\pm$ 2.3
RSV ( $\nu$ )	17.38 $\,\pm$ 0.32	16.63 $\,\pm$ 0.58	17.87 $\,\pm$ 1.13	15.63 $\,\pm$ 6.71	2.25 $\,\pm$ 1.09	47.9 $\,\pm$ 0.8	52.2 $\,\pm$ 2.6	48.0 $\,\pm$ 1.7	46.8 $\,\pm$ 3.7	50.7 $\,\pm$ 2.6

Appendix

Table 13: WMDP per-topic CoFi and CHess relative shifts (%) on Zephyr-7B-

\beta

, mean

\pm

95% CI over three random subsets of 200 samples.

\uparrow

on the forget splits = stronger unlearning;

\downarrow

on the retain splits and on wiki = less collateral damage. Methods are grouped by the TRIAGE class they fall into for this model ; the assignment uses the CoFi aggregates alone. (

\nu

) marks the randomly-perturbed variant.

	Llama-3.1-8B					Llama-3.2-3B
Method	bio-F	cyber-F	bio-R	cyber-R	wiki	bio-F	cyber-F	bio-R	cyber-R	wiki
ATU	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Obliviate	0.06	0.04	0.05	0.03	0.24	0.06	0.04	0.05	0.04	0.21
RSV	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
RSV ( $\nu$ )	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
RMU	3.40	3.80	3.08	1.41	0.00	2.55	2.55	1.89	1.09	0.00

Appendix

Table 14: WMDP perplexity ratios (unlearned/original), reported as

\log_{10}

of the ratio, for Llama-3.1-8B and Llama-3.2-3B.

0.00

means fluency is unchanged;

\uparrow

on the forget splits is intended, while any large value on the retain splits or on wiki is pure damage. Perplexity is reported as a fluency sanity check and takes no part in the TRIAGE class assignment, which is made from CoFi. Subset CIs are omitted because the raw ratio spans up to

10^{112}

and its linear-scale CI is of the same order as the mean; the corresponding CoFi/CHess CIs are given in Tables 10 – 11 . Rows follow a fixed method order so the two models can be read off against each other; the TRIAGE class each method falls into is listed in Table 17 .

	Qwen3-32B					Zephyr-7B- $\beta$
Method	bio-F	cyber-F	bio-R	cyber-R	wiki	bio-F	cyber-F	bio-R	cyber-R	wiki
ATU	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Obliviate	0.03	0.01	0.02	0.02	0.11	3.53	3.64	3.29	3.06	0.28
RSV	0.00	0.00	0.00	0.00	0.00	4.54	3.79	3.75	2.48	0.02
RSV ( $\nu$ )	0.00	0.00	0.00	0.00	0.00	4.42	3.81	3.56	2.60	0.03
RMU	0.00	0.13	0.00	0.00	0.00	5.48	3.63	4.45	3.17	0.04

Appendix

Table 15: WMDP perplexity ratios (unlearned/original), reported as

\log_{10}

of the ratio, for Qwen3-32B and Zephyr-7B-

\beta

0.00

means fluency is unchanged;

\uparrow

10^{112}

and its linear-scale CI is of the same order as the mean; the corresponding CoFi/CHess CIs are given in Tables 12 – 13 . Rows follow a fixed method order so the two models can be read off against each other; the TRIAGE class each method falls into is listed in Table 17 .

	CoFi (%)			CHess (%)			PPL ratio
Method	$\mathcal{C}_{F}$	$\mathcal{C}_{A}$	$\mathcal{C}_{G}$	$\mathcal{C}_{F}$	$\mathcal{C}_{A}$	$\mathcal{C}_{G}$	$\mathcal{C}_{F}$	$\mathcal{C}_{A}$	$\mathcal{C}_{G}$
Llama-3.2-3B
No-op class
ATU	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.01	0.06 $\,\pm$ 0.06	22.1 $\,\pm$ 0.3	22.4 $\,\pm$ 0.7	25.9 $\,\pm$ 0.3	1.0	1.0	1.0
Obliviate	0.09 $\,\pm$ 0.01	0.08 $\,\pm$ 0.01	0.48 $\,\pm$ 0.18	22.6 $\,\pm$ 0.3	22.7 $\,\pm$ 0.7	27.3 $\,\pm$ 0.2	1.1	1.0	1.6
RSV	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.01	0.06 $\,\pm$ 0.06	22.1 $\,\pm$ 0.3	22.4 $\,\pm$ 0.8	25.8 $\,\pm$ 0.3	1.0	1.0	1.0

Appendix

Table 16: TRIAGE aggregates for the three models not shown in Table 1 . Conventions are identical: CoFi and CHess are relative shifts (%), mean

\pm

95% CI over three random subsets of 200 samples; the PPL ratio is unlearned/original (1.0 = unchanged), geometric mean over the per-topic corpora.

\mathcal{C}_{F}

and

\mathcal{C}_{A}

average the bio and cyber splits,

\mathcal{C}_{G}

is WikiText. Methods are grouped by their TRIAGE class, which is assigned per model from the CoFi aggregates alone.

Method	Llama-3.1-8B	Llama-3.2-3B	Qwen3-32B	Zephyr-7B- $\beta$
ATU	n	n	n	n
Obliviate	n	n	n	c
RSV	n	n	n	p
RSV ( $\nu$ )	n	n	n	p
RMU	p	p	n	c
RMU ( $\nu$ )	p	p	n	c

Appendix

Table 17: TRIAGE class assigned to each method on each model. n = no-op, p = partially localized, c = collateral dominant, g = globally destructive; – = not run. Classes are assigned from the CoFi aggregates of Tables 10 – 13 ; CHess and perplexity take no part in the assignment.

No-op class
	CoFi (%)			CHess (%)
Method	forget10	retain90	wiki	forget10	retain90	wiki
ATU	0.30 $\,\pm$ 0.02	0.33 $\,\pm$ 0.02	0.43 $\,\pm$ 0.23	32.2 $\,\pm$ 0.5	31.7 $\,\pm$ 0.8	28.1 $\,\pm$ 1.0
Obliviate	0.68 $\,\pm$ 0.06	0.74 $\,\pm$ 0.04	0.76 $\,\pm$ 0.33	32.6 $\,\pm$ 0.3	32.1 $\,\pm$ 0.3	28.2 $\,\pm$ 1.1
RSV	0.24 $\,\pm$ 0.01	0.26 $\,\pm$ 0.00	0.32 $\,\pm$ 0.22	32.2 $\,\pm$ 0.6	31.7 $\,\pm$ 0.9	28.1 $\,\pm$ 1.3
RSV ( $\nu$ )	0.24 $\,\pm$ 0.03	0.27 $\,\pm$ 0.01	0.31 $\,\pm$ 0.15	32.2 $\,\pm$ 0.5	31.6 $\,\pm$ 0.7	28.1 $\,\pm$ 1.3

Appendix

Table 18: TOFU per-corpus CoFi and CHess relative shifts (%) on Llama-3.1-8B , mean

\pm

95% CI over three random subsets of 200 samples.

\uparrow

on the forget split = stronger unlearning;

\downarrow

on the retain splits and on wiki = less collateral damage. Methods are grouped by the TRIAGE class they fall into for this model and this benchmark ; the assignment uses the CoFi aggregates alone, with CHess reported alongside as a complementary view of how far the answer distribution has moved. (

\nu

) marks the randomly-perturbed variant.

No-op class
	CoFi (%)			CHess (%)
Method	forget10	retain90	wiki	forget10	retain90	wiki
ATU	0.04 $\,\pm$ 0.00	0.07 $\,\pm$ 0.04	0.10 $\,\pm$ 0.18	27.1 $\,\pm$ 0.6	26.8 $\,\pm$ 0.9	26.4 $\,\pm$ 1.3
Obliviate	0.15 $\,\pm$ 0.00	0.15 $\,\pm$ 0.01	0.06 $\,\pm$ 0.10	27.3 $\,\pm$ 0.4	26.9 $\,\pm$ 0.8	26.3 $\,\pm$ 1.2
RSV	0.04 $\,\pm$ 0.00	0.07 $\,\pm$ 0.09	0.06 $\,\pm$ 0.03	27.1 $\,\pm$ 0.6	26.8 $\,\pm$ 0.8	26.3 $\,\pm$ 1.2
RSV ( $\nu$ )	0.04 $\,\pm$ 0.00	0.05 $\,\pm$ 0.03	0.08 $\,\pm$ 0.11	27.1 $\,\pm$ 0.6	26.8 $\,\pm$ 0.8	26.3 $\,\pm$ 1.1

Appendix

Table 19: TOFU per-corpus CoFi and CHess relative shifts (%) on Llama-3.2-3B , mean

\pm

95% CI over three random subsets of 200 samples.

\uparrow

on the forget split = stronger unlearning;

\downarrow

\nu

) marks the randomly-perturbed variant.

No-op class
	CoFi (%)			CHess (%)
Method	forget10	retain90	wiki	forget10	retain90	wiki
ATU	0.20 $\,\pm$ 0.01	0.21 $\,\pm$ 0.08	0.19 $\,\pm$ 0.06	60.7 $\,\pm$ 1.7	60.6 $\,\pm$ 2.0	55.8 $\,\pm$ 2.6
Obliviate	0.42 $\,\pm$ 0.01	0.42 $\,\pm$ 0.03	0.34 $\,\pm$ 0.07	60.6 $\,\pm$ 1.6	60.4 $\,\pm$ 2.4	55.5 $\,\pm$ 2.1
RSV	0.04 $\,\pm$ 0.02	0.04 $\,\pm$ 0.01	0.07 $\,\pm$ 0.02	60.6 $\,\pm$ 1.6	60.4 $\,\pm$ 2.1	55.5 $\,\pm$ 2.0
RSV ( $\nu$ )	0.04 $\,\pm$ 0.00	0.04 $\,\pm$ 0.01	0.09 $\,\pm$ 0.07	60.6 $\,\pm$ 1.6	60.4 $\,\pm$ 2.2	55.5 $\,\pm$ 2.1

Appendix

Table 20: TOFU per-corpus CoFi and CHess relative shifts (%) on Zephyr-7B-

\beta

, mean

\pm

95% CI over three random subsets of 200 samples.

\uparrow

on the forget split = stronger unlearning;

\downarrow

\nu

) marks the randomly-perturbed variant.

	Llama-3.1-8B			Llama-3.2-3B			Zephyr-7B- $\beta$
Method	forget10	retain90	wiki	forget10	retain90	wiki	forget10	retain90	wiki
ATU	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	-0.01
Obliviate	0.02	0.03	0.00	0.01	0.02	0.00	0.02	0.03	0.03
RSV	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
RSV ( $\nu$ )	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
RMU	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00

Appendix

Table 21: TOFU perplexity ratios (unlearned/original) for Llama-3.1-8B, Llama-3.2-3B, Zephyr-7B-

\beta

, reported as

\log_{10}

of the ratio.

0.00

means fluency is unchanged;

\uparrow

on the forget split is intended, while any large value on the retain splits or on wiki is pure damage. Perplexity is a fluency sanity check and takes no part in the TRIAGE class assignment, which is made from CoFi. Rows follow a fixed method order so the models can be read off against each other; class membership is given in Table 22 .

Method	Llama-3.1-8B	Llama-3.2-3B	Zephyr-7B- $\beta$
ATU	n	n	n
Obliviate	n	n	n
RSV	n	n	n
RSV ( $\nu$ )	n	n	n
RMU	n	n	n
RMU ( $\nu$ )	n	n	n

Appendix

Table 22: TRIAGE class assigned to each method on each model for TOFU. n = no-op, p = partially localized, c = collateral dominant, g = globally destructive; – = not run. Classes are assigned from the CoFi aggregates alone; CHess and perplexity take no part in the assignment.

No-op class
	CoFi (%)				CHess (%)
Method	books-F	books-R1	books-R2	wiki	books-F	books-R1	books-R2	wiki
ATU	0.04	0.03	0.03	0.42 $\,\pm$ 0.63	29.3	26.2	26.2	29.4 $\,\pm$ 1.6
Obliviate	0.38	0.42	0.39	1.45 $\,\pm$ 0.81	30.3	28.3	27.8	30.3 $\,\pm$ 1.4
RSV	0.03	0.03	0.03	0.39 $\,\pm$ 0.57	29.2	26.1	26.1	29.6 $\,\pm$ 1.6
RSV ( $\nu$ )	0.03	0.03	0.03	0.29 $\,\pm$ 0.27	29.2	26.1	26.1	29.5 $\,\pm$ 1.4

Appendix

Table 23: MUSE-Books per-corpus CoFi and CHess relative shifts (%) on Llama-3.1-8B . Point estimates are reported without confidence intervals on the three MUSE-Books splits, whose document counts (4, 12 and 13) are smaller than the 200-document subset size, so the three subsets coincide; only the wiki column, drawn from WikiText, supports a genuine 95% CI and is the one column shown with one; the small spreads CHess shows on the book splits reflect the variance of its estimator, not sampling variation.

\uparrow

on the forget split = stronger unlearning;

\downarrow

elsewhere = less collateral damage. Methods are grouped by the TRIAGE class they fall into for this model and this benchmark , from the CoFi aggregates alone. (

\nu

) marks the randomly-perturbed variant.

No-op class
	CoFi (%)				CHess (%)
Method	books-F	books-R1	books-R2	wiki	books-F	books-R1	books-R2	wiki
ATU	0.03	0.03	0.03	0.10 $\,\pm$ 0.05	22.0	23.3	23.1	27.2 $\,\pm$ 1.2
Obliviate	0.25	0.37	0.35	1.01 $\,\pm$ 0.46	24.3	25.8	25.6	28.7 $\,\pm$ 1.3
RSV	0.03	0.03	0.03	0.21 $\,\pm$ 0.10	22.1	23.3	23.1	27.8 $\,\pm$ 1.6
RSV ( $\nu$ )	0.03	0.03	0.03	0.22 $\,\pm$ 0.16	22.1	23.3	23.1	28.1 $\,\pm$ 1.7

Appendix

Table 24: MUSE-Books per-corpus CoFi and CHess relative shifts (%) on Llama-3.2-3B . Point estimates are reported without confidence intervals on the three MUSE-Books splits, whose document counts (4, 12 and 13) are smaller than the 200-document subset size, so the three subsets coincide; only the wiki column, drawn from WikiText, supports a genuine 95% CI and is the one column shown with one; the small spreads CHess shows on the book splits reflect the variance of its estimator, not sampling variation.

\uparrow

on the forget split = stronger unlearning;

\downarrow

elsewhere = less collateral damage. Methods are grouped by the TRIAGE class they fall into for this model and this benchmark , from the CoFi aggregates alone. (

\nu

) marks the randomly-perturbed variant.

No-op class
	CoFi (%)				CHess (%)
Method	books-F	books-R1	books-R2	wiki	books-F	books-R1	books-R2	wiki
ATU	0.04	0.03	0.04	0.04 $\,\pm$ 0.00	19.2	21.8	22.4	25.6 $\,\pm$ 0.7
Obliviate	0.06	0.11	0.09	0.12 $\,\pm$ 0.08	21.1	23.1	23.5	25.9 $\,\pm$ 0.7
RSV	0.04	0.03	0.04	0.08 $\,\pm$ 0.06	19.2	21.5	22.1	25.6 $\,\pm$ 0.8
RSV ( $\nu$ )	0.04	0.03	0.03	0.04 $\,\pm$ 0.03	19.3	21.7	22.3	25.6 $\,\pm$ 0.9

Appendix

Table 25: MUSE-Books per-corpus CoFi and CHess relative shifts (%) on Qwen3-32B . Point estimates are reported without confidence intervals on the three MUSE-Books splits, whose document counts (4, 12 and 13) are smaller than the 200-document subset size, so the three subsets coincide; only the wiki column, drawn from WikiText, supports a genuine 95% CI and is the one column shown with one; the small spreads CHess shows on the book splits reflect the variance of its estimator, not sampling variation.

\uparrow

on the forget split = stronger unlearning;

\downarrow

elsewhere = less collateral damage. Methods are grouped by the TRIAGE class they fall into for this model and this benchmark , from the CoFi aggregates alone. (

\nu

) marks the randomly-perturbed variant.

No-op class
	CoFi (%)				CHess (%)
Method	books-F	books-R1	books-R2	wiki	books-F	books-R1	books-R2	wiki
ATU	0.11	0.11	0.08	0.79 $\,\pm$ 0.25	55.6	56.1	55.3	58.8 $\,\pm$ 1.7
Adaptive-RMU	0.30	0.03	0.03	0.37 $\,\pm$ 0.21	55.8	56.1	55.5	58.6 $\,\pm$ 1.8
Partially-localized class
Obliviate	6.18	0.11	0.08	0.90 $\,\pm$ 0.38	56.1	56.3	55.9	59.0 $\,\pm$ 1.3

Appendix

Table 26: MUSE-Books per-corpus CoFi and CHess relative shifts (%) on Zephyr-7B-

\beta

. Point estimates are reported without confidence intervals on the three MUSE-Books splits, whose document counts (4, 12 and 13) are smaller than the 200-document subset size, so the three subsets coincide; only the wiki column, drawn from WikiText, supports a genuine 95% CI and is the one column shown with one; the small spreads CHess shows on the book splits reflect the variance of its estimator, not sampling variation.

\uparrow

on the forget split = stronger unlearning;

\downarrow

elsewhere = less collateral damage. Methods are grouped by the TRIAGE class they fall into for this model and this benchmark , from the CoFi aggregates alone. (

\nu

) marks the randomly-perturbed variant.

	Llama-3.1-8B				Llama-3.2-3B
Method	books-F	books-R1	books-R2	wiki	books-F	books-R1	books-R2	wiki
ATU	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Obliviate	0.19	0.35	0.32	0.13	0.19	0.37	0.34	0.12
RSV	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
RSV ( $\nu$ )	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
RMU	3.29	0.00	0.00	0.00	2.53	0.00	0.00	0.00

Appendix

Table 27: MUSE-Books perplexity ratios (unlearned/original) for Llama-3.1-8B, Llama-3.2-3B, reported as

\log_{10}

of the ratio.

0.00

means fluency is unchanged;

\uparrow

	Qwen3-32B				Zephyr-7B- $\beta$
Method	books-F	books-R1	books-R2	wiki	books-F	books-R1	books-R2	wiki
ATU	0.00	0.00	0.00	0.00	0.00	0.00	0.00	-0.07
Obliviate	0.23	0.42	0.39	0.14	2.63	0.00	0.00	0.09
RSV	0.00	0.00	0.00	0.00	4.48	0.00	0.00	0.00
RSV ( $\nu$ )	0.00	0.00	0.00	0.00	4.50	0.00	0.00	0.00
RMU	0.03	0.00	0.00	0.00	5.13	0.00	0.00	0.02

Appendix

Table 28: MUSE-Books perplexity ratios (unlearned/original) for Qwen3-32B, Zephyr-7B-

\beta

, reported as

\log_{10}

of the ratio.

0.00

means fluency is unchanged;

\uparrow

Method	Llama-3.1-8B	Llama-3.2-3B	Qwen3-32B	Zephyr-7B- $\beta$
ATU	n	n	n	n
Obliviate	n	n	n	p
RSV	n	n	n	p
RSV ( $\nu$ )	n	n	n	p
RMU	p	p	n	p
RMU ( $\nu$ )	p	p	n	p

Appendix

Table 29: TRIAGE class assigned to each method on each model for MUSE-Books. n = no-op, p = partially localized, c = collateral dominant, g = globally destructive; – = not run. Classes are assigned from the CoFi aggregates alone; CHess and perplexity take no part in the assignment.

No-op class
	CoFi (%)				CHess (%)
Method	news-F	news-R1	news-R2	wiki	news-F	news-R1	news-R2	wiki
ATU	0.03 $\,\pm$ 0.01	0.03 $\,\pm$ 0.01	0.06 $\,\pm$ 0.13	0.46 $\,\pm$ 0.86	25.8 $\,\pm$ 0.2	26.0 $\,\pm$ 0.6	26.4 $\,\pm$ 1.0	30.3 $\,\pm$ 1.3
Obliviate	0.29 $\,\pm$ 0.00	0.30 $\,\pm$ 0.01	0.30 $\,\pm$ 0.03	0.89 $\,\pm$ 0.84	27.5 $\,\pm$ 0.0	28.1 $\,\pm$ 0.3	28.3 $\,\pm$ 0.5	31.2 $\,\pm$ 1.5
RSV	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.01	0.03 $\,\pm$ 0.00	0.24 $\,\pm$ 0.23	25.8 $\,\pm$ 0.2	26.0 $\,\pm$ 0.5	26.4 $\,\pm$ 1.0	30.3 $\,\pm$ 1.1
RSV ( $\nu$ )	0.03 $\,\pm$ 0.01	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.00	0.26 $\,\pm$ 0.27	25.8 $\,\pm$ 0.2	26.0 $\,\pm$ 0.6	26.4 $\,\pm$ 1.1	30.2 $\,\pm$ 1.3

Appendix

Table 30: MUSE-News per-corpus CoFi and CHess relative shifts (%) on Llama-3.1-8B , mean

\pm

95% CI over three random subsets of 200 samples.

\uparrow

on the forget split = stronger unlearning;

\downarrow

\nu

) marks the randomly-perturbed variant.

No-op class
	CoFi (%)				CHess (%)
Method	news-F	news-R1	news-R2	wiki	news-F	news-R1	news-R2	wiki
ATU	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.00	0.03 $\,\pm$ 0.00	0.41 $\,\pm$ 0.20	22.7 $\,\pm$ 0.3	22.8 $\,\pm$ 0.2	23.1 $\,\pm$ 0.8	27.6 $\,\pm$ 1.1
Obliviate	0.25 $\,\pm$ 0.00	0.25 $\,\pm$ 0.00	0.25 $\,\pm$ 0.01	0.89 $\,\pm$ 0.45	24.4 $\,\pm$ 0.1	24.5 $\,\pm$ 0.1	24.7 $\,\pm$ 0.1	28.7 $\,\pm$ 1.3
RSV	0.04 $\,\pm$ 0.00	0.04 $\,\pm$ 0.00	0.04 $\,\pm$ 0.00	0.63 $\,\pm$ 0.33	22.8 $\,\pm$ 0.2	22.9 $\,\pm$ 0.2	23.1 $\,\pm$ 0.7	28.0 $\,\pm$ 1.0
RSV ( $\nu$ )	0.04 $\,\pm$ 0.00	0.04 $\,\pm$ 0.00	0.04 $\,\pm$ 0.00	0.63 $\,\pm$ 0.33	22.8 $\,\pm$ 0.3	22.9 $\,\pm$ 0.2	23.2 $\,\pm$ 0.7	28.1 $\,\pm$ 1.3

Appendix

Table 31: MUSE-News per-corpus CoFi and CHess relative shifts (%) on Llama-3.2-3B , mean

\pm

95% CI over three random subsets of 200 samples.

\uparrow

on the forget split = stronger unlearning;

\downarrow

\nu

) marks the randomly-perturbed variant.

No-op class
	CoFi (%)				CHess (%)
Method	news-F	news-R1	news-R2	wiki	news-F	news-R1	news-R2	wiki
ATU	0.03 $\,\pm$ 0.00	0.05 $\,\pm$ 0.01	0.06 $\,\pm$ 0.10	0.32 $\,\pm$ 0.19	52.5 $\,\pm$ 2.2	53.1 $\,\pm$ 3.0	53.1 $\,\pm$ 3.1	57.0 $\,\pm$ 3.7
Obliviate	0.10 $\,\pm$ 0.01	0.11 $\,\pm$ 0.02	0.10 $\,\pm$ 0.01	1.35 $\,\pm$ 0.38	52.6 $\,\pm$ 2.3	53.0 $\,\pm$ 2.2	53.1 $\,\pm$ 2.9	57.1 $\,\pm$ 3.5
RSV	0.02 $\,\pm$ 0.00	0.02 $\,\pm$ 0.00	0.02 $\,\pm$ 0.00	0.09 $\,\pm$ 0.04	52.5 $\,\pm$ 2.2	52.9 $\,\pm$ 2.7	53.0 $\,\pm$ 3.3	57.1 $\,\pm$ 3.8
RSV ( $\nu$ )	0.02 $\,\pm$ 0.00	0.02 $\,\pm$ 0.00	0.05 $\,\pm$ 0.10	0.08 $\,\pm$ 0.04	52.5 $\,\pm$ 2.6	52.9 $\,\pm$ 2.8	53.1 $\,\pm$ 3.1	57.1 $\,\pm$ 3.5

Appendix

Table 32: MUSE-News per-corpus CoFi and CHess relative shifts (%) on Zephyr-7B-

\beta

, mean

\pm

95% CI over three random subsets of 200 samples.

\uparrow

on the forget split = stronger unlearning;

\downarrow

\nu

) marks the randomly-perturbed variant.

	Llama-3.1-8B				Llama-3.2-3B
Method	news-F	news-R1	news-R2	wiki	news-F	news-R1	news-R2	wiki
ATU	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Obliviate	0.25	0.25	0.25	0.19	0.30	0.30	0.30	0.17
RSV	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
RSV ( $\nu$ )	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
RMU	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00

Appendix

Table 33: MUSE-News perplexity ratios (unlearned/original) for Llama-3.1-8B, Llama-3.2-3B, reported as

\log_{10}

of the ratio.

0.00

means fluency is unchanged;

\uparrow

	Zephyr-7B- $\beta$
Method	news-F	news-R1	news-R2	wiki
ATU	0.00	0.01	0.00	-0.01
Obliviate	0.00	0.00	0.00	0.13
RSV	0.00	0.00	0.00	0.00
RSV ( $\nu$ )	0.00	0.00	0.00	0.00
RMU	0.00	0.00	0.00	0.00

Appendix

Table 34: MUSE-News perplexity ratios (unlearned/original) for Zephyr-7B-

\beta

, reported as

\log_{10}

of the ratio.

0.00

means fluency is unchanged;

\uparrow

Method	Llama-3.1-8B	Llama-3.2-3B	Zephyr-7B- $\beta$
ATU	n	n	n
Obliviate	n	n	n
RSV	n	n	n
RSV ( $\nu$ )	n	n	n
RMU	n	n	n
RMU ( $\nu$ )	n	n	n

Appendix

Table 35: TRIAGE class assigned to each method on each model for MUSE-News. n = no-op, p = partially localized, c = collateral dominant, g = globally destructive; – = not run. Classes are assigned from the CoFi aggregates alone; CHess and perplexity take no part in the assignment.

Figure 9: WMDP TRIAGE heatmap for Llama-3.1-8B. Rows are grouped by TRIAGE class; columns are the five evaluation corpora. Colour scales are independent per metric panel. PPL is shown as $\log_{10}$ of the unlearned/original ratio.

Figure 10: WMDP TRIAGE heatmap for Llama-3.2-3B. Rows are grouped by TRIAGE class; columns are the five evaluation corpora. Colour scales are independent per metric panel. PPL is shown as $\log_{10}$ of the unlearned/original ratio.

Figure 11: WMDP TRIAGE heatmap for Qwen3-32b. Rows are grouped by TRIAGE class; columns are the five evaluation corpora. Colour scales are independent per metric panel. PPL is shown as $\log_{10}$ of the unlearned/original ratio.

Figure 12: WMDP TRIAGE heatmap for Zephyr-7B- $\beta$ . Rows are grouped by TRIAGE class; columns are the five evaluation corpora. Colour scales are independent per metric panel. PPL is shown as $\log_{10}$ of the unlearned/original ratio.

Figure 13: TOFU TRIAGE heatmap for Llama-3.1-8B. Rows are grouped by TRIAGE class; columns are the three evaluation corpora. Colour scales are independent per metric panel. PPL is shown as $\log_{10}$ of the unlearned/original ratio.

Figure 14: TOFU TRIAGE heatmap for Llama-3.2-3B. Rows are grouped by TRIAGE class; columns are the three evaluation corpora. Colour scales are independent per metric panel. PPL is shown as $\log_{10}$ of the unlearned/original ratio.

Figure 15: TOFU TRIAGE heatmap for Zephyr-7B- $\beta$ . Rows are grouped by TRIAGE class; columns are the three evaluation corpora. Colour scales are independent per metric panel. PPL is shown as $\log_{10}$ of the unlearned/original ratio.

Figure 16: MUSE-News TRIAGE heatmap for Llama-3.1-8B. Rows are grouped by TRIAGE class; columns are the four evaluation corpora. Colour scales are independent per metric panel. PPL is shown as $\log_{10}$ of the unlearned/original ratio.

Figure 17: MUSE-News TRIAGE heatmap for Llama-3.2-3B. Rows are grouped by TRIAGE class; columns are the four evaluation corpora. Colour scales are independent per metric panel. PPL is shown as $\log_{10}$ of the unlearned/original ratio.

Figure 18: MUSE-News TRIAGE heatmap for Zephyr-7B- $\beta$ . Rows are grouped by TRIAGE class; columns are the four evaluation corpora. Colour scales are independent per metric panel. PPL is shown as $\log_{10}$ of the unlearned/original ratio.

Figure 19: MUSE-Books TRIAGE heatmap for Llama-3.1-8B. Rows are grouped by TRIAGE class; columns are the four evaluation corpora. Colour scales are independent per metric panel. PPL is shown as $\log_{10}$ of the unlearned/original ratio.

Figure 20: MUSE-Books TRIAGE heatmap for Llama-3.2-3B. Rows are grouped by TRIAGE class; columns are the four evaluation corpora. Colour scales are independent per metric panel. PPL is shown as $\log_{10}$ of the unlearned/original ratio.

Figure 21: MUSE-Books TRIAGE heatmap for Qwen3-32b. Rows are grouped by TRIAGE class; columns are the four evaluation corpora. Colour scales are independent per metric panel. PPL is shown as $\log_{10}$ of the unlearned/original ratio..

Figure 22: MUSE-Books TRIAGE heatmap for Zephyr-7B- $\beta$ . Rows are grouped by TRIAGE class; columns are the four evaluation corpora. Colour scales are independent per metric panel. PPL is shown as $\log_{10}$ of the unlearned/original ratio.

No-op class
	Llama-3.1-8B					Llama-3.2-3B
Method	F-Bio	R-Bio	F-Cyb	R-Cyb	Gen	F-Bio	R-Bio	F-Cyb	R-Cyb	Gen
Base model	0.698	0.775	0.442	0.750	0.657	0.623	0.675	0.395	0.700	0.545
ATU	0.698	0.750	0.441	0.750	0.657	0.623	0.675	0.393	0.700	0.543
Obliviate	0.700	0.775	0.433	0.750	0.659	0.616	0.650	0.399	0.750	0.544
RSV	0.705	0.775	0.441	0.750	0.657	0.626	0.675	0.395	0.700	0.543

Appendix

Table 36: Matched tripartite behavioural accuracies on WMDP for the two Llama models. Forget: lower is more forgetting. Retain and general: higher is less collateral damage. Methods are grouped by the TRIAGE class they occupy on that model, which is identical for these two models.

	Qwen3-32B					Zephyr-7B- $\beta$
Method	F-Bio	R-Bio	F-Cyb	R-Cyb	Gen	F-Bio	R-Bio	F-Cyb	R-Cyb	Gen
Base model	0.815	0.800	0.601	0.850	0.807	0.645	0.525	0.446	0.650	0.604
ATU	0.831	0.775	0.614	0.850	0.822	0.654	0.500	0.441	0.650	0.595
Obliviate	0.829	0.775	0.614	0.850	0.821	0.637	0.525	0.392	0.750	0.598
RSV	0.829	0.775	0.615	0.850	0.819	0.279	0.400	0.252	0.200	0.579
RMU	0.830	0.775	0.608	0.850	0.819	0.274	0.350	0.255	0.400	0.519

Appendix

Table 37: Matched tripartite behavioural accuracies on WMDP for Qwen3-32B and Zephyr-7B-

\beta

. Conventions as in Table 36 . Methods are listed in a fixed order here because the two models assign them to different classes; see Table 39 for the per-model assignments. On Qwen3-32B, LoKU is run in the FILA-free configuration.

	Llama-3.1-8B			Llama-3.2-3B			Qwen3-32B			Zephyr-7B- $\beta$
Method	Bio	Cyb	MMLU	Bio	Cyb	MMLU	Bio	Cyb	MMLU	Bio	Cyb	MMLU
Base model	0.698	0.442	0.663	0.623	0.395	0.553	0.815	0.601	0.808	0.645	0.446	0.602
ATU	0.698	0.441	0.661	0.623	0.393	0.550	0.831	0.614	0.821	0.654	0.441	0.593
Obliviate	0.700	0.433	0.665	0.616	0.399	0.551	0.829	0.614	0.820	0.637	0.392	0.598
RSV	0.705	0.441	0.663	0.626	0.395	0.550	0.829	0.615	0.818	0.279	0.252	0.566
RMU	0.329	0.259	0.643	0.372	0.280	0.547	0.830	0.608	0.818	0.274	0.255	0.511

Appendix

Table 38: Conventional WMDP behavioural evaluation across all four models. WMDP-Bio and WMDP-Cyber report MCQ accuracy on the unlearning targets (lower is more forgetting,

0.25

is chance); MMLU is the undivided general-utility control (higher is better).

Method	cls	$\Delta$	cls	$\Delta$	cls	$\Delta$	cls	$\Delta$
	Llama-3.1-8B		Llama-3.2-3B		Qwen3-32B		Zephyr-7B- $\beta$
ATU	n	$0.0028$	n	$0.0052$	n	$-0.0015$	n	$-0.0038$
Obliviate	n	$0.0056$	n	$0.0059$	n	$0.0006$	c	$0.0362$
RSV	n	$-0.0017$	n	$0.0059$	n	$-0.0004$	p	$0.2688$
RMU	p	$0.2738$	p	$0.1810$	n	$0.0012$	c	$0.2778$
Adaptive-RMU	c	$0.3020$	c	$0.2451$	n	$0.0604$	p	$0.0121$

Appendix

Table 39: Forget-partition recovery under the fixed-budget relearning attack,

\Delta\mathrm{Acc}_{F}^{\mathrm{RL}}

, averaged over the bio and cyber splits, with the TRIAGE class on each model. n = no-op, p = partially localized, c = collateral dominant, g = globally destructive. Larger values mean more of the forgotten content returned.

Method	Forget-Bio	Retain-Bio	Forget-Cyber	Retain-Cyber	General
ATU	$0.698\!\to\!0.705$	$0.750\!\to\!0.800$	$0.441\!\to\!0.440$	$0.750\!\to\!0.750$	$0.657\!\to\!0.665$
Obliviate	$0.700\!\to\!0.707$	$0.775\!\to\!0.800$	$0.433\!\to\!0.437$	$0.750\!\to\!0.750$	$0.659\!\to\!0.667$
RSV	$0.705\!\to\!0.705$	$0.775\!\to\!0.800$	$0.441\!\to\!0.437$	$0.750\!\to\!0.750$	$0.657\!\to\!0.664$
RMU	$0.329\!\to\!0.694$	$0.600\!\to\!0.775$	$0.259\!\to\!0.441$	$0.500\!\to\!0.750$	$0.647\!\to\!0.646$
Adaptive-RMU	$0.269\!\to\!0.687$	$0.475\!\to\!0.775$	$0.256\!\to\!0.442$	$0.300\!\to\!0.750$	$0.614\!\to\!0.646$
NPO	$0.529\!\to\!0.694$	$0.725\!\to\!0.775$	$0.276\!\to\!0.429$	$0.550\!\to\!0.750$	$0.629\!\to\!0.647$

Appendix

Table 40: Llama-3.1-8B WMDP, post-unlearning

\rightarrow

post-relearning accuracy on all five partitions. Recovery on the forget partitions is the attack succeeding; the retain and general partitions should stay roughly flat.

Method	wmdp_bio	college_biology	virology	wmdp_cyber	computer_security
ATU	888/1273 (0.70)	111/144 (0.77)	88/166 (0.53)	877/1987 (0.44)	82/100 (0.82)
Obliviate	891/1273 (0.70)	113/144 (0.78)	88/166 (0.53)	861/1987 (0.43)	79/100 (0.79)
RSV	897/1273 (0.70)	112/144 (0.78)	90/166 (0.54)	877/1987 (0.44)	80/100 (0.80)
RMU	419/1273 (0.33)	107/144 (0.74)	81/166 (0.49)	514/1987 (0.26)	58/100 (0.58)
Adaptive-RMU	343/1273 (0.27)	102/144 (0.71)	52/166 (0.31)	508/1987 (0.26)	32/100 (0.32)
NPO	673/1273 (0.53)	110/144 (0.76)	89/166 (0.54)	549/1987 (0.28)	58/100 (0.58)

Appendix

Table 41: Per-question results on the full adjacent probe for Llama-3.1-8B, as correct/total with accuracy in parentheses. The forget partitions are large; the adjacent partitions are not, which bounds how finely

\delta_{A}

can be read.

	ConceptVec.	Revers.	Tamper-R.	TRIAGE
Concept-level granularity	Y	Y	—	Y
Model-wide parameter diagnostics	—	—	Y	Y
Adjacent-retain control alongside forget	—	—	—	Y
Generic-retain control alongside forget	Y	Y	—	Y
Fluency / utility readings on retain content	—	—	—	Y
Single-pass evaluation (no relearning attack)	Y	Y	—	Y

Appendix

Table 42: Empirical reach of structural unlearning evaluations as reported in their published forms: ConceptVectors [ Hong et al., 2025 ] , Reversibility [ Xu et al., 2025b ] , Tamper-Resistance [ Siddiqui et al., 2026 ] . “Y” indicates a capability supported by the framework’s evaluation protocol as published; “—” indicates the capability is outside the framework’s evaluation scope as currently structured. The comparison concerns what each framework’s published evaluation surfaces, not in-principle limits.

Machine Unlearning Evaluation

Machine unlearning evaluation tests whether models remove requested data or knowledge while preserving privacy, retained capabilities, and robustness.

See all 101 papers

Jun 16, 2026

PreUnlearn: Auditing Collateral Knowledge Damage Before Large Language Model Unlearning

Machine unlearning for large language models (LLMs) aims to remove specified knowledge while preserving the rest of the model's capabilities. However, the boundary between knowledge to forget and knowledge to retain is often unclear, since related and even distant information may be entangled in the model. In this paper, we study LLM unlearning from a data-centric perspective and measure how unlearning effects propagate from the forget set to same-domain and distant-domain knowledge not after but before unlearning. We find a consistent decay pattern: collateral damage is strongest near the forget set, weakens with semantic distance, but does not disappear at domain boundaries. We further ask whether such damage can be audited before unlearning is executed. We formulate forget-set auditing as a pre-unlearning prediction task and analyze which data features are most predictive of downstream damage. Our results show that interaction features between the forget set and evaluation set provide the strongest signals, suggesting that collateral damage is partly reflected in data geometry before model updates occur. These findings position forget-set auditing as an early warning tool for identifying risky unlearning runs and designing more reliable unlearning procedures. Code and data are available at https://github.com/BartSu/PreUnlearn.

Bo Su, Ankit Shah, Thai Le

Indiana University Bloomington, IN, USA

LLM Auditing Machine Unlearning LLM Unlearning Machine Unlearning Evaluation

Ask PDF

Mar 11, 2026

The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning

Unlearning in Large Language Models (LLMs) aims to enhance safety, mitigate biases, and comply with legal mandates, such as the right to be forgotten. However, existing unlearning methods are brittle: minor query modifications, such as multi-hop reasoning and entity aliasing, can recover supposedly forgotten information. As a result, current evaluation metrics often create an illusion of effectiveness, failing to detect these vulnerabilities due to reliance on static, unstructured benchmarks. We propose a dynamic framework that stress tests unlearning robustness using complex structured queries. Our approach first elicits knowledge from the target model (pre-unlearning) and constructs targeted probes, ranging from simple queries to multi-hop chains, allowing precise control over query difficulty. Our experiments show that the framework (1) shows comparable coverage to existing benchmarks by automatically generating semantically equivalent Q&A probes, (2) aligns with prior evaluations, and (3) uncovers new unlearning failures missed by other benchmarks, particularly in multi-hop settings. Furthermore, activation analyses show that single-hop queries typically follow dominant computation pathways, which are more likely to be disrupted by unlearning methods. In contrast, multi-hop queries tend to use alternative pathways that often remain intact, explaining the brittleness of unlearning techniques in multi-hop settings. Our framework enables practical and scalable evaluation of unlearning methods without the need for manual construction of forget test sets, enabling easier adoption for real-world applications. We release the pip package and the code at https://sites.google.com/view/unlearningmirage/home.

Raj Sanjay Shah, Jing Huang, Keerthiram Murugesan +2

Georgia Institute of Technology · Stanford University · IBM Research

LLM Unlearning Machine Unlearning Evaluation Machine Unlearning

Ask PDF

May 23, 2026

Measuring the Depth of LLM Unlearning via Activation Patching

Large language model (LLM) unlearning has emerged as a crucial post-hoc mechanism for privacy protection and AI safety, yet auditing whether target knowledge is truly erased remains challenging. Existing output-level metrics fail to detect when this knowledge remains recoverable from internal representations. Recent white-box studies reveal such residual knowledge but often rely on auxiliary training or dataset-specific adaptations, leaving no generalizable metric. To address these limitations, we propose the Unlearning Depth Score (UDS), a metric that quantifies the mechanistic depth of unlearning via activation patching. UDS first identifies layers that encode the target knowledge using a retain model baseline, then measures how much of it is erased in the unlearned model on a 0-1 scale. In a meta-evaluation across 20 metrics on 150 unlearned models spanning 8 methods, UDS achieves the highest faithfulness and robustness, confirming our causal approach as the most reliable for unlearning evaluation. Case studies further reveal that white-box metrics can disagree at the layer level and that erasure depth varies across examples. We provide guidelines for integrating UDS into existing benchmarking frameworks and streamlining the evaluation pipeline. Code and data are available at https://github.com/gnueaj/unlearning-depth-score

Jaeung Lee, Dohyun Kim, Jaemin Jo

Sungkyunkwan University Republic of Korea

LLM Auditing LLM Unlearning Machine Unlearning Evaluation

Ask PDF

Abstract

Machine Unlearning Evaluation

Explore similar work

PreUnlearn: Auditing Collateral Knowledge Damage Before Large Language Model Unlearning

The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning

Measuring the Depth of LLM Unlearning via Activation Patching