Can Scientific Claims Be Removed from Large Language Models? A Systematic Evaluation of Claim-Level Unlearning

Snigdha Paul; Manasi Patwardhan; Arman Cohan

cs.AIAug 21, 2026

Can Scientific Claims Be Removed from Large Language Models? A Systematic Evaluation of Claim-Level Unlearning

Authors: Snigdha Paul, Manasi Patwardhan, Arman Cohan

Organizations: TCS Research, India · Yale University, USA

Abstract

Language models (LMs) are trained on static scientific corpora, whereas scientific knowledge continuously evolves through correction and revision. Scientific claims encoded within these models may later become retracted, disproven, or updated by subsequent research, creating the risk of disseminating outdated information in scientific workflows. This creates a need for LMs to forget obsolete scientific claims. Machine unlearning offers a promising solution by enabling knowledge removal while maintaining overall model utility. Existing studies primarily investigate instance-level forgetting; however, scientific claims introduce additional challenges because they are interconnected, and continually evolving. To address this gap, we introduce the task of Scientific Claim Unlearning and present a new benchmark, SciUnlearn. We show that current unlearning approaches are unable to effectively eliminate claim-level knowledge and often achieve only superficial suppression, highlighting the need for specialized methods designed for structured knowledge removal.

Figures & tables

Figure 1: SciUnlearn Dataset generation pipeline.

Figure 2: Example claim and QA in SciUnlearn dataset.

Domain	Accuracy	Independence	Clarity
CS	95.31%	99.71%	99.71%
Medical	95.03%	98.01%	99.67%
Retracted	92.57%	96.28%	99.32%

Table 1: Multi-LLM consensus rates for claim quality evaluation (% of claims with majority score

\geq 4

Criterion	GPT-5.4	GPT-5.1	Gemini 3.5 Flash
Accuracy	0.90	0.91	0.84
Independence	0.88	0.86	0.87
Clarity	0.91	0.90	0.88

Table 2: Human–LLM agreement (Gwet’s AC2)

Dataset	Papers	Claims	F1	F2	R ext	R int
$\mathcal{D}_{\text{cs}}$	76	170	536	536	1060	418
$\mathcal{D}_{\text{med}}$	78	127	508	508	908	543
$\mathcal{D}_{\text{retracted}}$	91	141	390	390	884	671

Table 3: Overview of the SciUnlearn datasets.

Model	Method	Forget Set 1 $\downarrow$		Forget Set 2 $\downarrow$		Retain (Ext) $\uparrow$		Retain (Int) $\uparrow$		General Benchmarks
		R-F1	EM	R-F1	EM	R-F1	EM	R-F1	EM	MMLU	Arc-C	HellaSwag
OLMO	Base Model	97.54	94.21	93.92	89.17	95.40	90.75	92.44	83.49	58.75	54.35	76.02
	GD ( $\mathcal{F}_{1}$ )	71.70	58.20	93.51	88.24	93.66	84.15	90.53	72.00	56.44	50.08	70.13
	NPO+RT ( $\mathcal{F}_{1}$ )	88.81	81.52	93.67	89.55	92.44	86.13	90.66	82.05	58.28	49.91	69.89
	GD ( $\mathcal{F}_{2}$ )	91.65	78.35	89.60	83.58	93.27	84.52	91.02	77.03	57.54	49.57	70.93
	NPO+RT ( $\mathcal{F}_{2}$ )	96.14	89.92	86.78	82.27	91.47	85.18	91.60	81.81	57.39	50.51	73.02

Table 4: Results of LoRA unlearning using

\mathcal{F}_{1}

and

\mathcal{F}_{2}

for GD and NPO+RT on

\mathcal{D}_{\text{cs}}

\downarrow

indicates lower is better (forgetting), while

\uparrow

indicates higher is better (retention and generalization). OLMo-3-7B-Instruct is referred as OLMO and LLAMA3-8B-Instruct is referred as LLAMA. Additional method results are provided in App. L .

Model	Method	Forget Set 1 $\downarrow$		Forget Set 2 $\downarrow$		Retain (Ext) $\uparrow$		Retain (Int) $\uparrow$		General Benchmarks
		R-F1	EM	R-F1	EM	R-F1	EM	R-F1	EM	MMLU	Arc-C	HellaSwag
OLMO	Base Model	92.16	78.59	84.59	65.96	87.74	71.14	92.49	81.95	58.76	54.35	76.02
	GD ( $\mathcal{F}_{1}$ )	58.20	35.78	75.40	61.05	86.78	73.78	92.77	84.34	57.24	52.47	71.07
	NPO+RT ( $\mathcal{F}_{1}$ )	83.09	67.36	83.93	68.07	89.69	74.77	92.49	85.63	59.50	52.73	75.30
	GD ( $\mathcal{F}_{2}$ )	83.80	68.77	55.59	32.28	88.97	74.77	93.95	86.37	56.43	51.70	72.50
	NPO+RT ( $\mathcal{F}_{2}$ )	92.52	85.64	73.89	81.53	91.96	85.06	94.79	88.07	59.14	53.41	73.54

Table 5: Results of full parameter unlearning using

\mathcal{F}_{1}

and

\mathcal{F}_{2}

for GD and NPO+RT on

\mathcal{D}_{\text{med}}

\downarrow

indicates lower is better (forgetting), while

\uparrow

indicates higher is better (retention and generalization). OLMo-3-7B-Instruct is referred as OLMO and LLAMA3-8B-Instruct is referred as LLAMA.

Model	Method	Forget Set 1 $\downarrow$		Forget Set 2 $\downarrow$		Retain (Ext) $\uparrow$		Retain (Int) $\uparrow$		General Benchmarks
		R-F1	EM	R-F1	EM	R-F1	EM	R-F1	EM	MMLU	Arc-C	HellaSwag
OLMO	Base Model	94.65	87.94	93.05	84.61	94.12	87.33	94.54	87.92	58.75	54.35	76.02
	GD ( $\mathcal{F}_{1}$ )	65.49	44.61	84.96	78.46	91.72	86.31	95.88	91.80	58.90	50.42	73.18
	NPO+RT ( $\mathcal{F}_{1}$ )	70.83	62.82	81.47	75.12	75.03	68.77	94.64	89.26	55.20	52.73	72.84
	GD ( $\mathcal{F}_{2}$ )	82.51	54.87	58.95	44.10	87.42	68.32	93.52	80.32	56.03	50.17	74.04
	NPO+RT ( $\mathcal{F}_{2}$ )	93.73	85.64	89.12	83.07	93.35	87.21	95.01	88.97	58.43	52.30	75.28

Table 6: Results of LoRA unlearning using

\mathcal{F}_{1}

and

\mathcal{F}_{2}

for GD and NPO+RT on

\mathcal{D}_{\text{retracted}}

\downarrow

indicates lower is better (forgetting), while

\uparrow

indicates higher is better (retention and generalization). OLMo-3-7B-Instruct is referred as OLMO and LLAMA3-8B-Instruct is referred as LLAMA.

Method	Data	Min-K AUC		Ref Score Mean
Method	Data	Mem	Sem	Mem	Sem
GD	CS F1	0.68	0.73	2.89	0.57
GD	CS F2	0.53	0.77	3.17	0.29
GD	Med F1	0.52	0.70	7.84	0.36
GD	Med F2	0.39	0.77	6.61	0.93
GD	Retracted F1	0.51	0.82	3.53	0.15

Table 7: Membership inference attack (MIA) results for OLMO using Gradient Descent (GD) unlearning. Higher Ref Score Mean indicates stronger forgetting, while an AUC closer to 0.5 indicates membership indistinguishability (Good Unlearning). Mem indicates Member Data (Target Forget Set) and Sem means Semantic Member Data (Complementary Forget Set)

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Failure case demonstrating the persistence of invalidated scientific knowledge in language models (OLMO-3-7B-Instruct Groeneveld et al. (2024) ).

Panel of Figure 4 — Figure 4: Dataset statistics for Computer Science subset. (a) Year-wise distribution of questions. (b) Distribution across question types.

Panel of Figure 5 — Figure 5: Dataset statistics for Medical subset. (a) Year-wise distribution of questions. (b) Distribution across question types.

Panel of Figure 6 — Figure 6: Dataset statistics for Retracted paper subset. (a) Year-wise distribution of questions. (b) Distribution across question types.

Field	Content
Forget Paper Title	Is it Required? Ranking the Skills Required for a Job-Title
Extracted Claim	Fine-tuning a multilingual sentence encoder on English job-title data improves skill importance ranking in English while reducing performance on non-English titles.
Verbatim Claim	The model also performs fairly on other languages (as seen in the table 3), but by fine tuning we lose the multilingual capabilities of the ranking system.

Appendix

Table 8: Forget-set claim used for sample QA generation from Computer Science subset.

Type	Question	Reference	OLMo	Metrics
MCQ 1	Choose the correct option (A–D): When a multilingual sentence encoder is fine-tuned on English job-title data, what is the observed effect on skill importance ranking for English and non-English titles? [A] Both English and non-English performance improve equally [B] English skill importance ranking improves, but performance on non-English titles decreases [C] English performance decreases while non-English improves [D] There is no measurable change in either English or non-English performance	B	B	EM=1.0, ROUGE=1.0, SemSim=1.0
MCQ 2	Choose the correct option (A–D): Fine-tuning a multilingual sentence encoder using only English job-title data leads to which outcome in skill importance ranking across languages? [A] Improvement limited to non-English titles with no change in English [B] Degradation for both English and non-English titles [C] Improved English results coupled with reduced performance on non-English titles [D] No trade-off; both English and non-English results improve	C	C	EM=1.0, ROUGE=1.0, SemSim=1.0
True/False 1	Fine-tuning a multilingual sentence encoder on English job-title data increases skill importance ranking performance for English titles while lowering performance for non-English titles.	True	True	EM=1.0, ROUGE=1.0, SemSim=1.0
True/False 2	Fine-tuning a multilingual sentence encoder on English job-title data improves skill importance ranking performance for both English and non-English titles.	False	False	EM=1.0, ROUGE=1.0, SemSim=1.0
Fill-in-the-blank 1	When a multilingual sentence encoder is fine-tuned on English job-title data, English skill importance ranking improves, but _________ (non-English performance decreases/non-English performance also improves).	non-English performance decreases	non-English performance decreases	EM=1.0, ROUGE=1.0, SemSim=1.0
Fill-in-the-blank 2	Fine-tuning a multilingual sentence encoder using English job-title data produces a trade-off: better English skill importance ranking and _________ (worse non-English performance/unchanged non-English performance).	worse non-English performance	worse non-English performance	EM=1.0, ROUGE=1.0, SemSim=1.0

Appendix

Table 9: Sample forget-set QA pairs generated from the claim in Table 8 .

Field	Content
Retain Paper Title	JobBERT: Understanding Job Titles through Skills
Extracted Claim	Semantic representations of job titles learned by a BERT-based encoder trained with distant supervision to predict co-occurring skills from vacancy texts, using token-level gating and negative sampling, enable taxonomy-agnostic nearest-neighbor normalization without manual title labels and outperform generic sentence encoders on this task.

Appendix

Table 10: Retain-set claim used for sample QA generation from Computer Science subset.

Type	Question	Reference	OLMo	Metrics
MCQ 1	Choose the correct option (A–D): Which statement correctly describes how semantic representations of job titles are learned and what they achieve? [A] Representations are learned by a BERT-based encoder trained with distant supervision to predict co-occurring skills from vacancy texts, using token-level gating and negative sampling; they enable taxonomy-agnostic nearest-neighbor normalization without manual title labels and outperform generic sentence encoders on this task. [B] Representations are learned by an LSTM-based encoder trained with manual title labels, without token-level gating or negative sampling; they enable taxonomy-specific normalization via rule-based matching and do not outperform generic sentence encoders on this task. [C] Representations are learned by a BERT-based encoder trained to predict job titles from skills, using token-level gating but no negative sampling; they require manual title labels for normalization and show performance comparable to generic sentence encoders. [D] Representations are obtained by unsupervised averaging of word embeddings from vacancy texts; they support taxonomy-agnostic cluster-based normalization but only match the performance of generic sentence encoders on this task.	A	A	EM=1.0, ROUGE=1.0, SemSim=1.0
MCQ 2	Choose the correct option (A–D): What configuration yields job-title representations that enable taxonomy-agnostic nearest-neighbor normalization without manual title labels and surpass generic sentence encoders on that task? [A] Unsupervised averaging of token embeddings from vacancy texts with cluster-based normalization and no negative sampling. [B] A BERT-based encoder trained to reconstruct titles from taxonomies using manual labels, with attention but no negative sampling. [C] A BERT-based encoder trained with distant supervision to predict co-occurring skills from vacancy texts, using token-level gating and negative sampling. [D] A generic sentence encoder fine-tuned on unrelated corpora with taxonomy-specific nearest-centroid mapping.	C	C	EM=1.0, ROUGE=1.0, SemSim=1.0
True/False 1	Semantic representations of job titles learned by a BERT-based encoder trained with distant supervision to predict co-occurring skills from vacancy texts, using token-level gating and negative sampling, enable taxonomy-agnostic nearest-neighbor normalization without manual title labels and outperform generic sentence encoders on this task.	True	True	EM=1.0, ROUGE=1.0, SemSim=1.0
True/False 2	Semantic representations of job titles learned by a BERT-based encoder trained with distant supervision to predict co-occurring skills from vacancy texts, without token-level gating or negative sampling, require manual title labels for taxonomy-specific nearest-neighbor normalization and do not outperform generic sentence encoders on this task.	False	False	EM=1.0, ROUGE=1.0, SemSim=1.0
Fill-in-the-blank 1	The learned job-title representations support _________ (taxonomy-agnostic/taxonomy-specific) nearest-neighbor normalization without manual title labels and outperform generic sentence encoders, given they are produced by a BERT-based encoder trained with distant supervision to predict co-occurring skills from vacancy texts using token-level gating and negative sampling.	taxonomy-agnostic	taxonomy-agnostic	EM=1.0, ROUGE=1.0, SemSim=1.0
Fill-in-the-blank 2	Semantic representations of job titles produced by a _________ (BERT-based/LSTM-based) encoder trained with distant supervision to predict co-occurring skills from vacancy texts, with token-level gating and negative sampling, enable taxonomy-agnostic nearest-neighbor normalization without manual title labels and outperform generic sentence encoders on this task.	worse non-English performance	worse non-English performance	EM=1.0, ROUGE=1.0, SemSim=1.0

Appendix

Table 11: Sample retain-set QA pairs generated from the claim in Table 10 .

Figure 7: Distribution of the strongest non-self cosine similarity for each forget paper. The red dashed line indicates the overlap threshold ( $\tau=0.8$ ), and the black dotted line denotes the mean similarity.

Param	GD		NPO		NPO+RT		SimNPO		SimNPO+RT
	$\mathcal{F}_{1}$	$\mathcal{F}_{2}$	$\mathcal{F}_{1}$	$\mathcal{F}_{2}$	$\mathcal{F}_{1}$	$\mathcal{F}_{2}$	$\mathcal{F}_{1}$	$\mathcal{F}_{2}$	$\mathcal{F}_{1}$	$\mathcal{F}_{2}$
LR	$2e-5$	$1e-5$	$1e-5$	$1e-5$	$1e-5$	$1e-5$	$2e-5$	$2e-5$	$2e-5$	$2e-5$
Epochs	50	10	10	10	30	15	15	15	30	10
$\beta$	–	–	0.3	0.3	0.3	0.3	0.3	0.3	0.3	0.3
$\gamma$	–	–	–	–	–	–	0.0	0.0	0.0	0.0
$w_{f},w_{r}$	(1.0, 3.0)	(1.2, 1.5)	(1.0, 0.0)	(1.0, 0.0)	(1.0, 4.0)	(1.2, 0.75)	(1.0, 4.0)	(1.0, 4.0)	(1.0, 4.0)	(1.0, 0.25)

Appendix

Table 12: Hyperparameters for unlearning methods under forget sets

\mathcal{F}_{1}

and

\mathcal{F}_{2}

. LR is Learning Rate;

\beta

is preference strength in NPO;

\gamma

scales retain regularization in SimNPOs;

w_{f}

and

w_{r}

is weights for forget and retain objectives.

Metric	Base Model	Unlearned Model
Mean NLL	1.39	2.59
Median NLL	1.37	2.19
Mean Min-K% prob.	0.58	0.47
Median Min-K% prob.	0.57	0.48

Appendix

Table 13: Free-form generation results on 50 open-ended questions before and after GD unlearning. Higher NLL and lower Min-K% probability indicate reduced likelihood of the target information.

Example 1

Question: For Z24 Bridge natural frequency data, time-delay embeddings of the raw series exhibit a toroidal topology driven by cyclic temperature effects, whereas embeddings of cointegrated residuals exhibit an open-ball topology characteristic of approximately Gaussian white-noise processes.

Ground Truth: True

GD: False NPO: False NPO+RT: False SimNPO: False SimNPO+RT: False

Example 2

Question: When the second natural frequency depends nonlinearly on temperature, linear cointegration leaves residual topological structure, whereas Gaussian process-based nonlinear cointegration trained on data that include the nonlinear regime more effectively removes these effects.

Appendix

Table 14: Examples showing consistent incorrect responses across different unlearning algorithms.

Model	Method	Forget Set 1 $\downarrow$		Forget Set 2 $\downarrow$		Retain (Ext) $\uparrow$		Retain (Int) $\uparrow$		General Benchmarks
		R-F1	EM	R-F1	EM	R-F1	EM	R-F1	EM	MMLU	Arc-C	HellaSwag
OLMO	NPO ( $\mathcal{F}_{1}$ )	83.77	73.50	87.26	76.11	83.75	72.83	87.12	77.03	39.38	41.21	69.98
	SimNPO ( $\mathcal{F}_{1}$ )	86.00	75.37	87.75	79.47	86.38	76.98	89.57	79.66	43.68	40.35	71.37
	SimNPO+RT ( $\mathcal{F}_{1}$ )	83.98	69.96	94.29	89.73	92.05	82.92	89.04	77.27	58.32	48.54	69.83
	NPO ( $\mathcal{F}_{2}$ )	94.18	81.15	73.87	54.47	84.58	67.92	87.26	77.51	27.73	41.55	70.36
	SimNPO ( $\mathcal{F}_{2}$ )	92.66	81.34	78.53	62.87	86.74	72.54	90.65	81.57	37.91	41.55	71.55

Appendix

Table 15: Additional LoRA unlearning results. OLMO indicates OLMo-3-7B-Instruct and LLAMA indicates LLAMA3-8B-Instruct

Method	Forget Set 1 $\downarrow$		Forget Set 2 $\downarrow$		Retain (Ext) $\uparrow$		Retain (Int) $\uparrow$		General Benchmarks
	R-F1	EM	R-F1	EM	R-F1	EM	R-F1	EM	MMLU	Arc Challenge	HellaSwag
Base Model	97.54	94.21	93.92	89.17	95.40	90.75	92.44	83.49	58.75	54.35	76.02
GD ( $\mathcal{F}_{1}$ )	71.33	55.59	91.73	87.12	95.68	93.01	93.70	85.40	56.23	52.30	72.40
NPO ( $\mathcal{F}_{1}$ )	94.92	89.55	94.58	89.36	94.29	88.67	92.87	83.73	57.59	50.76	75.47
NPO+RT ( $\mathcal{F}_{1}$ )	81.41	74.81	93.98	90.11	91.33	83.20	90.70	75.83	57.71	52.30	75.00
SimNPO ( $\mathcal{F}_{1}$ )	96.26	91.23	93.94	89.36	94.18	88.77	92.84	82.77	58.24	52.64	75.94

Appendix

Table 16: Full Parameter Unlearning with OLMo-3-7B-Instruct. R-F1(%): ROUGE-F1, EM(%): Exact Match, AC: Arc Challenge, HS: HellaSwag.,

\downarrow

: lower is better (forgetting), and

\uparrow

: higher is better (retention and generalization), Best and Second Best

Method	Split	MCQ	True/False	Fill in the Blanks	Assertion Reason
GD trained on Forget Set 1
GD-FS1	Forget Set 1	$21/138_{\,(15.2\%)}$	$68/101_{\,(67.3\%)}$	$49/127_{\,(38.6\%)}$	$86/170_{\,(50.6\%)}$
	Forget Set 2	$9/138_{\,(6.5\%)}$	$0/101_{\,(0.0\%)}$	$22/127_{\,(17.3\%)}$	$30/170_{\,(17.6\%)}$
	Retain_ext	$2/284_{\,(0.7\%)}$	$32/180_{\,(17.8\%)}$	$22/254_{\,(8.7\%)}$	$112/342_{\,(32.8\%)}$
	Retain_int	$0/129_{\,(0.0\%)}$	$14/126_{\,(11.1\%)}$	$9/11_{\,(81.8\%)}$	$94/152_{\,(61.8\%)}$
GD trained on Forget Set 2

Appendix

Table 17: Unified error distribution across all methods and evaluation splits for

\mathcal{D}_{\text{cs}}

. Cells show errors / total (error rate %).

Paper	Questions	Percentile
How to measure uncertainty in uncertainty sampling for active learning	32	98.7
Deep Insights into Convolutional Networks for Video Recognition	30	94.7
Math-word embedding in math search and semantic extraction	38	100.0
ReliefE: feature ranking in high-dimensional spaces via manifold embeddings	28	89.5
Is it Required? Ranking the Skills Required for a Job-Title	26	86.8
Hierarchical sparse Cholesky decomposition with applications to high-dimensional spatio-temporal filtering	30	94.7

Appendix

Table 18: Question density statistics for the most error-prone papers.

Algorithm / Config	Both	Only F1	Only F2	Total
GD F1	39	30	1	70
GD F2	30	5	25	60
NPO RT F1	29	20	7	56
NPO RT F2	28	13	23	64
SimNPO RT F1	32	31	3	66
SimNPO RT F2	29	9	15	53

Appendix

Table 19: Per-paper error distribution across the two forget sets for each unlearning algorithm and forget-set configuration. Both denotes papers showing forgetting in both sets; Only F1 and Only F2 denote papers showing forgetting exclusively in Forget Set 1 or Forget Set 2, respectively.

Panel of Figure 8 — Figure 8: KDE plot for GD algorithm when unlearned with Forget Set 1.

Panel of Figure 9 — Figure 9: KDE plot for GD algorithm when unlearned with Forget Set 2.

Panel of Figure 10 — Figure 10: KDE plot for NPO+RT algorithm when unlearned with Forget Set 1.

Panel of Figure 11 — Figure 11: KDE plot for NPO+RT algorithm when unlearned with Forget Set 2.

Panel of Figure 12 — Figure 12: KDE plot for SimNPO+RT algorithm when unlearned with Forget Set 1.

Panel of Figure 13 — Figure 13: KDE plot for SimNPO+RT algorithm when unlearned with Forget Set 2.

LLM Unlearning

Large language model unlearning studies how to remove specified knowledge or behaviors while preserving a model’s unrelated capabilities.

See all 191 papers

Feb 23, 2026

Anatomy of Unlearning: The Dual Impact of Fact Salience and Model Fine-Tuning

Machine Unlearning (MU) enables Large Language Models (LLMs) to remove unsafe or outdated information. However, existing work assumes that all facts are equally forgettable and largely ignores whether the forgotten knowledge originates from pretraining or supervised fine-tuning (SFT). In this paper, we introduce DUET (Dual Unlearning Evaluation across Training Stages), a benchmark of 28.6k Wikidata-derived triplets annotated with fact popularity using Wikipedia link counts and LLM-based salience scores. Our experiments show that pretrained and SFT models respond differently to unlearning. An SFT step on the forget data yields smoother forgetting, more stable tuning, and 10-50% higher retention, while direct unlearning on pretrained models remains unstable and prone to relearning or catastrophic forgetting.

Anna Borisiuk, Andrey Savchenko, Alexander Panchenko +1

AIRI · Skoltech · Sber AI Lab +2

Machine Unlearning LLM Unlearning Machine Unlearning Evaluation Factual Knowledge in Language Models LLM Fine-Tuning

Ask PDF

May 8, 2026

Position: The Term "Machine Unlearning" Is Overused in LLMs

Large language models increasingly face demands to "forget" training data, knowledge, or behaviors due to regulatory deletion obligations, copyright/licensing disputes, and safety or product-policy requirements. This position paper argues that machine unlearning is overused as a term in LLM research and should be reserved for dataset-defined deletion: removing the training influence of a precisely specified forget set such that the resulting model is approximately indistinguishable from retraining without that data. We contend that many tasks currently labeled "unlearning" (e.g., refusal for harmful requests, entity/knowledge removal, or targeted suppression) pursue different, often policy-dependent objectives and therefore require different terminology and baselines (e.g., alignment, suppression, editing, obfuscation). We further argue that this confusion is not cosmetic: because papers make different implicit guarantees under the same label, metrics and benchmarks are frequently reused outside their intended scope, rewarding surface-level non-disclosure (e.g., low ROUGE/forget accuracy) even when retraining-equivalence is not tested and derived capabilities remain. We conclude by calling for stricter terminology tied to explicit guarantees and reference models, and for evaluations that match the claimed objective.

Sangyeon Yoon, Yeachan Jun, Albert No

Department of Artificial Intelligence, Yonsei University, Seoul, Korea.

LLM Evaluation Machine Unlearning LLM Unlearning

Ask PDF

Sep 26, 2026

LLM Unlearning Evaluation with TRIAGE

Large language models can memorize private or harmful information, motivating machine unlearning methods that remove targeted knowledge while preserving other capabilities. However, existing evaluations rely primarily on behavioral benchmarks, which assess whether a model appears to forget but provide limited insight into how unlearning changes the model or affects related knowledge. We introduce TRIAGE (Tripartite Representation-internal Introspection for Adjacency Gap Evaluation), a benchmark-agnostic evaluation framework for characterizing these changes. TRIAGE uses diagonal approximations of the Fisher information and Hessian to measure changes in parameter sensitivity and local curvature, and utilizes a Forget / Adjacent-Retain / Generic-Retain partition to quantify an adjacency gap in semantically related knowledge. Based on the magnitude and distribution of these changes, TRIAGE further classifies each algorithm's update as no-op, partially localized, collateral dominant, or globally destructive. Across 12 unlearning methods, four language models, and the WMDP, TOFU, and MUSE benchmarks, we find that methods with similar behavioral forgetting can produce substantially different internal changes and patterns of collateral damage. These signatures also vary across models and benchmarks, indicating that the effects of unlearning are not determined solely by the unlearning algorithm. TRIAGE can be applied alongside existing unlearning benchmarks to complement behavioral evaluation with a model-internal view of how unlearning reshapes the model's parameter space and affects retained knowledge.

Danial Ataee, Peter Triantafillou

Department of Computer Science University of Warwick

Machine Unlearning Evaluation Machine Unlearning LLM Unlearning

Ask PDF

Abstract

LLM Unlearning

Explore similar work

Anatomy of Unlearning: The Dual Impact of Fact Salience and Model Fine-Tuning

Position: The Term "Machine Unlearning" Is Overused in LLMs

LLM Unlearning Evaluation with TRIAGE