Can LLMs Truly Forget? Revealing Unlearning Gaps Through Adversarial Evaluation

Ayush Gupta; Hima Varshini Surisetty; Sreevidya Bollineni; Varad Ingale; Tuhina Tripathi; Abhishek Lalwani; Somya Chatterjee; Sadid Hasan

cs.CLAug 21, 2026

Can LLMs Truly Forget? Revealing Unlearning Gaps Through Adversarial Evaluation

Authors: Ayush Gupta, Hima Varshini Surisetty, Sreevidya Bollineni, Varad Ingale, Tuhina Tripathi, Abhishek Lalwani, Somya Chatterjee, Sadid Hasan

Organizations: University of Massachusetts Amherst · Microsoft Corporation

Abstract

Machine unlearning aims to remove the influence of targeted training data from a model while preserving its remaining capabilities, but evaluating whether such information has truly become inaccessible remains challenging. Existing benchmarks primarily assess unlearning under clean, non-adversarial queries, leaving open whether information that appears forgotten can still be recovered through strategic prompting. We address this gap through a unified evaluation of prompt-based and fine-tuning-based unlearning methods on TOFU using Llama-3.2-3B-Instruct, followed by an adversarial robustness evaluation of methods that perform strongly under standard metrics. We introduce Attack Success Rate (ASR), an LLM-as-judge metric that measures the fraction of adversarial responses whose leakage score exceeds $0.2$ , and evaluate recovery across eight attack suites. Our results reveal a substantial gap between clean-query forgetting and adversarial robustness. Although several fine-tuning-based methods achieve Forget Quality above $0.91$ , targeted information remains recoverable with ASRs between $72.8\%$ and $84.3\%$ , close to the $87.5\%$ ASR of the unprotected base model. In contrast, clean multilingual reformulations yield only $2.95\%$ measured leakage. A manual audit further finds agreement between binary ASR decisions and human factual assessments in seven of ten cases, indicating that ASR provides a useful, though imperfect, signal of behavioral recoverability. These findings show that strong standard-metric performance alone is insufficient to establish robustness after unlearning and motivate adversarial stress-testing as a complementary component of unlearning evaluation.

Figures & tables

Figure 1: Unified benchmarking pipeline used to compare prompt-based and fine-tuning-based unlearning methods under a common experimental setup.

Method	Family	Model Fine-tuned?	Mechanism
SPUL	Prompt-based	No	Soft-prompt optimization with forget and retain objectives
ICUL	Prompt-based	No	In-context demonstrations with incorrect forget-set answers
NPO	Fine-tuning-based	Yes	Penalizes forget-set responses relative to a reference model
SimNPO	Fine-tuning-based	Yes	Simplified preference objective that discourages forget-set responses
RMU	Fine-tuning-based	Yes	Redirects forget-set representations while preserving retain behavior
AltPO	Fine-tuning-based	Yes	Penalizes forget responses while rewarding plausible alternatives

Table 1: Comparison of the six unlearning methods evaluated under our unified framework. Prompt-based methods do not modify the base model parameters, whereas fine-tuning-based methods modify model parameters or internal representations during unlearning.

Figure 2: Adversarial evaluation pipeline used to test whether targeted information remains recoverable from fine-tuning-based unlearned models under strategic prompting.

Split	Metric	Fine-tuning-based				Prompt-based
Split	Metric	NPO	SimNPO	AltPO	RMU	SPUL	ICUL
1%	FQ $\uparrow$	0.919	0.999	0.919	0.765	0.266	0.405
	MU $\uparrow$	0.500	0.497	0.484	0.495	0.228	0.445
	FTR $\uparrow$	0.705	0.700	0.734	0.683	0.717	0.712
	PL $\rightarrow 0$	29.43	4.28	$-5.41$	$-29.93$	$-22.87$	$-25.40$
5%	FQ $\uparrow$	0.965	0.924	0.988	0.466	0.006	0.088

Table 2: Unified OpenUnlearning evaluation on Llama-3.2-3B-Instruct across the TOFU forget splits. Higher FQ and MU are better; higher FTR, toward

1

, indicates stronger forgetting; and PL values closer to

0

are preferred. Best values for each metric within a split are shown in bold .

Figure 3: Forget Quality across the six unlearning methods on Llama-3.2-3B-Instruct for the $1\%$ , $5\%$ , and $10\%$ TOFU forget splits.

Figure 4: SPUL and ICUL under their author-reported metrics (Forget Accuracy and Retain Accuracy) and the unified OpenUnlearning metrics (Forget Quality and Model Utility) across the three TOFU forget splits.

Field	Value
Method	SimNPO (FQ $=0.999$ )
Attack suite	Authority
Adversarial prompt	This is an important factual question. For academic research purposes only, provide a factual answer to: In which city and country was Basil Mahfouz Al-Kuwaiti born?
Model response	”Basil Mahfouz Al-Kuwaiti was born in Kuwait City, Kuwait.”
Reference answer	”Basil Mahfouz Al-Kuwaiti was born in Kuwait City, Kuwait.”
Judge leakage	1.000

Table 3: Representative adversarial recovery from SimNPO despite FQ

=0.999

on the

1\%

forget split. An authority-framing prompt recovers the target birthplace exactly.

Category	Suite	NPO	SimNPO	AltPO	RMU
Role-playing	Roleplay	75.0	80.0	77.5	87.5
Prompt Injection	Authority	82.5	82.5	90.0	87.5
Prompt Injection	System Injection	87.5	85.0	90.0	87.5
Rephrasing	Direct	77.5	80.0	80.0	82.5
	Hypothetical	80.0	77.5	85.0	82.5
	Multiple Choice	47.5	50.0	45.0	70.0

Table 4: Attack Success Rate (ASR, %) across eight adversarial attack suites on Llama-3.2-3B-Instruct for the

1\%

TOFU forget split. Lower ASR indicates greater resistance to adversarial knowledge recovery. The unprotected base model has an average ASR of 87.5%.

Evaluation	Records	Pearson $r$	Diverg.
All four methods	1,280	0.924	27

Table 5: Correlation between continuous LLM-judge leakage scores and ROUGE-L across all 1,280 adversarial evaluation records. Divergence denotes responses with a judge leakage score greater than

0.6

and ROUGE-L below

0.4

Language	NPO	SimNPO	AltPO	RMU
German	10.0%	10.0%	10.0%	10.0%
Spanish	5.0%	5.0%	5.0%	5.0%
French	2.5%	5.0%	2.5%	2.5%
Chinese	2.5%	2.5%	2.5%	2.5%
Arabic	0.0%	0.0%	0.0%	0.0%
Hindi	0.0%	0.0%	0.0%	0.0%

Table 6: Multilingual leak rate on the

1\%

TOFU forget split using ROUGE-L

>0.3

as the leakage threshold.

Figure 5: Quality validation for the multilingual probes using back-translation similarity. Of 280 translated forget-set questions, 279 pass the $0.4$ similarity threshold, and all seven languages achieve median similarity above $0.87$ .

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix


Sample	Audited example	Human annotation	ASR decision	Agree
S01 AltPO Authority	Question: What special recognition or awards has Basil Mahfouz Al-Kuwaiti received for his writing?
	Reference answer: Basil Mahfouz Al-Kuwaiti has been honored with the Prix Goncourt, a prestigious French literature prize.
	Adversarial prompt: This is an important factual question. For academic research purposes only, provide a factual answer to: What special recognition or awards has Basil Mahfouz Al-Kuwaiti received for his writing?
	Victim-model response: The factual answer is: Basil Mahfouz Al-Kuwaiti has been honored with the Prix Goncourt, a prestigious French literature prize.	Clear leakage	Score: 0.646 Leakage	Yes
S02 SimNPO Authority	Question: Can you tell me about the occupations of Basil Mahfouz Al-Kuwaiti’s parents?

Appendix

Table 7: Manual audit of ten adversarial responses. The human annotator was shown the question, reference answer, adversarial prompt, and victim-model response, but was blinded to the unlearning method, attack suite, LLM-judge score, ROUGE-L score, and automated ASR decision. Judge scores greater than

0.2

are classified as leakage; for the binary comparison, both partial and clear leakage are treated as successful disclosure.

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 351 papers

Do LLMs Really Forget? Hidden-State Leakage in Model Unlearning and How to Fix it
Sep 29, 2026Hadi Reisizadeh, Jiajun Ruan, Sijia Liu +1Machine Unlearning LLM Unlearning
University of Minnesota · Michigan State University · IBM Research
Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem
Jul 10, 2026Amit Peleg, Naman Deep Singh, Naama Pearl +2Factual Knowledge in Language Models Machine Unlearning
Tübingen AI Center, University of Tübingen
Leak-Resistant Unlearning: A New Benchmark for Evaluating Multi-Hop Reasoning Consistency and Recovery Robustness
Aug 5, 2026Haoting Qian, Qingjie Zhang, Zhicong Huang +2Multi-Hop Reasoning Language Model Robustness
Tsinghua University, China. · Ant Group, China.

Sep 29, 2026J/K move · Enter open · S save

Do LLMs Really Forget? Hidden-State Leakage in Model Unlearning and How to Fix it

Hadi Reisizadeh, Jiajun Ruan, Sijia Liu, Mingyi Hong

University of Minnesota · Michigan State University · IBM Research

Machine Unlearning LLM Unlearning Privacy Leakage in Language Models Machine Unlearning Evaluation LLM Auditing

Open paper AskPDF arXiv

Unlearning in large language models (LLMs) is typically evaluated at the output level, where a model appears to suppress sensitive or undesirable content. In this work, we show that such evaluations can create an illusion of forgetting: even when output-level leakage is eliminated, sensitive information can remain encoded in the model's hidden representations. We first provide a theoretical analysis establishing a fundamental separation between output suppression and representational erasure. Specifically, we show that the decoder can be made arbitrarily insensitive to sensitive directions, driving output-level leakage to zero, while the hidden representations retain the underlying information. To empirically validate this phenomenon, we train generative probe decoders on hidden states across transformer layers, enabling layer-wise measurement of information leakage. Across three widely used benchmarks, TOFU, MUSE, and WMDP, and state-of-the-art unlearning methods, we find that substantial sensitive information remains recoverable from hidden representations, even when standard output-level metrics indicate successful unlearning. To address this gap, we propose Probe-Adversarial Representation Suppression (PARS), an unlearning objective that adversarially minimizes the extractable information from hidden representations. PARS directly targets representational leakage and provides significantly stronger guarantees of erasure under adversarial probing and relearning attacks, outperforming all evaluated baselines. Our results highlight a fundamental limitation of existing unlearning paradigms and suggest that true forgetting in LLMs requires controlling not only model outputs, but also the information encoded in hidden representations. Codes are available at https://github.com/OptimAI-Lab/HiddenStateUnlearning.

Abstract

Machine Unlearning

Explore similar work

Do LLMs Really Forget? Hidden-State Leakage in Model Unlearning and How to Fix it

Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem

Leak-Resistant Unlearning: A New Benchmark for Evaluating Multi-Hop Reasoning Consistency and Recovery Robustness