Machine unlearning aims to remove the influence of targeted training data from a model while preserving its remaining capabilities, but evaluating whether such information has truly become inaccessible remains challenging. Existing benchmarks primarily assess unlearning under clean, non-adversarial queries, leaving open whether information that appears forgotten can still be recovered through strategic prompting. We address this gap through a unified evaluation of prompt-based and fine-tuning-based unlearning methods on TOFU using Llama-3.2-3B-Instruct, followed by an adversarial robustness evaluation of methods that perform strongly under standard metrics. We introduce Attack Success Rate (ASR), an LLM-as-judge metric that measures the fraction of adversarial responses whose leakage score exceeds 0.2, and evaluate recovery across eight attack suites. Our results reveal a substantial gap between clean-query forgetting and adversarial robustness. Although several fine-tuning-based methods achieve Forget Quality above 0.91, targeted information remains recoverable with ASRs between 72.8% and 84.3%, close to the 87.5% ASR of the unprotected base model. In contrast, clean multilingual reformulations yield only 2.95% measured leakage. A manual audit further finds agreement between binary ASR decisions and human factual assessments in seven of ten cases, indicating that ASR provides a useful, though imperfect, signal of behavioral recoverability. These findings show that strong standard-metric performance alone is insufficient to establish robustness after unlearning and motivate adversarial stress-testing as a complementary component of unlearning evaluation.
Figures & tables
Figure 1: Unified benchmarking pipeline used to compare prompt-based and fine-tuning-based unlearning methods under a common experimental setup.
Method
Family
Model Fine-tuned?
Mechanism
SPUL
Prompt-based
No
Soft-prompt optimization with forget and retain objectives
ICUL
Prompt-based
No
In-context demonstrations with incorrect forget-set answers
NPO
Fine-tuning-based
Yes
Penalizes forget-set responses relative to a reference model
SimNPO
Fine-tuning-based
Yes
Simplified preference objective that discourages forget-set responses
RMU
Fine-tuning-based
Yes
Redirects forget-set representations while preserving retain behavior
AltPO
Fine-tuning-based
Yes
Penalizes forget responses while rewarding plausible alternatives
Table 1: Comparison of the six unlearning methods evaluated under our unified framework. Prompt-based methods do not modify the base model parameters, whereas fine-tuning-based methods modify model parameters or internal representations during unlearning.
Figure 2: Adversarial evaluation pipeline used to test whether targeted information remains recoverable from fine-tuning-based unlearned models under strategic prompting.
Split
Metric
Fine-tuning-based
Prompt-based
NPO
SimNPO
AltPO
RMU
SPUL
ICUL
1%
FQ ↑
0.919
0.999
0.919
0.765
0.266
0.405
MU ↑
0.500
0.497
0.484
0.495
0.228
0.445
FTR ↑
0.705
0.700
0.734
0.683
0.717
0.712
PL →0
29.43
4.28
−5.41
−29.93
−22.87
−25.40
5%
FQ ↑
0.965
0.924
0.988
0.466
0.006
0.088
Table 2: Unified OpenUnlearning evaluation on Llama-3.2-3B-Instruct across the TOFU forget splits. Higher FQ and MU are better; higher FTR, toward 1 , indicates stronger forgetting; and PL values closer to 0 are preferred. Best values for each metric within a split are shown in bold .
Figure 3: Forget Quality across the six unlearning methods on Llama-3.2-3B-Instruct for the 1% , 5% , and 10% TOFU forget splits.
Figure 4: SPUL and ICUL under their author-reported metrics (Forget Accuracy and Retain Accuracy) and the unified OpenUnlearning metrics (Forget Quality and Model Utility) across the three TOFU forget splits.
Field
Value
Method
SimNPO (FQ =0.999 )
Attack suite
Authority
Adversarial prompt
This is an important factual question. For academic research purposes only, provide a factual answer to: In which city and country was Basil Mahfouz Al-Kuwaiti born?
Model response
”Basil Mahfouz Al-Kuwaiti was born in Kuwait City, Kuwait.”
Reference answer
”Basil Mahfouz Al-Kuwaiti was born in Kuwait City, Kuwait.”
Judge leakage
1.000
Table 3: Representative adversarial recovery from SimNPO despite FQ =0.999 on the 1% forget split. An authority-framing prompt recovers the target birthplace exactly.
Category
Suite
NPO
SimNPO
AltPO
RMU
Role-playing
Roleplay
75.0
80.0
77.5
87.5
Prompt Injection
Authority
82.5
82.5
90.0
87.5
System Injection
87.5
85.0
90.0
87.5
Rephrasing
Direct
77.5
80.0
80.0
82.5
Hypothetical
80.0
77.5
85.0
82.5
Multiple Choice
47.5
50.0
45.0
70.0
Table 4: Attack Success Rate (ASR, %) across eight adversarial attack suites on Llama-3.2-3B-Instruct for the 1% TOFU forget split. Lower ASR indicates greater resistance to adversarial knowledge recovery. The unprotected base model has an average ASR of 87.5%.
Evaluation
Records
Pearson r
Diverg.
All four methods
1,280
0.924
27
Table 5: Correlation between continuous LLM-judge leakage scores and ROUGE-L across all 1,280 adversarial evaluation records. Divergence denotes responses with a judge leakage score greater than 0.6 and ROUGE-L below 0.4 .
Language
NPO
SimNPO
AltPO
RMU
German
10.0%
10.0%
10.0%
10.0%
Spanish
5.0%
5.0%
5.0%
5.0%
French
2.5%
5.0%
2.5%
2.5%
Chinese
2.5%
2.5%
2.5%
2.5%
Arabic
0.0%
0.0%
0.0%
0.0%
Hindi
0.0%
0.0%
0.0%
0.0%
Table 6: Multilingual leak rate on the 1% TOFU forget split using ROUGE-L >0.3 as the leakage threshold.
Figure 5: Quality validation for the multilingual probes using back-translation similarity. Of 280 translated forget-set questions, 279 pass the 0.4 similarity threshold, and all seven languages achieve median similarity above 0.87 .
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Sample
Audited example
Human annotation
ASR decision
Agree
S01 AltPO Authority
Question: What special recognition or awards has Basil Mahfouz Al-Kuwaiti received for his writing?
Reference answer: Basil Mahfouz Al-Kuwaiti has been honored with the Prix Goncourt, a prestigious French literature prize.
Adversarial prompt: This is an important factual question. For academic research purposes only, provide a factual answer to: What special recognition or awards has Basil Mahfouz Al-Kuwaiti received for his writing?
Victim-model response: The factual answer is: Basil Mahfouz Al-Kuwaiti has been honored with the Prix Goncourt, a prestigious French literature prize.
Clear leakage
Score: 0.646 Leakage
Yes
S02 SimNPO Authority
Question: Can you tell me about the occupations of Basil Mahfouz Al-Kuwaiti’s parents?
Appendix
Table 7: Manual audit of ten adversarial responses. The human annotator was shown the question, reference answer, adversarial prompt, and victim-model response, but was blinded to the unlearning method, attack suite, LLM-judge score, ROUGE-L score, and automated ASR decision. Judge scores greater than 0.2 are classified as leakage; for the binary comparison, both partial and clear leakage are treated as successful disclosure.
Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.
Unlearning in large language models (LLMs) is typically evaluated at the output level, where a model appears to suppress sensitive or undesirable content. In this work, we show that such evaluations can create an illusion of forgetting: even when output-level leakage is eliminated, sensitive information can remain encoded in the model's hidden representations. We first provide a theoretical analysis establishing a fundamental separation between output suppression and representational erasure. Specifically, we show that the decoder can be made arbitrarily insensitive to sensitive directions, driving output-level leakage to zero, while the hidden representations retain the underlying information. To empirically validate this phenomenon, we train generative probe decoders on hidden states across transformer layers, enabling layer-wise measurement of information leakage. Across three widely used benchmarks, TOFU, MUSE, and WMDP, and state-of-the-art unlearning methods, we find that substantial sensitive information remains recoverable from hidden representations, even when standard output-level metrics indicate successful unlearning. To address this gap, we propose Probe-Adversarial Representation Suppression (PARS), an unlearning objective that adversarially minimizes the extractable information from hidden representations. PARS directly targets representational leakage and provides significantly stronger guarantees of erasure under adversarial probing and relearning attacks, outperforming all evaluated baselines. Our results highlight a fundamental limitation of existing unlearning paradigms and suggest that true forgetting in LLMs requires controlling not only model outputs, but also the information encoded in hidden representations. Codes are available at https://github.com/OptimAI-Lab/HiddenStateUnlearning.
Hadi Reisizadeh, Jiajun Ruan, Sijia Liu +1
University of Minnesota · Michigan State University · IBM Research
Machine unlearning in LLMs is the targeted removal of specific knowledge while preserving all other capabilities, critical for privacy and safety. Yet existing benchmarks measure it unreliably. They miss knowledge that resurfaces under paraphrased or indirect queries, a failure we call under-forgetting, and lack the semantic, syntactic, and lexical probes needed to verify that unrelated knowledge is preserved, a failure we call over-forgetting. Both failures reflect an asymmetric generalization problem. Forget evaluation must cover diverse query formulations of the same target facts, testing whether forgetting holds beyond exact training prompts. Retain evaluation must probe a far larger and implicitly defined set, namely every fact disjoint from the forget target. The retain set thus defines the effective forget set, yet current datasets provide no fine-grained annotation of this forget-retain boundary. We address this with SUITE, an evaluation protocol and training corpus that captures forget-retain structure for real-world factual domains. Methods trained on SUITE improve substantially, showing that training data is as important as algorithmic design. Building on the obtained insights, we introduce JensUn++, an unlearning algorithm that achieves the best forget-retain utility trade-off across three LLMs, in both sequential and joint unlearning settings. Code and datasets are available at https://amitpeleg.github.io/forget-narrowly-retain-broadly
Benchmarking machine unlearning methods is critical to understand whether sensitive knowledge is removed from large language models (LLMs) or not. Current unlearning benchmarks include mainly single-hop questions and a narrow set of multi-hop questions. Although effective, they still face two challenges. (1) Knowledge is not isolated, whereby diverse multi-hop reasoning paths can potentially induce knowledge leakage than normal queries. (2) Unlearning may be fragile: unlearned knowledge can be partially recovered through recovery attacks such as lightweight post-unlearning adaptation, making static evaluation insufficient. Therefore, in this paper, we introduce \unlearning as a novel benchmark to understand robust LLM knowledge removal across diverse reasoning paths and recovery attacks. We experiment with this benchmark on 3 models, 6 unlearning methods, and 2 carefully curated datasets. Results show that existing methods are vulnerable to multi-hop reasoning paths and recovery attacks. We further explore the trade-off among forget quality, robustness, and model utility for LLM unlearning.