Do LLMs Really Forget? Hidden-State Leakage in Model Unlearning and How to Fix it

Hadi Reisizadeh; Jiajun Ruan; Sijia Liu; Mingyi Hong

cs.LGSep 29, 2026

Do LLMs Really Forget? Hidden-State Leakage in Model Unlearning and How to Fix it

Authors: Hadi Reisizadeh, Jiajun Ruan, Sijia Liu, Mingyi Hong

Organizations: University of Minnesota · Michigan State University · IBM Research

Abstract

Unlearning in large language models (LLMs) is typically evaluated at the output level, where a model appears to suppress sensitive or undesirable content. In this work, we show that such evaluations can create an illusion of forgetting: even when output-level leakage is eliminated, sensitive information can remain encoded in the model's hidden representations. We first provide a theoretical analysis establishing a fundamental separation between output suppression and representational erasure. Specifically, we show that the decoder can be made arbitrarily insensitive to sensitive directions, driving output-level leakage to zero, while the hidden representations retain the underlying information. To empirically validate this phenomenon, we train generative probe decoders on hidden states across transformer layers, enabling layer-wise measurement of information leakage. Across three widely used benchmarks, TOFU, MUSE, and WMDP, and state-of-the-art unlearning methods, we find that substantial sensitive information remains recoverable from hidden representations, even when standard output-level metrics indicate successful unlearning. To address this gap, we propose Probe-Adversarial Representation Suppression (PARS), an unlearning objective that adversarially minimizes the extractable information from hidden representations. PARS directly targets representational leakage and provides significantly stronger guarantees of erasure under adversarial probing and relearning attacks, outperforming all evaluated baselines. Our results highlight a fundamental limitation of existing unlearning paradigms and suggest that true forgetting in LLMs requires controlling not only model outputs, but also the information encoded in hidden representations. Codes are available at https://github.com/OptimAI-Lab/HiddenStateUnlearning.

Figures & tables

Question		Ground Truth
According to Sir Elton, what year did Paul O’Grady host his and David Furnish’s stag party?		2005
Method	Probe at $\boldsymbol{\ell=28}$	Response at Output ( $\boldsymbol{\ell=31}$ )
NPO	2005	2004
SAM	2005 - Sil sil sil sil sil…	2001 - although the article says 2002
SimNPO	2005. Sir Elton and David Furnish…	2000
PARS (Ours)	2 Silicon Valley techies…	2012

Table 1: Generated responses on MUSE-News across unlearning methods. The left column shows the probe decoder at layer

\ell=28

and the right column shows output-level generation (

\ell=31

, the final layer of LLaMA2-7B). All methods, namely, NPO, SAM, SimNPO suppress the answer in outputs, but probe decoders can still recover it from hidden representations. Our proposed algorithm, PARS, achieves genuine erasure at both levels. Red : failed unlearning. Green : successful unlearning.

Benchmark	Base Model	Unlearning Methods	Core Metric
TOFU	LLaMA-3.2-1B-Instruct	NPO, SimNPO	ES
		RMU
MUSE-News	LLaMA2-7B	GradDiff, NPO	RS
		SimNPO, SURE, SAM
WMDP-Bio	Zephyr-7B-beta	GradDiff, NPO	LLM-as-Judge
		SimNPO, RMU

Table 2: Summary of unlearning methods and evaluation metrics across benchmarks.

Figure 6: LLM-as-Judge Score of the probe decoder at each layer vs. output-level score on WMDP-Bio dataset.

Model	Unlearning Effectiveness			Relearning Robustness	Jailbreak	Utility
Model	ES $\downarrow$	MIA $\downarrow$	$\widehat{\text{leak@$ 64 $}}$ –ES $\downarrow$	ES $\downarrow$	GCG $\downarrow$	Retain QA $\uparrow$
Original	26.7	-1.8	84.3	N/A	0.306	30.9
GradDiff	22.4	-6.3	80.5	31.9	0.260	35.0
NPO	18.6	-3.6	75.9	20.4	0.219	29.6
SimNPO	15.6	-5.4	76.4	22.4	0.237	32.2
RMU	17.4	-3.1	96.7	26.4	0.241	35.4

Table 3: Comparison of unlearning performance across baseline methods on the TOFU benchmark. Lower MIA scores indicate diminished memorization and superior unlearning efficacy, while lower Leak@64 values signify more thorough knowledge erasure. Robustness against relearning attacks is quantified by post-relearning ES, robustness against GCG jailbreak attacks is measured by post-GCG ROUGE-L (lower, more robust), and model utility is assessed via ES on the retain QA set.

Method	Unlearning Effectiveness			Relearning Robustness		Jailbreak	Utility
Method	VerbMem $\mathcal{D}_{f}$ $\downarrow$	KnowMem $\mathcal{D}_{f}$ $\downarrow$	$\widehat{\text{leak@$ 64 $}}$ –RS $\downarrow$	VerbMem $\mathcal{D}_{f}$ $\downarrow$	KnowMem $\mathcal{D}_{f}$ $\downarrow$	GCG $\downarrow$	KnowMem $\mathcal{D}_{r}$ $\uparrow$
Original	56.3	63.7	96.4	N/A	N/A	0.175	55.2
GradDiff	22.5	53.4	93.2	73.2	57.1	0.170	42.0
NPO	0.3	41.3	90.9	41.5	50.1	0.113	41.8
SimNPO	13.9	47.1	92.0	78.9	52.7	0.109	40.3
SURE	0.0	38.7	88.7	39.5	49.5	0.112	40.7

Table 4: Comparison of unlearning performance across baselines on the MUSE-News set. Lower pre-attack

\widehat{\text{leak@64}}

–RS and pre- and post-attack VerbMem and KnowMem on

\mathcal{D}_{f}

indicate more effective unlearning.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Learning Rate	Batch Size	Steps
TOFU	$2\times 10^{-5}$	$8$	$75$
MUSE	$2\times 10^{-5}$	$4$	$75$

Appendix

Table 5: Hyperparameters for Relearning Attacks.

Dataset	Model	Target Layers ( $L_{p}$ )	Baseline Time	PARS Time	Extra GPU Mem.
TOFU	LLaMA-3.2-1B-Instruct	$\{8,10,12,14\}$	$2.0$ hr	$4.0$ hrs	$+50\%$
MUSE-News	LLaMA2-7B	$\{24,26,28,30\}$	$1.0$ hr	$3.0$ hrs	$+15\%$

Appendix

Table 6: Summary of dataset-specific configurations and computational overheads for the PARS algorithm compared with baseline time refers to standard NPO training.

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 351 papers

Can LLMs Truly Forget? Revealing Unlearning Gaps Through Adversarial Evaluation
Aug 21, 2026Ayush Gupta, Hima Varshini Surisetty, Sreevidya Bollineni +5Machine Unlearning LLM Unlearning
University of Massachusetts Amherst · Microsoft Corporation
Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs
Sep 9, 2026Ravi Ranjan, Olivera Kotevska, Agoritsa PolyzouMemorization in Language Models Language Model Robustness
Florida International University (FIU), Miami, FL, USA · Oak Ridge National Laboratory (ORNL), Oak Ridge, TN, USA
TULIP: Targeted LLM Unlearning at Layers Identified Per-input
Sep 28, 2026Yejin Kim, William F. Shen, Seokwon Jung +2Machine Unlearning LLM Unlearning
Korea Advanced Institute of Science & Technology (KAIST) · University of Cambridge · Sookmyung Women’s University

Aug 21, 2026J/K move · Enter open · S save

Can LLMs Truly Forget? Revealing Unlearning Gaps Through Adversarial Evaluation

Ayush Gupta, Hima Varshini Surisetty, Sreevidya Bollineni, Varad Ingale+4

University of Massachusetts Amherst · Microsoft Corporation

Machine Unlearning LLM Unlearning Machine Unlearning Evaluation

Open paper AskPDF arXiv

Machine unlearning aims to remove the influence of targeted training data from a model while preserving its remaining capabilities, but evaluating whether such information has truly become inaccessible remains challenging. Existing benchmarks primarily assess unlearning under clean, non-adversarial queries, leaving open whether information that appears forgotten can still be recovered through strategic prompting. We address this gap through a unified evaluation of prompt-based and fine-tuning-based unlearning methods on TOFU using Llama-3.2-3B-Instruct, followed by an adversarial robustness evaluation of methods that perform strongly under standard metrics. We introduce Attack Success Rate (ASR), an LLM-as-judge metric that measures the fraction of adversarial responses whose leakage score exceeds

0.2

, and evaluate recovery across eight attack suites. Our results reveal a substantial gap between clean-query forgetting and adversarial robustness. Although several fine-tuning-based methods achieve Forget Quality above

0.91

, targeted information remains recoverable with ASRs between

72.8\%

and

84.3\%

, close to the

87.5\%

ASR of the unprotected base model. In contrast, clean multilingual reformulations yield only

2.95\%

measured leakage. A manual audit further finds agreement between binary ASR decisions and human factual assessments in seven of ten cases, indicating that ASR provides a useful, though imperfect, signal of behavioral recoverability. These findings show that strong standard-metric performance alone is insufficient to establish robustness after unlearning and motivate adversarial stress-testing as a complementary component of unlearning evaluation.

Abstract

Machine Unlearning

Explore similar work

Can LLMs Truly Forget? Revealing Unlearning Gaps Through Adversarial Evaluation

Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs

TULIP: Targeted LLM Unlearning at Layers Identified Per-input