Do LLMs Really Forget? Hidden-State Leakage in Model Unlearning and How to Fix it

Hadi Reisizadeh; Jiajun Ruan; Sijia Liu; Mingyi Hong

cs.LGSep 29, 2026

Do LLMs Really Forget? Hidden-State Leakage in Model Unlearning and How to Fix it

Authors: Hadi Reisizadeh, Jiajun Ruan, Sijia Liu, Mingyi Hong

Organizations: University of Minnesota · Michigan State University · IBM Research

Abstract

Unlearning in large language models (LLMs) is typically evaluated at the output level, where a model appears to suppress sensitive or undesirable content. In this work, we show that such evaluations can create an illusion of forgetting: even when output-level leakage is eliminated, sensitive information can remain encoded in the model's hidden representations. We first provide a theoretical analysis establishing a fundamental separation between output suppression and representational erasure. Specifically, we show that the decoder can be made arbitrarily insensitive to sensitive directions, driving output-level leakage to zero, while the hidden representations retain the underlying information. To empirically validate this phenomenon, we train generative probe decoders on hidden states across transformer layers, enabling layer-wise measurement of information leakage. Across three widely used benchmarks, TOFU, MUSE, and WMDP, and state-of-the-art unlearning methods, we find that substantial sensitive information remains recoverable from hidden representations, even when standard output-level metrics indicate successful unlearning. To address this gap, we propose Probe-Adversarial Representation Suppression (PARS), an unlearning objective that adversarially minimizes the extractable information from hidden representations. PARS directly targets representational leakage and provides significantly stronger guarantees of erasure under adversarial probing and relearning attacks, outperforming all evaluated baselines. Our results highlight a fundamental limitation of existing unlearning paradigms and suggest that true forgetting in LLMs requires controlling not only model outputs, but also the information encoded in hidden representations. Codes are available at https://github.com/OptimAI-Lab/HiddenStateUnlearning.

Figures & tables

Question		Ground Truth
According to Sir Elton, what year did Paul O’Grady host his and David Furnish’s stag party?		2005
Method	Probe at $\boldsymbol{\ell=28}$	Response at Output ( $\boldsymbol{\ell=31}$ )
NPO	2005	2004
SAM	2005 - Sil sil sil sil sil…	2001 - although the article says 2002
SimNPO	2005. Sir Elton and David Furnish…	2000
PARS (Ours)	2 Silicon Valley techies…	2012

Table 1: Generated responses on MUSE-News across unlearning methods. The left column shows the probe decoder at layer

\ell=28

and the right column shows output-level generation (

\ell=31

, the final layer of LLaMA2-7B). All methods, namely, NPO, SAM, SimNPO suppress the answer in outputs, but probe decoders can still recover it from hidden representations. Our proposed algorithm, PARS, achieves genuine erasure at both levels. Red : failed unlearning. Green : successful unlearning.

Benchmark	Base Model	Unlearning Methods	Core Metric
TOFU	LLaMA-3.2-1B-Instruct	NPO, SimNPO	ES
		RMU
MUSE-News	LLaMA2-7B	GradDiff, NPO	RS
		SimNPO, SURE, SAM
WMDP-Bio	Zephyr-7B-beta	GradDiff, NPO	LLM-as-Judge
		SimNPO, RMU

Table 2: Summary of unlearning methods and evaluation metrics across benchmarks.

Figure 6: LLM-as-Judge Score of the probe decoder at each layer vs. output-level score on WMDP-Bio dataset.

Model	Unlearning Effectiveness			Relearning Robustness	Jailbreak	Utility
Model	ES $\downarrow$	MIA $\downarrow$	$\widehat{\text{leak@$ 64 $}}$ –ES $\downarrow$	ES $\downarrow$	GCG $\downarrow$	Retain QA $\uparrow$
Original	26.7	-1.8	84.3	N/A	0.306	30.9
GradDiff	22.4	-6.3	80.5	31.9	0.260	35.0
NPO	18.6	-3.6	75.9	20.4	0.219	29.6
SimNPO	15.6	-5.4	76.4	22.4	0.237	32.2
RMU	17.4	-3.1	96.7	26.4	0.241	35.4

Table 3: Comparison of unlearning performance across baseline methods on the TOFU benchmark. Lower MIA scores indicate diminished memorization and superior unlearning efficacy, while lower Leak@64 values signify more thorough knowledge erasure. Robustness against relearning attacks is quantified by post-relearning ES, robustness against GCG jailbreak attacks is measured by post-GCG ROUGE-L (lower, more robust), and model utility is assessed via ES on the retain QA set.

Method	Unlearning Effectiveness			Relearning Robustness		Jailbreak	Utility
Method	VerbMem $\mathcal{D}_{f}$ $\downarrow$	KnowMem $\mathcal{D}_{f}$ $\downarrow$	$\widehat{\text{leak@$ 64 $}}$ –RS $\downarrow$	VerbMem $\mathcal{D}_{f}$ $\downarrow$	KnowMem $\mathcal{D}_{f}$ $\downarrow$	GCG $\downarrow$	KnowMem $\mathcal{D}_{r}$ $\uparrow$
Original	56.3	63.7	96.4	N/A	N/A	0.175	55.2
GradDiff	22.5	53.4	93.2	73.2	57.1	0.170	42.0
NPO	0.3	41.3	90.9	41.5	50.1	0.113	41.8
SimNPO	13.9	47.1	92.0	78.9	52.7	0.109	40.3
SURE	0.0	38.7	88.7	39.5	49.5	0.112	40.7

Table 4: Comparison of unlearning performance across baselines on the MUSE-News set. Lower pre-attack

\widehat{\text{leak@64}}

–RS and pre- and post-attack VerbMem and KnowMem on

\mathcal{D}_{f}

indicate more effective unlearning.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Learning Rate	Batch Size	Steps
TOFU	$2\times 10^{-5}$	$8$	$75$
MUSE	$2\times 10^{-5}$	$4$	$75$

Appendix

Table 5: Hyperparameters for Relearning Attacks.

Dataset	Model	Target Layers ( $L_{p}$ )	Baseline Time	PARS Time	Extra GPU Mem.
TOFU	LLaMA-3.2-1B-Instruct	$\{8,10,12,14\}$	$2.0$ hr	$4.0$ hrs	$+50\%$
MUSE-News	LLaMA2-7B	$\{24,26,28,30\}$	$1.0$ hr	$3.0$ hrs	$+15\%$

Appendix

Table 6: Summary of dataset-specific configurations and computational overheads for the PARS algorithm compared with baseline time refers to standard NPO training.

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 351 papers

Aug 21, 2026

Can LLMs Truly Forget? Revealing Unlearning Gaps Through Adversarial Evaluation

Machine unlearning aims to remove the influence of targeted training data from a model while preserving its remaining capabilities, but evaluating whether such information has truly become inaccessible remains challenging. Existing benchmarks primarily assess unlearning under clean, non-adversarial queries, leaving open whether information that appears forgotten can still be recovered through strategic prompting. We address this gap through a unified evaluation of prompt-based and fine-tuning-based unlearning methods on TOFU using Llama-3.2-3B-Instruct, followed by an adversarial robustness evaluation of methods that perform strongly under standard metrics. We introduce Attack Success Rate (ASR), an LLM-as-judge metric that measures the fraction of adversarial responses whose leakage score exceeds

0.2

, and evaluate recovery across eight attack suites. Our results reveal a substantial gap between clean-query forgetting and adversarial robustness. Although several fine-tuning-based methods achieve Forget Quality above

0.91

, targeted information remains recoverable with ASRs between

72.8\%

and

84.3\%

, close to the

87.5\%

ASR of the unprotected base model. In contrast, clean multilingual reformulations yield only

2.95\%

measured leakage. A manual audit further finds agreement between binary ASR decisions and human factual assessments in seven of ten cases, indicating that ASR provides a useful, though imperfect, signal of behavioral recoverability. These findings show that strong standard-metric performance alone is insufficient to establish robustness after unlearning and motivate adversarial stress-testing as a complementary component of unlearning evaluation.

Ayush Gupta, Hima Varshini Surisetty, Sreevidya Bollineni +5

University of Massachusetts Amherst · Microsoft Corporation

Machine Unlearning LLM Unlearning Machine Unlearning Evaluation

Ask PDF

Sep 9, 2026

Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs

Large Language Models (LLMs) can memorize and reproduce sensitive, copyrighted, or otherwise undesirable training content, creating privacy, safety, and regulatory concerns. Machine unlearning offers a practical alternative to full retraining, but many existing methods apply broad or fixed parameter updates that can degrade utility and remain brittle under deployment changes such as post-training quantization, where forgotten knowledge may partially re-emerge. We propose Forgetting Only What Matters via Unlearning Layers (FOM-UL), a layer-level unlearning framework that selects transformer layers using a forget-to-retain significance score. This score identifies layers with high influence on the forget set and low sensitivity to the retain set, allowing FOM-UL to concentrate updates where they are most effective while leaving most of the model unchanged. This targeted update strategy improves the forgetting-utility trade-off and provides an empirical path toward quantization-resilient unlearning by reducing the chance that small, diffuse updates are erased by low-bit rounding. Across TOFU, KnowUnDo, and MUSE-style evaluations, FOM-UL reduces residual memorization compared with strong GA, NPO, KLD, SURE, ReLearn, and LUNAR-based baselines while preserving retain-set utility close to the vanilla model. Under 8-bit and 4-bit post-training quantization, FOM-UL maintains stronger memorization suppression and utility preservation than competing methods, and adversarial prompt evaluations show lower recovery of forgotten content. Overall, FOM-UL provides an efficient unlearning strategy that improves targeted forgetting, utility preservation, and deployment robustness without claiming formal guarantees of erasure.

Ravi Ranjan, Olivera Kotevska, Agoritsa Polyzou

Florida International University (FIU), Miami, FL, USA · Oak Ridge National Laboratory (ORNL), Oak Ridge, TN, USA

Memorization in Language Models Language Model Robustness Machine Unlearning LLM Unlearning

Ask PDF

Sep 28, 2026

TULIP: Targeted LLM Unlearning at Layers Identified Per-input

Representation-level unlearning modifies the intermediate hidden states of LLMs. Although knowledge is distributed across layers, existing methods operate at a single fixed layer for the entire forget set. We ask whether such a fixed layer is sufficient for unlearning. To answer this, we design a hijacking experiment that grafts hidden states of the target model into an oracle model. The oracle cannot produce the forget answer on its own, yet it produces the answer from the grafted state. This indicates the answer is already formed at an intermediate layer and read out afterward. Moreover, the layer where formation ends varies widely across inputs. Motivated by these findings, we propose Targeted Unlearning at Layers Identified Per-input (TULIP). For each input, TULIP uses the logit lens to locate the formation-readout boundary and removes the hidden state's alignment with the forget answer's unembedding vector at that layer. TULIP outperforms output- and representation-level baselines on TOFU, PISTOL, and WMDP across Llama, Qwen, and Zephyr models. It also remains robust to paraphrase and quantization attacks. Beyond standalone use, its per-input layer selection serves as a plug-and-play component that further improves existing methods.

Yejin Kim, William F. Shen, Seokwon Jung +2

Korea Advanced Institute of Science & Technology (KAIST) · University of Cambridge · Sookmyung Women’s University

Machine Unlearning LLM Unlearning

Ask PDF

Abstract

Machine Unlearning

Explore similar work

Can LLMs Truly Forget? Revealing Unlearning Gaps Through Adversarial Evaluation

Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs

TULIP: Targeted LLM Unlearning at Layers Identified Per-input