The Geometry of Ignorance: Detecting Suppressed Knowledge in LLMs through Activation Fragmentation

Adar Peleg; Dvir Alsheich; Tomer Ashuach; Avi Mendelson

cs.AINov 3, 2026

The Geometry of Ignorance: Detecting Suppressed Knowledge in LLMs through Activation Fragmentation

Authors: Adar Peleg, Dvir Alsheich, Tomer Ashuach, Avi Mendelson

Organizations: Technion - Israel Institute of Technology

Abstract

Evaluating whether a large language model (LLM) genuinely knows a concept remains a fundamental problem, since behavioral metrics can be fooled by suppression: a model that refuses to answer may still harbor recoverable knowledge. The framework measures the geometric coherence of anon-vectors, the differences in hidden states between a concept-specific query and its anonymized paraphrase, and shows that genuine knowledge forms a tight, low-entropy cluster while ignorance forces geometric fragmentation. Across ten instruction-tuned LLMs from three model families, all fragmentation metrics reliably separate known from unknown concepts, and post-unlearning, fragmentation geometry correlates with adversarial recovery across four attack types and five unlearning algorithms, though the direction of that relationship is shown to depend on attack methodology. Furthermore, a layer-resolved analysis reveals that different adversarial mechanisms exploit this geometry at varying depths, with high attack-specific correlations that are invisible to behavioral evaluation. Together, these results constitute a single-phase, attack-free proxy for unlearning robustness that captures aspects of parametric knowledge inaccessible to output-level metrics. A reference implementation is available at https://github.com/AdarPeleg/geometry-of-ignorance

Figures & tables

Panel of Figure 1 — Figure 1: Activation fragmentation isolates conceptual knowledge. Subtracting the mean hidden state of a cloud of anonymized variants cancels shared linguistic syntax. Plotting the resulting anon-difference vectors ( $\Delta v_{i}$ ) from the origin reveals that parametric knowledge manifests as tight, low-entropy covariance clusters of aligned vectors, whereas ignorance forces geometric fragmentation across high-variance directions.

Metric	$\|\delta\|$	Gap [95% CI]	$p_{\text{adj}}$
Dispersion ( $\downarrow$ HP expected; 10/10 models)
AUSS ${}_{\cos^{2}}$	$21.51$	$0.686$ $[0.667,\,0.705]$	$0.002^{**}$
AUSS ${}_{\text{L2}}$	$18.91$	$1.203$ $[1.166,\,1.239]$	$0.002^{**}$
NormAnonDiff $S_{2}^{\star}$	$4.21$	$1.206$ $[1.022,\,1.332]$	$0.002^{**}$
AnonDiff $S_{2}^{\star}$	$3.50$	$0.804$ $[0.671,\,0.941]$	$0.002^{**}$
Reg $S_{2}^{\dagger}$	$0.92$	$0.117$ $[0.054,\,0.200]$	$0.002^{**}$

Table 1: RQ1: Metric discriminability (HP known vs. TOFU unknown) across 10 LLMs, evaluated at the per-metric layer maximising HP–TOFU separation, sorted by Cohen’s

|d|

. Gap: mean

|\text{HP}{-}\text{TOFU}|

with 95% bootstrap CI.

p_{\text{adj}}

: Wilcoxon signed-rank, BH-corrected (

n{=}10

);

{}^{**}p{<}0.01

. ⋆ : to concept-differential vectors. † : raw-activation baseline, no anon-vector subtraction. Per-model values in Appendix (Tables 5 – 6 ).

Figure 2: RQ2: Mean attack success per unlearning method across four attack types (2

\times

2 grid), averaged over three concepts (error bars = std). Bars are grouped by unlearning method; colour indicates model (LLaMA-2-7B: blue, LLaMA-3-8B: green, Qwen2.5-7B: orange-red). Dashed lines mark chance level (0.5 for AUC-based attacks; 0 for ICL

\Delta

). GA is the only method that consistently drives Steer-AUC near chance across all three models. LLaMA-3 (green) reaches GCG Acc

=0

under every method, revealing strong model-family variation in gradient-attack vulnerability. ICL

\Delta

is near zero across all methods and models; the geometric correlation with ICL recoverability emerges from the layer-resolved analysis (Table 11 ), not from this aggregate view. Full per-run values in Appendix D (Tables 13 – 15 ).

Metric	ICL	GCG	Steering
AUSS ${}_{\text{L2}}$	$-0.51^{**}$	–	–
AUSS ${}_{\cos^{2}}$	$-0.50^{**}$	–	–
$\\|\bar{\Delta\hat{v}}\\|$	$+0.51^{**\dagger}$	–	–
Reg $\overline{\cos}$	–	$-0.51^{**}$	–
Reg $S_{2}$	–	$+0.45^{**}$	–
AnonDiff $S_{2}$	–	–	$+0.48^{**}$

Table 2: RQ2: BH-significant geometry predictors per attack (Spearman

\rho

n{=}44

–

45

non-Base non-WHP runs with complete attack data, BH-corrected within each attack’s 9-metric column). All metrics shown survive BH at

q{<}0.05

; full 9-metric table (all attacks, including non-significant cells) in Appendix Table 9 .

Figure 3: Spearman

\rho

by layer, GCG attack. Each line is one AUSS fragmentation metric; the peak at L29 (AUSS

{}_{\text{Jac}}

\rho{=}{-}0.853

) is marked. The strongest single predictor overall is NormAnonDiff

S_{2}

at L30 (

\rho{=}{-}0.832

, entropy family; Appendix D.5 ). GCG recoverability is governed by late-layer geometry, close to where ICL’s own peak falls (Appendix D.5 ). (Full four-attack sweep in Appendix D.5 .)

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	$B$	$d$ ( $p$ )
$\\|\overline{\Delta\hat{v}}\\|$ / Global_Coh.	2	0.14 (.82)
	5	0.64 (.13)
	10	0.72 (.13)
	20	0.92 (.04)
	50	1.13 (.02)
Batch_SecondMoment_TopEig	2	0.19 (.73)

Appendix

Table 3: Batch-size sensitivity: Cohen’s

d

(Wilcoxon

p

) for HP vs. TOFU separation at

B\in\{2,5,10,20,50\}

n{=}9

models, direction-corrected fresh extraction. All cells are in the hypothesized direction.

Metric	$r$	$d$
$\\|\overline{\Delta\hat{v}}\\|$	0.2	0.42
	0.3	0.67
	0.4	1.34
	0.5	1.51
	0.6	1.72
	0.7	2.21

Appendix

Table 4: Fixed relative-depth sweep (no per-model search),

n{=}9

models, Cohen’s

d

(

p

shown only at

r{=}0.8

, all

p{\leq}0.012

there); sign shown per metric’s own coherence/dispersion convention, all cells correct-direction at

r{=}0.8

. LOMO transfer: leave-one-model-out transfer of the median peak depth from the other 8 models, all 8 metrics, all 9 held-out models (72/72 pairs), identical to the 72/72 oracle per-model search.

	Harry Potter (known)				TOFU (unknown, QA = 0.00)
Model	QA ${}_{\text{HP}}$	$\\|\overline{\Delta\hat{v}}\\|{\uparrow}$	AUSS ${}_{\text{L2}}{\downarrow}$	AUSS ${}_{\cos^{2}}{\downarrow}$	$\\|\overline{\Delta\hat{v}}\\|$	AUSS ${}_{\text{L2}}$	AUSS ${}_{\cos^{2}}$
Qwen family
Qwen-7B-Chat	0.083	0.752	0.952	0.577	0.639	1.309	0.683
Qwen2.5-7B	0.250	0.684	1.150	0.501	0.601	1.402	0.408
Qwen-14B-Chat	0.135	0.804	0.780	0.553	0.688	1.164	0.696
Qwen2.5-14B	0.323	0.697	1.116	0.522	0.689	1.151	0.440

Appendix

Table 5: Fragmentation metrics (HP known vs. TOFU unknown) for all ten LLMs at peak layer

\ell^{*}{=}18

. QA

{}_{\text{HP}}

: QA accuracy on HP; TOFU QA is 0.00 for every model.

\|\overline{\Delta\hat{v}}\|{\uparrow}

: centroid norm (coherence;

\uparrow

= model knows). AUSS

{}_{\text{L2}}{\downarrow}

/ AUSS

{}_{\cos^{2}}{\downarrow}

: dispersion (

\downarrow

= model knows). Bold : HP entry in the theoretically expected direction.

	Harry Potter (known)					TOFU (unknown, QA = 0.00)
	Reg		AnonDiff			Reg		AnonDiff
Model	$S_{2}$	$\overline{\cos}$	$S_{2}$	$\bar{L}_{2}$	$\overline{\cos}$	$S_{2}$	$\overline{\cos}$	$S_{2}$	$\bar{L}_{2}$	$\overline{\cos}$
Qwen family
Qwen-7B-Chat	0.007	0.995	3.808	20.4	0.138	0.043	0.969	4.769	16.9	0.098
Qwen2.5-7B	0.002	0.999	3.715	29.5	0.118	0.009	0.996	4.614	24.7	0.121
Qwen-14B-Chat	0.018	0.990	3.544	21.5	0.166	0.090	0.954	4.795	16.7	0.088

Appendix

Table 6: Gram-matrix entropy (

S_{2}

, Rényi order 2) and companion statistics at per-metric peak layer

\ell^{*}

for ten LLMs. Reg : raw hidden states; AnonDiff : difference vectors

\Delta V=h_{\text{reg}}-h_{\text{anon}}

; applying Gram entropy to

\Delta V

is our adaptation of Skean et al. (2025) .

S_{2}{\downarrow}

: lower = more compressed = model knows concept.

\bar{L}_{2}{\uparrow}

\overline{\cos}{\uparrow}

: higher = stronger/more coherent representation. Bold : HP entry consistent across all 10 models. Reg

\bar{L}_{2}

omitted (

p_{\text{adj}}{=}0.43

, mixed direction).

Metric	vs. TOFU	$d$ ( $p$ )
$\\|\overline{\Delta\hat{v}}\\|$ / Global_Coh.	HP-forget	$+1.58$ (.0039)
	HP-retain	$+3.13$ (.0039)
	Star Wars	$+1.69$ (.0039)
Batch_SecondMoment_TopEig	HP-forget	$+2.57$ (.0039)
	HP-retain	$+6.20$ (.0039)
	Star Wars	$+4.62$ (.0039)

Appendix

Table 7: Four-way domain comparison: Cohen’s

d

(

p

, BH-corrected) for each condition vs. TOFU,

n{=}9

models. All 24 cells are BH-significant (

q\in[0.0063,0.0260]

	Logistic Regr.	MLP (8 hidden)
Accuracy	100%	94.4%
AUC	1.00	1.00
Correct	18/18	17/18
Models fully correct	9/9	8/9

Appendix

Table 8: Classification performance on known (HP) vs. unknown (TOFU) domains using standardized 8-dimensional AUSS features, LOMO-CV,

n{=}9

models (18 samples).

Metric	Steering	ICL	GCG	MIA
AUSS dispersion
AUSS ${}_{\text{L2}}$	$+0.00$	$\mathbf{-0.51}^{**}$	$-0.06$	$+0.04$
AUSS ${}_{\cos^{2}}$	$+0.06$	$\mathbf{-0.50}^{**}$	$-0.01$	$+0.10$
$\\|\overline{\Delta\hat{v}}\\|$	$-0.00$	$\mathbf{+0.51}^{**}$	$+0.06$	$-0.04$
AnonDiff
AnonDiff $\overline{\cos}$	$-0.24$	$+0.27$	$-0.09$	$-0.15$

Appendix

Table 9: Spearman

\rho

between all fragmentation metrics and attack-success rate across

n{=}44

–

45

post-unlearning runs (excluding Base and WHP). ∗∗ / ∗ / † : BH-adjusted

p<0.01/0.05/0.10

within each attack column (9 metrics). ⋆ : adaptation of Gram-matrix entropy to concept-differential vectors ( Skean et al., 2025 ) . Bold : survives BH at

p<0.05

. Only bold, starred cells (

q{<}0.05

) represent statistically robust findings; unmarked values are shown for completeness so the full 9-metric

\times

4-attack grid is auditable, and should not be read as evidence of a relationship. None of MIA’s AUSS/AnonDiff correlations survive BH correction; see Section 4 .

Family	Metric	Attack	$\rho$	$p$	$q_{\text{global}}$
AUSS	Centroid_Norm	ICL	+0.551	0.00002	0.0001 **
AUSS	AUSS ${}_{\cos^{2}}$	ICL	$-$ 0.572	0.00001	0.0001 **
AUSS	AUSS L2	ICL	$-$ 0.551	0.00002	0.0001 **
AUSS	Global_Coherence	ICL	+0.551	0.00002	0.0001 **
AUSS	Batch_Cov_Trace	ICL	$-$ 0.551	0.00002	0.0001 **
AUSS	Batch_SecMom_TopEig	ICL	+0.557	0.00001	0.0001 **

Appendix

Table 10: Full Xu et al. head-to-head comparison, sorted by

q_{\text{global}}

{}^{**}/{}^{*}

q{<}0.01/0.05

Figure 4: Spearman

\rho

by layer for all four attacks, leader metrics. Solid lines: AUSS family (AUSS

{}_{\text{L2}}

, AUSS

{}_{\cos^{2}}

, AUSS

{}_{\text{Jac}}

). Dashed lines: Entropy family (Reg

S_{2}

, AnonDiff

\overline{\cos}

, NormAnonDiff

S_{2}

). Dotted red vertical line: attack-specific peak layer among the six plotted metrics (GCG→L29, ICL→L00 for this restricted set; see Table 11 for the full-metric-universe peaks used in the main text, GCG→L29, ICL→L28). GCG and ICL both concentrate their strongest signal late in the network; MIA and Steering show weaker, non-BH-significant layer-specific signal in this dataset.

n{=}10

–

15

runs per attack (complete data).

Attack	Family	Best layer	Best metric	$\rho$	$p$	$q_{\text{attack}}$ / $q_{\text{global}}$
GCG	AUSS	L29	AUSS ${}_{\text{Jac}}$	$-0.853$	$0.0008$	$0.057$ / $0.046^{*}$
GCG	Entropy	L30	NormAnonDiff $S_{2}$ $\star$	$-0.832$	$0.0015$	$0.057$ / $0.046^{*}$
ICL	AUSS	L28	Batch_Cov_TopEig $\star$	$-0.886$	${<}0.0001$	$0.006^{}$ / $0.009^{}$
ICL	Entropy	L00	AnonDiff $S_{2}$	$-0.861$	$0.0002$	$0.011^{}$ / $0.037^{}$
MIA	AUSS	L31	Centroid_Norm	$-0.446$	$0.170$	$0.999$ / $0.581$
MIA	Entropy	L32	Reg $\bar{L}_{2}$ $\star$	$+0.627$	$0.039$	$0.999$ / $0.254$

Appendix

Table 11: Best (layer, metric) predictor per attack in the layer-resolved analysis.

n

: number of runs with complete data at that layer. AUSS and entropy families are shown separately; the overall best metric across both is marked

\star

q_{\text{attack}}

: BH correction within the attack’s own column (8 AUSS + entropy tests);

q_{\text{global}}

: BH correction across all 4,224 (layer, metric, attack) tests at once.

{}^{*}q{<}0.05

Attack	Metrics with 0 LOO sign flips	BH-significant ( $q{<}0.05$ )
ICL	8/8	7/8
Steering	8/8	0/8
GCG	7/8	0/8
MIA	2/8	0/8

Appendix

Table 12: Leave-one-out sign-flip stability, 8 AUSS metrics per attack,

n{=}26

–

27

verified runs.

			Forget	Geometry $\ell^{*}$			Attack Success $\downarrow$
Method	Cpt	$\checkmark$	ROUGE-L $\downarrow$	$\\|\overline{\Delta\hat{v}}\\|{\uparrow}$	AUSS ${}_{\text{L2}}{\downarrow}$	AUSS ${}_{\cos^{2}}{\downarrow}$	Steer-AUC	Steer-WF	ICL $\Delta$	MIA	GCG
Base	HP		0.131	0.878	0.507	0.439	0.428	0.265	+0.000	0.827	0.000
	SW		0.152	0.876	0.516	0.447	0.449	0.296	+0.120	0.955	0.000
	WS		0.090	0.911	0.379	0.341	0.503	0.274	+0.040	0.837	0.000
GA	HP	✓	0.000	0.823	0.717	0.553	0.500	0.000	+0.000	0.678	0.000
	SW	✓	0.000	0.875	0.521	0.442	0.500	0.000	+0.000	0.695	0.000

Appendix

Table 13: RQ2: Full post-unlearning results, LLaMA-2-7B-chat . ROUGE-L

\downarrow

: forget-set ROUGE-L after unlearning.

\checkmark

: verified (ROUGE-L

<0.10

& retain

\geq 0.80\times

base).

\|\overline{\Delta\hat{v}}\|{\uparrow}

: centroid norm; AUSS

{}_{\text{L2}}{\downarrow}

, AUSS

{}_{\cos^{2}}{\downarrow}

: dispersion (per-method peak layer

\ell^{*}

). Steer-AUC

\downarrow

: steering AUC; Steer-WF

\downarrow

: mean steered word-frequency score. ICL

\Delta{\uparrow}

: ICL accuracy minus baseline. MIA

\downarrow

: membership inference AUC; GCG

\downarrow

: GCG ASR. Blank cells: checkpoint unavailable.

			Forget	Geometry $\ell^{*}$			Attack Success $\downarrow$
Method	Cpt	$\checkmark$	ROUGE-L $\downarrow$	$\\|\overline{\Delta\hat{v}}\\|{\uparrow}$	AUSS ${}_{\text{L2}}{\downarrow}$	AUSS ${}_{\cos^{2}}{\downarrow}$	Steer-AUC	Steer-WF	ICL $\Delta$	MIA	GCG
Base	HP		0.062	0.856	0.594	0.493	0.423	0.437	+0.000	0.915	0.000
	SW		0.113	0.873	0.529	0.455	0.424	0.379	+0.040	0.987	0.000
	WS		0.068	0.901	0.416	0.370	0.425	0.347	+0.040	0.952	0.000
GA	HP	✓	0.000	0.872	0.534	0.439	0.500	0.000	+0.000	0.700	0.000
	SW	✓	0.000	0.801	0.797	0.591	0.500	0.000	+0.000	0.102	0.000

Appendix

Table 14: RQ2: Full post-unlearning results, Meta-LLaMA-3-8B-Instruct . ROUGE-L

\downarrow

: forget-set ROUGE-L after unlearning.

\checkmark

: verified (ROUGE-L

<0.10

& retain

\geq 0.80\times

base).

\|\overline{\Delta\hat{v}}\|{\uparrow}

: centroid norm; AUSS

{}_{\text{L2}}{\downarrow}

, AUSS

{}_{\cos^{2}}{\downarrow}

: dispersion (per-method peak layer

\ell^{*}

). Steer-AUC

\downarrow

: steering AUC; Steer-WF

\downarrow

: mean steered word-frequency score. ICL

\Delta{\uparrow}

: ICL accuracy minus baseline. MIA

\downarrow

: membership inference AUC; GCG

\downarrow

: GCG ASR. Blank cells: checkpoint unavailable.

			Forget	Geometry $\ell^{*}$			Attack Success $\downarrow$
Method	Cpt	$\checkmark$	ROUGE-L $\downarrow$	$\\|\overline{\Delta\hat{v}}\\|{\uparrow}$	AUSS ${}_{\text{L2}}{\downarrow}$	AUSS ${}_{\cos^{2}}{\downarrow}$	Steer-AUC	Steer-WF	ICL $\Delta$	MIA	GCG
Base	HP		0.106	0.872	0.534	0.439	0.433	0.373	+0.040	0.982	0.000
	SW		0.098	0.866	0.557	0.473	0.482	0.261	+0.080	0.982	0.000
	WS		0.080	0.907	0.395	0.352	0.495	0.373	+0.000	0.938	0.000
GA	HP	✓	0.000	0.936	0.275	0.253	0.500	0.000	+0.000	0.119	0.000
	SW	✓	0.000	0.901	0.417	0.370	0.500	0.000	+0.000	0.461	0.000

Appendix

Table 15: RQ2: Full post-unlearning results, Qwen2.5-7B-Instruct . ROUGE-L

\downarrow

: forget-set ROUGE-L after unlearning.

\checkmark

: verified (ROUGE-L

<0.10

& retain

\geq 0.80\times

base).

\|\overline{\Delta\hat{v}}\|{\uparrow}

: centroid norm; AUSS

{}_{\text{L2}}{\downarrow}

, AUSS

{}_{\cos^{2}}{\downarrow}

: dispersion (per-method peak layer

\ell^{*}

). Steer-AUC

\downarrow

: steering AUC; Steer-WF

\downarrow

: mean steered word-frequency score. ICL

\Delta{\uparrow}

: ICL accuracy minus baseline. MIA

\downarrow

: membership inference AUC; GCG

\downarrow

: GCG ASR. Blank cells: checkpoint unavailable.

Factual Knowledge in Language Models

Factual knowledge in language models concerns what facts they store, how reliably they retrieve them, and how that knowledge can be tested or changed.

See all 204 papers

Can LLMs Truly Forget? Revealing Unlearning Gaps Through Adversarial Evaluation
Aug 21, 2026Ayush Gupta, Hima Varshini Surisetty, Sreevidya Bollineni +5Machine Unlearning LLM Unlearning
University of Massachusetts Amherst · Microsoft Corporation
Do LLMs Really Forget? Hidden-State Leakage in Model Unlearning and How to Fix it
Sep 29, 2026Hadi Reisizadeh, Jiajun Ruan, Sijia Liu +1Machine Unlearning LLM Unlearning
University of Minnesota · Michigan State University · IBM Research
PreUnlearn: Auditing Collateral Knowledge Damage Before Large Language Model Unlearning
Jun 16, 2026Bo Su, Ankit Shah, Thai LeLLM Auditing Machine Unlearning
Indiana University Bloomington, IN, USA

Aug 21, 2026J/K move · Enter open · S save

Can LLMs Truly Forget? Revealing Unlearning Gaps Through Adversarial Evaluation

Ayush Gupta, Hima Varshini Surisetty, Sreevidya Bollineni, Varad Ingale+4

University of Massachusetts Amherst · Microsoft Corporation

Machine Unlearning LLM Unlearning Machine Unlearning Evaluation

Open paper AskPDF arXiv

Machine unlearning aims to remove the influence of targeted training data from a model while preserving its remaining capabilities, but evaluating whether such information has truly become inaccessible remains challenging. Existing benchmarks primarily assess unlearning under clean, non-adversarial queries, leaving open whether information that appears forgotten can still be recovered through strategic prompting. We address this gap through a unified evaluation of prompt-based and fine-tuning-based unlearning methods on TOFU using Llama-3.2-3B-Instruct, followed by an adversarial robustness evaluation of methods that perform strongly under standard metrics. We introduce Attack Success Rate (ASR), an LLM-as-judge metric that measures the fraction of adversarial responses whose leakage score exceeds

0.2

, and evaluate recovery across eight attack suites. Our results reveal a substantial gap between clean-query forgetting and adversarial robustness. Although several fine-tuning-based methods achieve Forget Quality above

0.91

, targeted information remains recoverable with ASRs between

72.8\%

and

84.3\%

, close to the

87.5\%

ASR of the unprotected base model. In contrast, clean multilingual reformulations yield only

2.95\%

measured leakage. A manual audit further finds agreement between binary ASR decisions and human factual assessments in seven of ten cases, indicating that ASR provides a useful, though imperfect, signal of behavioral recoverability. These findings show that strong standard-metric performance alone is insufficient to establish robustness after unlearning and motivate adversarial stress-testing as a complementary component of unlearning evaluation.

Abstract

Factual Knowledge in Language Models

Explore similar work

Can LLMs Truly Forget? Revealing Unlearning Gaps Through Adversarial Evaluation

Do LLMs Really Forget? Hidden-State Leakage in Model Unlearning and How to Fix it

PreUnlearn: Auditing Collateral Knowledge Damage Before Large Language Model Unlearning