Whitening Inverts the Hierarchy: What the Norm of a Whitened Embedding Measures

Mohammed Ahnouch; Lotfi Elaachak

cs.LGSep 19, 2026

Whitening Inverts the Hierarchy: What the Norm of a Whitened Embedding Measures

Authors: Mohammed Ahnouch, Lotfi Elaachak

Organizations: Université Paris 1 Paris, France · Faculty of Science and Technology of Tangier Abdelmalek Essaadi University Tangier, Morocco

Abstract

Whitening a foundation-model embedding and using its squared norm as a training-free likelihood surrogate is motivated by the observation that whitened coordinates often appear approximately standard normal. We show that this observation follows from the projection central limit theorem and therefore does not imply a Gaussian joint distribution. Across multiple encoders and three training objectives, we find systematic over-dispersion of the whitened radius relative to the Gaussian reference, including against distributional clones with identical mean and covariance. We further show that the commonly reported agreement between empirical and theoretical norm statistics is an algebraic consequence of in-sample whitening and does not constitute evidence for Gaussianity. We identify the mechanism behind this behavior: whitening reverses the encoder's spectral hierarchy, shifting the contribution to the squared norm toward near-degenerate directions that encode predominantly noise. In these directions, the dominant variability is governed by a single input-dependent scale. We estimate this scale from two moments and use it to predict, without additional free parameters, the cross-dependence between disjoint spectral halves. These results indicate that the squared whitened norm is better interpreted as a Mahalanobis measure of semantic atypicality than as a log-likelihood. This interpretation explains both its practical effectiveness and its calibration failures: the statistic can rank and detect atypical samples consistently with nonparametric density estimates and across encoders trained with different objectives, while Gaussian tail thresholds can be inaccurate by orders of magnitude. etc.

Figures & tables

Panel of Figure 1 — Figure 1: The polar decomposition of a whitened embedding, and the division this paper is about. The directional factor is what a contrastive loss optimises; the radial factor is the whole of the likelihood surrogate, and Theorem 5 shows the objective cannot determine it.

	images			captions
	published	measured	Gaussian	published	measured	Gaussian
radial mean	$27.43$	$27.43$	$27.7038$	$28.49$	$27.07$	$27.7038$
radial sd	$3.94$	$3.92$	$0.649$	$5.72$	$5.18$	$0.649$
AD pass	$98.3\%$	$98.7\%$	$100.0\%$	$90.1\%$	$92.3\%$	$100.0\%$
mean $A^{2}$	$0.4890$	$0.4951$	$0.3885$	$0.5926$	$0.5802$	$0.3885$
DP pass	$99.3\%$	$99.3\%$	$100.0\%$	$99.2\%$	$99.2\%$	$100.0\%$

Table 1: Published summary statistics against the same statistics on our own embeddings, in-sample whitening at the published group size

250

. “Gaussian” is an exact Gaussian of the same shape through the same pipeline, its radial sd the mean of

30

draws, spread

\pm 0.006

statistic	modality	published	exact Gaussian null	$z$	measured
mean Anderson–Darling $A^{2}$	images	$0.4890$	$0.3891\pm 0.0016$	$+62$	$0.4951$
	captions	$0.5926$	$0.3891\pm 0.0016$	$+126$	$0.5802$
mean D’Agostino–Pearson $p$	images	$0.3624$	$0.5052\pm 0.0021$	$-67$	$0.3630$
	captions	$0.2568$	$0.5052\pm 0.0021$	$-117$	$0.2741$

Table 2: Published grand averages against the null law of the protocol itself, obtained by simulating the entire protocol on exact Gaussians

24

times, with our measured values alongside.

	statistic	data	Gaussian clone	spherical clone
images	radial sd	$5.99$	$1.02$	$4.78$
	kurtosis, principal axes	$+0.5423\pm 0.0516$	$-0.0020\pm 0.0028$	$+0.2514\pm 0.0149$
	kurtosis, encoder’s own axes	$+0.0216\pm 0.0031$	$+0.0025\pm 0.0045$	$+0.2476\pm 0.0132$
	$\mathrm{corr}(R,\|\langle u,v_{1}\rangle\|)$	$-0.4533\pm 0.0229$	$+0.0036\pm 0.0110$	$-0.0005\pm 0.0179$
captions	radial sd	$8.09$	$1.01$	$6.39$
	kurtosis, principal axes	$+0.9532\pm 0.0609$	$-0.0024\pm 0.0026$	$+0.4354\pm 0.0136$

Table 3: Data against covariance-matched clones; split-half whitening, axes from the reference half, five splits. A spherical law has the same coordinate kurtosis in every basis and a radius independent of direction. Both clones do; the data does neither.

encoder	$d$	ends in	raw cv	over	$\gamma_{2}$ raw	$\gamma_{2}$ PC	$\mathrm{corr}(R,\|u_{1}\|)$
CLIP ViT-L/14, openai	768	linear projection	0.0444	5.90 $\times$	+0.022	+0.542	-0.453
CLIP ViT-B/32, openai	512	linear projection	0.0528	6.30 $\times$	+0.009	+1.523	-0.435
CLIP ViT-B/32, LAION-2B	512	linear projection	0.0505	6.64 $\times$	-0.008	+1.438	-0.421
CLIP ViT-B/32, DataComp	512	linear projection	0.0442	6.62 $\times$	-0.012	+1.494	-0.413
SigLIP ViT-B/16, WebLI	768	attention pool	0.0710	7.55 $\times$	+0.039	+0.922	-0.512
DINOv2 ViT-B/14	768	LayerNorm	0.0199	6.42 $\times$	+0.007	+0.687	-0.505

Table 4: Six encoders, the same

5{,}000

images, the same battery. “raw cv” is the coefficient of variation of

\lVert z\rVert

before whitening; “over” is the whitened radial sd divided by a Gaussian clone’s under the split-half protocol of Table 3 , which is why CLIP-L/14 reads

5.90\times

here and

6.04\times

in Table 1 .

	$k$	head share		over	halves	$V$	$\rho_{s}$	$V$ , one	tail $\gamma_{2}$		tail
		raw	whitened					scale	$3V$	measured	var
CLIP-L/14 images	95	66%	12%	42.5	+0.76	0.137	0.79	0.123	0.41	0.49	96%
CLIP-L/14 captions	67	69%	9%	73.9	+0.85	0.221	0.88	0.207	0.66	0.70	101%
CLIP-B/32 images	68	67%	13%	50.7	+0.41	0.312	0.42	0.223	0.94	1.37	99%
SigLIP images	73	66%	10%	64.7	+0.71	0.211	0.73	0.183	0.63	0.77	99%
DINOv2 images	205	69%	27%	46.4	+0.91	0.164	0.96	0.161	0.49	0.45	82%

Table 5: The inversion and the scale model of Proposition 4 , in-sample PCA whitening. “head share” is the share of the norm carried by the top-

k

directions before and after whitening. “over” is the tail-energy variance relative to a Gaussian clone and “halves” the correlation between the energies of the two halves of the tail; these two measurements determine

(V,\rho_{s})

jointly through eq. 10 . “

V

, one scale” is what the over-dispersion alone would give under

\rho_{s}=1

, shown to expose the bias. “tail

\gamma_{2}

” is the tail coordinate kurtosis, as implied by the fitted scale (

3V

) and as measured; the excess above

3V

is scale variation within a half, which

\rho_{s}

does not capture. “tail var” is the share of

\mathrm{Var}(\lVert w\rVert^{2})

carried by the tail.

Panel of Figure 2 — Figure 2: The mechanism in three panels. (a) Whitening amplifies the encoder’s least-informative directions, and that is where the non-Gaussianity is; the Gaussian clone is flat at zero. (b) Two disjoint halves of the tail have energies that move together for real embeddings and not for the clone: one per-input scale multiplies the whole tail. (c) That scale is semantic atypicality.

	COCO	CIFAR-100	pets	flowers	DTD	SVHN	within / between
CLIP ViT-B/32	5.54 $\times$	5.75 $\times$	6.72 $\times$	6.47 $\times$	6.02 $\times$	8.11 $\times$	$0.105$ / $0.929$
DINOv2 ViT-B/14	6.18 $\times$	4.18 $\times$	6.08 $\times$	6.13 $\times$	7.82 $\times$	7.62 $\times$	$0.149$ / $1.312$

Table 6: Over-dispersion by corpus, two encoders. The within-corpus figure is the standard deviation of a single block, estimated from disjoint pairs of the same distribution.

score	CLIP ViT-B/32		DINOv2 ViT-B/14
	COCO in	CIFAR-100 in	COCO in	CIFAR-100 in
Mahalanobis, $\lVert w\rVert^{2}$	0.895	0.947	0.974	0.907
tail term only	0.896	0.949	0.973	0.916
head term only	0.414	0.801	0.167	0.212
Mahalanobis ++	0.853	0.944	0.973	0.954
$k$ -NN cosine distance	0.688	0.895	0.957	0.939

Table 7: AUROC on a corpus-shift benchmark, averaged over five out-distributions, with the in-distribution fitted on one block of

2{,}500

images and evaluated on a disjoint block. “tail” and “head” are the parts of

\lVert w\rVert^{2}

beyond and within the effective rank. Mahalanobis ++ is Mahalanobis after

L^{2}

normalisation.

nominal	images		captions
	Gaussian	recalibrated	Gaussian	recalibrated
$10\%$	$38.3\%$	$10.1\%$	$38.4\%$	$10.0\%$
$5\%$	$35.8\%$	$5.0\%$	$36.5\%$	$5.1\%$
$1\%$	$31.2\%$	$1.0\%$	$33.0\%$	$1.1\%$
$0.1\%$	$26.5\%$	$0.1\%$	$29.3\%$	$0.1\%$

Table 8: Actual exceedance of a threshold set from the Gaussian surrogate, and from the one-dimensional recalibration eq. 11 , when the truth is the measured radial law.

Feature Whitening

See all 10 papers

In Defense of Cosine Similarity: Normalization Eliminates the Gauge Freedom
Feb 23, 2026Taha BouhsineHyperspherical Representation Learning
Whitening Improves Robustness to Spurious Correlations in Linear Probes
Sep 30, 2026Floris Holstege, Bram Wouters, Noud van Giersbergen +1Linear Probing Feature Whitening
University of Amsterdam, Department of Quantitative Economics · Tinbergen Institute
Calibrating WEAT Against Anisotropy: ZCA Whitening as a Geometric Pre-Processing Step for Embedding Association Tests
Aug 7, 2026Seitaro Ono, Senna Ross, Jun SaikiNeural Representation Geometry Text Embedding Evaluation
Graduate School of Human and Environmental Studies, Kyoto University, Kyoto, Japan · Brock University, St. Catharines, Ontario, Canada

Feb 23, 2026J/K move · Enter open · S save

In Defense of Cosine Similarity: Normalization Eliminates the Gauge Freedom

Taha Bouhsine

Hyperspherical Representation Learning

Open paper AskPDF arXiv

Steck, Ekanadham, and Kallus [arXiv:2403.05440] demonstrate that cosine similarity of learned embeddings from matrix factorization models can be rendered arbitrary by a diagonal

gauge'' matrix $D$. Their result is correct and important for practitioners who compute cosine similarity on embeddings trained with dot-product objectives. However, we argue that their conclusion, cautioning against cosine similarity in general, conflates the pathology of an incompatible training objective with the geometric validity of cosine distance on the unit sphere. We prove that when embeddings are constrained to the unit sphere $\mathbb{S}^{d-1}$ (either during or after training with an appropriate objective), the $D$-matrix ambiguity vanishes identically, and cosine distance reduces to exactly half the squared Euclidean distance. This monotonic equivalence implies that cosine-based and Euclidean-based neighbor rankings are identical on normalized embeddings. The

problem'' with cosine similarity is not cosine similarity, it is the failure to normalize.

Abstract

Feature Whitening

Explore similar work

In Defense of Cosine Similarity: Normalization Eliminates the Gauge Freedom

Whitening Improves Robustness to Spurious Correlations in Linear Probes

Calibrating WEAT Against Anisotropy: ZCA Whitening as a Geometric Pre-Processing Step for Embedding Association Tests