Whitening Improves Robustness to Spurious Correlations in Linear Probes

Floris Holstege; Bram Wouters; Noud van Giersbergen; Cees Diks

cs.LGSep 30, 2026

Whitening Improves Robustness to Spurious Correlations in Linear Probes

Authors: Floris Holstege, Bram Wouters, Noud van Giersbergen, Cees Diks

Organizations: University of Amsterdam, Department of Quantitative Economics · Tinbergen Institute

Abstract

Deep neural networks tend to rely on simple features that may be spurious and thus fail to generalize. We study this problem in the setting of linear probes, where a (generalized) linear model is fitted on the representations of a (pretrained) model. We use the connection of these models to the max-margin classifier, and show they favor directions associated with large eigenvalues of the covariance matrix. Whitening removes this preference by equalizing the eigenvalues of the covariance matrix. This observation motivates whitening as a preprocessing step that can reduce reliance on spurious correlations without requiring prior knowledge of their presence or labeled data. We examine the effect of whitening on a synthetic data-generating process and standard spurious correlation benchmarks, and find that it improves robustness. We also find that whitening can improve robustness when added to existing approaches.

Figures & tables

Figure 1: Predictions for ERM (left) and ERM after whitening (right) for $d=2000$ , plotted against the core coordinate $x_{c}$ and spurious coordinate $x_{s}$ , for 1000 sampled data points. Colors show predictions using all $d$ coordinates, and marker shapes show true classes. Background colors show the probability of predicting class $1$ , averaged over the simulations, based on $x_{c}$ and $x_{s}$ .

Figure 2: OOD test accuracy as a function of the noise-to-sample ratio $q/n$ for $n=1000$ . We compare ERM with three preprocessing choices: no preprocessing, standardization, and whitening, in terms of overall accuracy (left) and worst-group accuracy (right), where groups are defined by the pair $(y,a)$ . Points represent means over 100 simulations, shaded bands are 95% confidence intervals.

Figure 3: Worst-group (left) and equal-group (right) test accuracy for ERM with three preprocessing choices: no preprocessing, standardization, and whitening (top), together with mean within-seed paired differences for whitening minus each reference (bottom). Bars and points represent averages across seeds (ten for ResNet-50, five for BERT), and error bars are 95% confidence intervals.

Figure 4: Variance explained by the top $k$ eigenvalues, up to $k=500$ (left), and the (normalized) Rayleigh quotient for different classifiers (right). We take the classifier $\bm{w}$ after ERM with different preprocessing procedures and convert the classifier to the original coordinates, before calculating the Rayleigh quotient. Points and bars represent the average across seeds (ten for ResNet-50 and five for BERT), and error bars the 95% confidence intervals.

Figure 5: Worst-group test accuracy for DFR, AFR and NT for three preprocessing variants: no preprocessing, standardization, and whitening (top), together with mean within-seed paired differences for whitening minus each reference (bottom). Bars and points represent averages across seeds (ten for ResNet-50, five for BERT), and error bars are 95% confidence intervals.

Figure 6: Normalized Rayleigh quotient for Deep Feature Reweighting (DFR), Automatic Feature Reweighting (AFR) and NeuronTune (NT) for three preprocessing variants: no preprocessing, standardization, and whitening. Points and bars represent the average across seeds (ten for ResNet-50 and five for BERT), and error bars are 95% confidence intervals.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Estimated core coefficient $\hat{w}_{c}$ (left), spurious coefficient $\hat{w}_{s}$ (center), and normalized squared noise-coefficient norm $\lVert\hat{\bm{w}}_{\epsilon}\rVert_{2}^{2}/n$ (right) as functions of the training sample size $n$ for $q/n=1.2$ . Solid black curves are simulation averages, vertical error bars are 95% confidence intervals, and dashed lines are the corresponding limits in Theorem 1 .

Figure 8: Fitted core coefficient $\hat{w}_{c}$ (left), spurious coefficient $\hat{w}_{s}$ (center), and normalized squared noise-coefficient norm $\lVert\hat{\bm{w}}_{\epsilon}\rVert_{2}^{2}/n$ (right) as functions of the training sample size $n$ for $q/n=0.8$ . For each simulated sample, logistic regression is fitted after uncentered empirical whitening and the original-coordinate coefficients are normalized to unit minimum training margin. Solid black curves are simulation averages, vertical error bars are 95% confidence intervals, and dashed lines are the exact core-only values from Proposition 1 .

Figure 10: OOD test accuracy as a function of the noise-to-sample ratio $q/n$ for $n=1000$ . We compare ERM with ERM after empirical whitening in terms of overall accuracy (left) and worst-group accuracy (right), where groups are defined by the pair $(y,a)$ . Points are means over 100 simulations, shaded bands are 95% confidence intervals, and the dotted line marks $q/n=1$ .

Figure 11: Worst-group (left) and equal-group (right) test accuracy for ERM with empirical whitening and whitening with the nonlinear shrinkage estimator (top), together with mean within-seed paired differences for whitening with the nonlinear shrinkage estimator minus empirical whitening (bottom). Bars and points represent averages across seeds (ten for ResNet-50, five for BERT), and error bars are 95% confidence intervals.

Figure 12: Worst-group (left) and equal-group (right) test accuracy for DINO ViT-B/16 on Waterbirds and CelebA and DeBERTa-v3-base on MultiNLI under three preprocessing variants: no preprocessing, standardization, and whitening (top), together with mean within-seed paired differences for whitening minus each reference (bottom). Bars and points represent averages across three seeds, and error bars are 95% confidence intervals.

Figure 13: Equal-group test accuracy for Deep Feature Reweighting (DFR), Automatic Feature Reweighting (AFR), and NeuronTune (NT) under three preprocessing variants: no preprocessing, standardization, and whitening (top), together with mean within-seed paired differences for whitening minus no preprocessing and whitening minus standardization (bottom). Bars and points show averages across seeds (ten for ResNet-50, five for BERT), and error bars show 95% confidence intervals.

Statistic	Waterbirds	CelebA	MultiNLI
Train total	4,795	162,770	206,175
Train per class	$y=0$ : 3,670 $y=1$ : 1,125	$y=0$ : 138,503 $y=1$ : 24,267	$y=0$ : 68,656 $y=1$ : 68,897 $y=2$ : 68,622
Train per group	$g=1$ : 3,486 $g=2$ : 184 $g=3$ : 56 $g=4$ : 1,069	$g=1$ : 66,874 $g=2$ : 71,629 $g=3$ : 1,387 $g=4$ : 22,880	$g=0$ : 57,498 $g=1$ : 11,158 $g=2$ : 67,376 $g=3$ : 1,521 $g=4$ : 66,630 $g=5$ : 1,992
Val. total	1,199	19,867	82,462
Val. per class	$y=0$ : 945 $y=1$ : 254	$y=0$ : 16,811 $y=1$ : 3,056	$y=0$ : 27,448 $y=1$ : 27,562 $y=2$ : 27,452
Val. per group	$g=1$ : 898 $g=2$ : 47 $g=3$ : 13 $g=4$ : 241	$g=1$ : 8,276 $g=2$ : 8,535 $g=3$ : 182 $g=4$ : 2,874	$g=0$ : 22,814 $g=1$ : 4,634 $g=2$ : 26,949 $g=3$ : 613 $g=4$ : 26,655 $g=5$ : 797

Appendix

Table 1: Training and validation split sizes for Waterbirds, CelebA, and MultiNLI. The table reports total observations and counts by class

y

and group

g

. For Waterbirds and CelebA, groups are indexed by

g=2y+a+1

, where

a

is the binary spurious attribute. For MultiNLI, groups are indexed by

g=2y+a

Explore similar work

Spectral Overfitting in Noisy Linear Probing of Pretrained Representations
Mar 2, 2026Zice Wang, Zhenyu ZhangNoisy Labels Overfitting
School of Future Technology, Northeastern University, Shenyang, China.
Comparing Linear Probes with Mahalanobis Cosine Similarity
Jun 17, 2026Zhuofan Josh Ying, Peter Hase, Nikolaus KriegeskorteCosine Similarity Interpretability
Departments of 1Psychology, 2Neuroscience, 3Electrical Engineering · 4Zuckerman Mind Brain Behavior Institute; Columbia University, New York, NY · 5Stanford University, Stanford, CA +1
Mitigating Spurious Correlations with Memorization-Guided Dataset De-Biasing
Jun 1, 2026Arda Fazla, Abolfazl HashemiSpurious Correlations Selection Bias
School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47907, USA.

Mar 2, 2026, cs.LGJ/K move · Enter open · S save

Spectral Overfitting in Noisy Linear Probing of Pretrained Representations

Zice Wang, Zhenyu Zhang

School of Future Technology, Northeastern University, Shenyang, China.

Noisy Labels Overfitting Principal Component Analysis Pretraining Frozen Backbone

Open paper AskPDF arXiv

Frozen pretrained features are often treated as a safe interface for downstream learning: only a small linear readout is trained, while the backbone is fixed. We show that this readout can still overfit noisy labels in a structured way. A label-blind PCA rank sweep reveals a sharp spectral pattern: under label noise, exposing all pretrained directions can hurt clean accuracy, and intermediate ranks often recover much of the lost performance. Rank-matched random projections help less, and measured between-class signal is strongly concentrated in leading PCs. The pattern appears across three ImageNet-pretrained backbones on CIFAR-10, with gains up to

36.0\pm0.8

points over the default full-rank probe at 40% noise. Tuned full-rank probes outperform validation-selected PCA probes, so we present the sweep as a diagnostic of spectral overfitting rather than a competitive noisy-label method.

Abstract

Explore similar work

Spectral Overfitting in Noisy Linear Probing of Pretrained Representations

Comparing Linear Probes with Mahalanobis Cosine Similarity

Mitigating Spurious Correlations with Memorization-Guided Dataset De-Biasing