Diagonal Attenuation: A Finite-Sample Correction for PCA

Qiang Sun

stat.MLSep 5, 2026

Diagonal Attenuation: A Finite-Sample Correction for PCA

Authors: Qiang Sun

Organizations: University of Toronto and MBZUAI

Abstract

Principal component analysis (PCA) can rotate away from its population target when a covariance matrix is estimated from limited data. We introduce diagonal attenuation, which preserves sample cross-covariances while reducing coordinatewise sample variances. The method is revealed exactly by averaging a linear full-output reconstruction loss over random input masks; studying the correction directly extends it beyond the range attainable by masking. We isolate the part of the random coupling between retained and omitted population directions that is contributed by sample-variance errors, and show how attenuation can reduce the resulting rotation. Under balanced marginal variances, we derive an explicit expected-risk theorem, uniform over the attenuation path for all sufficiently large finite samples, and obtain the asymptotically risk-minimizing strength. For general covariances, we characterize when attenuation leaves the population PCA subspace unchanged and give a risk theorem that also accounts for changing eigengaps and the population cost when the target moves. Simulations track this tradeoff from exact preservation back to PCA. Across local image patches, speech spectra, and smartphone acceleration, both mask-derived and direct attenuation improve PCA under two fitting-sample budgets, and one of them has the largest mean gain among seven methods in every data--budget cell. The full path selects strengths beyond the mask-derived boundary on $63\%$ -- $95\%$ of the subsamples.

Figures & tables

Figure 1: Method overview. Left: averaging the full-output reconstruction loss in ( 1.1 ) over random input masks yields ( 1.2 ). Center: input masking produces the first part of the attenuation path, while direct attenuation extends the same correction from PCA to diagonal deletion. Right: the oracle strength balances the reduction in sampling-induced rotation against any population cost caused by moving the target, and may lie beyond the mask-derived range.

Figure 2: Testing the risk theory from preserved to moving targets. (a) Balanced marginal variances and (b) unequal marginal variances whose diagonal preserves the PCA target. Points and bars are Monte Carlo means and 95% intervals; dashed curves are the theoretical risk curve from ( 3.20 ); the vertical scale is the

n

-scaled expected clean excess risk. Blue shading marks the mask-derived range

0\leq\alpha<1/2

, with

\alpha=1/2

shown as its limiting endpoint. (c) Along ( 4.2 ), the upper axis compares the strengths selected by the theoretical risk curve (line) and Monte Carlo mean risk (points), and the lower axis reports the corresponding risk reductions relative to PCA. The horizontal coordinate is

\|P_{*,\phi}^{\perp}D_{\phi}P_{*,\phi}\|_{\mathrm{F}}

, so moving right means greater population target movement.

Figure 3: Seven-method fixed fitting-sample-budget comparison across three real-data modalities. Each row gives the mean percentage of PCA excess reconstruction risk removed by one alternative; positive values favor that method. Filled circles and open diamonds correspond to

N=64

and

128

, respectively, and the shaded rows are the two attenuation paths. Optdigits intervals use ten patch-location means, each averaging 100 training-subsample fits; Phoneme and HAR intervals use 100 training subsamples. Within each fit, all methods use the same sampled observations, validation folds, and official test set. HeteroPCA means beyond the displayed range are marked at the left boundary and labeled by their values.

	Optdigits patches		Phoneme		HAR
Method	$N=64$	$N=128$	$N=64$	$N=128$	$N=64$	$N=128$
PCA	$0.00$	$0.00$	$0.00$	$0.00$	$0.00$	$0.00$
Mask-derived DA	$4.32$	$4.18$	$0.84$	$\mathbf{0.72}$	$0.65$	$0.93$
Direct DA	$\mathbf{6.74}$	$\mathbf{6.40}$	$\mathbf{1.09}$	$0.70$	$\mathbf{0.76}$	$\mathbf{1.20}$
Correlation PCA	$5.84$	$5.41$	$-0.14$	$-0.52$	$0.36$	$1.01$
HeteroPCA	$-25.79$	$-37.36$	$0.04$	$0.53$	$-13.94$	$-4.87$

Table 1: Comparison under the same fitting-sample budgets. Entries are the mean percentage of PCA test excess reconstruction risk removed, so PCA is zero and larger is better. DA denotes diagonal attenuation and MT denotes masked-target training. The best method in each data–budget cell is bold.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S.1: The moment risk formula beyond Gaussian observations. Solid lines and intervals are Monte Carlo estimates at

n=1024

; dashed lines are the corresponding fourth-moment predictions. The covariance and target rank are common across panels, and shading marks the mask-feasible interval.

Data	$N$	Mask-derived DA	Direct DA
Optdigits	$64$	$4.32$ $[4.08,4.57]$	$6.74$ $[6.30,7.17]$
Optdigits	$128$	$4.18$ $[3.85,4.51]$	$6.40$ $[5.80,7.00]$
Phoneme	$64$	$0.84$ $[0.65,1.03]$	$1.09$ $[0.78,1.40]$
Phoneme	$128$	$0.72$ $[0.51,0.93]$	$0.70$ $[0.36,1.03]$
HAR	$64$	$0.65$ $[0.45,0.85]$	$0.76$ $[0.48,1.05]$
HAR	$128$	$0.93$ $[0.78,1.07]$	$1.20$ $[0.97,1.44]$

Appendix

Table S.1: Fixed fitting-sample-budget results for diagonal attenuation. Entries are the mean percentage of PCA test excess reconstruction risk removed, with 95% intervals in brackets. Optdigits intervals treat the ten patch-location seeds as independent clusters, with each seed-level mean averaging 100 training subsamples; the other intervals use 100 training subsamples.

Data	$N$	Correlation PCA	HeteroPCA	Coordinate MT	Block MT
Optdigits	$64$	$5.84$ $[5.34,6.33]$	$-25.79$ $[-27.60,-23.99]$	$-1.63$ $[-2.01,-1.25]$	$-3.58$ $[-4.55,-2.61]$
Optdigits	$128$	$5.41$ $[4.99,5.82]$	$-37.36$ $[-41.93,-32.79]$	$-1.24$ $[-1.80,-0.68]$	$-1.27$ $[-1.98,-0.56]$
Phoneme	$64$	$-0.14$ $[-0.70,0.42]$	$0.04$ $[-0.53,0.60]$	$0.93$ $[0.57,1.28]$	$0.00$ $[0.00,0.00]$
Phoneme	$128$	$-0.52$ $[-1.06,0.01]$	$0.53$ $[0.42,0.64]$	$0.28$ $[-0.10,0.66]$	$0.00$ $[0.00,0.00]$
HAR	$64$	$0.36$ $[0.06,0.66]$	$-13.94$ $[-15.46,-12.43]$	$0.56$ $[0.28,0.83]$	$0.00$ $[0.00,0.00]$
HAR	$128$	$1.01$ $[0.74,1.28]$	$-4.87$ $[-6.23,-3.50]$	$0.93$ $[0.64,1.23]$	$0.00$ $[0.00,0.00]$

Appendix

Table S.2: Comparison methods across the three real-data studies. Each entry is the mean percentage of PCA test excess reconstruction risk removed, with its 95% interval in brackets. Optdigits intervals cluster over ten patch-location seeds, each averaging 100 training subsamples; the remaining intervals use 100 training subsamples. MT denotes masked-target training.

Data	$N$	Mask DA $\alpha$	Direct DA $\alpha$	Coordinate MT $m$	Block MT $m$
Optdigits	$64$	$0.474$ $(1.7\%)$	$0.927$ $(1.7\%)$	$0.432$ $(51.8\%)$	$0.121$ $(86.6\%)$
Optdigits	$128$	$0.479$ $(1.1\%)$	$0.931$ $(1.1\%)$	$0.323$ $(64.1\%)$	$0.037$ $(95.9\%)$
Phoneme	$64$	$0.432$ $(7.0\%)$	$0.800$ $(7.0\%)$	$0.589$ $(18.0\%)$	$0.000$ $(100.0\%)$
Phoneme	$128$	$0.434$ $(7.0\%)$	$0.764$ $(7.0\%)$	$0.479$ $(15.0\%)$	$0.000$ $(100.0\%)$
HAR	$64$	$0.351$ $(22.0\%)$	$0.615$ $(18.0\%)$	$0.370$ $(27.0\%)$	$0.000$ $(100.0\%)$
HAR	$128$	$0.418$ $(9.0\%)$	$0.756$ $(8.0\%)$	$0.563$ $(18.0\%)$	$0.000$ $(100.0\%)$

Appendix

Table S.3: Mean cross-validation-selected strengths under fitting-sample budget

N

. DA denotes diagonal attenuation and MT denotes masked-target training. Parentheses give the percentage of fits selecting the explicit PCA fallback. DA columns report

\alpha

; MT columns report the mask rate

m

Principal Component Analysis

Principal component analysis (PCA) finds orthogonal directions of greatest variance for data compression, visualization, denoising, and feature discovery.

See all 67 papers

Mean-Shift PCA by Knockoff Mean
May 25, 2026Mengda Li, Zeng Li, Jianfeng YaoDistribution Shift Robustness Principal Component Analysis
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · Department of Statistics and Data Science, Southern University of Science and Technology, Shenzhen, China
SuperPCA: subspace analysis and an efficient algorithm for high-dimensional PCA
Sep 22, 2026Irina-Beatrice Haas, Maike Meier, Yuji Nakatsukasa +1Principal Component Analysis Dimensionality Reduction
Mathematical Institute, University of Oxford, Oxford, OX2 6GG, UK · Bernoulli Institute for Mathematics, Computer Science and Artificial Intelligence, University of Groningen, 9700 AB, Groningen, The Netherlands · Institute of Mathematics, EPFL, 1015 Lausanne, Switzerland
The Risk Shadow of Principal Component Analysis: When 99.9999% Variance Preservation Causes Catastrophic Decision Errors
Jun 12, 2026Hamidou TembinePrincipal Component Analysis Dimensionality Reduction
Department of EECS, School of Engineering, UQTR, Canada. Learning and Game Theory Laboratory (LnG Lab), TIMADIE.

May 25, 2026J/K move · Enter open · S save

Mean-Shift PCA by Knockoff Mean

Mengda Li, Zeng Li, Jianfeng Yao

School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · Department of Statistics and Data Science, Southern University of Science and Technology, Shenzhen, China

Distribution Shift Robustness Principal Component Analysis Spectral Methods Noise Robustness

Open paper AskPDF arXiv

Removing noise is difficult, but adding noise is easy. In this work, we show how to eliminate mean-shift noisy components from PCA by deliberately introducing knockoff mean-shift perturbation. Standard PCA is highly sensitive to shifts in the sample mean: a small fraction of samples from a shifted distribution can cause large deviations in the leading principal components. In high-dimensional regimes, existing Robust PCA approaches cannot handle the mean-shift contamination structure inherent in the mixture model. Using tools from Random Matrix Theory, we prove that the mean-shift spikes are spectrally separable from the stable eigenvalues of the original covariance. Furthermore, the original eigenspace remains asymptotically invariant to the contamination, independent of the mixture weight. Exploiting this spectral stability, we propose a simple, two-stage PCA algorithm by adding knockoff mean that identifies and removes the mean-shift component using only standard PCA operations.

Abstract

Principal Component Analysis

Explore similar work

Mean-Shift PCA by Knockoff Mean

SuperPCA: subspace analysis and an efficient algorithm for high-dimensional PCA

The Risk Shadow of Principal Component Analysis: When 99.9999% Variance Preservation Causes Catastrophic Decision Errors