math.STJul 18, 2026

De-floored Principal Component Regression: When Rank Selection Alone Is Insufficient for Prediction

Authors: Peng Zhao

Organizations: Department of Applied Economics and Statistics, University of Delaware

Abstract

Principal component regression (PCR) regularizes high-dimensional prediction by choosing a spectral cutoff, but rank selection cannot correct systematic inflation of the retained empirical eigenvalues. We study clean Gaussian random designs in which the aggregate covariance tail creates a nearly scalar sample-space floor comparable to the predictive head scale. De-floored principal component regression (dPCR) retains the cutoff and subtracts an estimated floor from the retained denominators. We prove an ordinary-PCR prediction-risk lower bound uniform over all ranks and a high-probability dPCR upper bound. When the floor is sharp and inexpensive to remove in population prediction risk, the conditional risk of dPCR is asymptotically negligible relative to that of the best ordinary PCR rank. An exact risk decomposition explains the separation: denominator inflation is governed by first spectral mass, whereas the clean prediction cost of correction is governed by squared spectral mass. A same-sample trimmed-mean floor estimate attains the oracle dPCR upper-bound rate at a prespecified rank, and the separation persists under approximate predictive alignment when the tail prediction-energy fraction vanishes. Separate pointwise fixed-aspect formulas show that the risk-optimal positive scalar correction improves rank-11 PCR, whereas mean-floor subtraction is generally not optimal for a broad Marchenko--Pastur bulk.

Explore similar work

CardsList
Jun 12, 2026cs.LG

The Risk Shadow of Principal Component Analysis: When 99.9999% Variance Preservation Causes Catastrophic Decision Errors

Principal Component Analysis (PCA) preserves variance, not the information needed to detect rare catastrophic events. This paper proves the existence of a {\it Risk Shadow}: PCA can retain over 99.9999 percent of total variance while completely erasing all signal about rare, high-impact failures. When this happens, even the best possible classifier operating on the PCA representation reduces to a constant predictor. The root cause is a fundamental mismatch between variance maximization and tail risk awareness. To break the shadow, we introduce Expectile PCA (ExPCA) and Tail-Preserving PCA (TP-PCA), two methods that reweight the data covariance toward high-impact events. We prove theoretically that ExPCA strictly outperforms PCA in retaining rare-event information, and we validate our claims on synthetic data and a real-world credit card fraud detection benchmark. Our results call for a fundamental rethinking of variance-based dimensionality reduction in high-stakes decisions.
Hamidou Tembine
Principal Component AnalysisRare EventsVariance ReductionEmpirical Risk MinimizationFraud DetectionAdjustable Equalizer Curve
AskPDF
Jun 20, 2026cs.LG

On the Curse of Dimensionality in Private Sparse Covariance Estimation and PCA

We study high-dimensional differentially private (DP) covariance estimation in the operator norm, and principal component analysis (PCA), under kk-row-column sparsity (kk-RCS) of the covariance matrix. In the non-private setting, it is known that poly(k,log⁡d)\mathsf{poly}(k, \log d) samples suffice to solve both of these problems. However, the only comparable result known under DP (Wang et al. 2021) requires Ω(d)Ω(d) samples under standard parameterizations of the problem. We investigate when this curse of dimensionality is inherent for sparse covariance estimation tasks under DP. On the upper bound front, we show that a poly(k,log⁡d)\mathsf{poly}(k, \log d) sample complexity for PCA is possible under DP, if we also posit sparsity of the leading eigenvector. We complement this result with poly(d)\mathsf{poly}(d) lower bounds under DP for both sparse covariance estimation and PCA, establishing an exponential gap between the private and non-private variants of these problems when k=polylog(d)k = \mathsf{polylog}(d). To our knowledge, no such separation has previously been demonstrated for any sparse estimation problems in private high-dimensional statistics. Our techniques are flexible enough that they imply stronger lower bounds even for the well-studied problem of standard DP PCA, without sparsity assumptions.
Syamantak Kumar, Shourya Pandey, Purnamrita Sarkar +1
Differential PrivacyPrincipal Component AnalysisCovariance EstimationHigh-DimensionalOptimal Sample ComplexitySparsity
AskPDF
Jul 27, 2026math.ST

The Zero Pattern of a Design Matrix Drives Multiple Descent in Over-parameterized Regression

Over-parameterized linear regression has been widely studied over the last decade. However, most existing works assume that the covariates are independent and that their covariance matrices are non-degenerate. In this paper, we relax both assumptions and derive deterministic equivalents for the prediction risk in a vanishing-ridge regime. We show that degeneracy of the covariance matrices and dependence can lead to multiple descent, and characterize where the corresponding peaks can occur. Our proofs use a novel graph representation of the variance profile. We show that maximum matchings and the Dulmage--Mendelsohn decomposition of the associated bipartite graph identify the configurations at which the variance becomes singular.
Kevin Han Huang, Haoyu Ye, Somak Laha +1
Over-Parameterized Neural NetworksRegressionCovariance EstimationBipartite Perfect MatchingsZeroOutput and Model Diversity
AskPDF