Principal Component Regression Dominates all Monotone Spectral Filters for Linear Regression

Juno Kim; Hengyu Fu; Peter Bartlett; Jason D. Lee; Jingfeng Wu

stat.MLSep 30, 2026

Principal Component Regression Dominates all Monotone Spectral Filters for Linear Regression

Authors: Juno Kim, Hengyu Fu, Peter Bartlett, Jason D. Lee, Jingfeng Wu

Organizations: University of California, Berkeley · Google DeepMind

Abstract

We compare the instance-wise, finite-sample risks of monotone spectral filters for linear regression, a broad class of estimators including principal component regression (PCR), gradient descent (GD), and ridge regression. We show that PCR dominates all monotone spectral filters: compared to any such filter, the risk of optimally tuned PCR is no bigger by a constant factor for all problems. Furthermore, the dominance is strong if the filter is separated from step functions (e.g., GD and ridge): there exist problem instances for which the risk of PCR is smaller by a polynomial factor in sample size dependence. Our comparison results show that PCR is optimal and thus admissible among monotone filters, significantly extending Wu et al. (2026)'s result that GD strongly dominates ridge. From a technical perspective, we establish new upper and lower bounds for general spectral filters, which are instance-wise sharp when specialized to ridge or GD, recovering or improving the best-known bounds.

Figures & tables

well-specified linear regression problems $(\mathbb{L})$	$\mathrm{GD}\prec_{\mathbb{L}}\mathrm{ridge}$	Wu et al. (2026)
	$\mathrm{PCR}\prec_{\mathbb{L}}\mathrm{GD}$	Theorem 2.3
Gaussian linear regression problems $(\mathbb{G})$	$\mathrm{PCR}\preceq_{\mathbb{G}}$ all monotone filters $\mathrm{PCR}\prec_{\mathbb{G}}$ all non-step monotone filters	Theorem 2.1 Theorem 2.2

Table 1: Dominance results for linear regression. For a problem class

\mathbb{P}

, we say algorithm

\mathcal{A}

dominates

\mathcal{B}

, or

\mathcal{A}\preceq_{\mathbb{P}}\mathcal{B}

, if its (excess) risk is never more than a constant multiple of

\mathcal{B}

’s on every problem instance in

\mathbb{P}

. We say

\mathcal{A}

strongly dominates

\mathcal{B}

, or

\mathcal{A}\prec_{\mathbb{P}}\mathcal{B}

, if additionally its risk is polynomially smaller (w.r.t. sample size) for some instances. We establish that for all well-specified linear regression problems

\mathbb{L}

( 4 ) with bounded signal-to-noise ratio, PCR strongly dominates GD; previous work showed that GD strongly dominates ridge in the same class ( Wu et al., 2026 ) . Thus, GD and ridge are both inadmissible. For all linear regression problems with Gaussian design

\mathbb{G}

( 2 ), a subset of

\mathbb{L}

, we prove that PCR dominates all monotone spectral filters, a wide class of regularization-based methods including GD and ridge ( Definition 1 ). Thus, PCR is optimal and hence admissible among all such filters. We further prove that PCR strongly dominates all monotone filters that are uniformly separated from step functions (e.g., Example 1 ).

Panel of Figure 1 — Figure 1: An $(\epsilon,\delta)$ -non-step monotone filter.

Explore similar work

De-floored Principal Component Regression: When Rank Selection Alone Is Insufficient for Prediction
Jul 18, 2026Peng ZhaoPrincipal Component Analysis Kernel Ridge Regression
Department of Applied Economics and Statistics, University of Delaware
Beyond Negative-Ridge Endpoints: Mixed-Sign Spectral Regularization via Negative-Shifted Gradient Descent
Jul 24, 2026Peng ZhaoRegularization Quadratic Slack Penalty
Department of Applied Economics and Statistics · University of Delaware
Ridge Regression from Poisson Resetting: A Renewal Perspective on Spectral Regularization
May 28, 2026Petar JolakoskiKernel Ridge Regression Regularization
Research Center for Computer Science and Information Technologies, Macedonian Academy of Sciences and Arts, Bul. Krste Misirkov 2, 1000 Skopje, Macedonia

Jul 18, 2026, math.STJ/K move · Enter open · S save

De-floored Principal Component Regression: When Rank Selection Alone Is Insufficient for Prediction

Peng Zhao

Department of Applied Economics and Statistics, University of Delaware

Principal Component Analysis Kernel Ridge Regression Spectral

Open paper AskPDF arXiv

Principal component regression (PCR) regularizes high-dimensional prediction by choosing a spectral cutoff, but rank selection cannot correct systematic inflation of the retained empirical eigenvalues. We study clean Gaussian random designs in which the aggregate covariance tail creates a nearly scalar sample-space floor comparable to the predictive head scale. De-floored principal component regression (dPCR) retains the cutoff and subtracts an estimated floor from the retained denominators. We prove an ordinary-PCR prediction-risk lower bound uniform over all ranks and a high-probability dPCR upper bound. When the floor is sharp and inexpensive to remove in population prediction risk, the conditional risk of dPCR is asymptotically negligible relative to that of the best ordinary PCR rank. An exact risk decomposition explains the separation: denominator inflation is governed by first spectral mass, whereas the clean prediction cost of correction is governed by squared spectral mass. A same-sample trimmed-mean floor estimate attains the oracle dPCR upper-bound rate at a prespecified rank, and the separation persists under approximate predictive alignment when the tail prediction-energy fraction vanishes. Separate pointwise fixed-aspect formulas show that the risk-optimal positive scalar correction improves rank-

1

PCR, whereas mean-floor subtraction is generally not optimal for a broad Marchenko--Pastur bulk.

Abstract

Explore similar work

De-floored Principal Component Regression: When Rank Selection Alone Is Insufficient for Prediction

Beyond Negative-Ridge Endpoints: Mixed-Sign Spectral Regularization via Negative-Shifted Gradient Descent

Ridge Regression from Poisson Resetting: A Renewal Perspective on Spectral Regularization