stat.MLAug 28, 2026

Learning between the peaks: sharp asymptotics for kernel ridge regression under power-law anisotropy

Authors: Lorenzo Rizzi, Arie Wortsman Zurich, Bruno Loureiro

Organizations: Departement d’Informatique, École Normale Supérieure, PSL & CNRS · Dipartimento di Fisica e Astronomia, Università degli Studi di Padova · Scuola Galileiana di Studi Superiori, Università degli Studi di Padova

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We study kernel ridge regression under anisotropic Gaussian data, where the input covariance decays as a power law with exponent α≥0α\geq 0 for polynomial inner-product kernels. We derive asymptotically sharp expressions for the kernel spectrum and the generalization error in the polynomial high-dimensional regime n=Θ(dκ)n=Θ(d^κ), revealing how anisotropy reshapes the learning curves. For weak anisotropy (0<α<10<α<1), the problem remains effectively high-dimensional and retains some features of the isotropic case, while departing from it in others: the variance still peaks at integer sample complexities κ∈Nκ\in\mathbb{N}, but these peaks are progressively damped as αα grows; meanwhile, for targets strongly aligned with the data's principal directions, the bias drops at fractional sample complexities, decoupling the bias transitions from the interpolation peaks. For strong anisotropy (α>1α> 1), the effective dimension of the problem is constant, and the variance stops depending on sample size altogether, plateauing under ridgeless interpolation or vanishing at an explicit rate under fixed ridge penalty. The bias undergoes a sharp transition governed by the target's decay rate: below a threshold, learning is abrupt rather than gradual; above it, the bias decays as a power law that recovers the classical source and capacity rates. We finally specialize these results to single-index targets, showing how the alignment of the index with the data's principal directions determines the effect of anisotropy on learning. Together, our results clarify how the input geometry shapes the kernel features and fundamentally impacts its generalization properties.

Figures & tables

Figure 1: Numerical illustration of the asymptotic variance of KRR. We use a null target function ( f⋆=0f_{\star}=0 ) to isolate the variance component of the risk. The experimental parameters are h(t)=1+8t+3t2+0.1t3+t4h(t)=1+8t+3t^{2}+0.1t^{3}+t^{4} , λ=0\lambda=0 , ψ=0.8\psi=0.8 and σε2=1.52\sigma_{\varepsilon}^{2}=1.5^{2} . ( Left ) For fractional sample complexity ( κ=0.7\kappa=0.7 ), the empirical variance vanishes. The curve matches the theoretical power-law decay rate predicted by Eq. ( 4.10 ) (dashed line). ( Right ) For integer sample complexity, the variance instead plateaus to a finite, order-one constant given by Eq. ( 4.8 ).
Figure 2: Behaviour of τ\tau as a function of input covariance power-law exponent α∈[0,1)\alpha\in[0,1) , obtained by numerically solving the Kernel state lemma 4.1 . (Left) Different sample complexities n=ψdκn=\psi d^{\kappa} with ψ∈{0.1,2,4,5,10}\psi\in\{0.1,2,4,5,10\} , in the interpolation limit ( λ=0\lambda=0 ). (Right) Different regularization strengths λ∈{0,0.1,0.5,1,5}\lambda\in\{0,0.1,0.5,1,5\} at fixed sample complexity ( ψ=2\psi=2 ). In both figures, we take κ=2\kappa=2 ( n=ψd2n=\psi d^{2} ) under the fourth-order truncated exponential kernel h(t)=∑j=04tj/j!h(t)=\sum_{j=0}^{4}t^{j}/j! .
Figure 3: Deterministic equivalent of the variance as a function of the sample complexity for κ=2\kappa=2 , σε=4\sigma_{\varepsilon}=4 and polynomial kernel h(t)=1+t+12t2+16t3+124t4h(t)=1+t+\frac{1}{2}t^{2}+\frac{1}{6}t^{3}+\frac{1}{24}t^{4} . ( Left ) Fixed λ=0\lambda=0 for different values of anisotropy. The inset shows the variance curve for a fixed ψ=2\psi=2 ; ( Right ) Fixed α=0.5\alpha=0.5 for different regularization λ\lambda . As expected, stronger regularization suppresses the variance.
Figure 4: Residual bias η\eta as a function of the sample complexity ψ\psi for α=0.5\alpha=0.5 , κ=2\kappa=2 , σε=4\sigma_{\varepsilon}=4 and polynomial kernel h(t)=1+t+12t2+16t3+124t4h(t)=1+t+\frac{1}{2}t^{2}+\frac{1}{6}t^{3}+\frac{1}{24}t^{4} , evaluated by solving eqs. eq. 4.6 , ( 4.8 ), and ( 4.18 ). ( Left ) Varying ω\omega for fixed λ=0\lambda=0 . The inset shows a cross-section of η\eta as a function of ω\omega at a fixed load ψ=1.0\psi=1.0 , highlighting how a stronger target anisotropy accelerates the learning of the κ\kappa -th shell. ( Right ) Fixed ω=0.25\omega=0.25 for varying ridge λ\lambda . As expected, regularization increases the bias.
Figure 5: Schematic learning curves for the bias Bn\mathsf{B}_{n} in the weakly anisotropic regime, comparing different target functions with different ω\omega but same energies E(m)E(m) and fixed power-law exponent α=0.25\alpha=0.25 . When ω<1/2\omega<1/2 , polynomial features are still learned at integer transitions κ∈N\kappa\in\mathbb{N} . However, higher value of ω\omega causes the κ\kappa -th sector to be learned more rapidly. When ω>1/2\omega>1/2 , the model can learn higher-order features at fractional scaling exponents κ=m(1−α)=0.75m\kappa=m(1-\alpha)=0.75m . This accelerates the learning process: for example, given a sample size n=O(d2−ϵ)n=O(d^{2-\epsilon}) , the isotropic model has only learned the linear part of the target, whereas the anisotropic model can learn a fraction of the quadratic components of the target.
Figure 6: Schematic learning curves for the excess risk in the weakly anisotropic regime ( α<1\alpha<1 ). The figure compares a target function with a slow decay 0<ω<120<\omega<\frac{1}{2} (top panel) and a fast decay ω>12\omega>\frac{1}{2} (bottom panel) against the isotropic baseline ( ω=0\omega=0 ). Solid lines represent the total excess risk, dashed lines indicate the bias contribution, and dotted lines show the variance. ( Top ): For 0<ω<120<\omega<\frac{1}{2} , the variance peaks exactly at integer polynomial scaling ( n≍dmn\asymp d^{m} ), in-phase with the resolution of new polynomial features. Here, the target decay ω\omega accelerates the learning of the active κ\kappa -th shell. ( Bottom ): For ω>12\omega>\frac{1}{2} , a decoupling occurs. The bias drops at fractional scales n≍dm(1−α)n\asymp d^{m(1-\alpha)} , while the variance continues to peak at integer scalings.
Figure 7: Schematic learning curves for the bias B\mathsf{B} in the strongly anisotropic regime. ( Top ) 0≤ω<1/20\leq\omega<1/2 : In the ridgeless limit ( λ=0\lambda=0 , red), learning is instantaneous and occurs abruptly at n=Θ(dm+ϵ)n=\Theta(d^{m+\epsilon}) . In the regularized regime ( λ=Θ(1)\lambda=\Theta(1) , orange), the ridge penalty shifts the activation of the mm -th shell to a further delayed sample complexity n=Θ(dmα)n=\Theta(d^{m\alpha}) . ( Bottom ) ω>1/2\omega>1/2 : The bias collapses to zero as a power-law (using λ=0\lambda=0 ). Below the critical threshold ω<α+1/2\omega<\alpha+1/2 (light and forest green), the learning rate continuously accelerates as the target energy compresses. Once the threshold is crossed at ω=α+1/2\omega=\alpha+1/2 (dark green, red), the asymptotic decay rate saturates to −2κ-2\kappa
Figure 8: Phase diagram summarizing the asymptotic bias Bn\mathsf{B}_{n} in the ridgeless limit ( λ=0\lambda=0 ), highlighting the various regimes as α\alpha (data anisotropy) and ω\omega (target decay rate) are varied. The red dashed lines represent the critical values where new qualitative behaviours appear.
Figure 9: Mapping of the various Single-Index Model (SIM) scenarios onto the general phase diagram of Fig. 8 , assuming λ=0\lambda=0 . Power-law single-index models (PL-SIMs) with target decay γ\gamma yield an effective decay rate ω=α+γ2\omega=\frac{\alpha+\gamma}{2} , translating into a series of diagonal lines that intersect the different generalization regimes. The sparse SIM limit ( j=1j=1 ) corresponds to ω→∞\omega\to\infty , while the perfectly delocalized scenario is captured by ω=α/2\omega=\alpha/2 .
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Asymptotic behavior of the integration boundaries ymin⁡y_{\min} and ymax⁡y_{\max} derived in Eqs. ( D.7 ). Notably, when m=κm=\kappa or m=κ1−α∈Nm=\frac{\kappa}{1-\alpha}\in\mathbb{N} , the respective boundary converges to a strictly positive, macroscopic constant Θd(1)\Theta_{d}(1) .
Figure 11: Finite-size scaling of the variance learning curve. The empirical variance Vn\mathsf{V}_{n} is plotted as a function of the load parameter ψ=n/dκ\psi=n/d^{\kappa} for varying system dimensions dd (blue gradients). As the dimension increases, the empirical realizations progressively collapse onto the asymptotic theoretical prediction (black dashed line) derived from our state equations in Theorem 1 . Fixed parameters: κ=1\kappa=1 , α=0.3\alpha=0.3 , and σε=2\sigma_{\varepsilon}=2 , h(t)=1+8t+3t2+0.1t3+t4h(t)=1+8t+3t^{2}+0.1t^{3}+t^{4} , λ=0\lambda=0 , ψ=0.8\psi=0.8
Figure 12: Asymptotic behavior of the integration boundaries ymin⁡y_{\min} and ymax⁡y_{\max} derived in Eqs. ( E.33 ). At variance with Fig. 10 , when m=κm=\kappa the respective boundary does not converge to a constant Θ(1)\Theta(1) due to the diverging factor f(d)f(d) .
Figure 13: Kernel spectrum (left panel) and cumulative target function comparison (right panel) for different values of α\alpha . In the left panel, we observe that the asymptotic theoretical spectrum obtained from Eq. ( 3.4 ) closely matches the spectrum empirically extracted via empirical diagonalization. In the right panel, we see that the cumulative curves A(u)A(u) computed on the Hermite basis and on the kernel eigenbasis are in close agreement. Simulation parameter: d=35d=35 , polynomial kernel k(x,x′)=(2+⟨x,x′⟩)3k(x,x^{\prime})=(2+\langle x,x^{\prime}\rangle)^{3} , random index vv . The empirical diagonalization was performed using M=10000M=10000 Monte-Carlo samples
Figure 14: Left: Asymptotic versus empirical bias for a sparse SIM in the weakly anisotropic regime. Right: Asymptotic versus empirical bias for a sparse SIM in the strongly anisotropic regime. Simulation parameters: exponential kernel truncated at the fourth order, κ=1\kappa=1 , ψ=0.8\psi=0.8 . Each empirical KRR run is repeated 2020 times and averaged, with errors representing the standard deviations.
Figure 15: Left: Asymptotic versus empirical bias for a power-law SIM in the weakly anisotropic regime with an effective ω<0.5\omega<0.5 . Right: Asymptotic versus empirical bias for a power-law SIM in the weakly anisotropic regime for an effective ω>0.5\omega>0.5 . The link function gg is chosen such that its expansion is truncated at degree 33 ( g(t)=∑m=03gmHem(t)g(t)=\sum_{m=0}^{3}g_{m}\text{He}_{m}(t) ), allowing the kernel to fully learn it in the ω>12\omega>\frac{1}{2} regime. All other parameters match those of Fig. 14 .
Figure 16: Left: Asymptotic versus empirical bias for a power-law SIM in the strongly anisotropic regime when ω>α+12\omega>\alpha+\frac{1}{2} . Right: Asymptotic versus empirical bias for a power-law SIM in the strongly anisotropic regime when ω<α+12\omega<\alpha+\frac{1}{2} . All other parameters match those of Fig. 14 .
More in this topic

Kernel Ridge Regression

Kernel ridge regression estimates nonlinear relationships by fitting data with kernel similarities while regularizing model complexity.

See all 29 papers

Explore similar work

CardsList

  1. Predicting kernel regression learning curves from only raw data statistics

    Oct 16, 2025Dhruva Karkada, Joseph Turnbull, Yuxi Liu +1Kernel RegressionKernel Methods

    UC Berkeley · Imbue

  2. Large Dimensional Kernel Ridge Regression: Extending to Product Kernels

    May 14, 2026Yang Zhou, Yicheng Li, Yuqian Cheng +1Generalization BoundsKernel Ridge Regression

    Department of Statistics and Data Science Tsinghua University Beijing, 100084, China · Department of Mathematical Science Tsinghua University Beijing, 100084, China

  3. On the Optimality of Random Features under General Conditions: Misspecified Cases and Robust Learning

    Oct 13, 2026Caixing WangRandom Feature MethodsKernel Ridge Regression

    School of Statistics and Data Science, Southeast University, Nanjing, China