The Impact of Anisotropic Covariance Structure on the Training Dynamics and Generalization Error of Linear Networks

Taishi Watanabe; Ryo Karakida; Jun-nosuke Teramae

stat.MLJan 11, 2026

The Impact of Anisotropic Covariance Structure on the Training Dynamics and Generalization Error of Linear Networks

Authors: Taishi Watanabe, Ryo Karakida, Jun-nosuke Teramae

Organizations: Graduate School of Informatics, Kyoto University · Artificial Intelligence Research Center, AIST · RIKEN AIP

Abstract

The success of deep neural networks largely depends on the statistical structure of the training data. While learning dynamics and generalization on isotropic data are well-established, the impact of pronounced anisotropy on these crucial aspects is not yet fully understood. We examine the impact of data anisotropy, represented by a spiked covariance structure, a canonical yet tractable model, on the learning dynamics and generalization error of a two-layer linear network in a linear regression setting. Our analysis reveals that the learning dynamics proceed in two distinct phases, governed initially by the input-output correlation and subsequently by other principal directions of the data structure. Furthermore, we derive an analytical expression for the generalization error, quantifying how the alignment of the spike structure of the data with the learning task improves performance. Our findings offer deep theoretical insights into how data anisotropy shapes the learning trajectory and final performance, providing a foundation for understanding complex interactions in more advanced network architectures.

Figures & tables

Panel of Figure 3 — Figure 3: Evolution of the normalized training loss for different settings of (a) spike magnitude $\rho$ and (b) spike-target alignment $A$ . The loss $\mathcal{L}$ is normalized by the empirical second moment of the labels, $\frac{1}{n}\sum_{i=1}^{n}y_{i}^{2}$ . Vertical dotted and dash-dotted lines indicate the characteristic timescales for the early and later learning phases, respectively, with the latter defined in Equation 23 using $\delta=3$ . Other parameters are the same as in Figure 2 .

Figure 4: Normalized generalization error $\hat{R}_{0}/{\bm{\beta}}^{\top}{\bm{\Sigma}}{\bm{\beta}}$ as a function of spike-target alignment $A$ and spike magnitude $\rho$ for a fixed ratio $\gamma=3$ .

Panel of Figure 5 — Figure 5: Normalized generalization error $\hat{R}_{0}/{\bm{\beta}}^{\top}{\bm{\Sigma}}{\bm{\beta}}$ as a function of the ratio $\gamma=d/n$ . Solid lines represent the theoretical prediction ( Equation 29 ) and points are results of numerical simulations. The empirical results are averaged over 10 independent trials, with error bars indicating the standard deviation. All simulations were performed with a two-layer linear network of input dimension $d=3000$ and width $m=3000$ , with $\sigma^{2}=1$ . The initialization procedure is the same as in Figure 2 .

Neural Network Generalization

Neural network generalization studies why trained models perform well on unseen data and how architecture, data, and training shape that ability.

See all 311 papers

Learning between the peaks: sharp asymptotics for kernel ridge regression under power-law anisotropy
Aug 28, 2026Lorenzo Rizzi, Arie Wortsman Zurich, Bruno LoureiroKernel Ridge Regression Kernel Methods
Departement d’Informatique, École Normale Supérieure, PSL & CNRS · Dipartimento di Fisica e Astronomia, Università degli Studi di Padova · Scuola Galileiana di Studi Superiori, Università degli Studi di Padova
Investigating the Early Signatures of Feature Learning in Deep Linear Networks
Oct 27, 2026Shaunak Bhandarkar, Boris HaninDeep Linear Networks Neural Network Generalization
Princeton University
A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay
Sep 7, 2026Yuqing Wang, Ioannis G. Kevrekidis, Mikhail BelkinNeural Network Generalization Neural Network Training Dynamics
Johns Hopkins University · University of California San Diego

Aug 28, 2026J/K move · Enter open · S save

Learning between the peaks: sharp asymptotics for kernel ridge regression under power-law anisotropy

Lorenzo Rizzi, Arie Wortsman Zurich, Bruno Loureiro

Departement d’Informatique, École Normale Supérieure, PSL & CNRS · Dipartimento di Fisica e Astronomia, Università degli Studi di Padova · Scuola Galileiana di Studi Superiori, Università degli Studi di Padova

Kernel Ridge Regression Kernel Methods Statistical Learning Theory High-Dimensional Asymptotics

Open paper AskPDF arXiv

We study kernel ridge regression under anisotropic Gaussian data, where the input covariance decays as a power law with exponent

α\geq 0

for polynomial inner-product kernels. We derive asymptotically sharp expressions for the kernel spectrum and the generalization error in the polynomial high-dimensional regime

n=Θ(d^κ)

, revealing how anisotropy reshapes the learning curves. For weak anisotropy (

0<α<1

), the problem remains effectively high-dimensional and retains some features of the isotropic case, while departing from it in others: the variance still peaks at integer sample complexities

κ\in\mathbb{N}

, but these peaks are progressively damped as

α

grows; meanwhile, for targets strongly aligned with the data's principal directions, the bias drops at fractional sample complexities, decoupling the bias transitions from the interpolation peaks. For strong anisotropy (

α> 1

), the effective dimension of the problem is constant, and the variance stops depending on sample size altogether, plateauing under ridgeless interpolation or vanishing at an explicit rate under fixed ridge penalty. The bias undergoes a sharp transition governed by the target's decay rate: below a threshold, learning is abrupt rather than gradual; above it, the bias decays as a power law that recovers the classical source and capacity rates. We finally specialize these results to single-index targets, showing how the alignment of the index with the data's principal directions determines the effect of anisotropy on learning. Together, our results clarify how the input geometry shapes the kernel features and fundamentally impacts its generalization properties.

Abstract

Neural Network Generalization

Explore similar work

Learning between the peaks: sharp asymptotics for kernel ridge regression under power-law anisotropy

Investigating the Early Signatures of Feature Learning in Deep Linear Networks

A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay