Understanding the Subspace Stabilization of the Hessian and Gradient Covariance Matrix

Fangshuo Liao; Anastasios Kyrillidis

cs.LGSep 25, 2026

Understanding the Subspace Stabilization of the Hessian and Gradient Covariance Matrix

Authors: Fangshuo Liao, Anastasios Kyrillidis

Organizations: Rice University

Abstract

The phenomenon of the top subspace stabilization of the Hessian matrix is an surprising and critical aspect in study of the second-order information of neural network training. Prior work argues that the top subspace of the Hessian stabilizes by measuring the overlap between the top subspaces of the step-wise Hessian, and explains this stabilization with diminishing parameter change in the late phase of training. In this paper, we define a new instability metric for the subspace evolution, and use it to detect subspace stabilization that is independent of the magnitude of parameter change. In the meantime, we observe that the gradient covariance matrix has a similar property of its top subspace to the Hessian. By using a between-class and within-class decomposition of the gradient covariance matrix, we identify an explicit form that gives a near-perfect approximation of the top- $(C-1)$ subspace of the Hessian and the gradient covariance matrix. In the gradient flow set-up, we show that the slow evolution of the idenfied approximation is due to the separation between the outlier and the bulk eigenvalues of the Hessian matrix, thus providing an explanation to the phenomenon of the top subspace stabilization of the Hessian matrix.

Figures & tables

Figure 1: Top-

K

subspace stabilization of the Hessian and the gradient covariance matrix, with

K\in\{5,9,10,15,20\}

shown in different colors. Column 1 and 3 shows the overlap metric, and column 2 and 4 shows the stability metric. First row: MLP on MNIST. Second row: 2-Layer CNN on CIFAR-10. Third row: 3-Layer CNN on CIFAR-10.

Figure 2: Structural similarity between the Hessian and the gradient covariance matrix. The three columns plots the Hessian eigenvalues, gradient covariance matrix’s eigenvalues, and the overlaop between the Hessian and the gradient covariance matrix’s top-

K

subspaces. First row: MLP on MNIST. Second row: 2-Layer CNN on CIFAR-10. Third row: 3-Layer CNN on CIFAR-10.

Figure 3: Overlaps between

{\bm{G}}

-span,

\hat{{\bm{G}}}

-span, and the top-

(C-1)

subspace of

\bm{\Sigma}\left(\bm{\theta}\right)

and

{\bm{H}}\left(\bm{\theta}\right)

throughout training. First row: MLP on MNIST. Second row: 2-Layer CNN on CIFAR-10. Third row: 3-Layer CNN on CIFAR-10.

Panel of Figure 4 — Figure 4: Left column: norm comparisons between ${\bm{g}}_{c}(\bm{\theta})$ s ( yellow to red for $c\in[C]$ ) and ${\bm{g}}\left(\bm{\theta}\right)$ ( teal ). Mid column: $\left\|\left({\bm{I}}-{\bm{P}}\right){\bm{H}}_{c}\right\|_{2}/\left\|{\bm{H}}_{c}\right\|_{2}$ for $c\in[C]$ ( yellow to red ), and $\left\|\left({\bm{I}}-{\bm{P}}\right){\bm{H}}\right\|_{2}/\left\|{\bm{H}}\right\|_{2}$ ( teal ). Right column: comparison of $\left\|\left({\bm{I}}-{\bm{P}}\right){\bm{H}}_{c}{\bm{g}}\right\|_{2}$ ( yellow to red ) and $\left\|\left({\bm{I}}-{\bm{P}}\right){\bm{H}}{\bm{g}}\right\|_{2}$ ( teal ) against $\sigma_{\min}\left({\bm{G}}(t)\right)$ ( purple ). The first and second row correspond to the setting of the first and second row in Figure 1 .

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	MNIST	CIFAR 2-layer	CIFAR 3-layer
Architecture	MLP	2-Layer CNN	3-Layer CNN
Train subset size	10,000	5,000	5,000
Minibatch size	100	50	50
Learning rate	0.01	0.001	0.001
Total SGD steps	5,000	12,000	6,000
Steps / epoch	100	100	100

Appendix

Table 5

Neural Network Training Dynamics

Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.

See all 520 papers

Worker Disagreement Reveals Sharp Directions in Local SGD
May 26, 2026Tolga Dimlioglu, Kristi Topollai, Anna ChoromanskaNeural Network Training Dynamics Loss Landscape Geometry
New York University, New York, USA
The Origin of Edge of Stability
Apr 22, 2026Elon LitmanEdge of Stability Gradient Descent
Stanford University
Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks
Jun 27, 2026Yuto Omae, Kazuki Sakai, Yohei Kakimoto +3Gradient Descent Neural Network Optimization
Nihon University (Japan) · National Institute of Technology, Nagaoka College (Japan) · Tokyo City University (Japan)

May 26, 2026J/K move · Enter open · S save

Worker Disagreement Reveals Sharp Directions in Local SGD

Tolga Dimlioglu, Kristi Topollai, Anna Choromanska

New York University, New York, USA

Neural Network Training Dynamics Loss Landscape Geometry Stochastic Gradient Descent

Open paper AskPDF arXiv

Deep neural network training often exhibits highly anisotropic loss geometry, where a few sharp dominant Hessian directions coexist with a large flatter bulk. Gradients tend to align disproportionately with these dominant directions, although stable progress often requires movement through flatter bulk directions. Estimating the dominant subspace is therefore useful but costly with direct Hessian-based methods. We show that standard Local SGD exposes this geometry through worker disagreement. We theoretically show that the worker-average gap covariance is shaped by stochastic-gradient noise and Hessian curvature, causing workers to disagree along sharp, curvature-sensitive directions. Thus, worker-average gaps provide a cheap Hessian-free estimator of the dominant subspace. Experiments on MLPs, CNNs, and Transformers show that subspaces formed by worker-average gaps capture a substantial fraction of the gradient component lying in the dominant Hessian eigenspace.

Abstract

Neural Network Training Dynamics

Explore similar work

Worker Disagreement Reveals Sharp Directions in Local SGD

The Origin of Edge of Stability

Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks