Sharpness of Minima in Deep Matrix Factorization

Anil Kamber; Rahul Parhi

stat.MLSep 30, 2025

Sharpness of Minima in Deep Matrix Factorization

Authors: Anil Kamber, Rahul Parhi

Organizations: Department of Electrical and Computer Engineering, University of California, San Diego, La Jolla, CA.

Abstract

Understanding the loss-landscape geometry near a minimum is key to explaining the implicit bias of gradient-based methods in non-convex optimization problems such as deep neural network training and deep matrix factorization. A central quantity for characterizing this geometry is the maximum eigenvalue of the Hessian of the loss. Currently, its precise role has been obfuscated because no exact expressions for this sharpness measure were known in general settings. In this paper, we present the first exact expression for the maximum eigenvalue of the Hessian of the squared-error loss at any minimizer in deep matrix factorization problems, resolving an open question posed by Mulayoff and Michaeli [2020]. This expression reveals a fundamental property of the loss landscape in deep matrix factorization: It is sufficient, but not necessary, for flatness that, across all layers, the product of the spectral norms of the left and right intermediate factors equals a constant determined by the depth and the spectral norm of the target matrix. Most notably, in both depth-2 matrix factorization and deep overparameterized scalar factorization, we show that this condition is both necessary and sufficient for flatness, which implies that flat minima are spectral-norm balanced and not necessarily balanced in depth-2 matrix factorization. To complement our theory, we provide empirical verification, relying on our exact expression, of an escape phenomenon during gradient-based training near a minimizer of a deep matrix factorization problem.

Figures & tables

Related Work	Depth	Input Dim.	Output Dim.	$\lambda_{\max}(\nabla^{2}\mathcal{L}(\mathbf{w}))$	Layers
Mulayoff and Michaeli (2020, Theorem 1)	$L$	$d_{0}$	$d_{L}$	$\mathbf{w}\in\Omega_{F}$	$\mathbb{R}^{a\times b}$
Zhu et al. (2023, Appendix B.1)	$2$	$1$	$1$	$\mathbf{w}\in\mathbb{R}^{N}$	$\mathbb{R}$
Singh and Hofmann (2024, Theorem 1)	$2$	$1$	$1$	$\mathbf{w}\in\mathbb{R}^{N}$	$\mathbb{R}^{a}$
Ghosh et al. (2025, Lemma 1)	$L$	$d_{0}$	$d_{L}$	$\mathbf{w}\in\Omega_{B}$	$\mathbb{R}^{a\times b}$
Theorem 3.1 (This Paper)	$L$	$d_{0}$	$d_{L}$	$\mathbf{w}\in\Omega$	$\mathbb{R}^{a\times b}$

Table 1: Closed-form expressions for the maximum Hessian eigenvalue in the literature.

\Omega

denotes the set of all global minimizers,

\Omega_{F}\subseteq\Omega

denotes the set of flat global minimizers, and

\Omega_{B}\subseteq\Omega

denotes the set of balanced global minimizers (see Definition 2.1 ). The Layers column specifies the form of the layers (factors) considered in each work: scalar-valued (

\mathbb{R}

), vector-valued (

\mathbb{R}^{a}

), or matrix-valued (

\mathbb{R}^{a\times b}

Panel of Figure 1 — Figure 1: The setting is a depth- $3$ matrix factorization of a random $10\times 20$ matrix with Gaussian factors $\mathbf{W}_{3}\in\mathbb{R}^{10\times 20}$ and $\mathbf{W}_{1},\mathbf{W}_{2}\in\mathbb{R}^{20\times 20}$ . $\mathbf{v}_{1}$ and $\mathbf{v}_{N}$ are the eigenvectors of $\nabla^{2}\mathcal{L}(\mathbf{w}^{*})$ that correspond to its maximum and minimum eigenvalues respectively. $\lambda_{\max}$ is computed via the closed-form expression in Theorem 3.1 . GD runs with different step sizes are initialized within $10^{-12}$ of the minimizer $\mathbf{w}^{*}$ along $\mathbf{v}_{1}$ . The contour plot in (c) projects the parameter space onto $\operatorname{span}\{\mathbf{v}_{1},\mathbf{v}_{N}\}$ (Appendix E.1 ); darker (brighter) regions indicate higher (lower) loss. See Appendix E.2 for details.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: The setting is a depth-2 matrix factorization of a random $10\times 20$ target matrix, where $\mathbf{L}\in\mathbb{R}^{10\times 30}$ and $\mathbf{R}\in\mathbb{R}^{20\times 30}$ are Gaussian factors. GD dynamics with different step sizes, indicated by different colors, are initialized within a radius of $10^{-15}$ in (a)–(c) and $10^{-9}$ in (d)–(f) from the minimizer $\mathbf{w}^{*}$ . The initialization direction is along the eigenvector of $\nabla^{2}\mathcal{L}(\mathbf{w}^{*})$ that corresponds to its maximum eigenvalue. The value of $\lambda_{\max}(\nabla^{2}\mathcal{L}(\mathbf{w}^{*}))$ is computed using the closed-form expression derived in Corollary 18 .

Panel of Figure 3 — Figure 3: The setting is a 20-layer overparameterized scalar factorization of a random scalar. GD dynamics with different step sizes, indicated by different colors, are initialized within a radius of $10^{-9}$ from the minimizer $\mathbf{w}^{*}$ . The initialization direction is along the eigenvector of $\nabla^{2}\mathcal{L}(\mathbf{w}^{*})$ that corresponds to its maximum eigenvalue. The value of $\lambda_{\max}(\nabla^{2}\mathcal{L}(\mathbf{w}^{*}))$ is computed using the closed-form expression derived in Corollary 3.2 .

Panel of Figure 4 — Figure 4: The setting is a 4-layer overparameterized scalar factorization of a random scalar. GD dynamics with different step sizes, indicated by different colors, are initialized within a radius of $10^{-12}$ from the minimizer $\mathbf{w}^{*}$ . The initialization direction is along the eigenvector of $\nabla^{2}\mathcal{L}(\mathbf{w}^{*})$ that corresponds to its maximum eigenvalue. The value of $\lambda_{\max}(\nabla^{2}\mathcal{L}(\mathbf{w}^{*}))$ is computed using the closed-form expression derived in Corollary 3.2 .

Panel of Figure 5 — Figure 5: The setting is a depth-2 matrix factorization of a random $25\times 20$ target matrix, where $\mathbf{L}\in\mathbb{R}^{25\times 30}$ and $\mathbf{R}\in\mathbb{R}^{20\times 30}$ are Gaussian factors. GD dynamics with different step sizes, indicated by different colors, are initialized within a radius of $10^{-9}$ from the minimizer $\mathbf{w}^{*}$ . The initialization direction is along the eigenvector of $\nabla^{2}\mathcal{L}(\mathbf{w}^{*})$ that corresponds to its maximum eigenvalue. The value of $\lambda_{\max}(\nabla^{2}\mathcal{L}(\mathbf{w}^{*}))$ is computed using the closed-form expression derived in Corollary 18 .

Loss Landscape Sharpness

See all 20 papers

On the Loss Landscape Geometry of Regularized Deep Matrix Factorization: Uniqueness and Sharpness
Mar 28, 2026Anil Kamber, Rahul ParhiLoss Landscape Loss Landscape Geometry
UC San Diego
Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima
Jul 9, 2026Lachlan Ewen MacDonald, René VidalGradient Descent Flat Minima
Innovation in Data Engineering and Science (IDEAS) · University of Pennsylvania Pennsylvania, PA 19104
Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks
Jun 27, 2026Yuto Omae, Kazuki Sakai, Yohei Kakimoto +3Gradient Descent Neural Network Optimization
Nihon University (Japan) · National Institute of Technology, Nagaoka College (Japan) · Tokyo City University (Japan)

Mar 28, 2026J/K move · Enter open · S save

On the Loss Landscape Geometry of Regularized Deep Matrix Factorization: Uniqueness and Sharpness

Anil Kamber, Rahul Parhi

UC San Diego

Loss Landscape Loss Landscape Geometry Matrix Factorization Weight Decay Deep Linear Networks

Open paper AskPDF arXiv

Weight decay is ubiquitous in training deep neural network architectures. Its empirical success is often attributed to capacity control; nonetheless, our theoretical understanding of its effect on the loss landscape and the set of minimizers remains limited. In this paper, we show that

\ell^2

-regularized deep matrix factorization/deep linear network training problems with squared-error loss admit a unique end-to-end minimizer for all target matrices subject to factorization, except for a set of Lebesgue measure zero formed by the depth and the regularization parameter. This observation reveals fundamental properties of the loss landscape of regularized deep matrix factorization problems: the Hessian spectrum is constant across all minimizers of the regularized deep scalar factorization problem with squared-error loss. Moreover, we show that, in regularized deep matrix factorization problems with squared-error loss, if the target matrix does not belong to the Lebesgue measure-zero set, then the Frobenius norm of each layer is constant across all minimizers. This, in turn, yields a global lower bound on the trace of the Hessian evaluated at any minimizer of the regularized deep matrix factorization problem. Furthermore, we establish a critical threshold for the regularization parameter above which the unique end-to-end minimizer collapses to zero.

Abstract

Loss Landscape Sharpness

Explore similar work

On the Loss Landscape Geometry of Regularized Deep Matrix Factorization: Uniqueness and Sharpness

Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima

Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks