Understanding the loss-landscape geometry near a minimum is key to explaining the implicit bias of gradient-based methods in non-convex optimization problems such as deep neural network training and deep matrix factorization. A central quantity for characterizing this geometry is the maximum eigenvalue of the Hessian of the loss. Currently, its precise role has been obfuscated because no exact expressions for this sharpness measure were known in general settings. In this paper, we present the first exact expression for the maximum eigenvalue of the Hessian of the squared-error loss at any minimizer in deep matrix factorization problems, resolving an open question posed by Mulayoff and Michaeli [2020]. This expression reveals a fundamental property of the loss landscape in deep matrix factorization: It is sufficient, but not necessary, for flatness that, across all layers, the product of the spectral norms of the left and right intermediate factors equals a constant determined by the depth and the spectral norm of the target matrix. Most notably, in both depth-2 matrix factorization and deep overparameterized scalar factorization, we show that this condition is both necessary and sufficient for flatness, which implies that flat minima are spectral-norm balanced and not necessarily balanced in depth-2 matrix factorization. To complement our theory, we provide empirical verification, relying on our exact expression, of an escape phenomenon during gradient-based training near a minimizer of a deep matrix factorization problem.
Figures & tables
Related Work
Depth
Input Dim.
Output Dim.
λmax(∇2L(w))
Layers
Mulayoff and Michaeli (2020, Theorem 1)
L
d0
dL
w∈ΩF
Ra×b
Zhu et al. (2023, Appendix B.1)
2
1
1
w∈RN
R
Singh and Hofmann (2024, Theorem 1)
2
1
1
w∈RN
Ra
Ghosh et al. (2025, Lemma 1)
L
d0
dL
w∈ΩB
Ra×b
Theorem 3.1 (This Paper)
L
d0
dL
w∈Ω
Ra×b
Table 1: Closed-form expressions for the maximum Hessian eigenvalue in the literature. Ω denotes the set of all global minimizers, ΩF⊆Ω denotes the set of flat global minimizers, and ΩB⊆Ω denotes the set of balanced global minimizers (see Definition 2.1 ). The Layers column specifies the form of the layers (factors) considered in each work: scalar-valued ( R ), vector-valued ( Ra ), or matrix-valued ( Ra×b ).
Figure 1: The setting is a depth- 3 matrix factorization of a random 10×20 matrix with Gaussian factors W3∈R10×20 and W1,W2∈R20×20 . v1 and vN are the eigenvectors of ∇2L(w∗) that correspond to its maximum and minimum eigenvalues respectively. λmax is computed via the closed-form expression in Theorem 3.1 . GD runs with different step sizes are initialized within 10−12 of the minimizer w∗ along v1 . The contour plot in (c) projects the parameter space onto span{v1,vN} (Appendix E.1 ); darker (brighter) regions indicate higher (lower) loss. See Appendix E.2 for details.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: The setting is a depth-2 matrix factorization of a random 10×20 target matrix, where L∈R10×30 and R∈R20×30 are Gaussian factors. GD dynamics with different step sizes, indicated by different colors, are initialized within a radius of 10−15 in (a)–(c) and 10−9 in (d)–(f) from the minimizer w∗ . The initialization direction is along the eigenvector of ∇2L(w∗) that corresponds to its maximum eigenvalue. The value of λmax(∇2L(w∗)) is computed using the closed-form expression derived in Corollary 18 .
Figure 3: The setting is a 20-layer overparameterized scalar factorization of a random scalar. GD dynamics with different step sizes, indicated by different colors, are initialized within a radius of 10−9 from the minimizer w∗ . The initialization direction is along the eigenvector of ∇2L(w∗) that corresponds to its maximum eigenvalue. The value of λmax(∇2L(w∗)) is computed using the closed-form expression derived in Corollary 3.2 .
Figure 4: The setting is a 4-layer overparameterized scalar factorization of a random scalar. GD dynamics with different step sizes, indicated by different colors, are initialized within a radius of 10−12 from the minimizer w∗ . The initialization direction is along the eigenvector of ∇2L(w∗) that corresponds to its maximum eigenvalue. The value of λmax(∇2L(w∗)) is computed using the closed-form expression derived in Corollary 3.2 .
Figure 5: The setting is a depth-2 matrix factorization of a random 25×20 target matrix, where L∈R25×30 and R∈R20×30 are Gaussian factors. GD dynamics with different step sizes, indicated by different colors, are initialized within a radius of 10−9 from the minimizer w∗ . The initialization direction is along the eigenvector of ∇2L(w∗) that corresponds to its maximum eigenvalue. The value of λmax(∇2L(w∗)) is computed using the closed-form expression derived in Corollary 18 .
Weight decay is ubiquitous in training deep neural network architectures. Its empirical success is often attributed to capacity control; nonetheless, our theoretical understanding of its effect on the loss landscape and the set of minimizers remains limited. In this paper, we show that ℓ2-regularized deep matrix factorization/deep linear network training problems with squared-error loss admit a unique end-to-end minimizer for all target matrices subject to factorization, except for a set of Lebesgue measure zero formed by the depth and the regularization parameter. This observation reveals fundamental properties of the loss landscape of regularized deep matrix factorization problems: the Hessian spectrum is constant across all minimizers of the regularized deep scalar factorization problem with squared-error loss. Moreover, we show that, in regularized deep matrix factorization problems with squared-error loss, if the target matrix does not belong to the Lebesgue measure-zero set, then the Frobenius norm of each layer is constant across all minimizers. This, in turn, yields a global lower bound on the trace of the Hessian evaluated at any minimizer of the regularized deep matrix factorization problem. Furthermore, we establish a critical threshold for the regularization parameter above which the unique end-to-end minimizer collapses to zero.
An important quantity in the theory of gradient descent (GD) is the sharpness, defined as the largest eigenvalue of the objective Hessian. Classical analyses typically require the step size to be uniformly smaller than twice the reciprocal of the sharpness, but this condition is frequently violated in the training of deep neural networks. Recent work bridges this gap in the setting of overparametrised least-squares with a single scalar output, providing a normal form for large-step GD in a neighbourhood of an isolated flat minimum and establishing three corresponding convergence results. In this paper, we extend this theory in two directions: (1) to overparametrised least-squares with vector-valued outputs (including regression with arbitrarily many observations), and (2) to a neighbourhood of a manifold of flat minima (which we show is essential for applications such as matrix factorisation). We generalise both the normal form and all three convergence theorems of \cite{macdonaldeos} to this broader setting, overcoming several technical challenges, including the solution of a singular partial differential equation via a novel method that may be of independent interest. We further show that our framework applies to deep matrix factorisation under mild assumptions, yielding several new structural results. In particular, we prove that the set of flat minima forms a fibre bundle over a product of spheres, and that the sharpness is Morse-Bott along this manifold.
Lachlan Ewen MacDonald, René Vidal
Innovation in Data Engineering and Science (IDEAS) · University of Pennsylvania Pennsylvania, PA 19104
The flatness hypothesis suggests that flatness of the loss landscape, as measured by the eigenvalues of the loss Hessian, correlates with better neural network generalization. While various algorithms reduce these eigenvalues, most focus on procedural design, leaving it unclear how data distributions and NN parameters structurally determine directions toward flat minima. Characterizing these directions analytically is generally intractable. To overcome this mathematical difficulty, recent studies derived the Wolkowicz-Styan (WS) upper bound on the maximum eigenvalue of the cross-entropy loss Hessian in three-layer NNs. Although this upper bound is differentiable, its gradient was not derived. Therefore, we analytically derive the gradient of the WS upper bound to characterize directions leading to flat minima. Based on this, we propose Hessian Spectral Range (HSR) Regularization, which updates parameters along the steepest descent direction of the WS bound. Experiments demonstrate that HSR Regularization narrows the Hessian eigenvalue spectrum, avoids sharp minima and saddle points, and promotes convergence to flat minima. Although the applicability of this method is currently limited to cross-entropy loss and three-layer architectures, to the best of the authors' knowledge, this is the first study to report a closed-form gradient that promotes convergence to flat minima without numerical approximations. Therefore, the theoretical analysis of this gradient is expected to contribute to the further development of NNs.
Yuto Omae, Kazuki Sakai, Yohei Kakimoto +3
Nihon University (Japan) · National Institute of Technology, Nagaoka College (Japan) · Tokyo City University (Japan)