stat.MLMar 28, 2026

On the Loss Landscape Geometry of Regularized Deep Matrix Factorization: Uniqueness and Sharpness

Authors: Anil Kamber, Rahul Parhi

Organizations: UC San Diego

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Weight decay is ubiquitous in training deep neural network architectures. Its empirical success is often attributed to capacity control; nonetheless, our theoretical understanding of its effect on the loss landscape and the set of minimizers remains limited. In this paper, we show that ℓ2\ell^2-regularized deep matrix factorization/deep linear network training problems with squared-error loss admit a unique end-to-end minimizer for all target matrices subject to factorization, except for a set of Lebesgue measure zero formed by the depth and the regularization parameter. This observation reveals fundamental properties of the loss landscape of regularized deep matrix factorization problems: the Hessian spectrum is constant across all minimizers of the regularized deep scalar factorization problem with squared-error loss. Moreover, we show that, in regularized deep matrix factorization problems with squared-error loss, if the target matrix does not belong to the Lebesgue measure-zero set, then the Frobenius norm of each layer is constant across all minimizers. This, in turn, yields a global lower bound on the trace of the Hessian evaluated at any minimizer of the regularized deep matrix factorization problem. Furthermore, we establish a critical threshold for the regularization parameter above which the unique end-to-end minimizer collapses to zero.

Figures & tables

Figure 1: Behavior of the data-fitting term D(ρ)D(\rho) , the regularization term R(ρ)R(\rho) , and the optimization objective ϕ(ρ)\phi(\rho) for a depth- 33 factorization of 33 with λ=4\lambda=4 .
Figure 2: Behavior of ϕ(ρ)\phi(\rho) for a depth- 55 factorization of −3-3 under different regularization parameters. Threshold is computed as τ=(∣m∣/(1−q2)(1−q)(q−1)/(2−q))(2−q)\tau=\left(|m|/\left(1-\frac{q}{2}\right)\left(1-q\right)^{({q-1})/({2-q})}\right)^{(2-q)} , where q:=2/Lq:=2/L .
Figure 3: Frobenius norm of the converged point obtained after T=5000T=5000 steps of GD with regularization parameter λ=ατ\lambda=\alpha\tau applied to ( 2 ), where τ=(\normσ(M♮)∞/(1−q2)(1−q)(q−1)/(2−q))(2−q)\tau=\left(\norm{\sigma(\mathbf{M}^\natural)}_{\infty}/\left(1-\frac{q}{2}\right)\left(1-q\right)^{({q-1})/({2-q})}\right)^{(2-q)} .
More in this topic

Loss Landscape

See all 22 papers

Explore similar work

CardsList

  1. Implicit Regularization in Perturbed Deep Matrix Factorization: Spectral Conditions and Stability

    May 27, 2026Jingzhe Wang, Hung-Hsu ChouAlgorithmic StabilityLow-Rank Matrix Decomposition

    Department of Informatics and Networked Systems, University of Pittsburgh · Department of Mathematics, University of Pittsburgh

  2. Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization

    May 28, 2025Etienne Boursier, Matthew Bowditch, Matthias Englert +1Loss LandscapeOverparameterized Neural Network

    INRIA LMO, Université Paris-Saclay Orsay, France · Mathematics Institute University of Warwick Coventry, UK · Department of Computer Science University of Warwick Coventry, UK

  3. Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks

    Jun 27, 2026Yuto Omae, Kazuki Sakai, Yohei Kakimoto +3Gradient DescentNeural Network Optimization

    Nihon University (Japan) · National Institute of Technology, Nagaoka College (Japan) · Tokyo City University (Japan)