Stochastic gradient descent (SGD) is the cornerstone of modern deep learning, yet its theoretical convergence properties remain poorly understood in non-convex settings. To address this gap, we study SGD dynamics in deep linear neural networks - simplified models that retain key non-convex characteristics while remaining analytically tractable. Building on recent results for gradient descent, we extend convergence guarantees to SGD under regularized squared loss. We establish almost sure boundedness of SGD iterates. Using the framework of asymptotic pseudotrajectories from stochastic approximation theory, our analysis establishes then, for both deterministic and adaptive step-size choices, convergence of SGD to the set of critical points under mild conditions (without relying on Lipschitz properties or whitened data). To our knowledge, this is the first work providing almost sure convergence guarantees for SGD in deep linear networks, bridging the gap between theoretical optimization and practical stochastic training.
Figures & tables
Figure 1. The left panel illustrates the evolution of the regularized loss functions across networks with deterministic step sizes as specified in ( 27 ) . The right panel presents the Frobenius norms of individual weight matrices for networks with depths 3 and 4 during training.
Figure 2. The left panel displays the evolution of the regularized losses across networks with deterministic stepsizes that do not meet requirements in Theorem 13 . The right panel shows the Frobenius norms of individual matrices for networks of depths 2, 3, and 4 during training.
Figure 3. The left panel illustrates the evolution of the regularized loss across networks with adaptive stepsizes ( 28 ) . The right panel presents the Frobenius norms of individual weight matrices for networks of depth 3 and 4.
Figure 4. The left panel shows the evolution of the regularized loss across networks with the modified adaptive step sizes ( 29 ) . The right panel displays the Frobenius norms of individual weight matrices for networks of depth 2, 3, and 4 during training with the same adaptive step sizes.
Figure 5. Convergence comparison for deterministic versus adaptive stepsizes on a linear network of depth 3 with parameters β=5 , λ=5⋅10−5 , r=5 , and mini-batch size T=30 .
Figure 6. Training nonlinear neural networks: evolution of the regularized loss functions across networks under deterministic step sizes in ( 27 ) on the left panel. Frobenius norms of individual weight matrices of the corresponding networks for depth 3, and 4 on the right panel.
Figure 7. Training nonlinear neural networks: evolution of the regularized loss functions across networks under adaptive stepsizes in ( 28 ) on the left panel. Frobenius norms of individual weight matrices of the corresponding networks for depth 3 and 4 on the right panel.