We study stochastic gradient descent (SGD) for composite optimization problems with N sequential operators subject to perturbations in both the forward and backward passes. Unlike classical analyses that treat gradient noise as additive and localized, perturbations to intermediate outputs and gradients cascade through the computational graph, compounding geometrically with the number of operators. We present the first comprehensive theoretical analysis of this setting. Specifically, we characterize how forward and backward perturbations propagate and amplify within a single gradient step, derive convergence guarantees for both general non-convex objectives and functions satisfying the Polyak--Łojasiewicz condition, and identify conditions under which perturbations do not deteriorate the asymptotic convergence order. As a byproduct, our analysis furnishes a theoretical explanation for the gradient spiking phenomenon widely observed in deep learning, precisely characterizing the conditions under which training recovers from spikes or diverges. Experiments on logistic regression with convex and non-convex regularization validate our theories, illustrating the predicted spike behavior and the asymmetric sensitivity to forward versus backward perturbations.
Figures & tables
Figure 1: An illustration of SGD algorithm with perturbed forward and backward passes.
Figure 2: Comparison of gradient spike patterns leading to contrasting outcomes. The gradient norm (left) and loss (right) trajectories illustrate two scenarios: a large-magnitude spike (orange) that permits rapid recovery and convergence, versus a more moderate spike (blue) that triggers persistent deviation and eventual non-convergence.
Assumption
Convergence rate
Zero mean
Qδ
Qε
Non-convex
O(T1)
✓
O(T21)
O(T)
✗
O(T21)
O(T21)
PL condition
O(TlnT)
✓
O(1)
O(T)
✗
O(1)
O(1)
Table 1: Upper bounds on the number of O(1) -magnitude intermittent perturbation occurrences in the forward and backward passes that SGD can tolerate without deteriorating the convergence rate. “Zero mean” indicates that the intermittent perturbations have zero expectation.
Figure 3: The convergence performance with forward and backward computation error with different step size γ for the logistic regression task with non-convex regularization. (Left: σf=2.0 , σb=0.0 . Right: σf=0.0 , σb=2.0 .)
Figure 4: The relationship between the stable gradient norm and the step size γ with forward and backward computation error for the logistic regression task with non-convex regularization. (Left: Varied σf with σb=0 . Right: Varied σb with σf=0 .)
Figure 5: The relationship between the stable gradient norm and the step size γ with forward and backward computation perturbations for the logistic regression task with strongly convex (hence PL) regularization. (Left: Varied σf with σb=0 . Right: Varied σb with σf=0 .)
Figure 6: The relationship between the stable gradient norm and the step size γ with non-zero-mean backward computation perturbations.
Figure 7: The relationship between the stable gradient norm and the step size γ with intermittent forward perturbations for the logistic regression task in the non-convex scenario. (Left: σf=1.0 . Right: σf=2.0 .)
Figure 8: The relationship between the stable gradient norm and the step size γ with intermittent forward perturbations for the logistic regression task in the strongly convex (hence PL) scenario. (Left: σf=1.0 . Right: σf=2.0 .)
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 9: The norm of the parameter w1(t) obtained by SGD with Top-1 compressor and the comparison with the non-compression case. Note that the Top-1 cases all fail to converge. (Left: γ=0.1 and β=0 . Right: γ=0.05 and β=0.9 .)
Large loss spikes in stochastic gradient descent are studied through a rigorous large-deviations analysis for a shallow, fully connected network in the NTK scaling. In contrast to full-batch gradient descent, the catapult phase is shown to split into inflationary and deflationary regimes, determined by an explicit log-drift criterion. In both cases, large spikes are shown to be at least polynomially likely. In addition, these spikes are shown to be the dominant mechanism by which sharp minima are escaped and curvature is reduced, thereby favouring flatter solutions. Corresponding results are also obtained for certain ReLU networks, and implications for curriculum learning are derived.
Benjamin Gess, Daniel Heydecker
Technische Universität Berlin, Fakultät II 10623 Berlin, Germany · Max Planck Institute for Mathematics in the Sciences 04103 Leipzig, Germany · University of Oslo, Blindern 0316 Oslo, Norway
As an alternative to the standard geometric analyses, we give an exact, information-theoretic analysis of stochastic gradient descent (SGD) and its variants. We show that a preconditioned SGD step is the posterior-mean update of a Gaussian Bayes model, and that its one-step regret splits into an intrinsic-time cost and a change in comparator information. The split extends to an identity for the objective itself. Convex convergence, strict-saddle-point escape, the link between flatness and generalization, the standard learning-rate schedules, adaptive optimizers, and the noisy, momentum, heavy-tailed, and gradient-free variants of SGD each correspond to a term or a special case of this identity. We measure its terms on synthetic and real training runs. On real networks it attributes the slack of classical convergence bounds to the terms their derivations drop and separates optimizers that reach the same training loss. That separation follows the number and consistency of their steps. Its relation to which of them generalizes better differs between networks. For gradient-free SGD the identity determines how a curvature preconditioner should enter the update. The sharpness-based generalization certificate it yields, with a data-independent isotropic prior, is vacuous at network scale unless the curvature spectrum is nearly flat across all parameters.
Stochastic gradient descent (SGD) is the cornerstone of modern deep learning, yet its theoretical convergence properties remain poorly understood in non-convex settings. To address this gap, we study SGD dynamics in deep linear neural networks - simplified models that retain key non-convex characteristics while remaining analytically tractable. Building on recent results for gradient descent, we extend convergence guarantees to SGD under regularized squared loss. We establish almost sure boundedness of SGD iterates. Using the framework of asymptotic pseudotrajectories from stochastic approximation theory, our analysis establishes then, for both deterministic and adaptive step-size choices, convergence of SGD to the set of critical points under mild conditions (without relying on Lipschitz properties or whitened data). To our knowledge, this is the first work providing almost sure convergence guarantees for SGD in deep linear networks, bridging the gap between theoretical optimization and practical stochastic training.
Gabin Maxime Nguegnang, Bubacarr Bah, Holger Rauhut +1