On the Convergence of Stochastic Gradient Descent with Perturbed Forward-Backward Passes

Boao Kong; Hengrui Zhang; Kun Yuan

math.OCFeb 24, 2026

On the Convergence of Stochastic Gradient Descent with Perturbed Forward-Backward Passes

Authors: Boao Kong, Hengrui Zhang, Kun Yuan

Organizations: Peking University. · Sichuan University.

Abstract

We study stochastic gradient descent (SGD) for composite optimization problems with $N$ sequential operators subject to perturbations in both the forward and backward passes. Unlike classical analyses that treat gradient noise as additive and localized, perturbations to intermediate outputs and gradients cascade through the computational graph, compounding geometrically with the number of operators. We present the first comprehensive theoretical analysis of this setting. Specifically, we characterize how forward and backward perturbations propagate and amplify within a single gradient step, derive convergence guarantees for both general non-convex objectives and functions satisfying the Polyak--Łojasiewicz condition, and identify conditions under which perturbations do not deteriorate the asymptotic convergence order. As a byproduct, our analysis furnishes a theoretical explanation for the gradient spiking phenomenon widely observed in deep learning, precisely characterizing the conditions under which training recovers from spikes or diverges. Experiments on logistic regression with convex and non-convex regularization validate our theories, illustrating the predicted spike behavior and the asymmetric sensitivity to forward versus backward perturbations.

Figures & tables

Figure 1: An illustration of SGD algorithm with perturbed forward and backward passes.

Figure 2: Comparison of gradient spike patterns leading to contrasting outcomes. The gradient norm (left) and loss (right) trajectories illustrate two scenarios: a large-magnitude spike (orange) that permits rapid recovery and convergence, versus a more moderate spike (blue) that triggers persistent deviation and eventual non-convergence.

Assumption	Convergence rate	Zero mean	$Q_{\delta}$	$Q_{\varepsilon}$
Non-convex	$\mathcal{O}\left(\dfrac{1}{\sqrt{T}}\right)$	✓	$\mathcal{O}(T^{\frac{1}{2}})$	$\mathcal{O}(T)$
Non-convex	$\mathcal{O}\left(\dfrac{1}{\sqrt{T}}\right)$	✗	$\mathcal{O}(T^{\frac{1}{2}})$	$\mathcal{O}(T^{\frac{1}{2}})$
PL condition	$\mathcal{O}\left(\dfrac{\ln T}{T}\right)$	✓	$\mathcal{O}(1)$	$\mathcal{O}(T)$
PL condition	$\mathcal{O}\left(\dfrac{\ln T}{T}\right)$	✗	$\mathcal{O}(1)$	$\mathcal{O}(1)$

Table 1: Upper bounds on the number of

\mathcal{O}(1)

-magnitude intermittent perturbation occurrences in the forward and backward passes that SGD can tolerate without deteriorating the convergence rate. “Zero mean” indicates that the intermittent perturbations have zero expectation.

Panel of Figure 3 — Figure 3: The convergence performance with forward and backward computation error with different step size $\gamma$ for the logistic regression task with non-convex regularization. (Left: $\sigma_{f}=2.0$ , $\sigma_{b}=0.0$ . Right: $\sigma_{f}=0.0$ , $\sigma_{b}=2.0$ .)

Panel of Figure 4 — Figure 4: The relationship between the stable gradient norm and the step size $\gamma$ with forward and backward computation error for the logistic regression task with non-convex regularization. (Left: Varied $\sigma_{f}$ with $\sigma_{b}=0$ . Right: Varied $\sigma_{b}$ with $\sigma_{f}=0$ .)

Panel of Figure 5 — Figure 5: The relationship between the stable gradient norm and the step size $\gamma$ with forward and backward computation perturbations for the logistic regression task with strongly convex (hence PL) regularization. (Left: Varied $\sigma_{f}$ with $\sigma_{b}=0$ . Right: Varied $\sigma_{b}$ with $\sigma_{f}=0$ .)

Figure 6: The relationship between the stable gradient norm and the step size

\gamma

with non-zero-mean backward computation perturbations.

Panel of Figure 7 — Figure 7: The relationship between the stable gradient norm and the step size $\gamma$ with intermittent forward perturbations for the logistic regression task in the non-convex scenario. (Left: $\sigma_{f}=1.0$ . Right: $\sigma_{f}=2.0$ .)

Panel of Figure 8 — Figure 8: The relationship between the stable gradient norm and the step size $\gamma$ with intermittent forward perturbations for the logistic regression task in the strongly convex (hence PL) scenario. (Left: $\sigma_{f}=1.0$ . Right: $\sigma_{f}=2.0$ .)

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: The norm of the parameter $w_{1}^{(t)}$ obtained by SGD with Top-1 compressor and the comparison with the non-compression case. Note that the Top-1 cases all fail to converge. (Left: $\gamma=0.1$ and $\beta=0$ . Right: $\gamma=0.05$ and $\beta=0.9$ .)

Stochastic Optimization Convergence

Stochastic optimization convergence studies how noisy, sampled, or delayed updates approach solutions and how fast they do so.

See all 205 papers

Large Spikes in Stochastic Gradient Descent: A Large-Deviations View
Mar 10, 2026Benjamin Gess, Daniel HeydeckerLarge Deviation Principles Gradient Descent Dynamics
Technische Universität Berlin, Fakultät II 10623 Berlin, Germany · Max Planck Institute for Mathematics in the Sciences 04103 Leipzig, Germany · University of Oslo, Blindern 0316 Oslo, Norway
Exact information accounting for SGD methods
Sep 30, 2026Akshay BalsubramaniNeural Network Generalization Stochastic Optimization Convergence
Convergence of stochastic gradient descent for learning linear neural networks
Oct 20, 2026Gabin Maxime Nguegnang, Bubacarr Bah, Holger Rauhut +1Stochastic Optimization Convergence Stochastic Gradient Descent

Mar 10, 2026J/K move · Enter open · S save

Large Spikes in Stochastic Gradient Descent: A Large-Deviations View

Benjamin Gess, Daniel Heydecker

Technische Universität Berlin, Fakultät II 10623 Berlin, Germany · Max Planck Institute for Mathematics in the Sciences 04103 Leipzig, Germany · University of Oslo, Blindern 0316 Oslo, Norway

Large Deviation Principles Gradient Descent Dynamics Stochastic Gradient Descent

Open paper AskPDF arXiv

Large loss spikes in stochastic gradient descent are studied through a rigorous large-deviations analysis for a shallow, fully connected network in the NTK scaling. In contrast to full-batch gradient descent, the catapult phase is shown to split into inflationary and deflationary regimes, determined by an explicit log-drift criterion. In both cases, large spikes are shown to be at least polynomially likely. In addition, these spikes are shown to be the dominant mechanism by which sharp minima are escaped and curvature is reduced, thereby favouring flatter solutions. Corresponding results are also obtained for certain ReLU networks, and implications for curriculum learning are derived.

Abstract

Stochastic Optimization Convergence

Explore similar work

Large Spikes in Stochastic Gradient Descent: A Large-Deviations View

Exact information accounting for SGD methods

Convergence of stochastic gradient descent for learning linear neural networks