cs.LGOct 23, 2025

Convergence Analysis of SGD under Expected Smoothness

Authors: Yuta Kawamoto, Hideaki Iiduka

Organizations: Meiji University

Abstract

Stochastic gradient descent (SGD) is the workhorse of large-scale learning, yet classical analyses rely on assumptions that can be either too strong (bounded variance) or too coarse (uniform noise). The expected smoothness (ES) condition has emerged as a flexible alternative that ties the second moment of stochastic gradients to the objective value and the full gradient. This paper presents a self-contained convergence analysis of SGD under ES. We (i) refine ES with interpretations and sampling-dependent constants; (ii) derive bounds of the expectation of squared full gradient norm; and (iii) prove sublinear convergence rates with explicit residual errors for various step-size schedules. Furthermore, we empirically validate our theoretical framework by training a ResNet-18 model on the CIFAR-100 dataset. We uncover a two-phase dynamic in the gradient noise structure, transitioning from a suboptimality-dominated regime to a geometry-dominated regime. Our treatment unifies and extends recent theoretical threads in the literature.

Figures & tables

Explore similar work

CardsList
  1. Exact Theory and Optimal Stepsizes for Stochastic Gradient Descent in Smooth Convex Minimization

    Oct 30, 2026Benjamin Grimmer, Qining ShenSmooth Convex OptimizationStochastic Gradient Descent

  2. Near-Optimal Last-Iterate Convergence of SGD in the Smooth Interpolation Regime

    Oct 18, 2026Matan Schliserman, Tomer KorenSmooth Convex OptimizationLast-Iterate Convergence

  3. The Exact Time-Uniform Rate Frontier for Stochastic Gradient Descent on Smooth Convex Objectives

    Sep 8, 2026Ruijie Li, Kang Chen, Tianyu WangLearning Rate SchedulingLast-Iterate Convergence