math.OCDec 18, 2024

Projected gradient methods for nonconvex and stochastic smooth optimization: new complexities and auto-conditioned stepsizes

Authors: Guanghui Lan, Tianjiao Li, Yangyang Xu

Organizations: School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA 30332. · Department of Mathematical Sciences, Rensselaer Polytechnic Institute, Troy, NY 12180.

Abstract

We present a novel class of projected gradient (PG) methods for minimizing a smooth but not necessarily convex function over a convex compact set. We first provide a novel analysis of the constant-stepsize PG method, achieving a unified iteration complexity for finding an approximate stationary point for convex and nonconvex problems. We then develop an "auto-conditioned" projected gradient (AC-PG) variant that achieves the same iteration complexity without requiring the input of the Lipschitz constant of the gradient or any line search procedure. The key idea is to estimate the Lipschitz constant using first-order information gathered from the previous iterations, and to show that the error caused by underestimating the Lipschitz constant can be properly controlled. We then generalize the PG methods to the stochastic setting, by proposing a stochastic projected gradient (SPG) method and a variance-reduced stochastic gradient (VR-SPG) method, achieving new unified complexity bounds in different oracle settings. We also present auto-conditioned stepsize policies for both stochastic PG methods and establish comparable convergence guarantees.

Figures & tables

Explore similar work

CardsList
  1. Stochastic Auto-conditioned Fast Gradient Methods with Optimal Rates

    Apr 7, 2026Yao Ji, Guanghui LanStochastic OptimizationAccelerated Gradient Methods

  2. Near-Optimal Methods for Generalized Lipschitz Convex Optimization

    Nov 15, 2026Dmitry Kovalev, Grigory MalinovskyNonsmooth Convex OptimizationStochastic Optimization Convergence