math.OCOct 1, 2026

Lower Bounds for Stochastic First-Order Algorithms with Variance Reduction in Nonconvex--Concave Minimax Optimization

Authors: Jiayi Song, Zi Xu

Organizations: Department of Mathematics, College of Sciences, Shanghai University, Shanghai 200444, China

Abstract

We establish complexity lower bounds for stochastic first-order algorithms in nonconvex--concave minimax optimization, allowing algorithms to use variance reduction. Our main contribution is a lower bound for a zero-respecting algorithm class that permits variance reduction, extending beyond the algorithmic restrictions imposed by some existing lower bounds. We consider objectives with an LL-Lipschitz continuous joint gradient, a compact convex dual domain of Euclidean radius at most DYD_Y, and a primal value function, defined by maximizing the objective over the dual variable, with initial suboptimality at most ΔΔ. The target accuracy ε\varepsilon is measured by the gradient norm of the Moreau envelope of the constrained primal value function with parameter 1/(2L)1/(2L). Under an unbiased stochastic first-order oracle with variance at most σ2σ^2 and mean-square smoothness, we prove the lower bound Ω ⁣(L2DYΔε−3+L3DY2Δσ2ε−6)Ω\!\left(L^2D_YΔ\varepsilon^{-3}+L^3D_Y^2Δσ^2\varepsilon^{-6}\right). This result quantifies the dependence on accuracy, dual-domain radius, and oracle noise even when variance reduction is allowed. We also establish complementary lower bounds for nonconvex--strongly-concave minimax optimization. With dual strong-concavity parameter μ>0μ>0 and condition number κ:=L/μκ:=L/μ, we obtain Ω ⁣(LΔκ ε−2+LΔκσ2ε−4)Ω\!\left(LΔ\sqrtκ\,\varepsilon^{-2}+LΔκσ^2\varepsilon^{-4}\right) under the bounded-variance oracle model. Under the additional mean-square smoothness condition with constant Lˉ\bar L, we obtain Ω ⁣(LΔκ ε−2+ΔLˉσκ3/2ε−3)Ω\!\left(LΔ\sqrtκ\,\varepsilon^{-2}+Δ\bar Lσκ^{3/2}\varepsilon^{-3}\right). Together, these results identify complexity barriers across the concave and strongly concave regimes, with the main nonconvex--concave bound remaining valid for algorithms that use variance reduction.

Figures & tables

Explore similar work

Jun 18, 2024cs.LG

Accelerated Stochastic Min-Max Optimization Based on Bias-corrected Momentum

Lower-bound analyses for nonconvex strongly-concave minimax optimization problems have shown that stochastic first-order algorithms require at least O(ε−4)\mathcal{O}(\varepsilon^{-4}) sample complexity to find an ε\varepsilon-stationary point. Some works indicate that this complexity can be improved to O(ε−3)\mathcal{O}(\varepsilon^{-3}) when the stochastic loss gradient is Lipschitz continuous. The question of achieving enhanced convergence rates under distinct conditions, remains open. In this work, we address this question for optimization problems that are nonconvex in the minimization variable and strongly concave or Polyak-Lojasiewicz (PL) in the maximization variable. We introduce novel bias-corrected momentum algorithms utilizing efficient Hessian-vector products. We establish convergence conditions and demonstrate a lower iteration complexity of O(ε−3)\mathcal{O}(\varepsilon^{-3}) for the proposed algorithms. The effectiveness of the proposed method is validated through applications to robust logistic regression and robust adaptive cruise control.
Sep 8, 2026math.OC

How to Make the Gradient Mapping Small for Constrained Stochastic Min-Max Problems and Beyond

We study the stochastic first-order oracle complexity for constrained or regularized convex-concave min-max optimization and stochastic monotone variational inequalities. We focus on the case when suboptimality is measured in terms of the gradient mapping, also known as, forward-backward or natural residual, an optimality notion that generalizes the gradient norm for unconstrained problems. In this setting, under standard unbiased oracle access with now-standard variance assumptions, the best-known complexity for making the norm of the gradient mapping less than ε\varepsilon is O~(ε−4)\widetilde{O}(\varepsilon^{-4}), compared to the near-optimal O~(ε−2)\widetilde{O}(\varepsilon^{-2}) that is established in the unconstrained case. We bridge this gap to improve the gradient mapping complexity for constrained convex-concave min-max problems to O~(ε−2)\widetilde{O}(\varepsilon^{-2}). We then extend to prove the same complexity for problems without the bounded variance, by using the Blum-Gladyshev assumption.
Aug 10, 2026math.OC

A Tight Lower Bound for Smooth Nonconvex Stochastic Optimization with Bounded Gradient Noise

We prove a sharp lower bound for smooth nonconvex stochastic optimization with uniformly bounded gradient noise. In the K=1K=1 fresh-sample model, every randomized adaptive algorithm requires Ω(ΔLε2+ΔLσ2ε4)Ω\left( \frac{ΔL}{ε^2} + \frac{ΔLσ^2}{ε^4} \right) queries to find a point with expected gradient norm at most εε. This matches the standard upper bound and, to the best of our knowledge, resolves the question raised by [Arjevani et al. 2023] of whether almost-surely bounded oracle error permits a better rate than bounded variance. The proof was independently generated with GPT-5.6 Sol in Codex's Ultra mode during a two-hour session. The human author supplied the prompt and was responsible only forchecking the proof and revising and polishing the manuscript.