Nonconvex Stochastic Optimization

Overview

The field studies how to find useful approximate solutions when objectives may have many local minima and gradients are estimated from sampled data, as in neural-network training. Common approaches include stochastic gradient descent (SGD), momentum and variance reduction, adaptive preconditioning, and robust methods such as gradient clipping for heavy-tailed noise; related work handles constraints, minimax objectives, and federated or asynchronous training. Researchers compare methods using stationarity measures and oracle or communication complexity, alongside experiments on tasks such as image classification, language-model training, and distributionally robust learning. A central tension is that strong guarantees often depend on assumptions about smoothness and noise that practical training may violate, while methods that work well empirically can be difficult to analyze or tune.

Latest papers

CardsList

103 papers12 papers in the last four weeks, down 33% on the four weeks before. 0.1% of all new papers.

  1. Muon Attains O(ε−3)\mathcal{O}(\varepsilon^{-3}) Iteration Complexity under Shuffling

    Nov 13, 2026Kun Huang, Shi Pu, Karl Henrik JohanssonMuon OptimizerStochastic Optimization

  2. Learning Rate Decay Can Exponentially Accelerate SGD for Global Optimization of Nonconvex Functions

    Nov 9, 2026Guneykan Ozgul, Dylan Herman, Junhyung Lyle Kim +2Nonconvex Stochastic OptimizationGlobal Optimization

  3. Zeroth-Order Langevin Dynamics for Stochastic Nonconvex Optimization: Curvature Regularization and Non-Asymptotic Guarantees

    Nov 7, 2026Stefano Bruno, Dong-Young LimZeroth-Order OptimizationNonconvex Stochastic Optimization

  4. Coordinatewise Epsilon for Adam under Heavy-Tailed Noise

    Nov 6, 2026Jeonseong Kim, Chenglin FanStochastic Optimization ConvergenceAdaptive Gradient Methods

  5. Adaptive Federated Optimization Across Smooth and Nonsmooth Regimes

    Oct 29, 2026Zimeng Wang, Ali Dadras, Alp YurtseverFederated LearningAdaptive Gradient Methods

  6. Rennala NSGD: Asynchronous Stochastic Optimization Beyond Euclidean Geometry

    Oct 28, 2026Igor Sokolov, Alexander Tyurin, Peter RichtárikStochastic OptimizationStochastic Gradient Descent

  7. Sharpness-Aware Muon: Matrix-Aware Optimization with Momentum-Guided Perturbations

    Oct 27, 2026Zhiwei Zhai, Wenjing Yan, Ying-Jun Angela ZhangMuon OptimizerSharpness-Aware Minimization

  8. Understanding Practical Optimization in Machine Learning

    Oct 26, 2026Egor ShulginNonconvex Stochastic OptimizationLMO Methods

  9. Last-Iterate Stationarity of Muon with Finitely Many Newton-Schulz Iterations

    Oct 25, 2026Ping Li, Vincent Y. F. TanMuon OptimizerLast-Iterate Convergence

  10. On the Limits of Stochastic Push-Pull over Time-Varying Digraphs

    Oct 23, 2026Feiming Wang, Yutong He, Sulaiman Alghunaim +1Decentralized OptimizationStochastic Optimization Convergence

  11. Variance Reduction with Delayed Feedback for Non-Convex Optimization

    Oct 20, 2026Wei Jiang, Dingzhi Yu, Zechao LiVariance ReductionNonconvex Stochastic Optimization

  12. Minimax Optimality of Shampoo for Nonstationary Stochastic Optimization

    Oct 12, 2026Abir Sarkar, Nirmalya Mandal, Martin T. WellsStochastic OptimizationNonconvex Stochastic Optimization

  13. Partially Informed Stochastic Optimization via Zeroth-Order Residual Tracking

    Oct 9, 2026El Mahdi Chayti, Alireza Mirrokni, Martin JaggiStochastic OptimizationNonconvex Stochastic Optimization

  14. Adaptive Variance Reduction across Smoothness and Noise Conditions

    Oct 9, 2026Wei Jiang, Yibo Wang, Sifan Yang +2Nonconvex Stochastic OptimizationVariance Reduction

  15. Composite Online-to-Nonconvex Conversion with Optimal Oracle Complexity

    Oct 8, 2026Mingyi Li, Taira Tsuchiya, Kenji YamanishiNonconvex Stochastic OptimizationNonconvex Optimization

  16. Decentralized SGD under Heavy-Tailed Noise: Optimal Convergence Rates and the Role of Gradient Clipping

    Oct 7, 2026Aleksandar Armacki, Haoyuan Cai, Ali H. SayedDecentralized OptimizationStochastic Gradient Descent

  17. Dimension-Free Decentralized Nonsmooth Nonconvex Stochastic Optimization

    Oct 5, 2026Yuanyu Wan, Lan Xue, Haomin Bai +2Nonconvex Stochastic OptimizationDecentralized Optimization

  18. Smoothed Gradient Method for Nonconvex Federated Stochastic Bilevel Optimization

    Oct 4, 2026Xinwen Zhang, Peiran Yu, Zhaosong Lu +1Gradient DescentNonconvex Stochastic Optimization

  19. Optimal Stochastic Bilevel Optimization with First-Order Oracles

    Oct 1, 2026Linxuan Pan, Junchi YangStochastic Bilevel OptimizationOracle Complexity

  20. Convergence Analysis of STORM Under Different Geometries

    Oct 1, 2026Wei Jiang, Yibo Wang, Wenhao Yang +3Stochastic OptimizationLast-Iterate Convergence

  21. Optimal Momentum Methods for Stochastic Multilevel Compositional Optimization

    Oct 1, 2026Wei Jiang, Rui Yan, Sifan Yang +3Stochastic OptimizationMomentum Methods

  22. Convergence of Practical Muon

    Sep 27, 2026Haonan Wang, Yu Wu, Minghui Liwang +2Muon OptimizerStochastic Optimization Convergence

  23. Precise Convergence Speed of Clipped SGD

    Sep 24, 2026David A. R. RobinGradient ClippingNonconvex Stochastic Optimization