Gradient Descent

Ask about this topic

Overview

Gradient descent is a family of methods that repeatedly adjusts parameters in the direction that lowers an objective, making it a basic tool for training neural networks and fitting models such as logistic regression. Common approaches include full-batch and stochastic gradient descent, momentum and Nesterov acceleration, adaptive optimizers such as Adam, and geometry-aware variants such as natural, preconditioned, projected, or mirror descent. Theory and applications span deep-network training, matrix recovery, differential privacy, and decentralized or federated learning, with evaluations often measuring convergence, prediction quality, communication, memory, or privacy cost. A central tension is that larger or adaptive steps can speed training but also cause oscillations or instability, while the geometry, noise, and stopping time can change both which solution is reached and how well it generalizes.

Latest papers

CardsList

118 papers16 papers in the last four weeks, up 60% on the four weeks before. 0.1% of all new papers.

  1. Gradient descent reaches arbitrary accuracy in a fixed number of large steps on multiclass logistic regression

    Oct 19, 2026Andreea-Alexandra Musat, Nicolas BoumalGradient DescentLogistic Regression

    École Polytechnique Fédérale de Lausanne (EPFL), Institute of Mathematics

  2. Understanding Stability of Outer Nesterov Momentum on Stochastic Pseudo-Gradients

    Oct 15, 2026Nishant Sharma, Laura Gomezjurado Gonzalez, Ganesh Talluri +2Neural Network StabilityNeural Network Training Dynamics

    Chandar Research Lab · Mila – Quebec AI Institute · Université de Montréal +4

  3. Geometry-Aware Multi-Objective Optimization: Norm-Constrained Oracles with Entropic Weight Updates

    Oct 9, 2026El Mahdi Chayti, Alireza Mirrokni, Martin JaggiLMO MethodsMulti-Objective Optimization

    Machine Learning and Optimization Laboratory (MLO), EPFL

  4. Inference and learning in sparse autoencoders as natural gradient flow

    Oct 5, 2026Hadi Vafaii, Tejas Rao, David Chanin +6Gradient DescentSparse Autoencoders

    Astera Institute · UC Berkeley · UCL +2

  5. Interleaved Projected Gradient Descent for Safe Imitation Learning

    Oct 5, 2026Shengfan Cao, Francesco BorrelliGradient DescentImitation Learning

    University of California, Berkeley

  6. Improved Convergence of Large Stepsize Gradient Descent for Logistic Regression

    Oct 5, 2026Xiaochuan Gong, Ang LiLogistic RegressionGradient Descent

    University of Maryland, College Park

  7. Last-Iterate Convergence Rate of Normalized Gradient Descent under Hölder Smoothness

    Oct 5, 2026Yuki Takezawa, Eduard GorbunovLearning Rate SchedulingLast-Iterate Convergence

    Toyota Motor Corporation · MBZUAI

  8. Learning Decision-Stump Thresholds in Context: Dynamics of Softmax Attention

    Oct 5, 2026Hong Ha Le, Jackie Lok, Atsushi Nitanda +1Language Model PretrainingSoftmax Attention

    National University of Singapore · Princeton University · Nanyang Technological University

  9. Smoothed Gradient Method for Nonconvex Federated Stochastic Bilevel Optimization

    Oct 4, 2026Xinwen Zhang, Peiran Yu, Zhaosong Lu +1Gradient DescentNonconvex Stochastic Optimization

    Temple University · University of Texas at Arlington · University of Minnesota

  10. Differential Privacy of Gradient Descent on Perturbed Objectives

    Oct 2, 2026Austin Watkins, Raman AroraDifferential PrivacyGradient Descent

    Johns Hopkins University Baltimore, MD 21218

  11. CAGE-NAS: Certified Functional Descent for Efficient Model Growth

    Oct 1, 2026Santiago Florido Gomez, Stéphane RivaudGradient DescentNeural Network Optimization

    Institut Polytechnique de Paris Paris, France · Inria Saclay, TAU Team Paris, France

  12. Convergence of Practical Muon with Finite Newton-Schulz Iterations and Nesterov Momentum

    Sep 30, 2026Hanyng Peng, Hui Wang, Yue YuNewton-Schulz IterationGradient Descent

    Pengcheng Laboratory, Shenzhen, China

  13. Stochastic Heavy Ball with Polyak Step Size and Armijo Line Search: A General Convergence Analysis

    Sep 29, 2026Jiawei Zhang, Qitan Shi, Yuantao GuStochastic OptimizationStochastic Optimization Convergence

    Tsinghua University

  14. Convex Optimization Is Free When Accuracy Is Expensive

    Sep 28, 2026Arthur Paing, Arthur JacotStochastic OptimizationGradient Descent

    Ecole polytechnique Palaiseau, France · Courant Institute, NYU New York, USA

  15. LionMuon: Alternating Spectral and Sign Descent for Efficient Training

    Sep 28, 2026Arman Bolatov, Artem Riabinin, Nikita Kornilov +4Stochastic OptimizationLanguage Model Pretraining

    Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · Applied Artificial Intelligence Institute +1

  16. GradLev: Token-Parallel Test-Time Training Via Costate Prediction

    Sep 28, 2026Bo Liu, Qiang LiuGradient DescentTest-Time Training

    The University of Texas at Austin

  17. Overfitting of Spectral Gradient Descent: How Matrix Geometry shapes Generalization and Implicit Bias

    Sep 26, 2026Guillaume Braun, Ichiro Hashimoto, Masaaki ImaizumiGradient DescentNeural Network Generalization

    University of Toronto · University of Tokyo, RIKEN AIP, Kyoto University

  18. Averaged Mirror Descent and Dual Gradient Methods: Convergent Algorithms for Entropic Gromov-Wasserstein Problems

    Sep 25, 2026Joanna Marks, Gabriel Rioux, Riccardo PasseggeriGradient DescentMirror Descent

    ⋆Denotes equal contribution to this work.

  19. Continuous Optimization for p-adic Models

    Sep 21, 2026Julian Salazar, Dimitri Kanevsky, Matt Harvey +2Gradient Descent

    Google DeepMind

  20. Gradient Descent with Stochastic Subspaces via Persistence of Memory

    Sep 16, 2026Subhroshekhar Ghosh, Clement Z. Q. Ng, Pierre-Louis Poirion +1Gradient DescentStochastic Optimization Convergence

    Department of Mathematics, National University of Singapore, Singapore · Center for Advanced Intelligence Project, RIKEN, Tokyo, Japan · Department of Mathematical Informatics, The University of Tokyo, Tokyo, Japan

  21. Geometry of learning dynamics: Gradient descent versus natural gradient on the ridge of optimization

    Sep 15, 2026Akira TamamoriInformation GeometryGradient Descent

    Faculty of Information Science, Aichi Institute of Technology 1247 Yachigusa, Yakusa-cho, Toyota-shi, Aichi 470-0392, Japan

  22. Exact-Form Regret for Gradient Descent, Mirror Descent and Follow-the-Regularized-Leader

    Sep 8, 2026Ashkan Soleymani, Gabriele Farina, Patrick JailletGame TheoryGradient Descent

    MIT

  23. Silver Rate Is (Almost) Optimal for Gradient Descent

    Sep 8, 2026Yuhan Ye, Kaizhao LiuLearning Rate SchedulingOptimization Convergence Analysis

    MIT

  24. Stochastic Nonconvex Bilevel Optimization: Improved Rates Without Rare-Visit Assumption

    Sep 6, 2026Daniel Cortild, Mathias Staudigl, Juan Peypouquet +1Stochastic Bilevel OptimizationBilevel Optimization

    Mathematical Institute, University of Oxford, United Kingdom · Department of Business Analytics and Decision Sciences, University of Economics and Business, Vienna, Austria · Bernouilli Institute, University of Groningen, The Netherlands

  25. Improved Gradient Descent Lower Bounds Beyond Nesterov

    Sep 2, 2026Yuhan Ye, Kaizhao LiuGradient DescentLearning Rate Scheduling

    MIT

  26. Momentum in large-batch training: Polyak enlarges the critical batch size, Nesterov improves data efficiency

    Sep 2, 2026Jia-Nan Wang, Zixun Huang, Kairui Li +2Momentum MethodsGradient Descent

    Peking University · University of Pennsylvania · AI for Science Institute, Beijing

  27. Kähler landscapes for complex neural network descents and guarantees including a search and destroy of the Calabi-Yau manifold

    Aug 20, 2026Andrew GracykRiemannian OptimizationGradient Descent

    Department of Mathematics Purdue University West Lafayette, IN 47907, USA

  28. Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness

    Aug 11, 2026Siqiao Mu, Diego KlabjanGradient DescentNeural Network Optimization

  29. Efficient Hypergradient Descent for Inverse Reinforcement Learning

    Aug 11, 2026Nikita Sevriukov, Anna Barabanova, Uliana Gagarina +4Reinforcement LearningGradient Descent

    HSE University Moscow, Russia

  30. A lower bound for stepsize-based acceleration of gradient descent

    Aug 11, 2026Jianhao Ma, Yuxin ChenLearning Rate SchedulingLast-Iterate Convergence

    Department of Industrial Engineering, Tsinghua University · Department of Statistics and Data Science, Wharton School, University of Pennsylvania