Overview

Gradient descent is a family of methods that repeatedly adjusts parameters in the direction that lowers an objective, making it a basic tool for training neural networks and fitting models such as logistic regression. Common approaches include full-batch and stochastic gradient descent, momentum and Nesterov acceleration, adaptive optimizers such as Adam, and geometry-aware variants such as natural, preconditioned, projected, or mirror descent. Theory and applications span deep-network training, matrix recovery, differential privacy, and decentralized or federated learning, with evaluations often measuring convergence, prediction quality, communication, memory, or privacy cost. A central tension is that larger or adaptive steps can speed training but also cause oscillations or instability, while the geometry, noise, and stopping time can change both which solution is reached and how well it generalizes.

Latest papers

CardsList

122 papers4 papers in the last four weeks, down 79% on the four weeks before. 0.0% of all new papers.

  1. Local linear convergence of gradient methods for overparameterized Gaussian mixtures

    May 29, 2026Jingxing Wang, Vasileios Charisopoulos, Maryam FazelGradient DescentGaussian Mixture Models

  2. Convergence of Steepest Descent and Adam under Non-Uniform Smoothness

    May 28, 2026Sharan Vaswani, Yifan Sun, Reza BabanezhadOptimization Convergence AnalysisGradient Descent

  3. A Unified Framework for Gradient Aggregation in Multi-Objective Optimization

    May 28, 2026Zeou Hu, Kelvin Ho, Yaoliang YuGradient DescentStochastic Optimization Convergence

  4. Online Learning on Hidden-Convex Losses via Algorithmic Equivalence: Optimal Regret, Geometric Barrier, and Bandit Feedback

    May 25, 2026Anas Barakat, Andreas Kontogiannis, Vasilis Pollatos +2Online Convex OptimizationGradient Descent

  5. EMA-Nesterov: Stabilizing Nesterov's Lookahead for Accelerated Deep Learning Optimization

    May 25, 2026Chung-Yiu Yau, Dawei Li, Athanasios Glentis +3Deep Learning OptimizationGradient Descent

  6. Accelerated Gradient Descent for Faster Convergence with Minimal Overhead

    May 15, 2026Manuel Graca, L. Miguel Silveira, Arlindo Oliveira +1Deep Learning OptimizationGradient Descent

  7. Genetic algorithm vs. gradient descent for training a neural network architecture dedicated to low data regimes in small medical datasets

    May 13, 2026Amine Boukhari, Boglarka Ecsedi, Laszlo Papp +1Gradient DescentNeural Network Optimization

  8. Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence

    May 13, 2026Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong +3Momentum MethodsGradient Descent

  9. On the global convergence of gradient descent for wide shallow models with bounded nonlinearities

    May 11, 2026Romain Petit, Clarice Poon, Gabriel PeyréShallow Neural NetworksGradient Descent

  10. PowerStep: Memory-Efficient Adaptive Optimization via ℓp\ell_p-Norm Steepest Descent

    May 11, 2026Yao Lu, Dengdong Fan, Shixun Zhang +1Gradient DescentMemory-Efficient Optimization

  11. Muon-OGD: Muon-based Spectral Orthogonal Gradient Projection for LLM Continual Learning

    May 9, 2026Binghang Lu, Zheyuan Deng, Runyu Zhang +6Continual Learning for LLMsGradient Descent

  12. Broximal Gradient Descent: A Projection-Free Sister of Projected Gradient Descent

    May 9, 2026Peter Richtárik, Kaja Gruntkowska, Hanmin LiGradient DescentConstrained Optimization

  13. Control Your View: High-Resolution Global Semantic Manipulation in Learned Image Compression

    May 9, 2026Jiaming Liang, Chi-Man Pun, Weisi Lin +1Learned Image CompressionAdversarial Attacks

  14. A Unified Lyapunov-IQC Framework for Uniform Stability of Smooth Quadratic First-Order Accelerated Optimizers

    May 8, 2026Don Li, Dacian DaescuLyapunov StabilityAlgorithmic Stability

  15. Locally Near Optimal Piecewise Linear Regression in High Dimensions via Difference of Max-Affine Functions

    May 7, 2026Haitham Kanj, Kiryung LeeGradient DescentSample Complexity

  16. Muon with Nesterov Momentum: Heavy-Tailed Noise and (Randomized) Inexact Polar Decomposition

    May 7, 2026Sayantan Choudhury, Xiaoran Cheng, Martin Takáč +2Stochastic OptimizationMomentum Methods

  17. Transformers Efficiently Perform In-Context Logistic Regression via Normalized Gradient Descent

    May 7, 2026Chenyang Zhang, Yuan CaoLogistic RegressionSelf-Attention

  18. Pro-KLShampoo: Projected KL-Shampoo with Whitening Recovered by Orthogonalization

    May 7, 2026Ruotong Sun, Ermin WeiLanguage Model PretrainingDeep Learning Optimization

  19. From Dual Tracking to Clipping: Provably Faster Distributionally Robust Multi-Objective Optimization

    May 7, 2026Yufeng Yang, Fangning Zhuo, Ziyi Chen +2Gradient ClippingGradient Descent

  20. A Provably Convergent and Practical Algorithm for Gromov--Wasserstein Optimal Transport

    May 5, 2026Ling Liang, Lei YangGradient DescentNonconvex Optimization

  21. A Parameter-Free First-Order Algorithm for Non-Convex Optimization with  O~(ε−5/3)\tilde{\mkern1mu O}(ε^{-5/3}) Global Rate

    May 4, 2026Sichao Xiong, Sadok Jerad, Coralia CartisOptimization Convergence AnalysisGradient Descent

  22. Randomized Subspace Nesterov Accelerated Gradient

    May 1, 2026Gaku Omiya, Pierre-Louis Poirion, Akiko TakedaGradient DescentSmooth Convex Optimization

  23. MGDA-Decoupled: Geometry-Aware Multi-Objective Optimisation for DPO-based LLM Alignment

    Apr 22, 2026Andor Vári-Kakas, Ji Won Park, Natasa TagasovskaDirect Preference OptimizationGradient Descent

  24. The Origin of Edge of Stability

    Apr 22, 2026Elon LitmanEdge of StabilityGradient Descent

  25. Natural gradient descent with momentum

    Apr 16, 2026Anthony Nouy, Agustín SomacalDeep Learning OptimizationRiemannian Optimization

  26. Gradient Descent's Last Iterate is Often (slightly) Suboptimal

    Apr 15, 2026Guy Kornowski, Ohad ShamirLast-Iterate ConvergenceGradient Descent

  27. A short proof of near-linear convergence of adaptive gradient descent under fourth-order growth and convexity

    Apr 15, 2026Damek Davis, Dmitriy DrusvyatskiyGradient DescentAdaptive Gradient Methods