Overview

Gradient descent is a family of methods that repeatedly adjusts parameters in the direction that lowers an objective, making it a basic tool for training neural networks and fitting models such as logistic regression. Common approaches include full-batch and stochastic gradient descent, momentum and Nesterov acceleration, adaptive optimizers such as Adam, and geometry-aware variants such as natural, preconditioned, projected, or mirror descent. Theory and applications span deep-network training, matrix recovery, differential privacy, and decentralized or federated learning, with evaluations often measuring convergence, prediction quality, communication, memory, or privacy cost. A central tension is that larger or adaptive steps can speed training but also cause oscillations or instability, while the geometry, noise, and stopping time can change both which solution is reached and how well it generalizes.

Latest papers

CardsList

122 papers4 papers in the last four weeks, down 79% on the four weeks before. 0.0% of all new papers.

  1. Efficient Hypergradient Descent for Inverse Reinforcement Learning

    Aug 11, 2026Nikita Sevriukov, Anna Barabanova, Uliana Gagarina +4Reinforcement LearningGradient Descent

  2. A lower bound for stepsize-based acceleration of gradient descent

    Aug 11, 2026Jianhao Ma, Yuxin ChenLearning Rate SchedulingLast-Iterate Convergence

  3. Generalized Convexity and Smoothness via Conjugate Duality, with Applications to Deep Network Training

    Aug 10, 2026Binchuan QiGradient DescentNeural Network Optimization

  4. Minimax Optimal Early-Stopped Gradient Descent for Gaussian Mixture Classification

    Aug 6, 2026Alex Buna, Shirley Xiaoqi Liu, Patrick RebeschiniEarly StoppingGradient Descent

  5. Nesterov acceleration in optimizing over probability measures

    Jul 25, 2026Jiaqi Tang, Qin Li, Wilfrid GangboMomentum MethodsGradient Descent

  6. Beyond Negative-Ridge Endpoints: Mixed-Sign Spectral Regularization via Negative-Shifted Gradient Descent

    Jul 24, 2026Peng ZhaoSpectral RegularizationEarly Stopping

  7. Complexity Bounds and Approaches to Learning Projected Gradient Descent Solver Iterates

    Jul 24, 2026Anjian Li, Ryne BeesonGradient DescentLearning to Optimize

  8. Optimizing Regret

    Jul 21, 2026Irene AldridgeGradient DescentRegret Minimization

  9. Enhanced Neural Quantum State via Annealed Gradient Descent

    Jul 21, 2026Shiwei Zhou, Yiming Huang, Xiao Yuan +1Neural Quantum StatesGradient Descent

  10. Backpropagation-Free Trunk Training via the Split Forward Gradients

    Jul 18, 2026Tian Qin, Wei-Min HuangBackpropagationGradient Descent

  11. Same Loss, Same Noise, Opposite Schedules: Noise Structure and Optimizer Normalization Jointly Determine Whether Learning-Rate Cooldown Helps

    Jul 14, 2026Subham Singh, Ashutosh Mishra, Subha RautLearning Rate SchedulingGradient Descent

  12. Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima

    Jul 9, 2026Lachlan Ewen MacDonald, René VidalLarge-Step-Size Gradient DescentGradient Descent

  13. Hybrid Least Squares/Gradient Descent Methods for MIONets

    Jul 8, 2026Jun Choi, Chang-Ock Lee, Minam MoonGradient DescentNeural Network Optimization

  14. Differentially Private Natural Gradient Descent

    Jul 7, 2026Pan Li, Kai Chen, Shuai Chang +3Gradient DescentNatural Gradient Methods

  15. Directional Curvature from Armijo Backtracking: A Low-Cost Sharpness Probe and a Calibration-Free Learning-Rate Safeguard for Adam

    Jul 4, 2026Ashmitha R, Jörg FrochteGradient DescentAdaptive Gradient Methods

  16. A Gradient Flow Perspective on Minimum MMD Estimation

    Jul 4, 2026Sophia Seulkee Kang, Louis Sharrock, Xiaoyuan Cheng +2Parameter EstimationMaximum Mean Discrepancy

  17. Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks

    Jun 27, 2026Yuto Omae, Kazuki Sakai, Yohei Kakimoto +3Gradient DescentNeural Network Optimization

  18. Finite-Sample Performance of Gradient Descent in Logistic Regression with Gaussian Design

    Jun 19, 2026Junren Chen, Arya MazumdarLogistic RegressionGradient Descent

  19. Accelerated and Stable Convergence with Anchored Optimistic Method

    Jun 19, 2026Motahareh Sohrabi, Jianxin You, Simon Lacoste-Julien +2Last-Iterate ConvergenceGradient Descent

  20. On the Oracle Complexity of Interpolation-Based Gradient Descent

    Jun 18, 2026Dongmin Lee, William Lu, Anuran MakurOracle ComplexityGradient Descent

  21. Functional Gradient Descent with Adaptive Representations

    Jun 15, 2026Daniel Csillag, Rodrigo Schuller, Pedro Dall'Antonia +3Representation LearningGradient Descent

  22. Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning

    Jun 12, 2026Dmitriy Bystrov, Daniil Medyakov, Dmitry Bylinkin +1Gradient DescentZeroth-Order Optimization

  23. Uniform Stability and Generalization Error of GD and SGD on Fixed-Point Parameters

    Jun 5, 2026Jonghyun Shin, Sejun ParkAlgorithmic StabilityGradient Descent

  24. Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networks

    Jun 4, 2026Junyu Zhou, Puyu Wang, Yunwen Lei +2Neural Network GeneralizationReLU Neural Networks

  25. Flatland: The Adventures of Gradient Descent with Large Step Sizes

    Jun 4, 2026Leonardo Galli, Curtis Fox, Wiebke Bartolomaeus +2Gradient Descent DynamicsEdge of Stability