Gradient Descent

Ask about this topic

Momentum

21 papers in the last four weeks, up 133% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 151

All topics
CardsList

  1. Tight Transition Time Bounds for Separable Logistic Regression at the Edge of Stability

    Oct 1, 2026Haodong Wen, Kaiyue Wen, Jiaye TengLogistic RegressionOptimal Sample Complexity

    Stanford University · Shanghai University of Finance and Economics

  2. CAGE-NAS: Certified Functional Descent for Efficient Model Growth

    Oct 1, 2026Santiago Florido Gomez, Stéphane RivaudNeural Architecture SearchNeural Network

    Institut Polytechnique de Paris Paris, France · Inria Saclay, TAU Team Paris, France

  3. The Devil Is in the Reconstruction Loss Scale: Rethinking Optimization in LLM Quantization

    Oct 1, 2026Chao Li, Shigeng Wang, Anbang YaoPost-Training QuantizationLarge Language Model Quantization

    Intel Labs China

  4. Principal Component Regression Dominates all Monotone Spectral Filters for Linear Regression

    Sep 30, 2026Juno Kim, Hengyu Fu, Peter Bartlett +2Kernel Ridge RegressionSpectral Filtering

    University of California, Berkeley · Google DeepMind

  5. On the Two Faces of Adam in Separable Linear Classification

    Sep 27, 2026Chen Fan, Csaba SzepesváriAdamGradient Descent

    University of Alberta

  6. Benign Overfitting for General Norms and Distributions

    Sep 27, 2026Daniel Barzilai, Ohad ShamirOverfittingSpectral Norm

    Weizmann Institute of Science · University of Toronto & Vector Institute

  7. Local LMO is Secretly a Projection Method!

    Sep 27, 2026Peter Richtárik, Ammar MahranConvex OptimizationProjection

    King Abdullah University of Science and Technology, Thuwal, Saudi Arabia

  8. Precise Convergence Speed of Clipped SGD

    Sep 24, 2026David A. R. RobinGradient DescentConvergence

    LAMSADE, Université Paris-Dauphine, PSL Research University

  9. Complexities of Weak Proximal Oracle Methods for Composite Convex Optimization

    Sep 21, 2026Dan GarberFirst Order Oracle ComplexityStochastic Convex Optimization

    Faculty of Data and Decision Sciences Technion - Israel Institute of Technology

  10. Riemannian--Lorentz Fusion of Vision Transformers and State-Space Models

    Sep 16, 2026Badri N. Patro, Vijay S. AgneeswaranSelf-Supervised Vision TransformersVision Transformer

    Microsoft

  11. Gradient Descent with Stochastic Subspaces via Persistence of Memory

    Sep 16, 2026Subhroshekhar Ghosh, Clement Z. Q. Ng, Pierre-Louis Poirion +1Stochastic Gradient DescentGradient Descent

    Department of Mathematics, National University of Singapore, Singapore · Center for Advanced Intelligence Project, RIKEN, Tokyo, Japan · Department of Mathematical Informatics, The University of Tokyo, Tokyo, Japan

  12. Geometry of learning dynamics: Gradient descent versus natural gradient on the ridge of optimization

    Sep 15, 2026Akira TamamoriGradient DescentShape Optimization

    Faculty of Information Science, Aichi Institute of Technology 1247 Yachigusa, Yakusa-cho, Toyota-shi, Aichi 470-0392, Japan

  13. Federated stochastic bilevel optimization with fully first-order gradients

    Sep 14, 2026Yihan Zhang, Rohit Dhaipule, Chiu C Tan +2Bilevel OptimizationGradient Descent

    1Temple University · 2Stony Brook University

  14. Compute-Optimal Pretrain--Fine-tune in Ridge Gradient Descent

    Sep 14, 2026Alex Buna, Fanghui Liu, Patrick RebeschiniPretrainingModel Fine-Tuning

    Department of Statistics, University of Oxford, UK · School of Mathematical Sciences, Institute of Natural Sciences and MOE-LSC, Shanghai Jiao Tong University, China.

  15. Benign Loss Landscapes Can Coexist with Worst-Case Hardness

    Sep 14, 2026Zach Furman, Stephan Wäldchen, Yangda Bei +1Anisotropic Loss LandscapesTensor Networks

    University of Melbourne · Iliad

  16. Quantile-based Loss Filtering for Outlier-Robust Stochastic Gradient Descent

    Sep 14, 2026Jamie Haddock, Anna Ma, Elizaveta RebrovaConvex LossStochastic Gradient Descent

    Department of Mathematics, Harvey Mudd College, Claremont, CA 91711-3116, USA. · Department of Mathematics, University of California, Irvine, CA 92617, USA. · Mathematical Institute, University of Oxford, Oxford, OX1 2JD, UK.

  17. AdamX: Cosine similarity meets gradient descent

    Sep 10, 2026Francisco Caldas, Ruben Belo, Cláudia SoaresAdamGradient Descent

    NOVA School of Science of Technology Universidade Nova de Lisboa, Caparica, Portugal

  18. Generalization Analysis of Distributed Kernel-based Robust Gradient Descent Algorithms

    Sep 10, 2026Jun-Yi Meng, Zheng-Chu Guo, Yuan MaoDistributionally-Robust OptimizationStochastic Gradient Descent

    School of Mathematical Sciences, Zhejiang University, Hangzhou 310058, P. R. China · College of Informatics, Huazhong Agricultural University, Wuhan 430070, P. R. China

  19. Exact-Form Regret for Gradient Descent, Mirror Descent and Follow-the-Regularized-Leader

    Sep 8, 2026Ashkan Soleymani, Gabriele Farina, Patrick JailletMirror DescentLearning-Augmented Algorithms

    MIT

  20. Why shared attention vectors fail: a case for outcome-indexed tuning

    Sep 8, 2026Lenard DomeAttention LayersLinear Attention

    Department of Psychiatry and Psychotherapy, Faculty of Medicine University of Tübingen, Tübingen, Germany · German Center for Mental Health (DZPG), Tübingen, Germany

  21. A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay

    Sep 7, 2026Yuqing Wang, Ioannis G. Kevrekidis, Mikhail BelkinTwo-Layer Neural NetworksGeneralization Bounds

    Johns Hopkins University · University of California San Diego

  22. The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flow

    Sep 1, 2026Raphaël BerthierGradient DescentGradient

    Sorbonne Université, Inria, Centre Inria de Sorbonne Université, Paris, France

  23. Generalization as a robust performance property of learning-enabled dynamical systems

    Aug 31, 2026Filippo FabianiLearning DynamicsGeneralization Bounds

    IMT School for Advanced Studies Lucca, Piazza San Francesco 19, 55100, Lucca, Italy

  24. Convergence rates for the RMSprop optimizer with full control of the hyperparameters

    Aug 31, 2026Steffen Dereich, Arnulf JentzenAdamStochastic Convex Optimization

  25. Kähler landscapes for complex neural network descents and guarantees including a search and destroy of the Calabi-Yau manifold

    Aug 20, 2026Andrew GracykAnisotropic Loss LandscapesManifolds

    Department of Mathematics Purdue University West Lafayette, IN 47907, USA

  26. Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws

    Aug 13, 2026Liu Ziyin, Yizhou Xu, Tomaso Poggio +1Two-Layer Neural NetworksMultilayer Perceptrons

    1Massachusetts Institute of Technology · 2 ´Ecole Polytechnique F´ed´erale de Lausanne

  27. Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness

    Aug 11, 2026Siqiao Mu, Diego KlabjanGradient DescentLipschitz Continuity

  28. Efficient Hypergradient Descent for Inverse Reinforcement Learning

    Aug 11, 2026Nikita Sevriukov, Anna Barabanova, Uliana Gagarina +4Offline Reinforcement LearningReward Functions

    HSE University Moscow, Russia

  29. A lower bound for stepsize-based acceleration of gradient descent

    Aug 11, 2026Jianhao Ma, Yuxin ChenStep AccuracyGradient Descent

    Department of Industrial Engineering, Tsinghua University · Department of Statistics and Data Science, Wharton School, University of Pennsylvania

  30. Generalized Convexity and Smoothness via Conjugate Duality: Optimization Theory for Deep Neural Networks

    Aug 10, 2026Binchuan QiGradient DescentSmoothness

    College of Electronics and Information Engineering Tongji University Shanghai, 200092, China

  31. Adaptive Hybrid Particle Swarm Optimization with Gradient Descent

    Aug 7, 2026Aryan GurudeoParticle Swarm OptimizationGradient Descent

  32. Minimax Optimal Early-Stopped Gradient Descent for Gaussian Mixture Classification

    Aug 6, 2026Alex Buna, Shirley Xiaoqi Liu, Patrick RebeschiniEmpirical Risk MinimizationGaussian Mixture Models

    Department of Statistics, University of Oxford

  33. Non-asymptotic implicit bias of logistic regression at early-stage gradient descent dynamics

    Aug 5, 2026Han BaoGradient DescentOverparameterization

    The Institute of Statistical Mathematics · Tohoku University · RIKEN AIP

  34. Adaptivity via a Parallel Architecture for Stochastic Gradient Methods

    Jul 31, 2026Bin FuStochastic Gradient DescentGradient Descent

    University of Texas Rio Grande Valley

  35. Simultaneous Coverage and Efficiency Guarantee in Online Conformal Prediction

    Jul 29, 2026Rahul VazeConformal InferenceOnline Conformal Prediction

    Tata Institute of Fundamental Research, Mumbai

  36. Adversarial Deepfake Generation and an Investigation of Purification-Based Adversarial Detection

    Jul 28, 2026Junghyun Kim, Seunghyun Kim, Jiyoung WooDeepfake DetectionUnsupervised Detection

    Department of AI and Big Data Engineering, Soonchunhyang University, Asan, South Korea

  37. Data-Dependent Regret and Polyak Corrections for Constrained Online Convex Optimization

    Jul 28, 2026Wentao ZhangConvex Optimization\Widetilde{\Mathcal{O}}(\Sqrt{T})$ Regret

    Tsinghua Shenzhen International Graduate School · Tsinghua University

  38. Extracting Voice Styles from Frozen TTS Models via Gradient-Based Inverse Optimization

    Jul 28, 2026Gyeongmin KimText-To-Speech SynthesisWav2Vec

    Department of Computer Science, Hanyang University, Seoul, Korea

  39. Global Convergence of DGM and PINN Algorithms for Solving Nonlinear PDEs

    Jul 27, 2026Justin Sirignano, Konstantinos Spiliopoulos, Samuel CohenPartial Differential EquationsVariational Formulation

    Mathematical Institute, University of Oxford · Department of Mathematics and Statistics, Boston University

  40. Greedy dynamical meta-learning

    Jul 27, 2026Aria YomMeta-LearningGradient-Based Optimization

    Rhodora

  41. Sparse Gaussian-Mixture-Model Q-Functions via Hadamard Overparametrization for Online Reinforcement Learning

    Jul 26, 2026Minh Vu, Konstantinos SlavakisOffline Reinforcement LearningValue Functions

    Institute of Science Tokyo, Department of Information and Communications Engineering, Yokohama, Japan.

  42. Learning from the Descent Direction: Adaptive Gradient Descent under One-Sided Hölder Regularity

    Jul 24, 2026Arzu Ahmadova, Ismail HuseynovGradient DescentDescent

    Weierstrass Institute for Applied Analysis and Stochastics Berlin, Germany · Physikalisch-Technische Bundesanstalt (PTB), Germany · Technical University of Berlin, Germany

  43. Beyond Negative-Ridge Endpoints: Mixed-Sign Spectral Regularization via Negative-Shifted Gradient Descent

    Jul 24, 2026Peng ZhaoRegularizationQuadratic Slack Penalty

    Department of Applied Economics and Statistics · University of Delaware

  44. Complexity Bounds and Approaches to Learning Projected Gradient Descent Solver Iterates

    Jul 24, 2026Anjian Li, Ryne BeesonGradient DescentSolver Iterations

    Department of Electrical and Computer Engineering Princeton University, Princeton, NJ 08544, USA · Department of Mechanical and Aerospace Engineering Princeton University, Princeton, NJ 08544, USA

  45. Energy Manifold Natural Gradient Descent: Riemannian Optimization for Neural PDE Solvers

    Jul 24, 2026Zhangyong Liang, Huanhuan GaoNeural Partial Differential Equation SolversRiemannian Optimization

    National Center for Applied Mathematics Tianjin University Tianjin, 300072, China · School of Mechanical and Aerospace Engineering Jilin University Changchun, 130025, China

  46. Optimizing Regret

    Jul 21, 2026Irene AldridgeRegretPolicy Gradient

    Risk AI Center

  47. Enhanced Neural Quantum State via Annealed Gradient Descent

    Jul 21, 2026Shiwei Zhou, Yiming Huang, Xiao Yuan +1AnnealingPost-Quantum Cryptography

    Institute of High Energy Physics, Chinese Academy of Sciences, Beijing 100049, China · China Center of Advanced Science and Technology, Beijing 100190, China · Center on Frontiers of Computing Studies, School of Computer Science, Peking University, Beijing 100871, China

  48. Exact Network Surgery: Functional Invariance and Gradient Plasticity in Reactive Computational Graphs

    Jul 18, 2026Abdallah KhemaisDiscontinuous GatingGradient Descent

    ISITCOM, University of Sousse

  49. Domain Adaptation of Mismatched Proximal Denoiser for Plug-and-Play Image Reconstruction

    Jul 16, 2026Guixian Xu, Jinglai Li, Junqi TangGaussian DeblurringImage Reconstruction

    School of Mathematics, University of Birmingham

  50. Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initialization

    Jul 14, 2026Jiajie Zhao, Jianxing Wang, Junjie Yang +2Gradient Descent

    School of Mathematical Sciences, Shanghai Jiao Tong University · Institute of Natural Sciences, MOE-LSC, Shanghai Jiao Tong University · School of Artificial Intelligence, Shanghai Jiao Tong University

  51. LayerNorm as Implicit Gain Control in Looped Transformers

    Jul 12, 2026Matthias M. M. BuehlmaierBatch NormalizationRecurrent Model

    Faculty of Business and Economics, The University of Hong Kong · Pokfulam Road, Hong Kong

  52. Sharper Analysis of Single-Loop Methods for Bilevel Optimization

    Jul 11, 2026Yubo Zhou, Jun Shu, Luo Luo +4Bilevel OptimizationConvergence

    School of Mathematics and Statistics, Xi’an Jiaotong University · School of Data Science, Fudan University · 3SGIT AI Lab, State Grid Corporation of China +1

  53. Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima

    Jul 9, 2026Lachlan Ewen MacDonald, René VidalGradient DescentHessian

    Innovation in Data Engineering and Science (IDEAS) · University of Pennsylvania Pennsylvania, PA 19104

  54. Hybrid Least Squares/Gradient Descent Methods for MIONets

    Jul 8, 2026Jun Choi, Chang-Ock Lee, Minam MoonGradient DescentLeast Squares

    Department of Mathematical Sciences, KAIST, Daejeon 34141, KOREA · Department of Mathematics, Korea Military Academy, Seoul 01805, KOREA

  55. Converge to Surprise: Evolutionary Self-supervised Image Clustering

    Jul 8, 2026Canlin Zhang, Xiuwen LiuClusteringDeep Learning

    Independent Researcher · Department of Computer Science Florida State University

  56. Differentially Private Natural Gradient Descent

    Jul 7, 2026Pan Li, Kai Chen, Shuai Chang +3Standard Differential-PrivacyStochastic Gradient Descent

    Institute of Information Engineering, Chinese Academy of Sciences Beijing, China · Department of Computer Science, Metropolitan College, Boston University Boston, United States · Nanyang Technological University Singapore, Singapore

  57. The Map Behind the Flow: Finite-Step Gradient Descent as a Dynamical System

    Jul 6, 2026Thomas HofmannGradient DescentWasserstein Gradient Flows

    Department of Computer Science, ETH Zurich

  58. Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks

    Jul 5, 2026Shokhrukh Ibragimov, Arnulf JentzenNeural Network OptimizationGradient Descent

    Applied Mathematics: Institute for Analysis and Numerics, University of Münster, Germany · School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China

  59. Why can genetic algorithms work in high-dimensional search spaces?

    Jun 29, 2026Stephen WhitelamGenetic AlgorithmsGradient Descent

    Molecular Foundry, Lawrence Berkeley National Laboratory, 1 Cyclotron Road, Berkeley, CA 94720, USA

  60. ITSPACE: Monotone Gaussian Optimal Transport Updates

    Jun 29, 2026Woojoo Na, Jennifer DyCovarianceLatent Representation Alignment

    Department of Computer Engineering, Northeastern University, Boston, USA.