Adaptive Gradient Methods

Ask about this topic

Momentum

13 papers in the last four weeks, up 160% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 113

All topics
CardsList

  1. A Trust-region Framework for Moment Estimation

    Jul 26, 2026Oluwasegun A. SomefunLearning Rate SchedulingTrust-Region Optimization

  2. Hidden Boundary Motion in Transformer Optimization: Function-Space Orthogonalization of Affine Weight and Bias Updates

    Jul 24, 2026Zhang Gongyue, Sheng Yixuan, Liu donghan +3TransformerNeural Network Optimization

    arXiv:2607.22927v1

  3. Learning from the Descent Direction: Adaptive Gradient Descent under One-Sided Hölder Regularity

    Jul 24, 2026Arzu Ahmadova, Ismail HuseynovLearning Rate SchedulingNonconvex Optimization

    Weierstrass Institute for Applied Analysis and Stochastics Berlin, Germany · Physikalisch-Technische Bundesanstalt (PTB), Germany · Technical University of Berlin, Germany

  4. Optimizing the Preconditioner: A Black-box Online-to-Nonconvex Conversion with Static Regret Minimization Oracles

    Jul 20, 2026Haichen Hu, David Simchi-LeviNonconvex Stochastic OptimizationAdaptive Gradient Methods

    Center for Computational Science and Engineering, MIT · Department of Civil and Environmental Engineering, MIT · Institute for Data, Systems, and Society, MIT

  5. FlexiGrad: Adaptive Gradient Modulation for Hierarchical Fine-Grained Classification

    Jul 20, 2026Zilu Zhou, Dongliang Chang, Junhan Chen +1Hierarchical ClassificationGradient Interference

    Beijing University of Posts and Telecommunications, Beijing 100876, China · Beijing Key Laboratory of Multimodal Data Intelligent Perception and Governance, Beijing 100876, China

  6. Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers

    Jul 16, 2026Akhilesh GogikarNeural Network OptimizationAdaptive Gradient Methods

    1Independent Researcher.

  7. Reassessing Muon for Matrix Factorization

    Jul 14, 2026Ali Parviz, Gal Mishne, Alex CloningerDeep Learning OptimizationMuon Optimizer

    †. Halicioğlu Data Science Institute, UC San Diego. · ∗. Department of Mathematics, UC San Diego.

  8. M+Adam: Low-Precision Training via Additive-Multiplicative Optimization

    Jul 12, 2026Xiaoyuan Liang, Sebastian Loeschcke, Mads Toftrup +1Quantization-Aware TrainingAdaptive Gradient Methods

    California Institute of Technology · University of Copenhagen · Aarhus University

  9. Optimization Geometrodynamics: Variational Reduction and Interaction Curvature

    Jul 7, 2026Zavier LiRiemannian OptimizationSecond-Order Optimization

    Xidian University Xi’an, China

  10. Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks

    Jul 5, 2026Shokhrukh Ibragimov, Arnulf JentzenDeep Learning OptimizationNeural Network Optimization

    Applied Mathematics: Institute for Analysis and Numerics, University of Münster, Germany · School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China

  11. Directional Curvature from Armijo Backtracking: A Low-Cost Sharpness Probe and a Calibration-Free Learning-Rate Safeguard for Adam

    Jul 4, 2026Ashmitha R, Jörg FrochteGradient DescentAdaptive Gradient Methods

    Department of Artificial Intelligence and Data Science, Sri Ramakrishna Engineering College, Anna University, Coimbatore 641022, Tamil Nadu, India · Interdisciplinary Institute for Applied AI and Data Science Ruhr (AKIS), Department of Electrical Engineering and Computer Science, Bochum University of Applied Sciences, Am Hochschulcampus 1, 44801 Bochum, Germany

  12. On the Convergence of Adam, Revisited

    Jul 3, 2026Steven Heilman, Sampad MohantyOnline Convex OptimizationStochastic Optimization Convergence

    Department of Mathematics, University of Southern California, Los Angeles, CA 90089.

  13. Characterizing Optimizer-Dependent Training Dynamics Through Hessian Eigenvector Displacement and Localization

    Jun 29, 2026Marcelina Marjankowska, Valerio Modugno, Paolo BaruccaSpectral MethodsAdaptive Gradient Methods

    Financial Computing and Analytics Group, University College London · Robot Perception and Learning Lab, University College London

  14. Optimizer Memory Makes Shuffle Order a First-Order Source of Fine-Tuning Noise

    Jun 28, 2026John SweeneyStochastic OptimizationFine-Tuning

  15. Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks

    Jun 28, 2026Tejas Pradeep ShirodkarDeep Learning OptimizationNeural Network Optimization

    IIIT, Hyderabad

  16. Analysis of Adam Algorithms for Stochastic Dynamic Systems

    Jun 27, 2026Xin Zheng, Yifei Jin, Lei GuoStochastic OptimizationStochastic Optimization Convergence

    State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing 101408, China

  17. Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?

    Jun 22, 2026Dingzhi Yu, Hongyi Tao, Yuanyu Wan +2Stochastic OptimizationHeavy-Tailed Noise

    1Nanjing University · 2Zhejiang University · 3Fudan University

  18. Adam Converges in Nonsmooth Nonconvex Optimization

    Jun 21, 2026Zijian LiuHeavy-Tailed NoiseNonconvex Optimization

    Department of Technology, Operations, and Statistics Leonard N. Stern School of Business New York University New York, NY 10012, USA

  19. Central limit theorem for the averaged Adam optimizer

    Jun 19, 2026Steffen Dereich, Arnulf JentzenStochastic ApproximationStochastic Optimization Convergence

    Institute for Mathematical Stochastics, University of Münster, Germany · School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China · Applied Mathematics: Institute for Analysis and Numerics, University of Münster, Germany

  20. One-Step Generalization Ratio Guided Optimization for Domain Generalization

    Jun 15, 2026Sumin Cho, Dongwon Kim, Kwangsu KimDomain GeneralizationAdaptive Gradient Methods

    Department of Computer Science and Engineering, University of Sungkyunkwan, Suwon, Korea.

  21. Schattor: Schatten-family methods for deep learning optimization

    Jun 14, 2026Bohao Ma, Junyu Zhang, Chuan HeDeep Learning OptimizationStochastic Optimization Convergence

    School of Data Science, The Chinese University of Hong Kong (Shenzhen), Shenzhen, Guangdong, China · Department of Industrial Systems Engineering and Management, National University of Singapore, Singapore · Department of Mathematics, Linköping University, Sweden

  22. Beyond a Single Explanation of the Adam--SGD Gap

    Jun 12, 2026Chenxiang Zhang, Rustem Islamov, Enea Monzio Compagnoni +3Deep Learning OptimizationAdaptive Gradient Methods

    University of Luxembourg · MPI for Intelligent Systems · University of Basel +2

  23. Gefen: Optimized Stochastic Optimizer

    Jun 11, 2026Nadav Benedek, Tomer Koren, Ohad FriedDeep Learning OptimizationMemory-Efficient Optimization

    Reichman University · Tel Aviv University, Google Research

  24. Weibull Weight-Scale Parameter Evolution under AdamW Training Dynamics

    Jun 11, 2026Tiexin DingWeight-Space AnalysisAdaptive Gradient Methods

    Independent Researcher.

  25. Preserving Plasticity in Continual Learning via Dynamical Isometry

    Jun 8, 2026Andries Rosseau, Robert Müller, Ann NowéLoss of PlasticityContinual Learning

    AI Lab, Vrije Universiteit Brussel · Aganthos

  26. Adaptive directional gradients for parameterised quantum circuits

    Jun 8, 2026Brian Coyle, Snehal Raj, Virag Umathe +2Variational Quantum CircuitsAdaptive Gradient Methods

    School of Informatics, University of Edinburgh, Edinburgh, UK · LIP6, CNRS, Sorbonne Université, Paris, France · QC Ware, Palo Alto, USA and Paris, France +1

  27. OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality

    Jun 7, 2026Ganzhao YuanMomentum MethodsNonconvex Stochastic Optimization

    Faculty of Computer Science and Artificial Intelligence Shenzhen University of Advanced Technology (SUAT), China

  28. Large-scale empirical tuning and comparison of default optimizers for variational inference

    Jun 5, 2026Trevor Campbell, Jonathan H. Huggins, Kyurae Kim +1Stochastic OptimizationBayesian Inference

    Department of Statistics, UBC · Department of Mathematics & Statistics, Boston University · Faculty of Computing & Statistics, Boston University +1

  29. Flatland: The Adventures of Gradient Descent with Large Step Sizes

    Jun 4, 2026Leonardo Galli, Curtis Fox, Wiebke Bartolomaeus +2Edge of StabilityGradient Descent

    LMU Munich · Munich Center for Machine Learning · University of British Columbia +1

  30. DP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum

    Jun 3, 2026Naima Tasnim, Lalitha Sankar, Oliver KosutDifferentially Private Stochastic Gradient DescentGradient Clipping

    Arizona State University, Tempe, AZ, United States