Adaptive Gradient Methods

Ask about this topic

Momentum

13 papers in the last four weeks, up 160% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 113

All topics
CardsList

  1. Early Memory Selection for Balanced Adam

    Oct 6, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2Adaptive Gradient MethodsHyperparameter Optimization

    Universitat Politècnica de València Valencia, Spain · Universitat Jaume I Castelló de la Plana, Spain

  2. DGA-Muon: Decoupled Geometry-Aligned Adaptive Scaling for Muon

    Oct 5, 2026Wenpeng Zhang, Runsheng YuMuon OptimizerAdaptive Gradient Methods

    Independent Reasearcher

  3. Exact information accounting for SGD methods

    Sep 30, 2026Akshay BalsubramaniNeural Network GeneralizationStochastic Optimization Convergence

  4. Adam under Generalized Smoothness with Second-Moment-Type Stochastic Gradients

    Sep 29, 2026Ruinan Jin, Difei Cheng, Ling Chen +3Convergence AnalysisStochastic Optimization Convergence

    Mohamed bin Zayed University of Artificial Intelligence · Aerospace Information Technology University · The Ohio State University +2

  5. Second-Moment Stochastic Approximation Methods

    Sep 29, 2026Tao Jiang, Lin XiaoStochastic ApproximationMuon Optimizer

    Fundamental AI Research (FAIR), Meta Superintelligence Labs

  6. The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics

    Sep 28, 2026Yihe Zhou, Tongtian Zhu, Yingxiao Huo +4Stochastic OptimizationSign-Based Optimization

    Department of Computer Science, The University of Manchester, Manchester, UK · College of Computer Science and Technology, Zhejiang University, Hangzhou, China

  7. On the Two Faces of Adam in Separable Linear Classification

    Sep 27, 2026Chen Fan, Csaba SzepesváriImplicit BiasClassification

    University of Alberta

  8. VCMM: Variance-Calibrated Momentum for Multimodal Learning

    Sep 23, 2026Zhongjing Gu, Chenyang Huang, Yufa Feng +3Multimodal OptimizationAdaptive Gradient Methods

  9. AURA: Angular Update Rate Adaptation for training complex-valued neural networks

    Sep 22, 2026Enrico Ballini, Allan Peter Engsig-Karup, Tito AndriolloNeural Network OptimizationComplex-Valued Neural Networks

    Aarhus University, Aarhus, Denmark · Technical University of Denmark, Kongens Lyngby, Denmark

  10. Adaptive Forgetting for Nonstationary Optimization: Towards Robust EEG Decoding

    Sep 21, 2026Hongyu Zhu, Lin Chen, Jing Chen +2Cross-Subject EEG DecodingAdaptive Gradient Methods

    Chongqing Institute of Green Intelligent Technology, Chinese Academy of Sciences · Beijing University of Posts and Telecommunications · School of Resources and Environment, University of Electronic Science and Technology of China

  11. Beyond Quadratic Loss: The Stability Phase Diagram of Adam

    Sep 16, 2026Gaoxiang Tang, Huanran Chen, Ziming LiuAdaptive Gradient MethodsNeural Network Training Dynamics

    IIIS, Tsinghua University · College AI, Tsinghua University · Shanghai Qizhi Institute +1

  12. A Full Adam Theorem for Spectral Heavy-Tail Onset

    Sep 14, 2026Zongmin LiuAdaptive Gradient MethodsNeural Network Training Dynamics

    Stanford University

  13. AdamX: Cosine similarity meets gradient descent

    Sep 10, 2026Francisco Caldas, Ruben Belo, Cláudia SoaresDeep Learning OptimizationAdaptive Gradient Methods

    NOVA School of Science of Technology Universidade Nova de Lisboa, Caparica, Portugal

  14. When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay

    Sep 8, 2026Hasan Amin, Wei-Kai Chang, Rajiv KhannaLearning Rate SchedulingWeight Decay

    Purdue University

  15. Equivariance Breaks the Learning Rate

    Sep 8, 2026Andrei Manolache, Mathias NiepertEquivariant Neural NetworksEquivariant Representation Learning

    University of Stuttgart, Germany · Bitdefender, Romania

  16. Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetry

    Sep 2, 2026Sai Niranjan Ramachandran, Suvrit SraStochastic Gradient DescentAdaptive Gradient Methods

    School of Computation, Information and Technology, Technical University of Munich, Germany · Munich Center for Machine Learning (MCML)

  17. Convergence rates for the RMSprop optimizer with full control of the hyperparameters

    Aug 31, 2026Steffen Dereich, Arnulf JentzenStochastic ApproximationStochastic Optimization Convergence

  18. Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversal

    Aug 21, 2026Yiman Fong, Heng YangEdge of StabilityAdaptive Gradient Methods

    School of Engineering and Applied Sciences, Harvard University.

  19. Momentum as Residual-Driven Multiplier Correction for Deep Learning Optimization

    Aug 13, 2026Zhixin Ren, Yau Lyu, Congrong Li +2Deep Learning OptimizationMomentum Methods

    Department of Mathmatical Science, Tsinghua University · School of Vehicle and Mobility, Tsinghua University · College of AI, Tsinghua University

  20. Adaptive Bregman Proximal Stochastic Gradient with a Stabilized Barzilai--Borwein Step Size

    Aug 12, 2026Chenhan Jin, Shengze Xu, Binghui Xie +4Nonconvex Stochastic OptimizationAdaptive Gradient Methods

    1The Chinese University of Hong Kong · University of Utah · 3Beijing Normal-Hong Kong Baptist University +1

  21. Gradient Under Microscope: Benchmarking Resource Utilization of Memory-Efficient Gradient Computation Methods

    Aug 9, 2026Sarthak Mahapatra, Zihan Zhou, Khatoon Khedri +2Efficient Language Model TrainingAdaptive Gradient Methods

  22. The Loss Does Not See the Basis, but Adam Does

    Aug 5, 2026Devender SinghLow-Rank Matrix DecompositionMatrix Optimization

    Department of Computer Science Memorial University of Newfoundland

  23. AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization

    Aug 3, 2026Alok Kumar Pandey, Umang Chaturvedi, Aatish Rana +1Deep Learning OptimizationStochastic Optimization Convergence

  24. Adaptivity via a Parallel Architecture for Stochastic Gradient Methods

    Jul 31, 2026Bin FuStochastic OptimizationStochastic Optimization Convergence

    University of Texas Rio Grande Valley

  25. The Convergence Behavior of Adam under Heavy-Tailed Noise

    Jul 29, 2026Yijiang PangNonconvex Stochastic OptimizationHeavy-Tailed Noise

    Michigan State University, East Lansing, USA

  26. Parameter-Free Dynamic Regret under Heavy-Tailed Noise

    Jul 29, 2026Vaneet AggarwalOnline Convex OptimizationDynamic Regret Minimization

    Purdue University

  27. Adaptive Gradient-Based Methods for a Broader Class of Optimization Problems under Performative Prediction

    Jul 29, 2026Hiroki Hamaguchi, Yuya Hikima, Hiroshi Sawada +1Stochastic Optimization ConvergenceAdaptive Gradient Methods

    Graduate School of Information Science and Technology, The University of Tokyo · Department of Industrial Engineering and Economics, Institute of Science Tokyo · Communication Science Laboratories, NTT, Inc. +1

  28. Data-Dependent Regret and Polyak Corrections for Constrained Online Convex Optimization

    Jul 28, 2026Wentao ZhangOnline Convex OptimizationConstrained Optimization

    Tsinghua Shenzhen International Graduate School · Tsinghua University

  29. PYPM-GGD: Pitman-Yor Process Mixture with Generalized Gaussian Density using ADAM

    Jul 27, 2026Kart-Leong LimBayesian Mixture ModelsClustering

    Institute of Microelectronics, A*Star Singapore

  30. Low-Rank Dependence Decomposition via Accelerated Symmetric Non-negative Matrix Factorization

    Jul 27, 2026Lavinia Ghita, Dhruv Desai, Jake Goldberg +1Non-Negative Matrix FactorizationLow-Rank Matrix Decomposition

    NVIDIA

  31. A Trust-region Framework for Moment Estimation

    Jul 26, 2026Oluwasegun A. SomefunLearning Rate SchedulingTrust-Region Optimization

  32. Hidden Boundary Motion in Transformer Optimization: Function-Space Orthogonalization of Affine Weight and Bias Updates

    Jul 24, 2026Zhang Gongyue, Sheng Yixuan, Liu donghan +3TransformerNeural Network Optimization

    arXiv:2607.22927v1

  33. Learning from the Descent Direction: Adaptive Gradient Descent under One-Sided Hölder Regularity

    Jul 24, 2026Arzu Ahmadova, Ismail HuseynovLearning Rate SchedulingNonconvex Optimization

    Weierstrass Institute for Applied Analysis and Stochastics Berlin, Germany · Physikalisch-Technische Bundesanstalt (PTB), Germany · Technical University of Berlin, Germany

  34. Optimizing the Preconditioner: A Black-box Online-to-Nonconvex Conversion with Static Regret Minimization Oracles

    Jul 20, 2026Haichen Hu, David Simchi-LeviNonconvex Stochastic OptimizationAdaptive Gradient Methods

    Center for Computational Science and Engineering, MIT · Department of Civil and Environmental Engineering, MIT · Institute for Data, Systems, and Society, MIT

  35. FlexiGrad: Adaptive Gradient Modulation for Hierarchical Fine-Grained Classification

    Jul 20, 2026Zilu Zhou, Dongliang Chang, Junhan Chen +1Hierarchical ClassificationGradient Interference

    Beijing University of Posts and Telecommunications, Beijing 100876, China · Beijing Key Laboratory of Multimodal Data Intelligent Perception and Governance, Beijing 100876, China

  36. Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers

    Jul 16, 2026Akhilesh GogikarNeural Network OptimizationAdaptive Gradient Methods

    1Independent Researcher.

  37. Reassessing Muon for Matrix Factorization

    Jul 14, 2026Ali Parviz, Gal Mishne, Alex CloningerDeep Learning OptimizationMuon Optimizer

    †. Halicioğlu Data Science Institute, UC San Diego. · ∗. Department of Mathematics, UC San Diego.

  38. M+Adam: Low-Precision Training via Additive-Multiplicative Optimization

    Jul 12, 2026Xiaoyuan Liang, Sebastian Loeschcke, Mads Toftrup +1Quantization-Aware TrainingAdaptive Gradient Methods

    California Institute of Technology · University of Copenhagen · Aarhus University

  39. Optimization Geometrodynamics: Variational Reduction and Interaction Curvature

    Jul 7, 2026Zavier LiRiemannian OptimizationSecond-Order Optimization

    Xidian University Xi’an, China

  40. Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks

    Jul 5, 2026Shokhrukh Ibragimov, Arnulf JentzenDeep Learning OptimizationNeural Network Optimization

    Applied Mathematics: Institute for Analysis and Numerics, University of Münster, Germany · School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China

  41. Directional Curvature from Armijo Backtracking: A Low-Cost Sharpness Probe and a Calibration-Free Learning-Rate Safeguard for Adam

    Jul 4, 2026Ashmitha R, Jörg FrochteGradient DescentAdaptive Gradient Methods

    Department of Artificial Intelligence and Data Science, Sri Ramakrishna Engineering College, Anna University, Coimbatore 641022, Tamil Nadu, India · Interdisciplinary Institute for Applied AI and Data Science Ruhr (AKIS), Department of Electrical Engineering and Computer Science, Bochum University of Applied Sciences, Am Hochschulcampus 1, 44801 Bochum, Germany

  42. On the Convergence of Adam, Revisited

    Jul 3, 2026Steven Heilman, Sampad MohantyOnline Convex OptimizationStochastic Optimization Convergence

    Department of Mathematics, University of Southern California, Los Angeles, CA 90089.

  43. Characterizing Optimizer-Dependent Training Dynamics Through Hessian Eigenvector Displacement and Localization

    Jun 29, 2026Marcelina Marjankowska, Valerio Modugno, Paolo BaruccaSpectral MethodsAdaptive Gradient Methods

    Financial Computing and Analytics Group, University College London · Robot Perception and Learning Lab, University College London

  44. Optimizer Memory Makes Shuffle Order a First-Order Source of Fine-Tuning Noise

    Jun 28, 2026John SweeneyStochastic OptimizationFine-Tuning

  45. Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks

    Jun 28, 2026Tejas Pradeep ShirodkarDeep Learning OptimizationNeural Network Optimization

    IIIT, Hyderabad

  46. Analysis of Adam Algorithms for Stochastic Dynamic Systems

    Jun 27, 2026Xin Zheng, Yifei Jin, Lei GuoStochastic OptimizationStochastic Optimization Convergence

    State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing 101408, China

  47. Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?

    Jun 22, 2026Dingzhi Yu, Hongyi Tao, Yuanyu Wan +2Stochastic OptimizationHeavy-Tailed Noise

    1Nanjing University · 2Zhejiang University · 3Fudan University

  48. Adam Converges in Nonsmooth Nonconvex Optimization

    Jun 21, 2026Zijian LiuHeavy-Tailed NoiseNonconvex Optimization

    Department of Technology, Operations, and Statistics Leonard N. Stern School of Business New York University New York, NY 10012, USA

  49. Central limit theorem for the averaged Adam optimizer

    Jun 19, 2026Steffen Dereich, Arnulf JentzenStochastic ApproximationStochastic Optimization Convergence

    Institute for Mathematical Stochastics, University of Münster, Germany · School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China · Applied Mathematics: Institute for Analysis and Numerics, University of Münster, Germany

  50. One-Step Generalization Ratio Guided Optimization for Domain Generalization

    Jun 15, 2026Sumin Cho, Dongwon Kim, Kwangsu KimDomain GeneralizationAdaptive Gradient Methods

    Department of Computer Science and Engineering, University of Sungkyunkwan, Suwon, Korea.

  51. Schattor: Schatten-family methods for deep learning optimization

    Jun 14, 2026Bohao Ma, Junyu Zhang, Chuan HeDeep Learning OptimizationStochastic Optimization Convergence

    School of Data Science, The Chinese University of Hong Kong (Shenzhen), Shenzhen, Guangdong, China · Department of Industrial Systems Engineering and Management, National University of Singapore, Singapore · Department of Mathematics, Linköping University, Sweden

  52. Beyond a Single Explanation of the Adam--SGD Gap

    Jun 12, 2026Chenxiang Zhang, Rustem Islamov, Enea Monzio Compagnoni +3Deep Learning OptimizationAdaptive Gradient Methods

    University of Luxembourg · MPI for Intelligent Systems · University of Basel +2

  53. Gefen: Optimized Stochastic Optimizer

    Jun 11, 2026Nadav Benedek, Tomer Koren, Ohad FriedDeep Learning OptimizationMemory-Efficient Optimization

    Reichman University · Tel Aviv University, Google Research

  54. Weibull Weight-Scale Parameter Evolution under AdamW Training Dynamics

    Jun 11, 2026Tiexin DingWeight-Space AnalysisAdaptive Gradient Methods

    Independent Researcher.

  55. Preserving Plasticity in Continual Learning via Dynamical Isometry

    Jun 8, 2026Andries Rosseau, Robert Müller, Ann NowéLoss of PlasticityContinual Learning

    AI Lab, Vrije Universiteit Brussel · Aganthos

  56. Adaptive directional gradients for parameterised quantum circuits

    Jun 8, 2026Brian Coyle, Snehal Raj, Virag Umathe +2Variational Quantum CircuitsAdaptive Gradient Methods

    School of Informatics, University of Edinburgh, Edinburgh, UK · LIP6, CNRS, Sorbonne Université, Paris, France · QC Ware, Palo Alto, USA and Paris, France +1

  57. OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality

    Jun 7, 2026Ganzhao YuanMomentum MethodsNonconvex Stochastic Optimization

    Faculty of Computer Science and Artificial Intelligence Shenzhen University of Advanced Technology (SUAT), China

  58. Large-scale empirical tuning and comparison of default optimizers for variational inference

    Jun 5, 2026Trevor Campbell, Jonathan H. Huggins, Kyurae Kim +1Stochastic OptimizationBayesian Inference

    Department of Statistics, UBC · Department of Mathematics & Statistics, Boston University · Faculty of Computing & Statistics, Boston University +1

  59. Flatland: The Adventures of Gradient Descent with Large Step Sizes

    Jun 4, 2026Leonardo Galli, Curtis Fox, Wiebke Bartolomaeus +2Edge of StabilityGradient Descent

    LMU Munich · Munich Center for Machine Learning · University of British Columbia +1

  60. DP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum

    Jun 3, 2026Naima Tasnim, Lalitha Sankar, Oliver KosutDifferentially Private Stochastic Gradient DescentGradient Clipping

    Arizona State University, Tempe, AZ, United States