Adaptive Gradient Methods

Ask about this topic

Momentum

13 papers in the last four weeks, up 160% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 113

All topics
CardsList

  1. Early Memory Selection for Balanced Adam

    Oct 6, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2Adaptive Gradient MethodsHyperparameter Optimization

    Universitat Politècnica de València Valencia, Spain · Universitat Jaume I Castelló de la Plana, Spain

  2. DGA-Muon: Decoupled Geometry-Aligned Adaptive Scaling for Muon

    Oct 5, 2026Wenpeng Zhang, Runsheng YuMuon OptimizerAdaptive Gradient Methods

    Independent Reasearcher

  3. Exact information accounting for SGD methods

    Sep 30, 2026Akshay BalsubramaniNeural Network GeneralizationStochastic Optimization Convergence

  4. Adam under Generalized Smoothness with Second-Moment-Type Stochastic Gradients

    Sep 29, 2026Ruinan Jin, Difei Cheng, Ling Chen +3Convergence AnalysisStochastic Optimization Convergence

    Mohamed bin Zayed University of Artificial Intelligence · Aerospace Information Technology University · The Ohio State University +2

  5. Second-Moment Stochastic Approximation Methods

    Sep 29, 2026Tao Jiang, Lin XiaoStochastic ApproximationMuon Optimizer

    Fundamental AI Research (FAIR), Meta Superintelligence Labs

  6. The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics

    Sep 28, 2026Yihe Zhou, Tongtian Zhu, Yingxiao Huo +4Stochastic OptimizationSign-Based Optimization

    Department of Computer Science, The University of Manchester, Manchester, UK · College of Computer Science and Technology, Zhejiang University, Hangzhou, China

  7. On the Two Faces of Adam in Separable Linear Classification

    Sep 27, 2026Chen Fan, Csaba SzepesváriImplicit BiasClassification

    University of Alberta

  8. VCMM: Variance-Calibrated Momentum for Multimodal Learning

    Sep 23, 2026Zhongjing Gu, Chenyang Huang, Yufa Feng +3Multimodal OptimizationAdaptive Gradient Methods

  9. AURA: Angular Update Rate Adaptation for training complex-valued neural networks

    Sep 22, 2026Enrico Ballini, Allan Peter Engsig-Karup, Tito AndriolloNeural Network OptimizationComplex-Valued Neural Networks

    Aarhus University, Aarhus, Denmark · Technical University of Denmark, Kongens Lyngby, Denmark

  10. Adaptive Forgetting for Nonstationary Optimization: Towards Robust EEG Decoding

    Sep 21, 2026Hongyu Zhu, Lin Chen, Jing Chen +2Cross-Subject EEG DecodingAdaptive Gradient Methods

    Chongqing Institute of Green Intelligent Technology, Chinese Academy of Sciences · Beijing University of Posts and Telecommunications · School of Resources and Environment, University of Electronic Science and Technology of China

  11. Beyond Quadratic Loss: The Stability Phase Diagram of Adam

    Sep 16, 2026Gaoxiang Tang, Huanran Chen, Ziming LiuAdaptive Gradient MethodsNeural Network Training Dynamics

    IIIS, Tsinghua University · College AI, Tsinghua University · Shanghai Qizhi Institute +1

  12. A Full Adam Theorem for Spectral Heavy-Tail Onset

    Sep 14, 2026Zongmin LiuAdaptive Gradient MethodsNeural Network Training Dynamics

    Stanford University

  13. AdamX: Cosine similarity meets gradient descent

    Sep 10, 2026Francisco Caldas, Ruben Belo, Cláudia SoaresDeep Learning OptimizationAdaptive Gradient Methods

    NOVA School of Science of Technology Universidade Nova de Lisboa, Caparica, Portugal

  14. When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay

    Sep 8, 2026Hasan Amin, Wei-Kai Chang, Rajiv KhannaLearning Rate SchedulingWeight Decay

    Purdue University

  15. Equivariance Breaks the Learning Rate

    Sep 8, 2026Andrei Manolache, Mathias NiepertEquivariant Neural NetworksEquivariant Representation Learning

    University of Stuttgart, Germany · Bitdefender, Romania

  16. Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetry

    Sep 2, 2026Sai Niranjan Ramachandran, Suvrit SraStochastic Gradient DescentAdaptive Gradient Methods

    School of Computation, Information and Technology, Technical University of Munich, Germany · Munich Center for Machine Learning (MCML)

  17. Convergence rates for the RMSprop optimizer with full control of the hyperparameters

    Aug 31, 2026Steffen Dereich, Arnulf JentzenStochastic ApproximationStochastic Optimization Convergence

  18. Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversal

    Aug 21, 2026Yiman Fong, Heng YangEdge of StabilityAdaptive Gradient Methods

    School of Engineering and Applied Sciences, Harvard University.

  19. Momentum as Residual-Driven Multiplier Correction for Deep Learning Optimization

    Aug 13, 2026Zhixin Ren, Yau Lyu, Congrong Li +2Deep Learning OptimizationMomentum Methods

    Department of Mathmatical Science, Tsinghua University · School of Vehicle and Mobility, Tsinghua University · College of AI, Tsinghua University

  20. Adaptive Bregman Proximal Stochastic Gradient with a Stabilized Barzilai--Borwein Step Size

    Aug 12, 2026Chenhan Jin, Shengze Xu, Binghui Xie +4Nonconvex Stochastic OptimizationAdaptive Gradient Methods

    1The Chinese University of Hong Kong · University of Utah · 3Beijing Normal-Hong Kong Baptist University +1

  21. Gradient Under Microscope: Benchmarking Resource Utilization of Memory-Efficient Gradient Computation Methods

    Aug 9, 2026Sarthak Mahapatra, Zihan Zhou, Khatoon Khedri +2Efficient Language Model TrainingAdaptive Gradient Methods

  22. The Loss Does Not See the Basis, but Adam Does

    Aug 5, 2026Devender SinghLow-Rank Matrix DecompositionMatrix Optimization

    Department of Computer Science Memorial University of Newfoundland

  23. AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization

    Aug 3, 2026Alok Kumar Pandey, Umang Chaturvedi, Aatish Rana +1Deep Learning OptimizationStochastic Optimization Convergence

  24. Adaptivity via a Parallel Architecture for Stochastic Gradient Methods

    Jul 31, 2026Bin FuStochastic OptimizationStochastic Optimization Convergence

    University of Texas Rio Grande Valley

  25. The Convergence Behavior of Adam under Heavy-Tailed Noise

    Jul 29, 2026Yijiang PangNonconvex Stochastic OptimizationHeavy-Tailed Noise

    Michigan State University, East Lansing, USA

  26. Parameter-Free Dynamic Regret under Heavy-Tailed Noise

    Jul 29, 2026Vaneet AggarwalOnline Convex OptimizationDynamic Regret Minimization

    Purdue University

  27. Adaptive Gradient-Based Methods for a Broader Class of Optimization Problems under Performative Prediction

    Jul 29, 2026Hiroki Hamaguchi, Yuya Hikima, Hiroshi Sawada +1Stochastic Optimization ConvergenceAdaptive Gradient Methods

    Graduate School of Information Science and Technology, The University of Tokyo · Department of Industrial Engineering and Economics, Institute of Science Tokyo · Communication Science Laboratories, NTT, Inc. +1

  28. Data-Dependent Regret and Polyak Corrections for Constrained Online Convex Optimization

    Jul 28, 2026Wentao ZhangOnline Convex OptimizationConstrained Optimization

    Tsinghua Shenzhen International Graduate School · Tsinghua University

  29. PYPM-GGD: Pitman-Yor Process Mixture with Generalized Gaussian Density using ADAM

    Jul 27, 2026Kart-Leong LimBayesian Mixture ModelsClustering

    Institute of Microelectronics, A*Star Singapore

  30. Low-Rank Dependence Decomposition via Accelerated Symmetric Non-negative Matrix Factorization

    Jul 27, 2026Lavinia Ghita, Dhruv Desai, Jake Goldberg +1Non-Negative Matrix FactorizationLow-Rank Matrix Decomposition

    NVIDIA