Adam

Ask about this topic

Momentum

17 papers in the last four weeks, up 467% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 100

All topics
CardsList

  1. Early Memory Selection for Balanced Adam

    Oct 6, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2AdamParameter-Efficient Adaptation

    Universitat Politècnica de València Valencia, Spain · Universitat Jaume I Castelló de la Plana, Spain

  2. AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models

    Oct 1, 2026Arash Lagzian, Paniz Halvachi, Junming Zhang +2MuonAdam

    National University of Singapore, Singapore · Independent Researcher · LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China

  3. The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation

    Sep 30, 2026S. Aaron McClendon, Jorge Gallego-Feliciano, Antonios SaravanosWeight DecayAdam

    Aimpoint Digital Labs, Atlanta, GA, USA · New York University, New York, NY, USA

  4. What Pretraining and Midtraining Make Learnable from Rewards?

    Sep 29, 2026Chiwun Yang, Xiaoyu LiReinforcement Learning Post-TrainingPretraining

    City University of Hong Kong · University of New South Wales

  5. Adam under Generalized Smoothness with Second-Moment-Type Stochastic Gradients

    Sep 29, 2026Ruinan Jin, Difei Cheng, Ling Chen +3SmoothnessAdam

    Mohamed bin Zayed University of Artificial Intelligence · Aerospace Information Technology University · The Ohio State University +2

  6. EvE: An Alternate Optimizer to Adam

    Sep 28, 2026Shashank Raj, Kalyanmoy DebAdamEvolutionary Optimization Methods

    Department of Computer Science The University of Alabama Tuscaloosa, AL 35487, USA

  7. The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics

    Sep 28, 2026Yihe Zhou, Tongtian Zhu, Yingxiao Huo +4AdamMomentum Stochastic Gradient Descent

    Department of Computer Science, The University of Manchester, Manchester, UK · College of Computer Science and Technology, Zhejiang University, Hangzhou, China

  8. On the Two Faces of Adam in Separable Linear Classification

    Sep 27, 2026Chen Fan, Csaba SzepesváriAdamGradient Descent

    University of Alberta

  9. Low-Rank Friction for Memory-Efficient Transformer Pretraining

    Sep 24, 2026Rajit Rajpal, Benedict LeimkuhlerAdamTransformer Architectures

    School of Mathematics University of Edinburgh Edinburgh, UK EH9 3FD

  10. Continuous Optimization for p-adic Models

    Sep 21, 2026Julian Salazar, Dimitri Kanevsky, Matt Harvey +2Adam

    Google DeepMind

  11. A Lightweight Convolutional Neural Network for Real-Time Recognition of Hand-Drawn Geometric Shapes

    Sep 21, 2026Shahir AbdullahConvolutional Neural NetworksStatistical Shape Model

  12. Online Supervised Dimension Reduction with Random Features: Diagnostics and Computational Trade-offs

    Sep 17, 2026Zhenlin Yao, Wei XiongPrincipal Component AnalysisSubspace

    School of Statistics University of International Business and Economics Beijing, China

  13. Directions That Don't Drift: Stiefel Manifold Routing for Transformer Attention

    Sep 16, 2026Rubén Darío GuerreroRiemannian OptimizationAdam

    NeuroTechNet S.A.S., 1108831, Bogotá, Colombia

  14. Beyond Quadratic Loss: The Stability Phase Diagram of Adam

    Sep 16, 2026Gaoxiang Tang, Huanran Chen, Ziming LiuAdamAnisotropic Loss Landscapes

    IIIS, Tsinghua University · College AI, Tsinghua University · Shanghai Qizhi Institute +1

  15. A Full Adam Theorem for Spectral Heavy-Tail Onset

    Sep 14, 2026Zongmin LiuAdam

    Stanford University

  16. AdamX: Cosine similarity meets gradient descent

    Sep 10, 2026Francisco Caldas, Ruben Belo, Cláudia SoaresAdamGradient Descent

    NOVA School of Science of Technology Universidade Nova de Lisboa, Caparica, Portugal

  17. SkillAdam: Stable and Efficient Skill Evolution for Agents

    Sep 8, 2026Gaoyuan Li, Meihao Fan, Yizhe Liu +7Skill EvolutionSkills

    Renmin University of China · Tencent

  18. Equivariance Breaks the Learning Rate

    Sep 8, 2026Andrei Manolache, Mathias NiepertAdamEquivariant Neural Networks

    University of Stuttgart, Germany · Bitdefender, Romania

  19. Convergence rates for the RMSprop optimizer with full control of the hyperparameters

    Aug 31, 2026Steffen Dereich, Arnulf JentzenAdamStochastic Convex Optimization

  20. Beyond Dense Adam States: Adaptive Log-Space Quantization for Memory-Efficient Optimizers

    Aug 23, 2026Yan WangAdamRecurrent State

    Independent Researcher

  21. Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversal

    Aug 21, 2026Yiman Fong, Heng YangAdamLocal Curvature

    School of Engineering and Applied Sciences, Harvard University.

  22. Gradient Under Microscope: Benchmarking Resource Utilization of Memory-Efficient Gradient Computation Methods

    Aug 9, 2026Sarthak Mahapatra, Zihan Zhou, Khatoon Khedri +2GradientAdam

  23. The Loss Does Not See the Basis, but Adam Does

    Aug 5, 2026Devender SinghLow-Rank StructureAdam

    Department of Computer Science Memorial University of Newfoundland

  24. MESH: Memory-Efficient Sinkhorn Optimization for Mixture-of-Experts Training

    Aug 5, 2026Masato FujitakeMixture-Of-ExpertsAdam

    FA Research, Fast Accounting Co., Ltd.

  25. Joint Affine Spectral Shaping: Coupling Weight and Bias Updates Beyond Weight-Only Muon

    Aug 4, 2026Gongyue Zhang, Honghai LiuAffineMuon

    State Key Laboratory of Robotics and Systems, Harbin Institute of Technology, Shenzhen, Shenzhen, 518055, China.

  26. The Convergence Behavior of Adam under Heavy-Tailed Noise

    Jul 29, 2026Yijiang PangAdamStochastic Convex Optimization

    Michigan State University, East Lansing, USA

  27. Reconstructing Backpropagation from Forward Fluctuations in Noise-modulated Neural Networks

    Jul 29, 2026Shuhei IkemotoBackpropagationNeural Network

    Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology, Kitakyushu, Japan

  28. PYPM-GGD: Pitman-Yor Process Mixture with Generalized Gaussian Density using ADAM

    Jul 27, 2026Kart-Leong LimVariational InferenceAdam

    Institute of Microelectronics, A*Star Singapore

  29. Hidden Boundary Motion in Transformer Optimization: Function-Space Orthogonalization of Affine Weight and Bias Updates

    Jul 24, 2026Zhang Gongyue, Sheng Yixuan, Liu donghan +3AdamAffine

    arXiv:2607.22927v1

  30. Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training

    Jul 21, 2026Nuemaan MalikAdaptive OptimizersMixture-Of-Experts

    Independent Researcher

  31. Backpropagation-Free Trunk Training via the Split Forward Gradients

    Jul 18, 2026Tian Qin, Wei-Min HuangBackpropagationGradient

    Math department Lehigh University Bethlehem, PA 18015, USA

  32. When Does Muon Help Agentic Reinforcement Learning?

    Jul 17, 2026Kai Ruan, Jinghao Lin, Zihe Huang +4MuonAdam

    Gaoling School of Artificial Intelligence, Renmin University of China · Independent Researcher · Institute of Computing Technology, Chinese Academy of Sciences +3

  33. Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers

    Jul 16, 2026Akhilesh GogikarAdamTime Discretization

    1Independent Researcher.

  34. Reassessing Muon for Matrix Factorization

    Jul 14, 2026Ali Parviz, Gal Mishne, Alex CloningerMuonAdam

    †. Halicioğlu Data Science Institute, UC San Diego. · ∗. Department of Mathematics, UC San Diego.

  35. M+Adam: Low-Precision Training via Additive-Multiplicative Optimization

    Jul 12, 2026Xiaoyuan Liang, Sebastian Loeschcke, Mads Toftrup +1Mixed-Precision QuantizationAdam

    California Institute of Technology · University of Copenhagen · Aarhus University

  36. Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks

    Jul 5, 2026Shokhrukh Ibragimov, Arnulf JentzenNeural Network OptimizationGradient Descent

    Applied Mathematics: Institute for Analysis and Numerics, University of Münster, Germany · School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China

  37. Directional Curvature from Armijo Backtracking: A Low-Cost Sharpness Probe and a Calibration-Free Learning-Rate Safeguard for Adam

    Jul 4, 2026Ashmitha R, Jörg FrochteAdamSharpness-Aware Minimization

    Department of Artificial Intelligence and Data Science, Sri Ramakrishna Engineering College, Anna University, Coimbatore 641022, Tamil Nadu, India · Interdisciplinary Institute for Applied AI and Data Science Ruhr (AKIS), Department of Electrical Engineering and Computer Science, Bochum University of Applied Sciences, Am Hochschulcampus 1, 44801 Bochum, Germany

  38. On the Convergence of Adam, Revisited

    Jul 3, 2026Steven Heilman, Sampad MohantyAdam\Widetilde{\Mathcal{O}}(\Sqrt{T})$ Regret

    Department of Mathematics, University of Southern California, Los Angeles, CA 90089.

  39. Beyond Adam: SOAP and Muon for Faster, Label-Efficient Training of Machine Learning Interatomic Potentials

    Jul 2, 2026Gil Harari, Yoel Zimmermann, Ola Tangen Kulseng +4Interatomic PotentialsAdam

    John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA · Robert Bosch LLC Research and Technology Center, Watertown, MA, USA

  40. SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication

    Jul 2, 2026Mingkai Zheng, Junlin Chen, Haotian Xie +1Large Language Model TrainingSparsity

    Rutgers University

  41. Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks

    Jun 28, 2026Tejas Pradeep ShirodkarAdamDeep Network

    IIIT, Hyderabad

  42. Analysis of Adam Algorithms for Stochastic Dynamic Systems

    Jun 27, 2026Xin Zheng, Yifei Jin, Lei GuoAdamStochastic Dynamics

    State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing 101408, China

  43. Tensorion: A Tensor-Aware Generalization of the Muon Optimizer

    Jun 24, 2026Vladimir Bogachev, Vladimir Aletov, Alexander Molozhavenko +2AdamMuon

    HSE University · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab)

  44. Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?

    Jun 22, 2026Dingzhi Yu, Hongyi Tao, Yuanyu Wan +2AdamLong-Tailed Distribution

    1Nanjing University · 2Zhejiang University · 3Fudan University

  45. Adam Converges in Nonsmooth Nonconvex Optimization

    Jun 21, 2026Zijian LiuStochastic Convex OptimizationAdam

    Department of Technology, Operations, and Statistics Leonard N. Stern School of Business New York University New York, NY 10012, USA

  46. Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective

    Jun 19, 2026Tianqi Shen, Jinji Yang, Runze Shi +3MuonAdam

    Department of Computer Science City University of Hong Kong (CityUHK) · Department of Statistics and Data Science University of Pennsylvania · School of Statistics and Data Science Shanghai University of Finance and Economics

  47. Central limit theorem for the averaged Adam optimizer

    Jun 19, 2026Steffen Dereich, Arnulf JentzenStochastic ApproximationLarge Deviation Principle

    Institute for Mathematical Stochastics, University of Münster, Germany · School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China · Applied Mathematics: Institute for Analysis and Numerics, University of Münster, Germany

  48. Towards Robust Training in NNGPT AutoML Pipeline: A Loss-Optimizer Pairing Selection Study

    Jun 18, 2026Anton Abramochkin, Radu Timofte, Dmitry IgnatovNeural Architecture SearchAdam

    Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany

  49. NeuronFabric: A Software Reference Architecture for On-Chip Transformer Training with Local Adam

    Jun 15, 2026Evgeny UkladchikovTransformer ArchitecturesField-Programmable Gate Arrays

    Independent Researcher

  50. Beyond a Single Explanation of the Adam--SGD Gap

    Jun 12, 2026Chenxiang Zhang, Rustem Islamov, Enea Monzio Compagnoni +3AdamExplainable AI Methods

    University of Luxembourg · MPI for Intelligent Systems · University of Basel +2

  51. Preserving Plasticity in Continual Learning via Dynamical Isometry

    Jun 8, 2026Andries Rosseau, Robert Müller, Ann NowéPlasticityContinual Learning

    AI Lab, Vrije Universiteit Brussel · Aganthos

  52. Muon Learns More Robust and Transferable Features than Adam

    Jun 8, 2026Tianyu Ruan, Fengzhuo Zhang, Shuche Wang +1AdamFeature Learning

    Yale University · University of Chinese Academy of Sciences · Academy of Mathematics and Systems Science, CAS +1

  53. On solving symmetric multi-type orthogonal non-negative matrix tri-factorization problem

    Jun 6, 2026Rok Hribar, Gregor Papa, Janez Povh +1OrthogonalityAdam

    Computer Systems Department, Jožef Stefan Institute, Jamova cesta 39, Ljubljana, 1000, Slovenia. · Laboratory for Engineering Design, Faculty of Mechanical Engineering, University of Ljubljana, Aškerčeva cesta 6, Ljubljana, 1000, Slovenia. · Rudolfovo – Science and Technology Centre Novo mesto, Podbreznik 15, Novo mesto, 8000, Slovenia. +1

  54. Why Muon Outperforms Adam: A Curvature Perspective

    Jun 3, 2026Shuche Wang, Fengzhuo Zhang, Jiaxiang Li +2AdamMuon

    National University of Singapore · Yale University · University of Minnesota

  55. Beyond Gradient Descent: Adam for Analog Ising Machines

    Jun 2, 2026Stijn Van Vooren, Guy Van der Sande, Guy VerschaffeltSpatial Photonic Ising MachinesAdam

    Applied Physics research group, Vrije Universiteit Brussel, Pleinlaan 2, 1050 Brussels, Belgium∗

  56. MAdam: Metric-Aware Multi-Objective Adam

    Jun 2, 2026Fengbei Liu, Rachit Saluja, Sunwoo Kwak +5AdamMulti-Objective Optimization

    Cornell Tech · Delft University of Technology · Weill Cornell Medicine

  57. DECA: Decentralizing Block-Wise Adam for Efficient LLM Full-Parameter Fine-Tuning on Non-IID Data

    Jun 2, 2026Yunsheng Yuan, Shaowei Li, Kai Wang +5Parameter-Efficient Fine-Tuning MethodsLarge Language Model Fine-Tuning

    School of Computer Science and Technology, Shandong University, Qingdao China · School of Mathematical Science, Peking University, China · IEIT SYSTEM, China +2

  58. Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling

    May 30, 2026Qiao Xiao, Boqian Wu, Patrik Okanovic +6Large Language Model TrainingSparsity

    Eindhoven University of Technology · University of Luxembourg · University of Twente +1

  59. Convergence of Steepest Descent and Adam under Non-Uniform Smoothness

    May 28, 2026Sharan Vaswani, Yifan Sun, Reza BabanezhadGradient DescentSmoothness

    1Simon Fraser University · 2Stony Brook University · 3Samsung AI, Montreal.

  60. Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization

    May 28, 2026Ruoran Xu, Borong She, Xiaobo Jin +1Stochastic Convex OptimizationAdam

    Xi’an Jiaotong-Liverpool University