Deep Learning Optimization

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, up 133% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 134

All topics

Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks
Jul 5, 2026Shokhrukh Ibragimov, Arnulf JentzenDeep Learning Optimization Neural Network Optimization
Applied Mathematics: Institute for Analysis and Numerics, University of Münster, Germany · School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China
OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers
Jul 4, 2026Siyuan Li, Jiabao Pan, Yumou Liu +9Deep Learning Optimization Neural Network Optimization
Shanghai Artificial Intelligence Laboratory · Westlake University · Shanghai University +4
Class-Grouped Normalized Momentum and Faster Hyperparameter Exploration to Tackle Class Imbalance in Federated Learning
Jul 1, 2026Haemin Park, Diego Klabjan, Martin W. Braun +2Class-Imbalanced Learning Deep Learning Optimization
Department of Industrial Engineering & Management Sciences, Northwestern University, Evanston, IL, USA · Intel Corporation, Chandler, AZ, USA
From Approximation to Emergence: A Theory of Deep Learning
Jul 1, 2026Zhilin ZhaoNeural Network Generalization Deep Learning Optimization
Sun Yat-sen University
Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimization
Jun 29, 2026Haoming Meng, Anton Sugolov, Vardan PapyanDeep Learning Optimization Neural Network Optimization
Department of Mathematics, University of Toronto, Toronto, Canada · Vector Institute, Toronto, Canada
Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks
Jun 28, 2026Tejas Pradeep ShirodkarDeep Learning Optimization Neural Network Optimization
IIIT, Hyderabad
DMuon: Efficient Distributed Muon Training with Near-Adam Overhead
Jun 25, 2026Vincent Chen, Starrick Liu, Regis Cheng +8Deep Learning Optimization Distributed Optimization
X SQUARE ROBOT TEAM
Tensorion: A Tensor-Aware Generalization of the Muon Optimizer
Jun 24, 2026Vladimir Bogachev, Vladimir Aletov, Alexander Molozhavenko +2Deep Learning Optimization Muon Optimizer
HSE University · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab)
An Analysis of Posterior Collapse, Parameterization and Initialization in Variational Deep Gaussian Processes
Jun 24, 2026Francisco Javier Sáez-Maldonado, Juan Maroñas, Daniel Hernández-LobatoDeep Learning Optimization Posterior Collapse
Department of Computer Science and Artificial Intelligence Universidad de Granada Granada, Spain · Department of Quantitative Methods CUNEF Universidad Madrid, Spain · Machine Learning Group at Universidad Autónoma de Madrid +1
Training for the Model You Return: Improving Optimization for Iterate-Averaged Language Models
Jun 23, 2026Kwok Chun Au, Adam BlockDeep Learning Optimization LLM Training
Department of Computer Science, Columbia University · Department of Electrical Engineering, Columbia University
Fast and Slow Variational Continual Learning
Jun 22, 2026Subarnaduti Paul, Yohan Jung, Mohammad Emtiyaz Khan +3Continual Learning for LLMs Deep Learning Optimization
Faculty of Mathematics and Computer Science, University of Bremen, Bremen, Germany · Department of Computer Science & AI, Jeonbuk National University, Republic of Korea · Department of Computer Science, TU Darmstadt, Darmstadt, Germany +3
Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective
Jun 19, 2026Tianqi Shen, Jinji Yang, Runze Shi +3Deep Learning Optimization Muon Optimizer
Department of Computer Science City University of Hong Kong (CityUHK) · Department of Statistics and Data Science University of Pennsylvania · School of Statistics and Data Science Shanghai University of Finance and Economics
Breaking chains with trees: Deep learning with $\mathcal{O}(\log N)$ parallel time complexity
Jun 19, 2026Neeraj Mohan Sushma, Aditya Nagarsekar, Cabrel Teguemne Fokam +4Deep Learning Optimization Neural Network Optimization
Center for Cognitive Interaction Technology, Bielefeld University, Germany · Ruhr Universität Bochum, Germany · Birla Institute of Technology and Science, Pilani, Goa Campus +1
Towards Robust Training in NNGPT AutoML Pipeline: A Loss-Optimizer Pairing Selection Study
Jun 18, 2026Anton Abramochkin, Radu Timofte, Dmitry IgnatovDeep Learning Optimization Neural Network Optimization
Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany
eCNNTO: A Highly Generalizable ConvNet for Accelerating Topology Optimization
Jun 18, 2026Shengbiao Lu, Xiaodong WeiTopology Optimization Deep Learning Optimization
Global college, Shanghai Jiao Tong University, 800 Dongchuan Road, Shanghai, 200240, Shanghai, China
NeuronFabric: A Software Reference Architecture for On-Chip Transformer Training with Local Adam
Jun 15, 2026Evgeny UkladchikovDeep Learning Optimization Transformer
Independent Researcher
Schattor: Schatten-family methods for deep learning optimization
Jun 14, 2026Bohao Ma, Junyu Zhang, Chuan HeDeep Learning Optimization Stochastic Optimization Convergence
School of Data Science, The Chinese University of Hong Kong (Shenzhen), Shenzhen, Guangdong, China · Department of Industrial Systems Engineering and Management, National University of Singapore, Singapore · Department of Mathematics, Linköping University, Sweden
When to use what Schatten- $p$ norm in deep learning?
Jun 13, 2026Thomas PethickDeep Learning Optimization Muon Optimizer
Beyond a Single Explanation of the Adam--SGD Gap
Jun 12, 2026Chenxiang Zhang, Rustem Islamov, Enea Monzio Compagnoni +3Deep Learning Optimization Adaptive Gradient Methods
University of Luxembourg · MPI for Intelligent Systems · University of Basel +2
Gefen: Optimized Stochastic Optimizer
Jun 11, 2026Nadav Benedek, Tomer Koren, Ohad FriedDeep Learning Optimization Memory-Efficient Optimization
Reichman University · Tel Aviv University, Google Research
LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold
Jun 11, 2026Franz Louis Cesista, Katherine Crowson, Cédric Simal +1Deep Learning Optimization Riemannian Optimization
Ateneo de Manila University · EleutherAI · NaXys +1
Simplicity Suffices for Parameter Noise Injection in Stochastic Gradient Descent
Jun 10, 2026Benjamin Leblanc, Louis-Jacob Lebel, Teddy Kana +1Neural Network Generalization Deep Learning Optimization
Universit´e Laval
5% > 100%: Flatness Preference is All You Need for Multimodal Parameter-Efficient Fine-Tuning
Jun 9, 2026Yifan Zhu, Can Lin, Hangjie Yuan +4Deep Learning Optimization Flat Minima
School of Computer Sciences, Beijing University of Posts and Telecommunications, Beijing 100876, China · College of Computer Science and Technology, Zhejiang University, Zhejiang 310027, China · ETH Zürich, Switzerland
FOGO: Forgetting-aware Orthogonalization Optimizer
Jun 9, 2026Toan Nguyen, Yang Liu, Trung Le +2Deep Learning Optimization Continual Learning
School of Computer Science and Engineering University of New South Wales Sydney, Australia · Department of Data Science & AI Monash University Melbourne, Australia · DEVCOM Army Research Laboratory USA
Principles and Practice of Deep Representation Learning: or a Mathematical Theory of Memory
Jun 4, 2026Sam Buchanan, Druv Pai, Peng Wang +1Deep Learning Optimization Representation Learning
UC Berkeley · TTIC · University of Macau +2
Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss
Jun 4, 2026Thomas T. Zhang, Alok Shah, Yifei Zhang +3Deep Learning Optimization
University of Pennsylvania · Carnegie Mellon University · Amazon FAR
Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering
Jun 2, 2026Xianliang Li, Zihan Zhang, Weiyang Liu +1Deep Learning Optimization Momentum Methods
1The Institute of Statistical Mathematics · 2The Graduate Institute for Advanced Studies, SOKENDAI · 3National Institute of Informatics +3
FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoo
Jun 1, 2026Kyunghun Nam, Sumyeong AhnDeep Learning Optimization Optimization Convergence Analysis
KENTECH, Naju, Republic of Korea.
Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks
May 29, 2026Tianyu Pang, Vignesh Kothapalli, Shenyang Deng +3Deep Learning Optimization Neural Network Training Dynamics
Dartmouth College · Stanford University · Virginia Tech
Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handling
May 29, 2026Dmitrii Feoktistov, Timofey Belinsky, Andrey Veprikov +2Stochastic Optimization Sign-Based Optimization
HSE University, Moscow, Russia · BRAIn Lab, Moscow, Russia · Yandex Research, Moscow, Russia +3

Jul 5, 2026, math.OCJ/K move · Enter open · S save

Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks

Shokhrukh Ibragimov, Arnulf Jentzen

Applied Mathematics: Institute for Analysis and Numerics, University of Münster, Germany · School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China

Deep Learning Optimization Neural Network Optimization Convergence Analysis Adaptive Gradient Methods Gradient Descent Dynamics

Open paper AskPDF arXiv

Gradient based optimization methods are nowadays the methods of choice for training deep neural networks (DNNs) in artificial intelligence (AI) systems. In practically relevant DNN training problems, one does usually not apply the standard gradient descent (GD) optimization method but instead one employs suitable sophisticated GD optimization methods, which incorporate adaptivity and/or acceleration techniques, such as the famous Adam optimizer. It is a key contribution of this work to provide a general unified convergence analysis for GD optimization methods in the training of DNNs with analytic activations such as the softplus and the popular Gaussian error linear unit (GeLU) activation. Our general unified convergence result applies to a large class of gradient based optimization methods such as the standard GD, the momentum, the Nesterov accelerated gradient (NAG), the RMSprop, the Adam, the Adamax, the Nadam, the Nadamax, the Adan, the AdaBelief, the AMSGrad, and the Yogi optimizers. Our analysis employs the theory of Kurdyka-Łojasiewicz (KL) inequalities to establish convergence to critical points in the training of DNNs. To the best of our knowledge, the generality of our convergence analysis is also just in the special situation of the Adam optimizer a new contribution to the literature on the analysis of AI optimization algorithms.

Momentum

Related topics

Latest papers 134