Deep Learning Optimization

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, up 133% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 132

All topics

Class-Grouped Normalized Momentum and Faster Hyperparameter Exploration to Tackle Class Imbalance in Federated Learning
Jul 1, 2026Haemin Park, Diego Klabjan, Martin W. Braun +2Class-Imbalanced Learning Deep Learning Optimization
Department of Industrial Engineering & Management Sciences, Northwestern University, Evanston, IL, USA · Intel Corporation, Chandler, AZ, USA
From Approximation to Emergence: A Theory of Deep Learning
Jul 1, 2026Zhilin ZhaoNeural Network Generalization Deep Learning Optimization
Sun Yat-sen University
Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimization
Jun 29, 2026Haoming Meng, Anton Sugolov, Vardan PapyanDeep Learning Optimization Neural Network Optimization
Department of Mathematics, University of Toronto, Toronto, Canada · Vector Institute, Toronto, Canada
Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks
Jun 28, 2026Tejas Pradeep ShirodkarDeep Learning Optimization Neural Network Optimization
IIIT, Hyderabad
DMuon: Efficient Distributed Muon Training with Near-Adam Overhead
Jun 25, 2026Vincent Chen, Starrick Liu, Regis Cheng +8Deep Learning Optimization Distributed Optimization
X SQUARE ROBOT TEAM
Tensorion: A Tensor-Aware Generalization of the Muon Optimizer
Jun 24, 2026Vladimir Bogachev, Vladimir Aletov, Alexander Molozhavenko +2Deep Learning Optimization Muon Optimizer
HSE University · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab)
An Analysis of Posterior Collapse, Parameterization and Initialization in Variational Deep Gaussian Processes
Jun 24, 2026Francisco Javier Sáez-Maldonado, Juan Maroñas, Daniel Hernández-LobatoDeep Learning Optimization Posterior Collapse
Department of Computer Science and Artificial Intelligence Universidad de Granada Granada, Spain · Department of Quantitative Methods CUNEF Universidad Madrid, Spain · Machine Learning Group at Universidad Autónoma de Madrid +1
Training for the Model You Return: Improving Optimization for Iterate-Averaged Language Models
Jun 23, 2026Kwok Chun Au, Adam BlockDeep Learning Optimization LLM Training
Department of Computer Science, Columbia University · Department of Electrical Engineering, Columbia University
Fast and Slow Variational Continual Learning
Jun 22, 2026Subarnaduti Paul, Yohan Jung, Mohammad Emtiyaz Khan +3Continual Learning for LLMs Deep Learning Optimization
Faculty of Mathematics and Computer Science, University of Bremen, Bremen, Germany · Department of Computer Science & AI, Jeonbuk National University, Republic of Korea · Department of Computer Science, TU Darmstadt, Darmstadt, Germany +3
Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective
Jun 19, 2026Tianqi Shen, Jinji Yang, Runze Shi +3Deep Learning Optimization Muon Optimizer
Department of Computer Science City University of Hong Kong (CityUHK) · Department of Statistics and Data Science University of Pennsylvania · School of Statistics and Data Science Shanghai University of Finance and Economics
Breaking chains with trees: Deep learning with $\mathcal{O}(\log N)$ parallel time complexity
Jun 19, 2026Neeraj Mohan Sushma, Aditya Nagarsekar, Cabrel Teguemne Fokam +4Deep Learning Optimization Neural Network Optimization
Center for Cognitive Interaction Technology, Bielefeld University, Germany · Ruhr Universität Bochum, Germany · Birla Institute of Technology and Science, Pilani, Goa Campus +1
Towards Robust Training in NNGPT AutoML Pipeline: A Loss-Optimizer Pairing Selection Study
Jun 18, 2026Anton Abramochkin, Radu Timofte, Dmitry IgnatovDeep Learning Optimization Neural Network Optimization
Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany
eCNNTO: A Highly Generalizable ConvNet for Accelerating Topology Optimization
Jun 18, 2026Shengbiao Lu, Xiaodong WeiTopology Optimization Deep Learning Optimization
Global college, Shanghai Jiao Tong University, 800 Dongchuan Road, Shanghai, 200240, Shanghai, China
NeuronFabric: A Software Reference Architecture for On-Chip Transformer Training with Local Adam
Jun 15, 2026Evgeny UkladchikovDeep Learning Optimization Transformer
Independent Researcher
Schattor: Schatten-family methods for deep learning optimization
Jun 14, 2026Bohao Ma, Junyu Zhang, Chuan HeDeep Learning Optimization Stochastic Optimization Convergence
School of Data Science, The Chinese University of Hong Kong (Shenzhen), Shenzhen, Guangdong, China · Department of Industrial Systems Engineering and Management, National University of Singapore, Singapore · Department of Mathematics, Linköping University, Sweden
When to use what Schatten- $p$ norm in deep learning?
Jun 13, 2026Thomas PethickDeep Learning Optimization Muon Optimizer
Beyond a Single Explanation of the Adam--SGD Gap
Jun 12, 2026Chenxiang Zhang, Rustem Islamov, Enea Monzio Compagnoni +3Deep Learning Optimization Adaptive Gradient Methods
University of Luxembourg · MPI for Intelligent Systems · University of Basel +2
Gefen: Optimized Stochastic Optimizer
Jun 11, 2026Nadav Benedek, Tomer Koren, Ohad FriedDeep Learning Optimization Memory-Efficient Optimization
Reichman University · Tel Aviv University, Google Research
LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold
Jun 11, 2026Franz Louis Cesista, Katherine Crowson, Cédric Simal +1Deep Learning Optimization Riemannian Optimization
Ateneo de Manila University · EleutherAI · NaXys +1
Simplicity Suffices for Parameter Noise Injection in Stochastic Gradient Descent
Jun 10, 2026Benjamin Leblanc, Louis-Jacob Lebel, Teddy Kana +1Neural Network Generalization Deep Learning Optimization
Universit´e Laval
5% > 100%: Flatness Preference is All You Need for Multimodal Parameter-Efficient Fine-Tuning
Jun 9, 2026Yifan Zhu, Can Lin, Hangjie Yuan +4Deep Learning Optimization Flat Minima
School of Computer Sciences, Beijing University of Posts and Telecommunications, Beijing 100876, China · College of Computer Science and Technology, Zhejiang University, Zhejiang 310027, China · ETH Zürich, Switzerland
FOGO: Forgetting-aware Orthogonalization Optimizer
Jun 9, 2026Toan Nguyen, Yang Liu, Trung Le +2Deep Learning Optimization Continual Learning
School of Computer Science and Engineering University of New South Wales Sydney, Australia · Department of Data Science & AI Monash University Melbourne, Australia · DEVCOM Army Research Laboratory USA
Principles and Practice of Deep Representation Learning: or a Mathematical Theory of Memory
Jun 4, 2026Sam Buchanan, Druv Pai, Peng Wang +1Deep Learning Optimization Representation Learning
UC Berkeley · TTIC · University of Macau +2
Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss
Jun 4, 2026Thomas T. Zhang, Alok Shah, Yifei Zhang +3Deep Learning Optimization
University of Pennsylvania · Carnegie Mellon University · Amazon FAR
Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering
Jun 2, 2026Xianliang Li, Zihan Zhang, Weiyang Liu +1Deep Learning Optimization Momentum Methods
1The Institute of Statistical Mathematics · 2The Graduate Institute for Advanced Studies, SOKENDAI · 3National Institute of Informatics +3
FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoo
Jun 1, 2026Kyunghun Nam, Sumyeong AhnDeep Learning Optimization Optimization Convergence Analysis
KENTECH, Naju, Republic of Korea.
Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks
May 29, 2026Tianyu Pang, Vignesh Kothapalli, Shenyang Deng +3Deep Learning Optimization Neural Network Training Dynamics
Dartmouth College · Stanford University · Virginia Tech
Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handling
May 29, 2026Dmitrii Feoktistov, Timofey Belinsky, Andrey Veprikov +2Stochastic Optimization Sign-Based Optimization
HSE University, Moscow, Russia · BRAIn Lab, Moscow, Russia · Yandex Research, Moscow, Russia +3
Inconsistency-Aware Minimization: Improving Generalization with Unlabeled Data
May 29, 2026Hee-Sung Kim, Hyeonseong Kim, Sungyoon LeeNeural Network Generalization Deep Learning Optimization
Department of Computer Science, Hanyang University, Seoul, Korea · Agency for Defense Development, Daejeon, Korea
Is Backpropagation Optimal? When Synthetic Gradients Improve Sample Efficiency
May 27, 2026Yibo Jacky Zhang, Zeyu Tang, Sanmi KoyejoDeep Learning Optimization Backpropagation
Stanford University

Jul 1, 2026, cs.LGJ/K move · Enter open · S save

Class-Grouped Normalized Momentum and Faster Hyperparameter Exploration to Tackle Class Imbalance in Federated Learning

Haemin Park, Diego Klabjan, Martin W. Braun, Xiuqi Li+1

Department of Industrial Engineering & Management Sciences, Northwestern University, Evanston, IL, USA · Intel Corporation, Chandler, AZ, USA

Class-Imbalanced Learning Deep Learning Optimization Hyperparameter Optimization Federated Learning

Open paper AskPDF arXiv

Class imbalance poses a critical challenge in federated learning (FL), where underrepresented classes suffer from poor predictive performance yet cannot be addressed by standard centralized techniques due to privacy and heterogeneity constraints. We propose FedCGNM (Federated Class-Grouped Normalized Momentum), a client-side optimizer in FL that partitions classes into a small number of groups based on minimum within-group variance, maintains a momentum per group, normalizes each group momentum to unit length, and uses the summation of the normalized group momentums as an update direction. This design both equalizes gradient magnitude across majority and minority groups and mitigates the noise inherent in rare-class gradients. We further provide a theoretical convergence analysis explicitly accounting for time-varying resampling-rates. Additionally, to efficiently optimize these rates in small-client regimes, we introduce FedHOO, an X-armed-bandit (XAB) based algorithm that exploits federated parallelism that evaluates many combinations of two candidate rates per client at linear cost. Empirical evaluation on four public long-tailed benchmarks and a proprietary chip-defect dataset demonstrates that FedCGNM consistently outperforms baselines, with FedHOO yielding further gains in small-scale federations.

Momentum

Related topics

Latest papers 132