Deep Learning Optimization

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, up 133% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 132

All topics

DASH: Faster Shampoo via Batched Block Preconditioning and Efficient Inverse-Root Solvers
Feb 2, 2026Ionut-Vlad Modoranu, Philip Zmushko, Erik Schultheis +2Deep Learning Optimization GPU Acceleration
Institute of Science and Technology Austria · Lancaster University · Red Hat AI
Provable Learning of Random Hierarchy Models and Hierarchical Shallow-to-Deep Chaining
Jan 27, 2026Yunwei Ren, Yatin Dandi, Florent Krzakala +1Deep Learning Optimization Hierarchical Representation Learning
Princeton University · École Polytechnique Fédérale de Lausanne · University of California, Berkeley
A Constrained Optimization Perspective of Unrolled Transformers
Jan 24, 2026Javier Porras-Valenzuela, Samar Hadou, Alejandro RibeiroDeep Learning Optimization Transformer
Department of Electrical and Systems Engineering, University of Pennsylvania, Philadelphia, PA, United States of America.
ZENITH: Automated Gradient Norm Informed Stochastic Optimization
Jan 21, 2026Dhrubo SahaDeep Learning Optimization Learning Rate Scheduling
Georgia Institute of Technology
Ky Fan Norms and Beyond: Dual Norms and Combinations for Matrix Optimization
Dec 10, 2025Alexey Kravatskiy, Ivan Kozyrev, Nikolai Kozlov +3Deep Learning Optimization Muon Optimizer
MIRIAI · Russia · MIPT, INM RAS +3
Generalized Design Choices for Deepfake Detectors
Nov 26, 2025Lorenzo Pellegrini, Serafino Pandolfini, Davide Maltoni +3Deep Learning Optimization Deepfake Detection
Department of Computer Science and Engineering (DISI) University of Bologna, Italy · IdentifAI, Italy
Why Do We Need Warm-up? A Theoretical Perspective
Oct 3, 2025Foivos Alimisis, Rustem Islamov, Aurelien LucchiDeep Learning Optimization Learning Rate Scheduling
University of Basel, Switzerland
Geometrically Principled Randomized Optimization for Efficient LLM Training
Oct 2, 2025Sahar Rajabi, Nayeema Nonta, Sirisha RambhatlaLanguage Model Pretraining Deep Learning Optimization
CriticalML Lab, University of Waterloo
Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimization
Sep 3, 2025Wu Lin, Scott C. Lowe, Felix Dangel +3Deep Learning Optimization Neural Network Optimization
1Vector Institute, Toronto, Canada · University of Cambridge, Cambridge, United Kingdom · 3Microsoft, United States +1
Preconditioned Inexact Stochastic ADMM for Deep Model
Feb 15, 2025Shenglong Zhou, Ouya Wang, Ziyan Luo +2Stochastic Optimization Deep Learning Optimization
School of Mathematics and Statistics, Beijing Jiaotong University, China · ITP Lab, Department of EEE, Imperial College London, UK · National Mobile Communications Research Laboratory, Southeast University, China
Convergence Rate Analysis of LION
Nov 12, 2024Yiming Dong, Huan Li, Zhouchen LinSign-Based Optimization Deep Learning Optimization
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University, Beijing, 100871, China · Institute of Robotics and Automatic Information Systems, College of Artificial Intelligence, Nankai University, Tianjin, 300071, China · Pazhou Laboratory (Huangpu), Guangzhou, 510555, China
DP-Muon: Differentially Private Optimization via Matrix-Orthogonalized Momentum
Date pendingJihwan Kim, Chenglin FanDeep Learning Optimization Differentially Private Stochastic Gradient Descent
Seoul National University

Feb 2, 2026, cs.LGJ/K move · Enter open · S save

DASH: Faster Shampoo via Batched Block Preconditioning and Efficient Inverse-Root Solvers

Ionut-Vlad Modoranu, Philip Zmushko, Erik Schultheis, Mher Safaryan+1

Institute of Science and Technology Austria · Lancaster University · Red Hat AI

Deep Learning Optimization GPU Acceleration Distributed Optimization Second-Order Optimization

Open paper AskPDF arXiv

Shampoo is one of the leading approximate second-order optimizers: a variant of it has won the MLCommons AlgoPerf competition, and it has been shown to produce models with lower activation outliers that are easier to compress. Yet, applying Shampoo currently comes at the cost of significant computational slowdown, due to its expensive internal operations. In this paper, we take a significant step to address this shortcoming by proposing \method (for \textbf{D}istributed \textbf{A}ccelerated \textbf{SH}ampoo), a faster implementation of Distributed Shampoo based on two main new techniques: First, we show that preconditioner blocks can be stacked into 3D tensors to significantly improve GPU utilization; second, we introduce the Newton-DB iteration and the Chebyshev polynomial approximations as novel and faster approaches for computing the inverse matrix roots required by Shampoo. Along with these algorithmic contributions, we provide a first in-depth analysis of how matrix scaling critically affects Shampoo convergence. On the practical side, our GPU-aware implementation achieves up to

5.6\times

faster optimizer steps compared to the well-optimized Distributed Shampoo, while Newton-DB attains the lowest validation perplexity per iteration among all tested methods. Our code is available at https://github.com/IST-DASLab/DASH.

Momentum

Related topics

Latest papers 132