Deep Learning Optimization

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, up 133% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 132

All topics

The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training
May 26, 2026Hongtao Zhang, Wenjie Zhou, Chenxi Jia +2Language Model Pretraining Deep Learning Optimization
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China +1
Reparametrizing Shampoo and SOAP for Subspace Basis Updates and BFloat16 Storage
May 25, 2026Alan Milligan, Zikun Xu, Simon Lacoste-Julien +2Deep Learning Optimization Memory-Efficient Optimization
Mila & Université de Montréal · Microsoft · Concordia University & Mila +1
BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training
May 25, 2026Zili Zhang, Chengxu Yang, Shenglong Zhang +8Deep Learning Optimization Multimodal Large Language Models
1Peking University · †Work done during internships at Xiaohongshu, Inc. · 2Independent Researcher +2
EMA-Nesterov: Stabilizing Nesterov's Lookahead for Accelerated Deep Learning Optimization
May 25, 2026Chung-Yiu Yau, Dawei Li, Athanasios Glentis +3Deep Learning Optimization Gradient Descent
University of Minnesota · 2Amazon AGI · 3The Chinese University of Hong Kong
PILOT: Policy-Informed Learned Optimization for Adaptive Deep Network Training
May 23, 2026Sattam Altuuaim, Lama Ayash, Muhammad Mubashar +1Deep Learning Optimization Learning to Optimize
King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · University of Strathclyde, Glasgow, Scotland
Momentum Streams for Optimizer-Inspired Transformers
May 23, 2026Jingchu Gai, Nai-Chieh Huang, Jiayun WuDeep Learning Optimization Transformer
Carnegie Mellon University
Recursive Block-Diagonal Coupling for Resource-Efficient Training of Vision Models
May 22, 2026Maxim Henry, Adrien Deliège, Sébastien Piérard +1Deep Learning Optimization Efficient ViTs
Montefiore Institute, University of Liège, Liège, Belgium
Anytime Training with Schedule-Free Spectral Optimization
May 21, 2026Anuj Apte, Pranav Deshpande, Niraj Kumar +2Deep Learning Optimization Learning Rate Scheduling
Global Technology Applied Research, JPMorganChase New York, NY 10001, USA
AMUSE: Anytime Muon with Stable Gradient Evaluation
May 21, 2026Jueun Kim, Baekrok Shin, Jihun Yun +3Deep Learning Optimization Muon Optimizer
KAIST · KRAFTON · Seoul National University, KRAFTON
An Improved Adaptive PID Optimizer with Enhanced Convergence and Stability for Deep Learning
May 21, 2026Saurabh Saini, Kapil Ahuja, Thomas Wick +1Deep Learning Optimization Stochastic Optimization Convergence
Department of Computer Science & Engineering, Indian Institute of Technology Indore, India. · Leibniz Universität Hannover, Institut für Angewandte Mathematik, Germany. · National Remote Sensing Centre, Indian Space Research Organisation, India.
Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws
May 20, 2026Nandan Kumar Jha, Brandon ReagenDeep Learning Optimization Language Model Scaling Laws
New York University
HORST: Composing Optimizer Geometries for Sparse Transformer Training
May 20, 2026Tom Jacobs, Rohan Jain, Rebekka BurkholzDeep Learning Optimization Transformer
CISPA Helmholtz Center for Information Security
Ada2MS: A Hybrid Optimization Algorithm Based on Exponential Mixing of Elementwise and Global Second-Moment Estimates
May 19, 2026Meng Zhu, Quan Xiao, Weidong MinDeep Learning Optimization Momentum Methods
School of Information Management and Mathematics, Jiangxi University of Finance and Economics, Nanchang 330032, China · School of Mathematics and Computer Science, Nanchang University, Nanchang 330031, China · Institute of Metaverse, Nanchang University, Nanchang 330031, China +1
SMA-DP: Spectral Memory-Aware Differential Privacy for Deep Learning
May 19, 2026Mohammad Partohaghighi, Roummel MarciaDeep Learning Optimization Differentially Private Stochastic Gradient Descent
Department of Electrical Engineering and Computer Science University of California, Merced Merced, CA, USA · Department of Applied Mathematics University of California, Merced Merced, CA, USA
Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases
May 19, 2026Jingwen Liu, Ezra Edelman, Surbhi Goel +1Deep Learning Optimization Statistical Learning Theory
Columbia University · University of Pennsylvania · Kempner Institute, Harvard University
StableGrad: Backward Scale Control without Batch Normalization
May 19, 2026Jose I. Mestre, Alberto Fernández-Hernández, Cristian Pérez-Corral +2Deep Learning Optimization Neural Network Training Dynamics
Universitat Politècnica de València · Universitat Jaume I
MiMuon: Mixed Muon Optimizer with Improved Generalization for Large Models
May 19, 2026Feihu Huang, Yuning Luo, Songcan ChenNeural Network Generalization Deep Learning Optimization
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China · College of Design and Engineering, National University of Singapore, Singapore
Replacement Learning: Training Neural Networks with Fewer Parameters
May 19, 2026Yuming Zhang, Peizhe Wang, Tianyang Han +5Deep Learning Optimization Neural Network Compression
The University of Hong Kong, Hong Kong, China · Southeast University, Nanjing, China
Differentiable Optimization Layers for Guaranteed Fairness in Deep Learning
May 16, 2026David Troxell, Noah Roemer, Guido MontúfarDeep Learning Optimization Algorithmic Fairness
Department of Statistics & Data Science, University of California, Los Angeles, USA · Department of Mathematics, University of California, Los Angeles, USA · Max Planck Institute for Mathematics in the Sciences, Leipzig, Germany
Does Weight Decay Enhance Training Stability?
May 15, 2026Marius Saether, Amir Kolic, Tomaso Poggio +1Edge of Stability Deep Learning Optimization
NTNU, MIT · MIT
Accelerated Gradient Descent for Faster Convergence with Minimal Overhead
May 15, 2026Manuel Graca, L. Miguel Silveira, Arlindo Oliveira +1Deep Learning Optimization Gradient Descent
IST, ULisboa Lisbon, Portugal · Old Dominion University Norfolk, VA USA
Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered
May 15, 2026Sijia Liu, Yicheng Lang, Soumyadeep Pal +6Deep Learning Optimization Zeroth-Order Optimization
1OPTML Lab, Michigan State University, USA · 2Lawrence Livermore National Laboratory, USA
Rethinking Neural Network Learning Rates: A Stackelberg Perspective
May 15, 2026Sihan Zeng, Sujay Bhatt, Sumitra GaneshDeep Learning Optimization Neural Network Optimization
JPMorgan AI Research, United States.
Novel Dynamic Batch-Sensitive Adam Optimiser for Vehicular Accident Injury Severity Prediction
May 14, 2026Daniel Asare Kyei, Alimatu Saadia-Yussiff, Maame G. Asante-Mensah +3Class-Imbalanced Learning Deep Learning Optimization
Department of Computer Science and Information Technology, University of Cape Coast, Cape Coast, Ghana
Beyond What to Select: A Plug-and-play Oscillatory Data-Volume Scheduling for Efficient Model Training
May 14, 2026Suorong Yang, Hanqi Zhu, Hai Gan +4Deep Learning Optimization Data Selection
1National University of Singapore · 2Nanjing University · 3Hokkaido University +1
Efficient and provably convergent end-to-end training of deep neural networks with linear constraints
May 12, 2026Zonglin Yang, Zhexuan Gu, Yancheng YuanDeep Learning Optimization Backpropagation
Department of Applied Mathematics, The Hong Kong Polytechnic University, Hung Hom, Hong Kong
Error whitening: Why Gauss-Newton outperforms Newton
May 11, 2026Maricela Best McKay, Nathan P. Lawrence, Brian Wetton +1Gauss-Newton Optimization Deep Learning Optimization
University of British Columbia · University of California, Berkeley
Muon is Not That Special: Random or Inverted Spectra Work Just as Well
May 11, 2026Zakhar Shumaylov, Nathaël Da Costa, Peter Zaika +6Stochastic Optimization Deep Learning Optimization
University of Cambridge · University of Tübingen · University of Oxford
Optimistic Dual Averaging Unifies Modern Optimizers
May 11, 2026Thomas Pethick, Wanyun Xie, Roman Machacek +1Stochastic Optimization Deep Learning Optimization
Independent Researcher · EPFL (LIONS) · University of Bern
Elucidating Representation Degradation Problem in Diffusion Model Training
May 11, 2026Zhipeng Yao, Dazhou Li, Zitong Zhang +6Deep Learning Optimization Diffusion Model Acceleration
University of Louisville · Shenyang University of Chemical Technology · Florida International University +2

May 26, 2026, cs.LGJ/K move · Enter open · S save

The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training

Hongtao Zhang, Wenjie Zhou, Chenxi Jia, Wei Chen+1

State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China+1

Language Model Pretraining Deep Learning Optimization Neural Network Training Dynamics

Open paper AskPDF arXiv

Large language model pre-training typically exhibits a two-phase trajectory: a fast initial loss drop followed by a prolonged slow improvement. We identify an underlying spectral phenomenon, Stability of Singular Distribution (SoSD), where the trace-normalized singular value spectrum stabilizes early, even as parameter matrices continue to evolve. We demonstrate that synchronization between SoSD and the slow-descent regime is widely observed across diverse architectures (GPT-2, LLaMA) and settings, including various schedules (Step-wise, WSD, Cosine Decay), weight decays, and optimizers (AdamW, Muon). By analyzing a simplified Transformer, we prove that growing weight norms inevitably precipitate an early SoSD threshold, after which the rate of loss decrease becomes theoretically bounded by the variation in the singular distribution. We further interpret strategies like WSD and Muon through their ability to modulate the SoSD scale, offering a spectral lens for understanding efficient pre-training dynamics.

Momentum

Related topics

Latest papers 132