Neural Network Training Dynamics

Follow this topic Ask about this topic

Momentum

41 papers in the last four weeks, up 273% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 349

All topics

How Much Orthogonalization Does Muon Need?
May 29, 2026Hua HuangNewton-Schulz Iteration Neural Network Optimization
NVIDIA
Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks
May 29, 2026Tianyu Pang, Vignesh Kothapalli, Shenyang Deng +3Deep Learning Optimization Neural Network Training Dynamics
Dartmouth College · Stanford University · Virginia Tech
Dynamics and Representation Structure of Local Approximations to Gradient-Based Learning in Linear Recurrent Neural Networks
May 29, 2026Ezekiel Williams, Alexandre Payeur, Guillaume LajoieRecurrent Neural Networks Linear RNNs
Department of Mathematics and Statistics, Universit´e de Montr´eal, Canada · Mila Qu´ebec, Canada · Imperial College London, United Kingdom
Gradient Descent with Large Step Size Restores Symmetry in Deep Linear Networks with Multi-Pathway
May 29, 2026Hee-Sung Kim, Sungyoon LeeEdge of Stability Symmetry Breaking
Department of Computer Science, Hanyang University, Seoul, Korea.
Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tail
May 29, 2026Konstantin Nikolaou, Jonas Scheunemann, Sven Krippendorf +2Representation Learning Neural Network Training Dynamics
Institute for Computational Physics, University of Stuttgart, Stuttgart, Germany · Department of Applied Mathematics and Theoretical Physics and Department of Physics, University of Cambridge, Cambridge, UK
A2SG:Adaptive and Asymmetric Surrogate Gradients for Training Deep Spiking Neural Networks
May 29, 2026Yechan Kang, Yongjin Kweon, Mingyeong Seo +8Neural Network Optimization Spiking Neural Networks
Center for Semiconductor Technology, Korea Institute of Science and Technology (KIST), Seoul, Republic of Korea · Department of Computer Science and Engineering, Korea University, Seoul, Republic of Korea · LIG Defense&Aerospace, Seongnam, Republic of Korea +1
Augmented Lagrangian Predictive Coding
May 29, 2026Jeffrey Seely, Julian GouldBackpropagation Predictive Coding
Sakana AI
Interpreting FCDNNs via RG on Exponential Family
May 29, 2026Fuzhou Gong, Zigeng XiaNeural Network Interpretability Neural Network Training Dynamics
Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China. · School of Mathematics, Northwest University, Xi’an 710127, China.
Supervised Training Rapidly Degrades Early Visual Cortex Alignment Across Biologically Plausible Learning Rules
May 28, 2026Nils LeuteneggerNeural Representation Geometry Predictive Coding
Independent Researcher, Switzerland
Unveiling Multi-regime Patterns in SciML: Distinct Failure Modes and Regime-specific Optimization
May 27, 2026Yuxin Wang, Yuanzhe Hu, Xiaokun Zhong +7Neural Network Optimization Scientific ML
Dartmouth College · University of California, San Diego · University of California, Berkeley +3
Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias
May 27, 2026Mohua Das, Pierfrancesco Beneventano, Shibshankar Dey +2Neural Network Generalization Neural Network Initialization
MIT · Northwestern University
The Hamilton-Jacobi Theory of Deep Learning
May 27, 2026Jose Marie Antonio Miñoza, Erika Fille T. Legara, Christopher P. MonterolaNeural Network Generalization Neural Network Training Dynamics
Center for AI Research PH · Asian Institute of Management
Gradient-Flow Optimization as Dynamic Random-Effects Inference: Testing and Early Stopping with Applications to Deep Learning
May 27, 2026Minhao Yao, Ruoyu Wang, Xihong Lin +2Early Stopping Neural Network Training Dynamics
Centre for Biomedical Data Science, Duke-NUS Medical School, National University of Singapore · Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA, USA · Institute of Natural Sciences, MOE-LSC, School of Mathematical Sciences, CMA-Shanghai, SJTU-Yale2026 Joint Center of Biostatistics and Data Science, Shanghai Jiao Tong University +1
Worker Disagreement Reveals Sharp Directions in Local SGD
May 26, 2026Tolga Dimlioglu, Kristi Topollai, Anna ChoromanskaNeural Network Training Dynamics Loss Landscape Geometry
New York University, New York, USA
When do complex-valued neural networks help? A study of representation, geometry, and optimization
May 26, 2026Ashutosh KumarNeural Network Optimization Complex-Valued Neural Networks
Owl Autonomous Imaging, Inc.
How the Optimizer Shapes Learned Solutions in Equivariant Neural Networks
May 26, 2026Teodor-Mihai Stupariu, Andrei ManolacheNeural Representation Geometry Neural Network Optimization
1Bitdefender, Romania · 2Tudor Vianu High School of Com · University of Stuttgart, Germany +1
SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Training
May 26, 2026Mohammed Adnan, Rohan Jain, Tom Jacobs +4Neural Network Optimization Neural Network Training Dynamics
University of Calgary · 2Vector Institute · 3CISPA Helmholtz +2
Mildly Overparameterized ReLU Networks on Orthogonal Data: Incremental Learning and Implicit Bias
May 26, 2026James Town, Etienne Boursier, Ben Lewis +2ReLU Neural Networks Neural Network Optimization
University of Warwick · INRIA LMO, Université Paris-Saclay
Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descent
May 26, 2026Chi-Ning Chou, Oscar Uzdelewicz, Neng-Chun Chiu +2Double Descent Neural Network Generalization
Center for Computational Neuroscience, Flatiron Institute, New York, NY, USA. · Department of Physics, Harvard University, Cambridge, MA, USA. · Kempner Institute, Harvard University, Allston, MA, USA. +1
Architecture-driven Shift: towards a lightweight selector for capturing the trends of logit shift
May 26, 2026Zhong Ye, Yu Hu, Ruilin TangContinual Learning Neural Network Training Dynamics
School of Computer Science and Technology Guangdong University of Technology Guangzhou, China · School of Computer Science and Engineering2026 South China University of Technology Guangzhou, China
The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training
May 26, 2026Hongtao Zhang, Wenjie Zhou, Chenxi Jia +2Language Model Pretraining Deep Learning Optimization
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China +1
Stochastic Estimation of the Layer-wise Hessian Trace for Monitoring Neural-network Training
May 25, 2026Maxim Bolshim, Alexander KugaevskikhNeural Network Memorization Stochastic Approximation
ITMO University, Saint Petersburg, Russia · ITMO University, St. Petersburg, Russia
Towards the Connection between Activation Sparsity and Flat Minima
May 25, 2026Ze Peng, Jian Zhang, Lei Qi +2Efficient Neural Network Inference Activation Sparsity
State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China · Institute of Brain-Machine Interface, Nanjing University, Nanjing 210023, China · School of Computer Science and Engineering, Southeast University, Nanjing 211189, China
Mitigating Gradient Pathology in PINNs through Aligned Constraint
May 24, 2026Yichen Luo, Peiyu Zhu, Dongxiao Hu +5Neural PDE Solvers Gradient Interference
Department of Information Science and Engineering, KTH Royal Institute of Technology, Stockholm, Sweden · School of Advanced Manufacturing and Robotics, Peking University, Beijing, China · School of Advanced Technology, Xi’an Jiaotong-Liverpool University, Suzhou, China +2
On the Impact of Class Imbalance on the Learning Dynamics of Deep Neural Networks:An Intuitive Insight
May 24, 2026Ismail B. Mustapha, Shafaatunnur Hasan, Sunday O. Olatunji +1Class-Imbalanced Learning Neural Network Generalization
Faculty of Computing Universiti Teknologi Malaysia Johor, Malaysia · Faculty of Computing Adejkunle Ajasin University Akungba-Akoko, Nigeria
A lift for input-convex neural net training
May 22, 2026Ali SiahkoohiNeural Network Optimization Input Convex Neural Networks
Department of Computer Science University of Central Florida
Move on Muon : A Hamiltonian probability gradient flow perspective of Muon optimizer
May 22, 2026Aratrika Mustafi, Soumya Mukherjee, Bharath K. SriperumbudurMomentum Methods Muon Optimizer
Department of Statistics, Pennsylvania State University
Not All Transitions Matter: Evidence from PPO
May 22, 2026Ajhesh BasnetProximal Policy Optimization Neural Network Training Dynamics
Department of Artificial Intelligence and Data Science KPR Institute of Engineering and Technology, Coimbatore
Non-normal spectral signatures of instability in neural network training dynamics
May 22, 2026Souvik GhoshDynamical Systems Neural Network Optimization
Department of Physics, National Sun Yat-sen University, Kaohsiung 80424, Taiwan
Feature Lottery? A Bifurcation Theory of Concept Emergence
May 22, 2026Fuming YangDynamical Systems Representation Learning
MIT

May 29, 2026, cs.LGJ/K move · Enter open · S save

How Much Orthogonalization Does Muon Need?

Hua Huang

NVIDIA

Newton-Schulz Iteration Neural Network Optimization Muon Optimizer Second-Order Optimization Neural Network Training Dynamics Matrix Factorization

Open paper AskPDF arXiv

Muon optimizers improve neural-network training by replacing ill-conditioned momentum updates with approximately semi-orthogonal updates. This motivates a practical question: how much orthogonalization does Muon actually require? We study this question using a relaxed cubic Newton--Schulz schedule derived directly for Muon's low precision singular value band. The resulting five-step cubic construction uses ten dominant matrix multiplications, compared with fifteen for five quintic Newton--Schulz iterations. The cubic schedule is not intended as a more accurate polar solver; instead, it is a principled low-cost variant that lets us probe the relation between polar accuracy, spectral shaping, and training quality. Across synthetic diagnostics, NanoGPT ablations, and training experiments on hybrid MoE/Mamba models, we find that training quality is not governed monotonically by polar-decomposition accuracy: truncated Polar Express, Muon-Jordan, cubic Newton--Schulz, and an explicit FP32 SVD polar factor can reach nearly indistinguishable final loss on GPT-2 Small, and cubic5 matches the Muon-Jordan quintic update within about

10^{-3}

validation loss on hybrid MoE/Mamba models with one billion to four billion parameters. These results support cubic5 as a practical low-cost Muon orthogonalization variant, with empirical evidence of training-quality parity in the settings tested.

Momentum

Related topics

Latest papers 349