Neural Network Optimization

Follow this topic Ask about this topic

Neural network optimization asks how to adjust a model’s parameters so it learns useful patterns efficiently and reliably. Common approaches include stochastic gradient descent and Adam, matrix-aware and spectral optimizers such as Muon, curvature-based methods, learning-rate schedules, and techniques that reduce memory or computation. Researchers also analyze loss landscapes, scaling laws, and initialization, and explore alternatives such as local or forward-only learning and analytic solutions for restricted models. A central tension is that gains depend on architecture, data, and training budget, while faster loss reduction does not necessarily improve generalization; connecting practical performance to reliable theory remains an open challenge.

Latest papers

332 papers37 papers in the last four weeks, up 28% on the four weeks before. 0.2% of all new papers.

CacheMuon: Using Temporal Preconditioning To Approximate Polar Factor
Jun 15, 2026Bishnu Dev, Sushil Bohara, Martin Takáč +1Momentum Methods Neural Network Optimization
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Mohamed bin Zayed University of Artificial Intelligence
Z-Plane Neural Networks: Bounded Geometric Activation Replaces ReLU and LayerNorm
Jun 14, 2026Sungwoo Goo, Hwi-yeol Yun, Sangkeun JungNeural Representation Geometry Neural Network Optimization
College of Pharmacy, Chungnam National University, Daejeon, Republic of Korea · Department of Computer Science & Engineering, Chungnam National University, Daejeon, Republic of Korea
Can Neural Networks Achieve Optimal Computational-statistical Tradeoff? An Analysis on Single-Index Model
Jun 13, 2026Siyu Chen, Beining Wu, Miao Lu +2Representation Learning Single-Index Models
Department of Statistics and Data Science, Yale University · Department of Statistics, University of Chicago · Department of Management Science and Engineering, Stanford University +1
A Projection-Based Surrogate Gradient Interpretation for Neural Codec Wrappers
Jun 12, 2026Esteban Pesnel, Julien Le Tanou, Michael Ropert +2Neural Network Optimization Neural Video Compression
COMPACT · MediaKind, Rennes, France · INRIA, Rennes, France
Zeta: Dual Whitening for Matrix Optimization via Coordinate-Adaptive Preconditioning
Jun 12, 2026Kaiwen Chen, Shuhai Zhang, Zimo Liu +7Neural Network Optimization Feature Whitening
South China University of Technology · AIGCode · Hong Kong Baptist University
Muon $^p$ : Muon with Fractional Spectral Powers
Jun 11, 2026Yihe Dong, Will SawinNeural Network Optimization Muon Optimizer
Princeton University
Different Layers, Different Manifolds: Module-Wise Weight-Space Geometry in Transformer Optimization
Jun 11, 2026Kirato YoshiharaStiefel Manifold Optimization Neural Network Optimization
School of Engineering Science, The University of Osaka, Osaka, Japan.
Overcoming Rank Collapse in Feedback Alignment
Jun 9, 2026Gauthier Boeshertz, Razvan Pascanu, Claudia ClopathBackpropagation Neural Network Optimization
Department of Bioengineering Imperial College London · Mila
Conservation Laws from Data Symmetry in Neural Networks
Jun 9, 2026Jakob Galley, Vahid Shahverdi, Axel FlinthNeural Network Optimization Neural Network Training Dynamics
Umeå University, Umeå, Sweden
A Theory on Flow Matching with Neural Networks
Jun 8, 2026Yihan He, Qishuo Yin, Yuan Cao +2Neural Network Generalization Flow Matching
Princeton University, Princeton, NJ, USA. · The University of Hong Kong, Hong Kong. · Northwestern University, Evanston, IL, USA.
Optimizing Energy-based Neural Network Training with Coherent Ising Machine
Jun 8, 2026Chen-Rui Fan, Bo Lu, Zhi-Hong Zhang +3Equilibrium Propagation Neural Network Optimization
School of Artiﬁcial Intelligence, Beijing Normal University, Beijing, 100875, China. · Laboratory for Advanced Computing and Intelligence Engineering, Information Engineering University, Zhengzhou, 450001, China. · China Mobile (Suzhou) Software Technology Company Limited, Suzhou, 215163, China.
Understanding Quantization-Aware Training: Gradients at Quantized Weights Bias to the Low-Loss Basin
Jun 8, 2026Hanyang Li, Jianhao Ma, Ying CuiNeural Network Optimization Straight-Through Estimator
Department of IEOR, University of California, Berkeley · Department of Statistics and Data Science, University of Pennsylvania
PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training
Jun 4, 2026Senmiao Wang, Tiantian Fang, Haoran Zhang +4Language Model Pretraining Transformer
The Chinese University of Hong Kong, Shenzhen, China · Google LLC, Mountain View, CA, United States · Shenzhen International Center for Industrial and Applied Mathematics, Shenzhen2026 Research Institute of Big Data +1
On the training of physics-informed neural operators for solving parametric partial differential equations
Jun 4, 2026Nanxi Chen, Chuanjie Cui, Airong Chen +2Neural Network Optimization Gradient Interference
College of Civil Engineering, Tongji University, Shanghai 200092 · Department of Engineering Science, University of Oxford, Oxford OX1 3PJ · Institute for Foundations of Data Science, Yale University, New Haven, CT 06520
Multi-Column RBF Neural Network Using Adaptive and Non-Adaptive Particle Swarm Optimization
Jun 3, 2026Ammar Hoori, Yuichi MotaiMetaheuristic Optimization Neural Network Optimization
Department of Biomedical Engineering, Case Western Reserve University, Cleveland, OH 44106, USA · Department of Electrical and Computer Engineering, Virginia Commonwealth University, Richmond, VA 23284, USA
Curvature-aware dynamic precision approach for physics-informed neural networks
Jun 3, 2026Yingjie Shao, Ioannis N. Athanasiadis, George van Voorn +1Neural PDE Solvers Neural Network Optimization
Mathematical & Statistical Methods Group (Biometris), Wageningen University & Research, Wageningen, 6700AA, The Netherlands · Artificial Intelligence Group, Wageningen University & Research, Wageningen, 6700AA, The Netherlands
Why Muon Outperforms Adam: A Curvature Perspective
Jun 3, 2026Shuche Wang, Fengzhuo Zhang, Jiaxiang Li +2Loss Landscape Sharpness Neural Network Optimization
National University of Singapore · Yale University · University of Minnesota
When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks
Jun 3, 2026Berk Tinaz, Changzhi Xie, Mahdi SoltanolkotabiReLU Neural Networks Neural Network Optimization
Department of Electrical and Computer Engineering University of Southern California Los Angeles, CA, USA · Department of Computer Science University of Southern California Los Angeles, CA, USA
A Geometric Characterization of the Stationary Plateau for Two-Layer Neural Networks
Jun 3, 2026Tian Ding, Dawei Li, Ruoyu SunTwo-Layer Neural Networks Neural Network Optimization
Shenzhen International Center of Industrial and Applied Mathematics, Shenzhen Research Institute of Big Data, China; Shenzhen Loop Area Institute, China; AutoKernel, China. · Shenzhen Loop Area Institute, China · AutoKernel, China +3
FFR: Forward-Forward Learning for Regression
Jun 2, 2026Xinyang Liu, Xuanyu Liang, Shiqi Ding +4Neural Network Optimization Feedforward Neural Networks
University of Bristol · University College London · University of Cambridge
Unlocking Feature Learning in Gated Delta Networks at Scale
Jun 2, 2026Yifeng Liu, Quanquan GuNeural Network Optimization Hyperparameter Transfer
University of California, Los Angeles
Beyond $\ell_2$ -norm and $\ell_\infty$ -norm: A Curvature-Inspired $\ell_p$ -Norm Scheme for Deep Neural Networks
Jun 1, 2026Jianhao Xu, Zhuang YangNeural Network Optimization Nonconvex Stochastic Optimization
School of Computer Science and Technology, Soochow University, Suzhou, 215006, China
Fast Generalization after Interpolation via Critically Damped Momentum Optimization
Jun 1, 2026Luca Muscarnera, Silas Ruhrberg Estévez, Yuanzhang Xiao +1Neural Network Generalization Momentum Methods
University of Cambridge · University of Hawaii at Manoa
Exploiting weight-space symmetries for approximating curvature
May 30, 2026Artem Artemev, Rui Xia, Benjamin M. Boyd +4Neural Network Optimization Second-Order Optimization
MediaTek, Cambridge, UK · University of Cambridge, Cambridge, UK · Vector Institute, Toronto, Canada
How Much Orthogonalization Does Muon Need?
May 29, 2026Hua HuangMuon Optimizer Matrix Factorization
NVIDIA
A2SG:Adaptive and Asymmetric Surrogate Gradients for Training Deep Spiking Neural Networks
May 29, 2026Yechan Kang, Yongjin Kweon, Mingyeong Seo +8Neural Network Optimization Spiking Neural Networks
Center for Semiconductor Technology, Korea Institute of Science and Technology (KIST), Seoul, Republic of Korea · Department of Computer Science and Engineering, Korea University, Seoul, Republic of Korea · LIG Defense&Aerospace, Seongnam, Republic of Korea +1
LLMs Without Deep Neural Networks: New Architecture, Benefits and Case Study
May 28, 2026Vincent GranvilleNeural Network Optimization Radial Basis Function Networks
CAIO
On the Optimizer Dependence of Neural Scaling Laws
May 28, 2026Vansh Ramani, Shourya Vir JainNatural Gradient Methods Neural Network Optimization
Department of Computer Science and Engineering, Indian Institute of Technology Delhi · Department of Electrical Engineering, Indian Institute of Technology Delhi
Unveiling Multi-regime Patterns in SciML: Distinct Failure Modes and Regime-specific Optimization
May 27, 2026Yuxin Wang, Yuanzhe Hu, Xiaokun Zhong +7Loss Landscape Neural Network Optimization
Dartmouth College · University of California, San Diego · University of California, Berkeley +3
Learning High-Dimensional Parity Functions with Product Networks using Gradient Descent
May 27, 2026Guillaume Larue, Louis-Adrien Dufrène, Quentin Lampin +2Boolean Function Learning Neural Network Optimization
Orange Research, Meylan, France · Télécom Paris, Institut Polytechnique de Paris, Palaiseau, France · Télécom Paris, Institut Polytechnique de Paris, Palaiseau, France.

Jun 15, 2026J/K move · Enter open · S save

CacheMuon: Using Temporal Preconditioning To Approximate Polar Factor

Bishnu Dev, Sushil Bohara, Martin Takáč, Samuel Horváth

Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Mohamed bin Zayed University of Artificial Intelligence

Momentum Methods Neural Network Optimization Muon Optimizer

Open paper AskPDF arXiv

Muon is an optimizer that computes updates using the polar factor of the momentum matrix and has shown strong empirical performance across a range of training settings. A key component of Muon is the Newton-Schulz iteration used to compute this polar factor. Although this avoids the cost of an exact singular value decomposition, it remains expensive in practice because it is applied at every optimization step. At the same time, the momentum matrix changes smoothly over training, suggesting strong temporal correlation in the corresponding polar factors. In this paper, we exploit this structure and propose CacheMuon, a temporal preconditioning method that reuses information from previous optimization steps to approximate the polar factor at the current step. This reduces redundant orthogonalization computation across iterations. We analyze CacheMuon as an inexact Muon update, with error controlled by fresh-solver error and cache staleness. Empirically, CacheMuon provides a controllable quality-efficiency frontier: conservative thresholds closely match fresh Muon on language-model and vision training while reducing orthogonalization FLOPs, whereas more aggressive thresholds yield larger arithmetic savings at the cost of modest validation-quality degradation.