Neural Network Optimization

Follow this topic Ask about this topic

Neural network optimization asks how to adjust a model’s parameters so it learns useful patterns efficiently and reliably. Common approaches include stochastic gradient descent and Adam, matrix-aware and spectral optimizers such as Muon, curvature-based methods, learning-rate schedules, and techniques that reduce memory or computation. Researchers also analyze loss landscapes, scaling laws, and initialization, and explore alternatives such as local or forward-only learning and analytic solutions for restricted models. A central tension is that gains depend on architecture, data, and training budget, while faster loss reduction does not necessarily improve generalization; connecting practical performance to reliable theory remains an open challenge.

Latest papers

332 papers37 papers in the last four weeks, up 28% on the four weeks before. 0.2% of all new papers.

The Anatomy of Implicit Bias: Information Allocation in Neural Network Training
Jul 8, 2026Zhang Gongyue, Wang Zhiyong, Liu Donghan +3Neural Network Optimization Implicit Bias
1State Key Laboratory of Robotics and Systems, Harbin Institute of Technology Shenzhen, Shenzhen, 518055, China.
Hybrid Least Squares/Gradient Descent Methods for MIONets
Jul 8, 2026Jun Choi, Chang-Ock Lee, Minam MoonGradient Descent Neural Network Optimization
Department of Mathematical Sciences, KAIST, Daejeon 34141, KOREA · Department of Mathematics, Korea Military Academy, Seoul 01805, KOREA
K-ABENA: K-Adaptive Backpropagation with Error-based N-exclusion Algorithm : (Compensated Loss-Based Sample Exclusion with Unbiased Gradient Estimation)
Jul 7, 2026Jean-Francois BonbhelBackpropagation Neural Network Optimization
NeuroSoft IA, Qu´ebec City, Canada · YekoElite University, Brazzaville, Republic of Congo
Hyperparameter Transfer in Graph Neural Networks
Jul 6, 2026Gage DeZoort, Boris HaninGraph Neural Networks Neural Network Optimization
Department of Operations Research and Financial Engineering · School of Engineering and Applied Science · Princeton University +1
Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks
Jul 5, 2026Shokhrukh Ibragimov, Arnulf JentzenDeep Learning Optimization Neural Network Optimization
Applied Mathematics: Institute for Analysis and Numerics, University of Münster, Germany · School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China
OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers
Jul 4, 2026Siyuan Li, Jiabao Pan, Yumou Liu +9LMO Methods Deep Learning Optimization
Shanghai Artificial Intelligence Laboratory · Westlake University · Shanghai University +4
Beyond Adam: SOAP and Muon for Faster, Label-Efficient Training of Machine Learning Interatomic Potentials
Jul 2, 2026Gil Harari, Yoel Zimmermann, Ola Tangen Kulseng +4Machine Learning Interatomic Potentials Neural Network Optimization
John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA · Robert Bosch LLC Research and Technology Center, Watertown, MA, USA
Born Discrete, Made Smooth: Variational Formulation of Shallow Neural Networks
Jul 2, 2026Matej Benko, Pierre Bousquet, Iwona Chlebicka +1Shallow Neural Networks Neural Network Generalization
Institute of Mathematics, Faculty of Mechanical Engineering, Brno University of Technology · Technická 2896/2, 616 69, Brno, Czech Republic · Université de Toulouse, INSA Toulouse, CNRS, IMT, F-31062 Toulouse Cedex 9, France +2
Muon as a Residual Connection
Jul 1, 2026Hao HuangNeural Network Optimization Muon Optimizer
College of Computer Science and Technology Zhejiang University Hangzhou, Zhejiang, China
Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimization
Jun 29, 2026Haoming Meng, Anton Sugolov, Vardan PapyanDeep Learning Optimization Neural Network Optimization
Department of Mathematics, University of Toronto, Toronto, Canada · Vector Institute, Toronto, Canada
GPU Parallelization Strategies for Forward and Backward Propagation in Shallow Neural Networks: A CUDA-Based Comparative Study
Jun 29, 2026Rania Zitouni, Nadine Bousdjira, Sarah Hasnaoui +2GPU Kernel Optimization GPU Acceleration
E´ cole Nationale Supe´rieure d’Informatique (ESI) · High Performance Computing (HPC)
A Stochastic--Geometric Theory of Scaling Laws in Grokking
Jun 29, 2026Róisín Luo, Christian Gagné, Jonas Ngnawé +2Neural Network Generalization Neural Network Optimization
Research Ireland – Centre for Research Training in AI · University of Galway · Universit´e Laval +2
Scalar Representations of Neural Network Training Dynamics
Jun 29, 2026Pedro Jiménez-González, Miguel C. Soriano, Lucas LacasaNeural Network Optimization Chaotic Dynamical Systems
Institute for Cross-Disciplinary Physics and Complex Systems (IFISC, CSIC-UIB), Campus UIB, 07122 Palma de Mallorca, Spain
FiRe: Frequency Reparameterization as a Preconditioner for Periodic Implicit Neural Representations
Jun 28, 2026Harinandan Shukla, Rajarshi Verma, Jitin SinglaImplicit Neural Representations Neural Network Optimization
Department of Computer Science and Engineering, IIT Roorkee · Department of Biosciences and Bioengineering, IIT Roorkee
Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks
Jun 28, 2026Tejas Pradeep ShirodkarDeep Learning Optimization Neural Network Optimization
IIIT, Hyderabad
Knowing in Advance When an Evolutionary Outer Loop Will Not Help: A Pre-Registered Cheap-Baseline Screening Rule
Jun 28, 2026Ramchand KumaresanNeural Network Optimization
Murai Labs
Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks
Jun 27, 2026Yuto Omae, Kazuki Sakai, Yohei Kakimoto +3Gradient Descent Neural Network Optimization
Nihon University (Japan) · National Institute of Technology, Nagaoka College (Japan) · Tokyo City University (Japan)
Singular Learning and Occam's Razor in Deep Monomial Networks
Jun 26, 2026Kathlén Kohn, Giovanni Luca Marchetti, Farhan Shabir +2Neural Network Optimization Singular Learning Theory
KTH Stockholm · No Affiliation · Umeå University +1
Aurora: A Leverage-Aware Spectral Optimizer
Jun 26, 2026Alec Dewulf, Dhruv Pai, Li Yang +2Neural Network Optimization Muon Optimizer
Tilde Research
Hierarchical Muon: Tiled Newton-Schulz Updates for Efficient Muon Optimization
Jun 25, 2026Ziyuan Tang, Tianshi Xu, Yousef Saad +1Newton-Schulz Iteration Neural Network Optimization
Department of Computer Science and Engineering, University of Minnesota, Minneapolis · Department of Mathematics, Emory University, Atlanta, GA 30322
Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors
Jun 24, 2026Alexander Hägele, Alejandro Hernández-Cano, Atli Kosson +1Neural Network Optimization Neural Network Training Dynamics
Machine Learning and Optimization Lab, EPFL
Neural Network Quantization by Learning Low-Loss Subspaces
Jun 23, 2026Vladimir Protsenko, Mikhalina Kharkevich, Alexander Vashchilko +1Neural Network Optimization Neural Network Quantization
Huawei, Montreal, Canada
Convergence of Gradient Descent for General Neural Network Architectures Beyond the NTK Regime
Jun 22, 2026Yuqing WangGradient Descent Neural Network Optimization
Johns Hopkins University
SOAP-Bubbles: Structured Weight Uncertainty for Neural Networks
Jun 22, 2026Adrian Robert Minut, Nico Daheim, Marco Miani +3Bayesian Neural Networks Neural Posterior Estimation
Sapienza University of Rome, Rome, Italy · Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt · Technical University of Denmark, Lyngby, Denmark +3
Breaking chains with trees: Deep learning with $\mathcal{O}(\log N)$ parallel time complexity
Jun 19, 2026Neeraj Mohan Sushma, Aditya Nagarsekar, Cabrel Teguemne Fokam +4Deep Learning Optimization Neural Network Optimization
Center for Cognitive Interaction Technology, Bielefeld University, Germany · Ruhr Universität Bochum, Germany · Birla Institute of Technology and Science, Pilani, Goa Campus +1
Towards Robust Training in NNGPT AutoML Pipeline: A Loss-Optimizer Pairing Selection Study
Jun 18, 2026Anton Abramochkin, Radu Timofte, Dmitry IgnatovDeep Learning Optimization Neural Network Optimization
Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany
Convex training of Lipschitz-regularized shallow neural networks
Jun 17, 2026Chao Yin, Antoine Lesage-LandryAdversarial Training Shallow Neural Networks
Polytechnique Montr´eal, GERAD & Mila, Montr´eal, QC, Canada, H3T 1J4.
Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale
Jun 17, 2026Tejas Pradeep Shirodkar, P. J. NarayananInformation Geometry Transformer
IIIT, Hyderabad
MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization
Jun 16, 2026Da Chang, Ganzhao YuanNeural Network Optimization Stochastic Optimization Convergence
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · Pengcheng Laboratory · University of Chinese Academy of Sciences +1
Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization
Jun 15, 2026Kaiyue Wen, Xingyu Dang, Kaifeng Lyu +2Language Model Pretraining Weight Decay
†Stanford University. · ‡Princeton University. · §Tsinghua University.

Jul 8, 2026J/K move · Enter open · S save

The Anatomy of Implicit Bias: Information Allocation in Neural Network Training

Zhang Gongyue, Wang Zhiyong, Liu Donghan, Ren Weihong+2

1State Key Laboratory of Robotics and Systems, Harbin Institute of Technology Shenzhen, Shenzhen, 518055, China.

Neural Network Optimization Implicit Bias Neural Network Training Dynamics

Open paper AskPDF arXiv

Implicit bias is usually explained as the preference of an optimization process for certain final solutions and their geometry. This view helps explain where a model finally stops. It gives less direct explanation of how this bias is formed during training. This paper proposes a training-time information allocation view. Under this view, optimization forms a writing pattern for error signals across parameter paths, coordinate channels, and sample regions. This paper builds a set of observable allocation diagnostics. These diagnostics include gradient demand, actual update injection, coordinate gain induced by exponential moving averages, channel-level update ratios, and sample-wise loss distributions. To separate training progress from internal allocation, this paper introduces a collapse--persistence analysis. Under matched training loss, if external loss statistics collapse but internal allocation ratios remain separated, then the factor changes the internal allocation of the training signal. Overall, this paper extends the analysis of implicit bias from final-solution geometry to training-time signal allocation. The main claim is that implicit bias is not only reflected by the final solution. It is also reflected by which parameter paths, coordinate channels, and sample regions receive the error signal first and more strongly during training. Based on this view, this paper places different training factors into a unified information-allocation diagnostic framework. The framework gives a mechanism-level explanation of training-time implicit bias. It also provides a basis for future optimization methods that control training progress and signal allocation separately.