Neural Network Optimization

Ask about this topic

Overview

Neural network optimization asks how to adjust a model’s parameters so it learns useful patterns efficiently and reliably. Common approaches include stochastic gradient descent and Adam, matrix-aware and spectral optimizers such as Muon, curvature-based methods, learning-rate schedules, and techniques that reduce memory or computation. Researchers also analyze loss landscapes, scaling laws, and initialization, and explore alternatives such as local or forward-only learning and analytic solutions for restricted models. A central tension is that gains depend on architecture, data, and training budget, while faster loss reduction does not necessarily improve generalization; connecting practical performance to reliable theory remains an open challenge.

Latest papers

CardsList

332 papers37 papers in the last four weeks, up 28% on the four weeks before. 0.2% of all new papers.

  1. EdgeDiT: Hardware-Aware Diffusion Transformers for Efficient On-Device Image Generation

    Mar 30, 2026Sravanth Kodavanti, Manjunath Arveti, Sowmya Vajrala +2Diffusion TransformerDiffusion-Based Image Generation

    Samsung Research Institute Bangalore, India

  2. MuonEq: Balancing Before Orthogonalization with Lightweight Equilibration

    Mar 30, 2026Da Chang, Qiankun Shi, Lvgang Zhang +5Neural Network OptimizationMuon Optimizer

    Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · Pengcheng Laboratory · University of Chinese Academy of Sciences +4

  3. Preconditioned Attention: Enhancing Efficiency in Transformers

    Mar 28, 2026Hemanth SaratchandranAttention MechanismsNeural Network Optimization

    Australian Institute for Machine Learning (AIML), Adelaide University

  4. Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory

    Mar 27, 2026Juno Kim, Eshaan Nichani, Denny Wu +2Spectral OptimizationNeural Network Optimization

    UC Berkeley · Princeton University · New York University +1

  5. Second-Order, First-Class: A Composable Stack for Curvature-Aware Training

    Mar 26, 2026Mikalai Korbit, Mario ZanonCurvature-Aware OptimizationDeep Learning Optimization

    IMT School for Advanced Studies Lucca Lucca, Italy

  6. Improving Infinitely Deep Bayesian Neural Networks with Nesterov's Accelerated Gradient Method

    Mar 26, 2026Chenxu Yu, Wenqi FangBayesian Neural NetworksNeural Network Optimization

    Shenzhen Institutes of Advanced Technology Chinese Academy of Sciences Shenzhen, P.R.China

  7. A Systematic Empirical Study of Grokking: Depth, Architecture, Activation, and Regularization

    Mar 26, 2026Shalima Binta Manir, Anamika Paul RupaGrokkingNeural Network Training Dynamics

    Howard University

  8. Local learning for stable backpropagation-free neural network training towards physical learning

    Mar 25, 2026Yaqi Guo, Fabian Braun, Bastiaan Ketelaar +3Physical Neural NetworksNeural Network Optimization

    Department of Materials Science and Engineering, Delft University of Technology, Mekelweg 2, Delft, 2628CD, The Netherlands · Department of Precision and Microsystems Engineering, Delft University of Technology, Mekelweg 2, Delft, 2628CD, The Netherlands · Department of Intelligent Systems, Delft University of Technology, Van Mourik Broekmanweg 6, Delft 2628XE, The Netherlands

  9. Stochastic Dimension Zeroth-Order Estimator: Stable and Memory-Efficient Training of PINNs

    Mar 25, 2026Zhangyong Liang, Huanhuan GaoZeroth-Order OptimizationNeural Network Optimization

    National Center for Applied Mathematics, Tianjin University, Tianjin 300072, China. · School of Mechanical and Aerospace Engineering, Jilin University, Changchun 130025, China

  10. Unveiling Hidden Convexity in Deep Learning: a Sparse Signal Processing Perspective

    Mar 25, 2026Emi Zeger, Mert PilanciReLU Neural NetworksDeep Learning Optimization

    Department of Electrical Engineering, Stanford University

  11. On the Role of Batch Size in Stochastic Conditional Gradient Methods

    Mar 22, 2026Rustem Islamov, Roman Machacek, Aurelien Lucchi +3Stochastic OptimizationNeural Network Optimization

    University of Basel, Switzerland · MBZUAI, UAE · University of Bern, Switzerland +2

  12. Rethinking Plasticity in Deep Reinforcement Learning

    Mar 22, 2026Zhiqiang HeLoss of PlasticityNon-Stationary RL

    The University of Electro-Communications, Tokyo, Japan

  13. RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization

    Mar 20, 2026Shenyang Deng, Zhuoli Ouyang, Tianyu Pang +4Neural Network OptimizationMuon Optimizer

    Dartmouth College · International Computer Science Institute · University of California, Berkeley +1

  14. Does This Gradient Spark Joy?

    Mar 20, 2026Ian OsbandBandit LearningPolicy Gradient

    Google DeepMind

  15. Linearized Bregman Iterations for Sparse Spiking Neural Networks

    Mar 17, 2026Daniel Windhager, Bernhard A. Moser, Michael LunglmayrSpiking Neural NetworksSparse Neural Networks

    Silicon Austria Labs, Linz, Austria · Software Competence Center Hagenberg, Hagenberg, Austria · Institute of Signal Processing, Johannes Kepler University, Linz, Austria

  16. Prior-Informed Neural Network Initialization: A Spectral Approach for Function Parameterizing Architectures

    Mar 17, 2026David Orlando Salazar Torres, Diyar Altinses, Andreas SchwungNeural Network InitializationNeural Network Optimization

    Department of Automation Technology and Learning Systems, South Westphalia University of Applied Sciences, Soest, Germany

  17. Align Forward, Adapt Backward: Closing the Discretization Gap in Logic Gate Networks

    Mar 14, 2026Youngsung KimNeural Network OptimizationDifferentiable Logic Gate Networks

    Department of Artificial Intelligence and Department of Electrical and Computer Engineering, Inha University, Incheon, Korea.

  18. Scaling Laws and Pathologies of Single-Layer PINNs: Network Width and PDE Nonlinearity

    Mar 13, 2026Faris ChaudhryPhysics-Informed MLNeural Scaling Laws

    Department of Computer Science, Imperial College London

  19. SCORE: Replacing Layer Stacking with Contractive Recurrent Depth

    Mar 11, 2026Guillaume GodinResidual LearningRecurrent Neural Networks

    1Osmo Labs PBC New York, USA

  20. What do near-optimal learning rate schedules look like?

    Mar 11, 2026Hiroki Naganuma, Atish Agarwala, Priya Kasimbeg +1Learning Rate SchedulingNeural Network Optimization

    Mila · Université de Montréal · Google DeepMind

  21. On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfer

    Mar 10, 2026Ruihan Xu, Jiajin Li, Yiping LuNeural Network OptimizationHyperparameter Transfer

    University of Chicago · Sauder School of Business, University of British Columbia · Industrial Engineering & Management Science, Northwestern University

  22. Combining Adam and its Inverse Counterpart to Enhance Generalization of Deep Learning Optimizers

    Mar 7, 2026Tao Shi, Liangming Chen, Long Jin +1Neural Network OptimizationAdaptive Gradient Methods

    School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China · Helen and John C. Hartmann Department of Electrical and Computer Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA

  23. Weak-SIGReg: Covariance Regularization for Stable Deep Learning

    Mar 6, 2026Habibullah AkbarNeural Network OptimizationRepresentation Collapse

    Kreasof AI, Jakarta, Indonesia

  24. Multilevel Training for Kolmogorov Arnold Networks

    Mar 5, 2026Ben S. Southworth, Jonas A. Actor, Graham Harper +1Kolmogorov-Arnold NetworksNeural Network Optimization

    Theoretical Division, Los Alamos National Laboratory, USA. · Center for Computing Research, Sandia National Laboratories, USA.

  25. Exploiting Subgradient Sparsity in Max-Plus Neural Networks

    Mar 4, 2026Ikhlas Enaieh, Olivier FercoqNeural Network OptimizationGradient Descent

    LTCI, S2A · Image, Data, Signal Department (IDS), Telecom Paris, Institut Polytechnique de Paris, France. · S2A, LTCI

  26. Same Error, Different Function: The Optimizer as an Implicit Prior in Financial Time Series

    Mar 3, 2026Federico Vittorio Cortesi, Giuseppe Iannone, Giulia Crippa +2Neural Network OptimizationFinancial Time Series

    Massachusetts Institute of Technology, Cambridge, MA, USA · Princeton University, Princeton, NJ, USA.

  27. Geodesic Gradient Descent: A Generic and Learning-rate-free Optimizer on Objective Function-induced Manifolds

    Feb 28, 2026Liwei Hu, Guangyao Li, Wenyong Wang +2Neural Network OptimizationRiemannian Optimization

    the School of Information Science and Engineering, Hebei University of Science and Technology, Shijiazhuang 050091, China · the School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China · the International Institute of Next Generation Internet, Macau University of Science and Technology, Macau 519020

  28. MSINO: Curvature-Aware Sobolev Optimization for Manifold Neural Networks

    Feb 26, 2026Suresan ParethRiemannian Neural NetworksNeural Network Optimization

  29. When Learning Hurts: Fixed-Pole RNN for Real-Time Online Training

    Feb 25, 2026Alexander Morgan, Ummay Sumaya Khan, Lingjia Liu +1Recurrent Neural NetworksNeural Network Training Dynamics

    dept. of ECE, Virginia Tech, Alexandria, VA, USA

  30. Neural network optimization strategies and the topography of the loss landscape

    Feb 24, 2026Jianneng Yu, Alexandre V. MorozovNeural Network OptimizationStochastic Gradient Descent

    Department of Physics & Astronomy, Rutgers, The State University of New Jersey, 136 Frelinghuysen Rd., Piscataway, NJ 08854, U.S.A.