Neural Network Optimization

Ask about this topic

Overview

Neural network optimization asks how to adjust a model’s parameters so it learns useful patterns efficiently and reliably. Common approaches include stochastic gradient descent and Adam, matrix-aware and spectral optimizers such as Muon, curvature-based methods, learning-rate schedules, and techniques that reduce memory or computation. Researchers also analyze loss landscapes, scaling laws, and initialization, and explore alternatives such as local or forward-only learning and analytic solutions for restricted models. A central tension is that gains depend on architecture, data, and training budget, while faster loss reduction does not necessarily improve generalization; connecting practical performance to reliable theory remains an open challenge.

Latest papers

CardsList

332 papers37 papers in the last four weeks, up 28% on the four weeks before. 0.2% of all new papers.

  1. A new Uncertainty Principle in Machine Learning

    Feb 24, 2026V. Dolotin, A. MorozovTwo-Layer Neural NetworksNeural Network Optimization

    MIPT, Dolgoprudny, 141701, Russia · NRC “Kurchatov Institute", 123182, Moscow, Russia · Institute for Information Transmission Problems, Moscow 127994, Russia +1

  2. Scale-PINN: Learning Efficient Physics-Informed Neural Networks Through Sequential Correction

    Feb 23, 2026Pao-Hsiung Chiu, Jian Cheng Wong, Chin Chun Ooi +3Neural PDE SolversPhysics-Informed ML

    Institute of High Performance Computing, A*STAR, Singapore. · Centre for Frontier AI Research, A*STAR, Singapore. · School of Mechanical Engineering, Tianjin University, China. +1

  3. Limits of Convergence-Rate Control for Open-Weight Safety

    Feb 21, 2026Domenic Rosati, Xijie Zeng, Hong Huang +4Neural Network OptimizationSpectral Methods

    Dalhousie University · Vector Institute · Indian Institute of Management Bangalore

  4. OptiRoulette Optimizer: A New Stochastic Meta-Optimizer for up to 5.3x Faster Convergence

    Feb 20, 2026Stamatis MastromichalakisNeural Network OptimizationStochastic Optimization

    Independent Researcher, tmnetworks.gr

  5. Hardware-Friendly Input Expansion for Accelerating Function Approximation

    Feb 20, 2026Hu Lou, Yin-Jun Gao, Dong-Xiao Zhang +3Neural Network Approximation TheoryNeural Network Optimization

    Northwest Institute of Nuclear Technology, Xi’an, 710000, Shaanxi, China.

  6. Learning with Boolean threshold functions

    Feb 19, 2026Veit Elser, Manish Krishan LalBoolean Function LearningBinary Neural Networks

    Department of Physics, Cornell · Mathematics, Independent Researcher, India

  7. HAWX: A Hardware-Aware FrameWork for Fast and Scalable ApproXimation of DNNs

    Feb 18, 2026Samira Nazari, Mohammad Saeed Almasi, Mahdi Taheri +4Neural Network Hardware AccelerationNeural Network Optimization

    University of Zanjan, Zanjan, Iran · Brandenburg Technical University, Cottbus, Germany · Tallinn University of Technology, Tallinn, Estonia +1

  8. Conjugate Learning Theory: Uncovering the Mechanisms of Trainability and Generalization in Deep Neural Networks

    Feb 18, 2026Binchuan QiNeural Network GeneralizationNeural Network Optimization

    Tongji University, Siping Street, 200092 Shanghai, China · Zhejiang Yuying College of Vocational Technology, Sihao Street, Hangzhou, 310018, Zhejiang, China

  9. Algorithmic Simplification of Neural Networks with Mosaic-of-Motifs

    Feb 16, 2026Pedram Bakhtiarifard, Tong Chen, Jonathan Wenshøj +2Kolmogorov ComplexityNeural Network Compression

    Department of Computer Science, University of Copenhagen, Denmark

  10. Scale redundancy and soft gauge fixing in positively homogeneous neural networks

    Feb 16, 2026Rodrigo Carmo TerinNeural Network OptimizationNeural Network Training Dynamics

    King Juan Carlos University, Facuty of Experimental Sciences and Technology, Department of Applied Physics, Av. del Alcalde de Móstoles, 28933, Madrid, Spain

  11. Learning to Control: The iUzawa-Net for Nonsmooth Optimal Control of Linear PDEs

    Feb 12, 2026Yongcun Song, Xiaoming Yuan, Hangrui Yue +1Optimal ControlConstrained Optimization

    Division of Mathematical Sciences, School of Physical and Mathematical Sciences, Nanyang Technological University, 21 Nanyang Link, 637371, Singapore. · Department of Mathematics, The University of Hong Kong, Pok Fu Lam, Hong Kong SAR, China. · School of Mathematical Sciences, Nankai University, Tianjin 300071, China.

  12. μμpscaling small models: Principled warm starts and hyperparameter transfer

    Feb 11, 2026Yuxin Ma, Nan Chen, Mateo Díaz +3Neural Network OptimizationHyperparameter Transfer

    Department of Applied Mathematics and Statistics Johns Hopkins University

  13. ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling

    Feb 9, 2026Yilang Zhang, Bingcong Li, Niao He +1Residual LearningNeural Network Optimization

    ML Research, Morgan Stanley, New York, NY 10019 · Department of CS, ETH Zürich, 8092 Zürich, Switzerland · Department of ECE, University of Minnesota, Minneapolis, MN 55455

  14. Unrolled Neural Networks for Constrained Optimization

    Jan 24, 2026Samar Hadou, Alejandro RibeiroConstrained OptimizationLearning to Optimize

    Department of Electrical and Systems Engineering, University of Pennsylvania, Philadelphia, PA 19104 USA

  15. Linearized subspace refinement framework to expose hidden accuracy in trained neural networks

    Jan 20, 2026Wenbo Cao, Weiwei ZhangSubspace OptimizationNeural Network Optimization

    Institute of AI for Industries, Chinese Academy of Sciences, Nanjing 211135, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · School of Aeronautics, Northwestern Polytechnical University, Xi’an 710072, China +2

  16. Performance and Complexity Trade-off Optimization of Speech Models During Training

    Jan 20, 2026Esteban Gómez, Tom BäckströmModel CompressionNeural Network Optimization

    Department of Information and Communications Engineering, Aalto University, Espoo, Finland

  17. Turbo-Muon: Almost-Orthogonal Pre-Conditioning for Fast Muon Updates

    Dec 4, 2025Thibaut Boissin, Thomas Massena, Franck Mamalet +1Newton-Schulz IterationOrthogonal Optimization

    IRIT-MISFIT · Institut de Recherche Technologique Saint-Exupery, France · DTIPG - SNCF, IRIT-MISFIT +2

  18. Mean-Field Model for Two-Layer Neural Networks Trained with Consensus-Based Optimization

    Nov 26, 2025William De Deyn, Michael Herty, Giovanni SamaeyNeural Network OptimizationMean-Field Theory

    Institut für Geometrie und Praktische Mathematik, RWTH Aachen University, Germany · Department of Computer Science, KU Leuven, Belgium · Extraordinary Professor, Department of Mathematics and Applied Mathematics, University of Pretoria, South Africa

  19. Gradient Prediction with Control Variates in the Cheap-Forward Regime

    Nov 7, 2025Kamil Ciosek, Nicolò Felicioni, Juan Elenter +1Neural Network Optimization

    Spotify

  20. Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLU

    Oct 29, 2025Jingzhou LiuShallow Neural NetworksDynamical Systems

  21. SmartMixed: A Two-Phase Training Strategy for Adaptive Activation Function Learning in Neural Networks

    Oct 25, 2025Amin OmidvarNeural Network OptimizationNeural Network Activation Functions

    Independent Researcher Toronto, Ontario, Canada

  22. Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rules

    Sep 22, 2025Doğay AltınelDeep Learning OptimizationNeural Network Optimization

    Istanbul Medeniyet University

  23. Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Training

    Sep 15, 2025Chuan He, Zhanwang Deng, Zhaosong LuNeural Network OptimizationLow-Rank Approximation

    Department of Mathematics Linköping University, Sweden · Academy for Advanced Interdisciplinary Studies Peking University, Beijing, People’s Republic of China · Department of Industrial and Systems Engineering University of Minnesota, USA

  24. Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimization

    Sep 3, 2025Wu Lin, Scott C. Lowe, Felix Dangel +3Deep Learning OptimizationNeural Network Optimization

    1Vector Institute, Toronto, Canada · University of Cambridge, Cambridge, United Kingdom · 3Microsoft, United States +1

  25. Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initialization

    May 28, 2025Etienne Boursier, Matthew Bowditch, Matthias Englert +1Loss LandscapeOverparameterized Neural Network

    INRIA LMO, Université Paris-Saclay Orsay, France · Mathematics Institute University of Warwick Coventry, UK · Department of Computer Science University of Warwick Coventry, UK

  26. KO: Kinetics-inspired Neural Optimizer with PDE Simulation Approaches

    May 20, 2025Mingquan Feng, Yixin Huang, Yifan Fu +2Neural Network OptimizationNeural Network Training Dynamics

    Shanghai Jiao Tong University

  27. Forward Target Propagation: A Forward-Only Approach to Global Error Credit Assignment via Local Losses

    May 20, 2025Nazmus Saadat As-Saquib, A N M Nafiz Abeer, Hung-Ta Chien +3Neural Network OptimizationLocal Learning

    Texas A&M University, College Station, TX 77843 · Sandia National Laboratories, Livermore, CA 94550

  28. AYLA: Architecting a loss landscape in shallow neural networks to accelerate feature recovery

    Apr 2, 2025Behnam Gheshlaghi, Shahin AtakishiyevShallow Neural NetworksRepresentation Learning

  29. Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential

    Feb 26, 2025Shijun Zhang, Hongkai Zhao, Yimin Zhong +1Fourier Feature EmbeddingsNeural Network Optimization

    Department of Applied Mathematics Hong Kong Polytechnic University · Department of Mathematics Duke University · Department of Mathematics and Statistics Auburn University +1

  30. Koopman-informed recurrent neural networks

    Oct 30, 2024Erik Lien Bolager, Ana Čukarska, Iryna Burak +2Neural Network OptimizationTime Series Forecasting

    Munich Data Science Institute · School of Computation, Information and Technology, Technical University of Munich, Germany · Interdisciplinary Center for Scientific Computing (IWR) +1