Neural Network Optimization

Ask about this topic

Overview

Neural network optimization asks how to adjust a model’s parameters so it learns useful patterns efficiently and reliably. Common approaches include stochastic gradient descent and Adam, matrix-aware and spectral optimizers such as Muon, curvature-based methods, learning-rate schedules, and techniques that reduce memory or computation. Researchers also analyze loss landscapes, scaling laws, and initialization, and explore alternatives such as local or forward-only learning and analytic solutions for restricted models. A central tension is that gains depend on architecture, data, and training budget, while faster loss reduction does not necessarily improve generalization; connecting practical performance to reliable theory remains an open challenge.

Latest papers

CardsList

332 papers37 papers in the last four weeks, up 28% on the four weeks before. 0.2% of all new papers.

  1. Muon-C: Operator-Aligned Muon for Convolutional Kernels

    Sep 9, 2026Jiaxin Qing, Lexin LiNeural Network OptimizationMuon Optimizer

    University of California, Berkeley Berkeley, CA 94720-1776, USA

  2. Teacher Geometry Shapes Learnability in Teacher-Student Networks

    Sep 9, 2026Kai J. Sandbrink, Flavio Martinelli, Alexander van Meegen +2Neural Network OptimizationNeural Network Training Dynamics

    Department of Experimental Psychology, University of Oxford · Department of Physics, RWTH Aachen

  3. Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack

    Sep 8, 2026Sohir Maskey, Philipp Scholl, Jonas Knupp +2Language Model PretrainingNeural Network Optimization

    Aleph Alpha

  4. HyperTransfer: Understanding the Equivalence between Base Optimizer and Hyperball

    Sep 7, 2026Jinghui Yuan, Hongtao Zhang, Jade Zou +4Neural Network OptimizationNeural Network Training Dynamics

    StepFun · University of Chinese Academy of Sciences, StepFun · University of Chinese Academy of Sciences +2

  5. Optimizer Memory Schedules for Outscaling the Overtraining Axis

    Sep 4, 2026Katie Everett, Shikai QiuNeural Network OptimizationMuon Optimizer

    MIT CSAIL

  6. Gradient-Update Mismatch: Rethinking Conflict-Free Training of Physics-Informed Neural Networks

    Sep 1, 2026Jing Xiao, Xinhai Chen, Qinglin Wang +5Neural Network OptimizationGradient Interference

    Laboratory of Digitizing Software for Frontier Equipment, National University of Defense Technology, Changsha 410073, China · National Key Laboratory of Parallel and Distributed Computing, National University of Defense Technology, Changsha 410073, China · School of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China

  7. Subspace Levenberg Marquardt Algorithms in Training Neural Networks

    Sep 1, 2026M. Duc HoangNeural Network OptimizationKrylov Subspace Methods

    Department of Mathematics, University of California, Davis, Davis, CA, USA

  8. A Study of Hidden-State Optimization Order in Predictive Coding Networks

    Sep 1, 2026Xueyuan Li, Danilo Vasconcellos VargasPredictive CodingNeural Network Optimization

  9. SS-ESOAP: Self-Scaled Adaptive Preconditioning for Physics-Informed Learning

    Aug 29, 2026Guangyuan Wang, Mads Toftrup, Sebastian Loeschcke +2Neural Network OptimizationPhysics-Informed ML

    McGill University · California Institute of Technology · Aarhus University +1

  10. From the Loss Landscape to Diverse Feature Learning in Neural Networks

    Aug 28, 2026David Aram YunisLoss LandscapeMode Connectivity

    TOYOTA TECHNOLOGICAL INSTITUTE AT CHICAGO Chicago, Illinois

  11. Blog: Survey of Optimizers

    Aug 28, 2026Ruoran XuNeural Network OptimizationMemory-Efficient Optimization

  12. Residual-Guided Randomized Neural Networks

    Aug 28, 2026Mushir Akhtar, M. Tanveer, Mohd. ArshadNeural Network Optimization

    Department of Mathematics Indian Institute of Technology Indore

  13. Physics-Informed Stochastic Configuration Machine: A Backpropagation-Free Neural Network with Fast Training for Nonlinear Differential Equations

    Aug 27, 2026Yuehao Song, Zhong Chen, Lihui Cen +2Physics-Informed MLNeural Network Optimization

    School of Automation, Central South University, Changsha, China · School of Automation, Central South University, 932 South Lushan Road, Changsha 410083, China · Johns Hopkins University, Baltimore, USA +3

  14. Delayed Optimizer-State Transport Shapes Short-Horizon Training Decisions

    Aug 25, 2026Jinhui GuoNeural Network OptimizationNeural Network Training Dynamics

    School of Physics, Beihang University, Beijing 100191, China

  15. Physics-Informed Error Field Learning: A Post-Training Optimization Framework for Physics-Informed Neural Networks

    Aug 25, 2026Jiuyun Sun, Yong ZhangNeural PDE SolversPhysics-Informed ML

    College of Mathematics and Systems Science, Shandong University of Science and Technology, Qianwangang Road 579, Qingdao, 266590, Shandong, China

  16. A Physical Response-and-Memory Model for Muon Optimization

    Aug 24, 2026Yinze Hu, Hongjun Xiang, Xingao Gong +1Neural Network OptimizationMuon Optimizer

    Key Laboratory of Computational Physical Sciences (Ministry of Education), Institute of Computational Physical Sciences, State Key Laboratory of Surface Physics, and Department of Physics, Fudan University, Shanghai 200433, China · School of Physics Science and Engineering, Tongji University, Shanghai 200092, China

  17. A Commutator Framework for Selective Spectral Alignment in Deep Neural Networks

    Aug 24, 2026Kaj NyströmNeural Representation GeometryNeural Network Training Dynamics

  18. Blockwise Stabilized Adaptive Cubic Regularization with Subsolvers via Recurrence

    Aug 22, 2026Rodion PodorozhnySecond-Order OptimizationNeural Network Optimization

    Texas State University

  19. RODE: A Radial-Orthogonal Decoupled Engine for Optimization

    Aug 21, 2026Guoxiang Xu, Bince Qu, Qi Sun +1Neural Network OptimizationLLM Training

    Zhejiang University, Hangzhou, China

  20. Enhanced Artificial Neural Networks Using QHAdamW in Air Quality Forecasting

    Aug 20, 2026Mary Joy Daniel VinasAir Quality PredictionNeural Network Optimization

    Computer Studies Department, Technological University of the Philippines, City of Manila, Philippines.

  21. Finite-Horizon Input-Output Dynamics of Minibatch Perturbations in AdamW

    Aug 20, 2026Kang Liu, Suyan LiNeural Network OptimizationNeural Network Training Dynamics

    School of Future Technology, Xi’an Jiaotong University, Xi’an, China, 710049 · Department of Electrical and Computer Engineering, National University of Singapore, Singapore, 119077.

  22. Kähler landscapes for complex neural network descents and guarantees including a search and destroy of the Calabi-Yau manifold

    Aug 20, 2026Andrew GracykRiemannian OptimizationGradient Descent

    Department of Mathematics, Purdue University, West Lafayette, IN 47907, USA

  23. Activation-Keyed Momentum: An Anisotropic Momentum Update via the Delta Rule

    Aug 19, 2026Euijin Hong, Guannan QuDeep Learning OptimizationMomentum Methods

    Electrical and Computer Engineering, Carnegie Mellon University

  24. Quantum Tensor Network Learning with DMRG

    Aug 19, 2026Gustav J L Jäger, Martin B Plenio, Hans-Martin RieserTensor NetworksQuantum Machine Learning

    Deutsches Zentrum für Luft- und Raumfahrt e.V. – Institut für KI-Sicherheit Wilhelm-Runge-Straße 10, 89081 Ulm, Germany · Universität Ulm – Institut für Theoretische Physik and IQST, Albert-Einstein-Allee 11, 89081 Ulm, Germany

  25. Many Optimizers But Only One Training Path: Repeated Resampling for Adaptive Optimizer Selection

    Aug 19, 2026Ronald Richman, Mario V. WüthrichDeep Learning OptimizationNeural Network Optimization

    insureAI · ETH Zürich

  26. On the Principles Behind Neural Network Optimizers

    Aug 17, 2026Yushun ZhangNeural Network OptimizationAdaptive Gradient Methods

    School of Data Science, The Chinese University of Hong Kong, Shenzhen, China

  27. GradRepair-ODE: Certified Gradient Repair for Neural ODE Training

    Aug 14, 2026Ziqian Bi, Xin Liang ChiaNeural ODEsOrdinary Differential Equations

    Purdue University, West Lafayette, IN, USA · Rice University, Houston, TX, USA

  28. Where the Cost Falls: A Deployment-Aware Adoption Order for Stability Enhancements to Cycle-Consistent Adversarial Networks

    Aug 14, 2026Rowan Hussein, Mohamed OufGenerative Adversarial NetworksUnpaired Image-to-Image Translation

    University of Ottawa, Ottawa, Canada · Queen’s University, Kingston, Canada

  29. SAGE: Surrogate-gradient Adaptation via Attention-Guided Entropy for Spiking Transformers

    Aug 13, 2026Kiran Nair, Rodrigue Rizk, KC SantoshSpiking Neural NetworksNeural Network Optimization

    USD Artificial Intelligence Research Lab Department of Computer Science University of South Dakota Vermillion, SD 57069, USA

  30. Training Under Challenge: Executable Certificates and Challenge-Closed Optimality for Neural Networks

    Aug 12, 2026Farhang Yeganegi, Arian Eamaz, Mojtaba SoltanalianNeural Network VerificationNeural Network Optimization

    Department of Electrical and Computer Engineering University of Illinois Chicago Chicago, Illinois, USA