Neural Network Optimization

Ask about this topic

Overview

Neural network optimization asks how to adjust a model’s parameters so it learns useful patterns efficiently and reliably. Common approaches include stochastic gradient descent and Adam, matrix-aware and spectral optimizers such as Muon, curvature-based methods, learning-rate schedules, and techniques that reduce memory or computation. Researchers also analyze loss landscapes, scaling laws, and initialization, and explore alternatives such as local or forward-only learning and analytic solutions for restricted models. A central tension is that gains depend on architecture, data, and training budget, while faster loss reduction does not necessarily improve generalization; connecting practical performance to reliable theory remains an open challenge.

Latest papers

CardsList

332 papers37 papers in the last four weeks, up 28% on the four weeks before. 0.2% of all new papers.

  1. MOON: Multi-Objective OrthoNormalized Updates for Multitask Learning

    Aug 12, 2026Shiji Zhou, Kunlin Lyu, Lei Zhang +2Neural Network OptimizationMulti-Task Learning

    Institute of Artificial Intelligence, Beihang University · Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University · Beijing Academy of Artificial Intelligence (BAAI) +1

  2. Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness

    Aug 11, 2026Siqiao Mu, Diego KlabjanGradient DescentNeural Network Optimization

  3. Accelerated Learning of High Dimensional Functions with a Tensor-Featured Training Network

    Aug 11, 2026Karl Pierce, Yuehaw Khoo, Haizhao YangNeural Network OptimizationNeural Network Approximation Theory

    † The University of Maryland, College Park, College Park MD 20742 USA · ∗The University of Chicago, Chicago, IL 60637 USA

  4. Generalized Convexity and Smoothness via Conjugate Duality, with Applications to Deep Network Training

    Aug 10, 2026Binchuan QiGradient DescentNeural Network Optimization

    College of Electronics and Information Engineering Tongji University Shanghai, 200092, China · Sihao Street Zhejiang Yuying College of Vocational Technology Zhejiang, 310018, China

  5. ZeroLock: Concurrent Memory-Efficient LLM Training via Modular Update Decoupling

    Aug 8, 2026Wentao Dai, Xuanran Li, Yuxiang Zhang +2On-Device TrainingMemory-Efficient Fine-Tuning

    Department of Computer Science and Engineering, Southern University of Science and Technology, China · Department of Mathematics, Southern University of Science and Technology, China · School of Computing, Montclair State University, New Jersey, USA

  6. MESH: Memory-Efficient Sinkhorn Optimization for Mixture-of-Experts Training

    Aug 5, 2026Masato FujitakeNeural Network OptimizationMemory-Efficient Optimization

    FA Research, Fast Accounting Co., Ltd.

  7. Muon Meets Mamba: Spectral Optimization for State Space Models

    Aug 4, 2026Arslan Battalov, Karim Kramin, Alexander Markotenko +1Neural Network OptimizationMuon Optimizer

    HSE University · Moscow, Russia

  8. Joint Affine Spectral Shaping: Coupling Weight and Bias Updates Beyond Weight-Only Muon

    Aug 4, 2026Gongyue Zhang, Honghai LiuNeural Network OptimizationMuon Optimizer

    State Key Laboratory of Robotics and Systems, Harbin Institute of Technology, Shenzhen, Shenzhen, 518055, China.

  9. New Synchronous Computation Dynamics for Hopfield Networks

    Jul 30, 2026Francisco Requena-Domínguez, Rafaela Benítez-Rochel, Ezequiel López-RubioNeural Network OptimizationHopfield Networks

    Department of Computer Languages and Computer Science University of Málaga Bulevar Louis Pasteur 35, 29071 Málaga, Spain · ITIS Software, University of Málaga C/ Arquitecto Francisco Peñalosa 18, 29010 Málaga, Spain

  10. Greedy dynamical meta-learning

    Jul 27, 2026Aria YomMeta-LearningNeural Network Optimization

    Rhodora

  11. Scale Weight Decay and Train Better

    Jul 26, 2026Anuj ApteWeight DecayNeural Network Optimization

    Global Technology Applied Research, JPMorganChase, New York, NY 10001

  12. Hidden Boundary Motion in Transformer Optimization: Function-Space Orthogonalization of Affine Weight and Bias Updates

    Jul 24, 2026Zhang Gongyue, Sheng Yixuan, Liu donghan +3TransformerNeural Network Optimization

    arXiv:2607.22927v1

  13. Hyperball May Not Be a Free Lunch

    Jul 24, 2026Yihao Xiao, Jialong Sun, Zitian Gao +5Learning Rate SchedulingNeural Network Optimization

    1IQuest Research · 4Shenzhen University of Advanced Technology · 2Peking University +2

  14. Searching the Space of Feed-Forward Neural-Network Weight-Update Rules with Fixed Depth Symbolic Regression

    Jul 23, 2026Charles Brum, Edward FinkelsteinNeural Network OptimizationLearning to Optimize

    San Diego, California, USA · UCI, USA

  15. Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay

    Jul 23, 2026Xiaolong Li, Zhangchen Zhou, Zhi-Qin John XuWeight DecayNeural Network Optimization

    Institute of Natural Sciences, School of Mathematical Sciences, Shanghai Jiao Tong University.

  16. New Complexity-Theoretic Frontiers of Tractability for Neural Network Training

    Jul 23, 2026Cornelius Brand, Robert Ganian, Mathis RoctonReLU Neural NetworksNeural Network Optimization

    Algorithms & Complexity Group Vienna University of Technology Favoritenstraße 9-11, 1040 Vienna, Austria

  17. On the Depth Scalability of Logic Gate Networks

    Jul 19, 2026Taegun An, Dohun kim, Haebeom Lee +1Neural Network OptimizationCredit Assignment

    Jung Woonoh IT & General Education Center, Korea University 145, Anam-ro, Seongbuk-gu, Seoul 02841, Republic of Korea

  18. First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers

    Jul 18, 2026Irina Piontkovskaia, Sergey NikolenkoFine-TuningNeural Network Optimization

    1DAIMLD, Moscow, Russia · 2St. Petersburg Department of the Steklov Institute of Mathematics, St. Petersburg, Russia · 3St. Petersburg State University

  19. Effects of width-dependent model hyperparameters and ℓ2\ell_2-regularization on the loss landscape of two-layer ReLU networks

    Jul 18, 2026Haruka Eshima, Makoto YamadaReLU Neural NetworksWeight Decay

    Okinawa Institute of Science and Technology, Okinawa, Japan

  20. Backpropagation-Free Trunk Training via the Split Forward Gradients

    Jul 18, 2026Tian Qin, Wei-Min HuangBackpropagationGradient Descent

    Math department Lehigh University Bethlehem, PA 18015, USA

  21. Gradient-Energy Guided Block-Wise Perturbations for Sharpness-Aware Minimization

    Jul 17, 2026Zhen Huang, Jiaxin Deng, Junbiao PangSharpness-Aware MinimizationNeural Network Optimization

    Faculty of Information Technology, Beijing University of Technology, Beijing, China

  22. Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers

    Jul 16, 2026Akhilesh GogikarNeural Network OptimizationAdaptive Gradient Methods

    1Independent Researcher.

  23. LIGO-PINN: Learned Initialization via Gated Optimization to Alleviate Convergence Failures in Physics Informed Neural Networks

    Jul 15, 2026Nilay Anurag, Shital Adhikari, Taniya Kapoor +1Neural PDE SolversNeural Network Optimization

    Department of Computer Science, Stevens Institute of Technology · Artificial Intelligence Group, Wageningen University & Research

  24. How the Hessian-Spectrum of Neural Networks Depends on Data

    Jul 15, 2026Jasraj Singh, Enea Monzio Compagnoni, Antonio OrvietoLoss Landscape SharpnessNeural Network Optimization

    ELLIS Institute, Tübingen, Germany · Universität Basel, Basel, Switzerland · MPI-IS, Tübingen, Germany

  25. FastCentNN: Accelerating Centroid Neural Network with Entropy Proxy

    Jul 15, 2026Le-Anh TranNeural Network OptimizationUnsupervised Clustering

  26. Approximation of solutions of parameter-dependent problems by residual neural networks

    Jul 15, 2026Ana CarpioNeural Surrogate ModelingNeural Network Optimization

    Universidad Complutense de Madrid, Spain

  27. Learning in Curved Weight Space:Exponential-Linear Weight Reparameterization for Improved Optimization

    Jul 10, 2026Ethan SmithNeural Network OptimizationWeight-Space Learning

    Canva Research

  28. Fully Trainable Deep Differentiable Logic Gate Networks and Lookup Table Networks

    Jul 10, 2026Wout Mommen, Lars Keuninckx, Matthias Hartmann +2Neural Network OptimizationDifferentiable Logic Gate Networks

    imec, Kapeldreef 75, Leuven, 3001, , Belgium · Vrije Universiteit Brussel, Peinlaan 2, Elsene, 1050, , Belgium

  29. Systematic Evaluation of Learning Rate Scheduling Strategies Across Heterogeneous Architectures

    Jul 9, 2026Hafsa Mateen, Radu Timofte, Dmitry IgnatovLearning Rate SchedulingNeural Network Optimization

    Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany

  30. Reliable mechanistic operator recovery with biologically-informed neural networks: principles for architecture and optimisation design

    Jul 8, 2026Rebecca M. Crossley, Yuan Yin, Sarah L. Waters +1Neural Network OptimizationMechanistic Interpretability

    Mathematical Institute, University of Oxford, Oxford, United Kingdom, OX2 6GG