Neural Network Optimization

Overview

Neural network optimization asks how to adjust a model’s parameters so it learns useful patterns efficiently and reliably. Common approaches include stochastic gradient descent and Adam, matrix-aware and spectral optimizers such as Muon, curvature-based methods, learning-rate schedules, and techniques that reduce memory or computation. Researchers also analyze loss landscapes, scaling laws, and initialization, and explore alternatives such as local or forward-only learning and analytic solutions for restricted models. A central tension is that gains depend on architecture, data, and training budget, while faster loss reduction does not necessarily improve generalization; connecting practical performance to reliable theory remains an open challenge.

Latest papers

CardsList

355 papers29 papers in the last four weeks, down 28% on the four weeks before. 0.2% of all new papers.

  1. Barron Approximation and locally optimal weight densities for shallow neural networks

    Nov 15, 2026G. WelperShallow Neural NetworksNeural Network Optimization

  2. NAP-Wave: A Neural Adaptive Plane-Wave Architecture for High-Frequency Helmholtz Problems

    Nov 14, 2026Hamza Kamil, Mohammad Mahdi Abedi, David PardoNeural PDE SolversPDE Solving

  3. Grokking Can Be Steered: Selecting Desirable Solutions By Choice Of Learning Rule

    Nov 14, 2026Pascal Sager, Benjamin Grewe, Pau Vilimelis Aceituno +1GrokkingNeural Network Optimization

  4. Neural Dual Bounds: Valid-by-Construction JGLP Warm-Starts for MAP and Constrained MAP

    Nov 14, 2026Akshay Vyas, Shivvrat Arya, Brij Malhotra +3Probabilistic Graphical ModelsNeural Network Optimization

  5. CAFIN: CUDA-accelerated Adaptive Fisher Information Matrix Network Optimization via GPU-Native Kernel Fusion

    Nov 10, 2026Maryam Arshad, Muhammad Khuram ShahzadNeural Network OptimizationGPU Kernel Optimization

  6. Spectral Unlearning: Transformer Structure-Preserving Updates for Language Models

    Nov 9, 2026Sung Il Choi, Junhao Cai, Dohun Kim +1Machine UnlearningLLM Unlearning

  7. On Lipschitz Explosion in Deep Neural Networks with Normalization: Consequences for Optimization and Adversarial Robustness

    Nov 5, 2026Ashkan Soleymani, Reyhaneh Hosseinpourkhoshkbari, Hadi Daneshmand +1Batch NormalizationLipschitz Neural Networks

  8. An Educated Guess: Deriving Statistically Aligned Gradient Estimates for Zeroth-Order Optimisation

    Nov 5, 2026Justus Huebotter, Serge Thill, Marcel van Gerven +1Zeroth-Order OptimizationNeural Network Optimization

  9. Module-Aware Optimization for Graph Neural Networks

    Nov 5, 2026Guy Hadad, Haggai Roitman, Moshe Eliasof +1Neural Network OptimizationGraph Neural Networks

  10. Projection-Free Gradient Tracking Neurodynamic for Distributed Nonnegative Sparse Optimization

    Nov 5, 2026Juntao Li, Qihang Guo, Deyuan MengDistributed OptimizationConstrained Optimization

  11. A Bilevel Neurodynamic Approach for Distributed SCAD-Regularized Sparse Least Squares

    Nov 5, 2026Juntao Li, Qihang Guo, Deyuan MengSparse RecoveryDistributed Optimization

  12. Positional Encoding is All You Need for Scalable Equivariance Constraint Relaxation

    Nov 3, 2026Hagay Michaeli, Haggai Maron, Daniel SoudryEquivariant Neural NetworksNeural Network Optimization

  13. Non-Interfering Asymmetric AdamW Updates for Constrained Exploration of Equifinality in Hybrid Models

    Nov 3, 2026Ekaterina Gikalo, Zavud Baghirov, Christian Reimers +3Neural Network OptimizationAdaptive Gradient Methods

  14. Theoretical Analysis of Muon on Transformers for Next-Token Prediction

    Nov 3, 2026Shengwei Wu, Yixuan Yang, Gao Huang +1Transformer Training DynamicsMuon Optimizer

  15. Variational Functional Neural Networks (vfNN)

    Nov 2, 2026Chinmay Katke, C. Nadir KaplanNeural Network Optimization

  16. Gradient Descent on Two ReLU Neurons: Global Landscape and Bifurcation Dynamics

    Oct 30, 2026Binghua Li, Mengzhe Li, Denny Wu +1Neural Network Training DynamicsNeural Network Optimization

  17. Optimization-based lane-changing trajectory planning in mixed traffic with embedded input convex neural networks

    Oct 28, 2026Yuanxiang Yang, Yu Liu, Fabricio Oliveira +2Motion PlanningAutonomous Driving Planning

  18. Nonlinear Direct Feedback Alignment for Scalable Backpropagation-Free Training

    Oct 27, 2026Wu Lin, Yan Fan, Chenxiang Ma +4Neural Network Optimization

  19. The proton structure one mode at a time: a Neural Tangent Kernel analysis of a DIS fit

    Oct 26, 2026Amedeo Chiefa, Luigi Del Debbio, Richard KenwayNeural Network Training DynamicsNeural Tangent Kernel

  20. Reactive Optimizers

    Oct 24, 2026Xingjian Bai, Jia-Hao Liang, Jayden Teoh +1Neural Network OptimizationLarge Language Model-Guided Optimization

  21. Optimizing Physical Neural Networks through Device Variable Selection

    Oct 24, 2026Sjoerd van den Belt, Manuel Escudero, Mohamadreza Zolfagharinejad +2Physical Neural NetworksNeural Network Optimization

  22. Training Mixed-Monotone Neural Networks for Verification Without Set Propagation

    Oct 22, 2026Jonas Fingerhut, Lukas Koller, Matthias AlthoffNeural Network VerificationNeural Network Optimization

  23. Forward-Only Learning for Shared-Kernel Wavelet Neural Networks without Backpropagation

    Oct 21, 2026Yongfei Guo, Xuan Sun, Ang Li +2Neural Network OptimizationForward-Forward Learning

  24. Do Neural Models Prefer Their Submodels?

    Oct 20, 2026Alvaro Ribot, Yulia Alexandr, Anna SeigalNeural Network OptimizationMultilayer Perceptrons

  25. Dynamical Batching: How Class-Cycling Training Shapes the Internal Structure of Neural Networks

    Oct 19, 2026Nicolás Ratier-Werbin, Samuel Lozano, Miguel Ruiz-GarcíaNeural Network Training DynamicsNeural Network Optimization

  26. Representations Have Trajectories: Characterizing Temporal Representation Dynamics in Neural Networks

    Oct 17, 2026Shivaprasad B. Gowda, Sparsh MittalLow-Rank Representation LearningNeural Representation Geometry