Neural Network Optimization

Overview

Neural network optimization asks how to adjust a model’s parameters so it learns useful patterns efficiently and reliably. Common approaches include stochastic gradient descent and Adam, matrix-aware and spectral optimizers such as Muon, curvature-based methods, learning-rate schedules, and techniques that reduce memory or computation. Researchers also analyze loss landscapes, scaling laws, and initialization, and explore alternatives such as local or forward-only learning and analytic solutions for restricted models. A central tension is that gains depend on architecture, data, and training budget, while faster loss reduction does not necessarily improve generalization; connecting practical performance to reliable theory remains an open challenge.

Latest papers

CardsList

355 papers29 papers in the last four weeks, down 28% on the four weeks before. 0.2% of all new papers.

  1. Theory-Guided Instance-Dependent Early Stopping via Second-Order Loss Dynamics

    Oct 17, 2026Ahad Bin Islam Shoeb, Jamal Uddin Tanvin, Kamrul Hasan +2Early StoppingNeural Network Optimization

  2. A Minimal ReLU Classifier on Non-Linearly Separable Data: Convergence and Implicit Bias at and beyond the Edge of Stability

    Oct 16, 2026Quan Nguyen, Robin Yadav, Param Mody +1Neural Network OptimizationReLU Neural Networks

  3. Towards Hyperparameter Transfer for Differentially Private Optimization

    Oct 16, 2026Tianze Wang, Zhiqi Bu, Linjun ZhangNeural Network OptimizationHyperparameter Transfer

  4. Scalable Levenberg-Marquardt Training Framework for Deep Learning

    Oct 16, 2026Xianghao Wang, Johan Barthélemy, John TaylorNeural Network OptimizationSecond-Order Optimization

  5. Measuring the local shape of learning-rate transfer curves

    Oct 15, 2026Satoki Ishikawa, Rio Yokota, Ryo KarakidaNeural Network OptimizationNeural Network Training Dynamics

  6. Approaching Backpropagation Accuracy with Weight Perturbation on MNIST

    Oct 15, 2026Patrick InoueGradient-Free OptimizationNeural Network Optimization

  7. When Is Muon Optimal? Radial Fragility and Minimax Geometry

    Oct 15, 2026Ao XuMuon OptimizerNeural Network Optimization

  8. One Size Does Not Fit All: Curvature-Aware Adaptive Perturbation for SAM

    Oct 14, 2026Alessia Rossi, Felipe Campelo, Gabriel L. OliveiraCurvature-Aware OptimizationSharpness-Aware Minimization

  9. Which Optimizer Leaks Most? Optimizer Implicit Bias as a Training-Data Privacy Controller

    Oct 13, 2026Sagi Habani, Yehonathan Refael, Ofir LindenbaumTraining Data Reconstruction AttacksModel Inversion Attacks

  10. Which Geometry on Which Layer? Measured Sign-Spectral Assignment in Mixed-Optimizer Training

    Oct 12, 2026Hiroki Naganuma, Laura Gomezjurado, Mahdi Ghaznavi +5Spectral OptimizationDeep Learning Optimization

  11. A Multi-Agent LLM System for Automated Tuning of Specialized Neural Network Models

    Oct 9, 2026Ivan Moloshnikov, Dmitry Kunitsyn, Alexander Sboev +3Neural Network OptimizationLarge Language Model-Guided Optimization

  12. A Scalable Learned Iterative Solver for Nonlinear MPC via Temporal Decomposition

    Oct 9, 2026Lukas Lüken, Sergio LuciaLearning to OptimizeNonlinear MPC

  13. Correlational Training of Morphological Neural Networks

    Oct 8, 2026Konstantinos Fotopoulos, Petros MaragosNeural Network Optimization

  14. Optimizing Large Language Models with Chained LMOs

    Oct 7, 2026Sungyoon Kim, Kaan Ozkara, Youngsuk ParkLMO MethodsDeep Learning Optimization

  15. DeltaTTT: Layerwise Optimization for Nonlinear Recurrent Memory

    Oct 6, 2026Yining Li, Dongchen Han, Jie Fu +1Deep Learning OptimizationNeural Network Optimization

  16. How Bregman Divergences Shape Shampoo

    Oct 6, 2026Bing Liu, Wenjie Zhou, Chengcheng Zhao +4Deep Learning OptimizationFunctional Bregman Divergences

  17. Singular parameters and missing limits in neural PDE solvers

    Oct 5, 2026Daniel FernándezNeural PDE SolversNeural Network Optimization

  18. Muon Is Theoretically Wrong For Convolutions, But Empirically Effective

    Oct 5, 2026Thibaut Boissin, Thomas Massena, Mathieu Serrurier +1Newton-Schulz IterationNeural Network Optimization

  19. Hamiltonian Metric Learning and Energy-Based Training: A Dissipative Geometric Framework for Optimization

    Oct 4, 2026Sparsho Chakraborty, Mohammad Alamgir, Nishanth M +3Neural Network OptimizationRiemannian Optimization

  20. Selective Backpropagation for Efficient Few-Shot Class-Incremental Learning

    Oct 2, 2026Eeham Khan, Abdulmoumen Al-Atrash, Ali AyubClass-Incremental LearningNeural Network Optimization

  21. Removing spurious minima for planar features by skip connections

    Oct 1, 2026Jakob Paul Zimmermann, Andrei Balakin, Moritz Grillo +1Shallow Neural NetworksReLU Neural Networks

  22. CAGE-NAS: Certified Functional Descent for Efficient Model Growth

    Oct 1, 2026Santiago Florido Gomez, Stéphane RivaudGradient DescentNeural Network Optimization

  23. Preconditioned Physics-Informed Neural Operator Training

    Sep 28, 2026Shizheng Wen, Siddhartha Mishra, Marius ZeinhoferPhysics-Informed Neural OperatorsPDE Operator Learning

  24. Predicting Block-Coordinate Performance via Cross-Curvature

    Sep 27, 2026Shengkun Zhu, Jinshan Zeng, Zhiqiang Kou +2Neural Network OptimizationBlock Coordinate Descent

  25. The Price of Locality: Why Forward-Forward Underperforms Backpropagation?

    Sep 27, 2026Zhaoxian Wu, Haichuan Liu, Tianyi ChenForward-Forward LearningNeural Network Optimization