Neural Network Optimization

Ask about this topic

Overview

Neural network optimization asks how to adjust a model’s parameters so it learns useful patterns efficiently and reliably. Common approaches include stochastic gradient descent and Adam, matrix-aware and spectral optimizers such as Muon, curvature-based methods, learning-rate schedules, and techniques that reduce memory or computation. Researchers also analyze loss landscapes, scaling laws, and initialization, and explore alternatives such as local or forward-only learning and analytic solutions for restricted models. A central tension is that gains depend on architecture, data, and training budget, while faster loss reduction does not necessarily improve generalization; connecting practical performance to reliable theory remains an open challenge.

Latest papers

CardsList

332 papers37 papers in the last four weeks, up 28% on the four weeks before. 0.2% of all new papers.

  1. Nonlinear Direct Feedback Alignment for Scalable Backpropagation-Free Training

    Oct 27, 2026Wu Lin, Yan Fan, Chenxiang Ma +4Neural Network Optimization

    College of Computer Science and Software Engineering, Shenzhen University · School of Artificial Intelligence, Tianjin University · School of Artificial Intelligence, Shenzhen University +2

  2. The proton structure one mode at a time: a Neural Tangent Kernel analysis of a DIS fit

    Oct 26, 2026Amedeo Chiefa, Luigi Del Debbio, Richard KenwayNeural Network Training DynamicsNeural Tangent Kernel

    The Higgs Centre for Theoretical Physics, School of Physics and Astronomy, The University of Edinburgh, Peter Guthrie Tait Road, Edinburgh EH9 3FD, United Kingdom

  3. Reactive Optimizers

    Oct 24, 2026Xingjian Bai, Jia-Hao Liang, Jayden Teoh +1Neural Network OptimizationLarge Language Model-Guided Optimization

    Massachusetts Institute of Technology

  4. Optimizing Physical Neural Networks through Device Variable Selection

    Oct 24, 2026Sjoerd van den Belt, Manuel Escudero, Mohamadreza Zolfagharinejad +2Physical Neural NetworksNeural Network Optimization

    BRAINS Center for Brain-Inspired Computing, University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlands · MESA+ Institute, University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlands · Institute of Physics, University of Münster, 48149 Münster, Germany

  5. Training Mixed-Monotone Neural Networks for Verification Without Set Propagation

    Oct 22, 2026Jonas Fingerhut, Lukas Koller, Matthias AlthoffNeural Network VerificationNeural Network Optimization

    Technical University of Munich

  6. Forward-Only Learning for Shared-Kernel Wavelet Neural Networks without Backpropagation

    Oct 21, 2026Yongfei Guo, Xuan Sun, Ang Li +2Neural Network OptimizationForward-Forward Learning

    Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China · Chinese Academy of Sciences, Beijing, China

  7. How Does Orthogonalization Adapt to the Neural Network Hessian Structure? A Gradient Self Outer-Product Analysis at Initialization

    Oct 20, 2026Shenyang Deng, Garvit Goyal, Tianyu Pang +2Neural Network OptimizationOrthogonal Optimization

    Dartmouth College · Meta

  8. A General Topological Symmetry Index for Critical Orbits In Neural Network Loss Landscapes

    Oct 20, 2026Jingzhou LiuLoss Landscape GeometryNeural Network Optimization

  9. Do Neural Models Prefer Their Submodels?

    Oct 20, 2026Alvaro Ribot, Yulia Alexandr, Anna SeigalNeural Network OptimizationMultilayer Perceptrons

    Harvard University · University of Melbourne

  10. Dynamical Batching: How Class-Cycling Training Shapes the Internal Structure of Neural Networks

    Oct 19, 2026Nicolás Ratier-Werbin, Samuel Lozano, Miguel Ruiz-GarcíaNeural Network Training DynamicsNeural Network Optimization

    Departamento de Estructura de la Materia, Física Térmica y Electrónica, Universidad Complutense de Madrid, 28040 Madrid, Spain · Grupo Interdisciplinar de Sistemas Complejos, Universidad Complutense de Madrid, 28040 Madrid, Spain

  11. Representations Have Trajectories: Characterizing Temporal Representation Dynamics in Neural Networks

    Oct 17, 2026Shivaprasad B. Gowda, Sparsh MittalLow-Rank Representation LearningNeural Representation Geometry

    Indian Institute of Information Technology, Nagpur · Indian Institute of Technology, Roorkee

  12. Theory-Guided Instance-Dependent Early Stopping via Second-Order Loss Dynamics

    Oct 17, 2026Ahad Bin Islam Shoeb, Jamal Uddin Tanvin, Kamrul Hasan +2Early StoppingNeural Network Optimization

    University of Dhaka · Military Institute of Science and Technology · Tennessee State University +1

  13. A Minimal ReLU Classifier on Non-Linearly Separable Data: Convergence and Implicit Bias at and beyond the Edge of Stability

    Oct 16, 2026Quan Nguyen, Robin Yadav, Param Mody +1Neural Network OptimizationReLU Neural Networks

    University of British Columbia · Canada CIFAR AI Chair (Amii)

  14. Towards Hyperparameter Transfer for Differentially Private Optimization

    Oct 16, 2026Tianze Wang, Zhiqi Bu, Linjun ZhangNeural Network OptimizationHyperparameter Transfer

    Department of Statistics, Rutgers University · Independent Researcher

  15. Scalable Levenberg-Marquardt Training Framework for Deep Learning

    Oct 16, 2026Xianghao Wang, Johan Barthélemy, John TaylorNeural Network OptimizationSecond-Order Optimization

    School of Computing Australian National University Canberra, Australia · NVIDIA CA, United States

  16. Measuring the local shape of learning-rate transfer curves

    Oct 15, 2026Satoki Ishikawa, Rio Yokota, Ryo KarakidaNeural Network OptimizationNeural Network Training Dynamics

    Institute of Science Tokyo · AIST

  17. Approaching Backpropagation Accuracy with Weight Perturbation on MNIST

    Oct 15, 2026Patrick InoueGradient-Free OptimizationNeural Network Optimization

    The MNIST experiments originate from the author’s master’s thesis, submitted under the name Patrick Stricker at Albstadt-Sigmaringen University in March 2023 [4].

  18. When Is Muon Optimal? Radial Fragility and Minimax Geometry

    Oct 15, 2026Ao XuMuon OptimizerNeural Network Optimization

    Jilin University, Changchun 130012, China · Zhongguancun Academy, Beijing 100094, China

  19. One Size Does Not Fit All: Curvature-Aware Adaptive Perturbation for SAM

    Oct 14, 2026Alessia Rossi, Felipe Campelo, Gabriel L. OliveiraCurvature-Aware OptimizationSharpness-Aware Minimization

    School of Engineering Mathematics and Technology, University of Bristol, Bristol, Lovelace Building, Ada, Bristol, BS8 1TW, UK

  20. Trainable at Depth, Solvable at Small Size: What the Loss Surface of a Polynomial Activation Dictates for Designing, Initializing and Training Networks

    Oct 14, 2026Xiuyi YangNeural Network OptimizationNeural Network Training Dynamics

  21. Which Optimizer Leaks Most? Optimizer Implicit Bias as a Training-Data Privacy Controller

    Oct 13, 2026Sagi Habani, Yehonathan Refael, Ofir LindenbaumTraining Data Reconstruction AttacksModel Inversion Attacks

    Ben-Gurion University · Tel Aviv University · Bar Ilan University

  22. Which Geometry on Which Layer? Measured Sign-Spectral Assignment in Mixed-Optimizer Training

    Oct 12, 2026Hiroki Naganuma, Laura Gomezjurado, Mahdi Ghaznavi +5Spectral OptimizationDeep Learning Optimization

    Mila · Université de Montréal · Stanford University +4

  23. A Multi-Agent LLM System for Automated Tuning of Specialized Neural Network Models

    Oct 9, 2026Ivan Moloshnikov, Dmitry Kunitsyn, Alexander Sboev +3Neural Network OptimizationLarge Language Model-Guided Optimization

    National Research Centre “Kurchatov Institute”, 1 Akademika Kurchatova Square, Moscow, 123182, Moscow, Russia · National Research Nuclear University MEPhI, 31 Kashirskoye Shosse, Moscow, 115409, Moscow, Russia · Moscow Institute of Physics and Technology, 9 Institutskiy Lane, Dolgoprudny, 141701, Moscow, Russia +1

  24. A Scalable Learned Iterative Solver for Nonlinear MPC via Temporal Decomposition

    Oct 9, 2026Lukas Lüken, Sergio LuciaLearning to OptimizeNonlinear MPC

    Chair of Process Automation Systems, TU Dortmund University, Emil-Figge Straße 70, Dortmund, 44227, North Rhine-Westphalia, Germany

  25. Correlational Training of Morphological Neural Networks

    Oct 8, 2026Konstantinos Fotopoulos, Petros MaragosNeural Network Optimization

    HERON - Hellenic Robotics Center of Excellence, Athena Research Center, Marousi, Greece · Institute of Robotics, Athena Research Center, Marousi, Greece · School of ECE, National Technical University of Athens, Athens, Greece

  26. Optimizing Large Language Models with Chained LMOs

    Oct 7, 2026Sungyoon Kim, Kaan Ozkara, Youngsuk ParkLMO MethodsDeep Learning Optimization

    Department of Electrical Engineering, Stanford University · Amazon Annapurna Labs

  27. DeltaTTT: Layerwise Optimization for Nonlinear Recurrent Memory

    Oct 6, 2026Yining Li, Dongchen Han, Jie Fu +1Deep Learning OptimizationNeural Network Optimization

    Qiuzhen College, Tsinghua University · LeapLab, Tsinghua University · IQuest Research

  28. How Bregman Divergences Shape Shampoo

    Oct 6, 2026Bing Liu, Wenjie Zhou, Chengcheng Zhao +4Deep Learning OptimizationFunctional Bregman Divergences

    Zhejiang University · University of the Chinese Academy of Sciences · Peking University +2

  29. Singular parameters and missing limits in neural PDE solvers

    Oct 5, 2026Daniel FernándezNeural PDE SolversNeural Network Optimization

    Chair for Dynamics, Control, Machine Learning, and Numerics (Alexander von Humboldt Professorship), Department of Mathematics, Friedrich-Alexander-Universität Erlangen-Nürnberg, 91058 Erlangen, Germany.

  30. Muon Is Theoretically Wrong For Convolutions, But Empirically Effective

    Oct 5, 2026Thibaut Boissin, Thomas Massena, Mathieu Serrurier +1Newton-Schulz IterationNeural Network Optimization

    IRIT · IRIT, DTIPG - SNCF, UT3 · Institut de Recherche Technologique Saint-Exupery, France