Neural Network Optimization

Ask about this topic

Overview

Neural network optimization asks how to adjust a model’s parameters so it learns useful patterns efficiently and reliably. Common approaches include stochastic gradient descent and Adam, matrix-aware and spectral optimizers such as Muon, curvature-based methods, learning-rate schedules, and techniques that reduce memory or computation. Researchers also analyze loss landscapes, scaling laws, and initialization, and explore alternatives such as local or forward-only learning and analytic solutions for restricted models. A central tension is that gains depend on architecture, data, and training budget, while faster loss reduction does not necessarily improve generalization; connecting practical performance to reliable theory remains an open challenge.

Latest papers

CardsList

332 papers37 papers in the last four weeks, up 28% on the four weeks before. 0.2% of all new papers.

  1. A Non-Monotone Preconditioned Trust-Region Method for Neural Network Training

    May 14, 2026Andrea Angino, Bindi Çapriqi, Shega Likaj +2Neural Network OptimizationSecond-Order Optimization

    UniDistance Suisse · King Abdullah University of Science and Technology · Universit`a della Svizzera italiana

  2. Genetic algorithm vs. gradient descent for training a neural network architecture dedicated to low data regimes in small medical datasets

    May 13, 2026Amine Boukhari, Boglarka Ecsedi, Laszlo Papp +1Gradient DescentNeural Network Optimization

    Laboratory of medical information processing (LaTIM), INSERM, UMR 1101,university of western brittany · Center for Medical Physics and Biomedical Engineering, Medical University of Vienna · Georgia Institute of Technology

  3. NeuroRisk: Physics-Informed Neural Optimization for Risk-Aware Traffic Engineering

    May 13, 2026Yingming Mao, Ximeng Liu, Jingyi Cheng +8Stochastic OptimizationNeural Network Optimization

    Xi’an Jiaotong University · Shanghai Innovation Institute · Shanghai Jiao Tong University +2

  4. Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation

    May 12, 2026Kexuan Shi, Hanxuan Li, Zeju Qiu +3Neural Network Optimization

    1The Chinese University of Hong Kong · 2Max Planck Institute for Intelligent Systems · 3Westlake University

  5. Adaptive Multi-Scale Goodness Aggregation for Forward-Forward Learning

    May 11, 2026Salar Beigzad, Vansh VermaNeural Network OptimizationLocal Learning

    Computer Engineering University of St. Thomas Minnesota, USA

  6. OUIDecay: Adaptive Layer-wise Weight Decay for CNNs Using Online Activation Patterns

    May 11, 2026Alberto Fernández-Hernández, Jose I. Mestre, Cristian Pérez-Corral +3Deep Learning OptimizationWeight Decay

    Universitat Politècnica de València, Valencia, Spain · Universitat Jaume I, Castelló de la Plana, Spain · Openchip & Software Technologies S.L., Spain

  7. Rethinking Constraint Awareness for Efficient State Embedding of Neural Routing Solver

    May 11, 2026Canhong Yu, Changliang Zhou, Rongsheng Chen +2Neural Network OptimizationCombinatorial Optimization

    College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China · School of Automation and Intelligent Manufacturing, Southern University of Science and Technology, Shenzhen, China · Guangdong Provincial Key Laboratory of Fully Actuated System Control Theory and Technology, Southern University of Science and Technology, Shenzhen, China +1

  8. Optimizer-Induced Mode Connectivity: From AdamW to Muon

    May 11, 2026Fangzhao Zhang, Sungyoon Kim, Erica Zhang +2Mode ConnectivityNeural Network Optimization

    Stanford University

  9. Chebyshev Center-Based Direction Selection for Multi-Objective Optimization and Training PINNs

    May 11, 2026Hoyeol Yoon, Seoungbin Bae, Nam Ho-Nguyen +1Neural PDE SolversNeural Network Optimization

    Department of Industrial & Systems Engineering, KAIST · Discipline of Business Analytics, The University of Sydney · Department of Mathematical Sciences, Seoul National University

  10. A Controlled Diagnostic Study of Hardware-Induced Distortions in Hardware-Aware Training

    May 10, 2026Yunxuan Fang, Xinhe WangCompute-in-MemoryNeural Network Optimization

    Beihang University Beijing, China

  11. Learning-Augmented Scalable Linear Assignment Problem Optimization via Neural Dual Warm-Starts

    May 10, 2026Ilay Yavlovich, Jad Agbaria, Muhamed Mhamed +2Neural Network OptimizationCombinatorial Optimization

    Department of Electrical and Computer Engineering, Technion – Israel Institute of Technology, Haifa, Israel.

  12. Fitting Multilinear Polynomials for Logic Gate Networks

    May 9, 2026Youngsung KimBoolean Function LearningNeural Network Optimization

    Department of AI; and Department of ECE, Inha University

  13. Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit

    May 8, 2026Konstantin Riedl, Konstantinos Spiliopoulos, Justin SirignanoOverparameterized Neural NetworkNeural Network Optimization

    Mathematical Institute University of Oxford · Department of Mathematics & Statistics Boston University

  14. Convex Optimization of Spiking Neural Networks with Global and Approximation Guarantees

    May 8, 2026Himanshu Udupi, Calvin Ang, Sungyoon Kim +2Neural Network OptimizationRecurrent Neural Networks

    University of Illinois Urbana-Champaign

  15. Adaptive Regularization for Sparsity Control in Bregman-Based Optimizers

    May 8, 2026Ahmad Aloradi, Tim Roith, Emanuël A. P. Habets +1Neural Network OptimizationSparse Neural Networks

    Department of Data Science, FAU Erlangen-Nürnberg, Erlangen, Germany · 2International Audiolabs, FAU Erlangen-Nürnberg, Erlangen, Germany · School of Computation, Information and Technology, Technical University of Munich +1

  16. OrScale: Orthogonalised Optimization with Layer-Wise Trust-Ratio Scaling

    May 8, 2026Yuxuan Lou, Yang YouOrthogonal OptimizationNeural Network Optimization

    National University of Singapore

  17. Hessian Surgery: Class-Targeted Post-Hoc Rebalancing via Hessian Spike Perturbation

    May 8, 2026Hugo Vigna, Samuel BontempsClass-Imbalanced LearningNeural Network Optimization

    CentraleSupélec – Université Paris-Saclay · ESILV – Léonard de Vinci

  18. Weight-Decay Turns Transformer Loss Landscapes Villani: Functional-Analytic Foundations for Optimization and Generalization

    May 7, 2026Abhijit Das, Sayantan DuttaWeight DecayNeural Network Optimization

    GE HealthCare, Bangalore 560066, Karnataka, India

  19. Fast Gauss-Newton for Multiclass Cross-Entropy

    May 7, 2026Mikalai Korbit, Mario ZanonGauss-Newton OptimizationNeural Network Optimization

    IMT School for Advanced Studies Lucca

  20. Estimating Implicit Regularization in Deep Learning

    May 6, 2026Joseph H. Rudoler, Kevin Tan, Giles Hooker +1Neural Network OptimizationImplicit Bias

    University of Pennsylvania

  21. Exact Dual Geometry of SOC-ICNN Value Functions

    May 6, 2026Kang Liu, Jianchen Hu, Wei PengSecond-Order Cone ProgrammingNeural Network Optimization

    School of Future Technology Xi’an Jiaotong University Xi’an, China · School of Future Technology School of Automation Science and Engineering Xi’an Jiaotong University Xi’an, China · School of Automation Science and Engineering Xi’an Jiaotong University Xi’an, China

  22. Gradient Scaling Effects in Adaptive Spectral PINNs for Stiff Nonlinear ODEs

    May 6, 2026Isabela M. Yepes, Pavlos ProtopapasDynamical SystemsNeural Network Optimization

    Harvard University

  23. Nora: Normalized Orthogonal Row Alignment for Scalable Matrix Optimizer

    May 5, 2026Jinghui Yuan, Jiaxuan Zou, Shuo Wang +2Neural Network OptimizationMatrix Optimization

    1. School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University. · 2. School of Mathematics and Statistics, Xi’an Jiaotong University. · 4. Gaoling School of Artificial Intelligence, Renmin University of China. +1

  24. Communication Dynamics Neural Networks: FFT-Diagonalized Layers for Improved Hessian Conditioning at Reduced Parameter Count

    May 4, 2026Lurong PanNeural Network OptimizationFeedforward Neural Networks

  25. Universal Smoothness via Bernstein Polynomials: A Constructive Approximation Approach for Activation Functions

    May 4, 2026Wentao Zhang, Yutong Zhang, Yifan Zhu +1Efficient Neural Network InferenceNeural Network Optimization

    Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · College of Computer Science, Sichuan University, Chengdu, China

  26. Training Non-Differentiable Networks via Optimal Transport

    May 3, 2026An T. LeBlack-Box OptimizationNeural Network Optimization

    Center for AI Research, VinUniversity, Vietnam · Intelligent Autonomous Systems, TU Darmstadt, Germany

  27. A Theory of Saddle Escape in Deep Nonlinear Networks

    May 2, 2026Divit Rawal, Michael R. DeWeeseNeural Network OptimizationNeural Network Training Dynamics

    Department of Physics University of California, Berkeley Berkeley, CA 94720

  28. NLPOpt-Net: A Learning Method for Nonlinear Optimization with Feasibility Guarantees

    Apr 30, 2026Bimol Nath Roy, Rahul Golder, MM Faruque HasanNeural Network OptimizationLearning to Optimize

    Texas A&M University

  29. NeuroPlastic: A Plasticity-Modulated Optimizer for Biologically Inspired Learning Dynamics

    Apr 29, 2026Douglas Jiang, Yuechen Wang, Jiayi Wang +3Neural Network OptimizationAdaptive Gradient Methods

    Department of Neurology, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA 02115, USA · Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA 02115, USA · Equal contributions +2

  30. Scaling Probabilistic Transformer via Efficient Cross-Scale Hyperparameter Transfer

    Apr 28, 2026Penghao Kuang, Haoyi Wu, Kewei TuTransformerNeural Network Optimization

    School of Information Science and Technology, ShanghaiTech University