Deep Learning Optimization

Ask about this topic

Overview

It studies how parameter updates, learning-rate and batch schedules, regularization, and model structure shape neural-network training. Common approaches range from SGD and AdamW to adaptive preconditioners such as Shampoo, curvature-aware and sharpness-aware methods, and matrix or spectral optimizers such as Muon; theory examines convergence, stability, and geometry, while systems work reduces memory, communication, and step time. Evaluation spans vision and tabular models, language-model pretraining, diffusion, and continual learning, measuring accuracy and loss alongside time to target and memory use. Optimizer rankings depend on architecture, batch size, training horizon, and compute budget, leaving a tension between per-step progress, wall-clock efficiency, generalization, and reliable tuning.

Latest papers

CardsList

208 papers39 papers in the last four weeks, up 457% on the four weeks before. 0.2% of all new papers.

  1. Learn&Drop: Fast Learning of CNNs based on Layer Dropping

    Apr 25, 2026Giorgio Cruciata, Luca Cruciata, Liliana Lo Presti +2Deep Learning OptimizationConvolutional Neural Networks

    Engineering Department, University of Palermo, V.le delle Scienze, Ed. 6, Palermo, 90128, Italy. · Computer Vision Lab, Delft University of Technology, Van Mourik Broekmanweg 6,2026 Delft, 2628 XE, The Netherlands.

  2. A Layer Separation Optimization Framework for Cross-Entropy Training in Deep Learning

    Apr 25, 2026Yaru Liu, Michael K. Ng, Yiqi GuDeep Learning OptimizationNonconvex Optimization

    School of Mathematical Sciences, University of Electronic Science and Technology of China, Sichuan 611731, China · Department of Mathematics, Hong Kong Baptist University, Kowloon Tong, Hong Kong, China

  3. Training Time Prediction for Mixed Precision-based Distributed Training

    Apr 17, 2026Minchul Kang, Changyong Shin, Jinwoo Jeong +5Deep Learning OptimizationDistributed Training

    Korea University · KT Corporation

  4. Natural gradient descent with momentum

    Apr 16, 2026Anthony Nouy, Agustín SomacalDeep Learning OptimizationRiemannian Optimization

    Centrale Nantes, Nantes Universit´e, Laboratoire de Math´ematiques Jean Leray UMR CNRS 6629

  5. Benchmarking Optimizers for MLPs in Tabular Deep Learning

    Apr 16, 2026Yury Gorishniy, Ivan Rubachev, Dmitrii Feoktistov +1Multilayer PerceptronsDeep Learning Optimization

    α Yandex · β HSE University

  6. CLion: Efficient Cautious Lion Optimizer with Enhanced Generalization

    Apr 16, 2026Feihu Huang, Guanyi Zhang, Songcan ChenStochastic OptimizationSign-Based Optimization

    College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China

  7. Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations

    Apr 14, 2026Tong Zhang, Jiangning Zhang, Zhucun Xue +9Deep Learning OptimizationAdaptive Gradient Methods

    Zhejiang University, APRIL Lab · Fudan University · Shanghai Jiaotong University +1

  8. Evolution-Inspired Sample Competition for Deep Neural Network Optimization

    Apr 14, 2026Ying Zheng, Yiyi Zhang, Yi Wang +1Instance ReweightingDeep Learning Optimization

    Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, China. · Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China.

  9. VISTA: Validation-Informed Trajectory Adaptation via Self-Distillation

    Apr 13, 2026Eli Corn, Daphna WeinshallNeural Network GeneralizationDeep Learning Optimization

    Department of Computer Science, Hebrew University of Jerusalem, Jerusalem, Israel.

  10. A Faster Path to Continual Learning

    Apr 13, 2026Wei Li, Hangjie Yuan, Zixiang Zhao +3Continual LearningNeural Network Optimization

    College of Computer Science, Sichuan University, China · College of Computer Science and Technology, Zhejiang University, China · Photogrammetry and Remote Sensing Lab, ETH Zürich, Switzerland +3

  11. AdaCubic: An Adaptive Cubic Regularization Optimizer for Deep Learning

    Apr 10, 2026Ioannis Tsingalis, Constantine Kotropoulos, Corentin BriatDeep Learning Optimization

    Aristotle University of Thessaloniki, Greece · University of Applied Sciences Northwestern Switzerland, Muttenz, Switzerland

  12. Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima

    Apr 10, 2026Huanran Chen, Huaqing Zhang, Xiao Li +3Language Model PretrainingNeural Network Generalization

    Tsinghua University · ByteDance Seed

  13. A Self-Attentive Meta-Optimizer with Group-Adaptive Learning Rates and Weight Decay

    Apr 10, 2026JiangBo Zhao, ZhaoXin LiuDeep Learning OptimizationLearning to Optimize

  14. Product-Stability: Provable Convergence for Gradient Descent on the Edge of Stability

    Apr 3, 2026Eric GanDeep Learning OptimizationEdge of Stability

    Independent Researcher.

  15. Sven: Singular Value Descent as a Computationally Efficient Natural Gradient Method

    Apr 1, 2026Samuel Bright-Thonney, Thomas R. Harvey, Andre Lukas +1Deep Learning OptimizationNatural Gradient Methods

    Department of Physics, Massachusetts Institute of Technology · The NSF Institute for Artificial Intelligence and Fundamental Interactions · Rudolf Peierls Centre for Theoretical Physics, University of Oxford +2

  16. Second-Order, First-Class: A Composable Stack for Curvature-Aware Training

    Mar 26, 2026Mikalai Korbit, Mario ZanonCurvature-Aware OptimizationDeep Learning Optimization

    IMT School for Advanced Studies Lucca Lucca, Italy

  17. Unveiling Hidden Convexity in Deep Learning: a Sparse Signal Processing Perspective

    Mar 25, 2026Emi Zeger, Mert PilanciReLU Neural NetworksDeep Learning Optimization

    Department of Electrical Engineering, Stanford University

  18. dynActivation: A Trainable Activation Family for Adaptive Nonlinearity

    Mar 23, 2026Alois BachmannNeural Network Activation FunctionsNeural Network Training Dynamics

    Ruprecht-Karls-Universität Heidelberg

  19. Democratizing AI: A Comparative Study in Deep Learning Efficiency and Future Trends in Computational Processing

    Mar 21, 2026Lisan Al Amin, Md Ismail Hossain, Rupak Kumar Das +2GPU AccelerationDeep Learning Optimization

    Cognitive Links, Maryland, USA · North South University, Bangladesh · Pennsylvania State University, USA +2

  20. LassoFlexNet: Flexible Neural Architecture for Tabular Data

    Mar 21, 2026Kry Yik Chau Lui, Cheng Chi, Kishore Basu +1Tabular MLTabular Representation Learning

    RBC Borealis · RBC

  21. Beyond Muon: MUD (MomentUm Decorrelation) for Faster Transformer Training

    Mar 18, 2026Ben S. Southworth, Stephen ThomasDeep Learning OptimizationMuon Optimizer

    Theoretical Division, Los Alamos National Laboratory, USA. · Lehigh University, Computer Science

  22. Ghosts of Softmax: Complex Singularities That Limit Safe Step Sizes in Cross-Entropy

    Mar 13, 2026Piyush SaoDeep Learning OptimizationConvergence Analysis

    Oak Ridge National Laboratory, Oak Ridge, TN, USA

  23. A Geometrically-Grounded Drive for MDL-Based Optimization in Deep Learning

    Mar 12, 2026Ming Lei, Shufan Wu, Christophe BaehrDeep Learning OptimizationMinimum Description Length

    School of Aeronautics and Astronautics, Shanghai JiaoTong University, Shanghai China · Météo-France/CNRS CNRM/GAME UMR 3589 and the Mathematical Institute of Toulouse, France

  24. Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning

    Mar 10, 2026Yechen Zhang, Shuhao Xing, Junhao Huang +5Deep Learning OptimizationMuon Optimizer

    Shanghai Jiao Tong University · Shanghai AI Laboratory · Fudan University

  25. Improving Continual Learning for Gaussian Splatting based Environments Reconstruction on Commercial Off-the-Shelf Edge Devices

    Mar 9, 2026Ivan Zaino, Matteo Risso, Daniele Jahier Pagliari +33DGS Training3D Gaussian Splatting

    Politecnico di Torino · VERSES

  26. Beyond the Markovian Assumption: Robust Optimization via Fractional Weyl Integrals in Imbalanced Data

    Mar 9, 2026Gustavo A. DorregoDeep Learning OptimizationClass-Imbalanced Learning

    Department of Mathematics, Universidad Nacional del Nordeste

  27. Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementation

    Mar 9, 2026Jianlong Chen, Zhiming ZhouSharpness-Aware MinimizationDeep Learning Optimization

    Key Laboratory of Interdisciplinary Research of Computation and Economics Shanghai University of Finance and Economics

  28. Dynamic Momentum Recalibration in Online Gradient Learning

    Mar 6, 2026Zhipeng Yao, Rui Yu, Guisong Chang +3Deep Learning OptimizationStochastic Gradient Descent

    Shenyang University of Chemical Technology · University of Louisville · Northeastern University

  29. When to restart? Exploring escalating restarts on convergence

    Mar 4, 2026Ayush K. Varshney, Šarūnas Girdzijauskas, Konstantinos Vandikas +1Deep Learning OptimizationLearning Rate Scheduling

    Ericsson Research, Ericsson AB, Sweden · KTH Royal Institute of Technology, Stockholm, Sweden · RISE Research Institutes of Sweden, Stockholm, Sweden

  30. Leveraging Arbitrary Data Sources for AI-Generated Image Detection Without Sacrificing Generalization

    Feb 28, 2026Qinghui He, Haifeng Zhang, Xiuli Bi +3AI-Generated Image DetectionDeep Learning Optimization

    Chongqing Key Laboratory of Image Cognition, School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, China · Chongqing Key Laboratory of Image Cognition, School of Artificial Intelligence, Chongqing University of Posts and Telecommunications, Chongqing, China · Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China