Deep Learning Optimization

Overview

It studies how parameter updates, learning-rate and batch schedules, regularization, and model structure shape neural-network training. Common approaches range from SGD and AdamW to adaptive preconditioners such as Shampoo, curvature-aware and sharpness-aware methods, and matrix or spectral optimizers such as Muon; theory examines convergence, stability, and geometry, while systems work reduces memory, communication, and step time. Evaluation spans vision and tabular models, language-model pretraining, diffusion, and continual learning, measuring accuracy and loss alongside time to target and memory use. Optimizer rankings depend on architecture, batch size, training horizon, and compute budget, leaving a tension between per-step progress, wall-clock efficiency, generalization, and reliable tuning.

Latest papers

CardsList

217 papers13 papers in the last four weeks, down 65% on the four weeks before. 0.1% of all new papers.

  1. Predator-Prey Model: Driven Hunt for Accelerated Grokking

    Sep 10, 2025I. A. Lopatin, S. V. Kozyrev, A. N. PechenDeep Learning OptimizationGrokking

  2. Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimization

    Sep 3, 2025Wu Lin, Scott C. Lowe, Felix Dangel +3Deep Learning OptimizationNeural Network Optimization

  3. Position: Adopt Constraints Over Fixed Penalties in Deep Learning

    May 27, 2025Juan Ramirez, Meraj Hashemizadeh, Simon Lacoste-JulienDeep Learning OptimizationConstrained Optimization

  4. Preconditioned Inexact Stochastic ADMM for Deep Model

    Feb 15, 2025Shenglong Zhou, Ouya Wang, Ziyan Luo +2Stochastic OptimizationDeep Learning Optimization

  5. Convergence Rate Analysis of LION

    Nov 12, 2024Yiming Dong, Huan Li, Zhouchen LinSign-Based OptimizationDeep Learning Optimization