Deep Learning Optimization

Ask about this topic

Overview

It studies how parameter updates, learning-rate and batch schedules, regularization, and model structure shape neural-network training. Common approaches range from SGD and AdamW to adaptive preconditioners such as Shampoo, curvature-aware and sharpness-aware methods, and matrix or spectral optimizers such as Muon; theory examines convergence, stability, and geometry, while systems work reduces memory, communication, and step time. Evaluation spans vision and tabular models, language-model pretraining, diffusion, and continual learning, measuring accuracy and loss alongside time to target and memory use. Optimizer rankings depend on architecture, batch size, training horizon, and compute budget, leaving a tension between per-step progress, wall-clock efficiency, generalization, and reliable tuning.

Latest papers

CardsList

208 papers39 papers in the last four weeks, up 457% on the four weeks before. 0.2% of all new papers.

  1. UTA: Universal Training Adaptive Learning Rate Decay

    Oct 26, 2026Anda Tang, Junjie Wang, Xuanlin Yang +2Learning Rate SchedulingDeep Learning Optimization

    the State Key Lab of General AI School of Intelligence Science and Technology Peking University Beijing, China · the Pazhou Laboratory (Huangpu) · the Institute for Artificial Intelligence

  2. Where Rounding Happens Matters: Rounding-Location Alignment Stabilizes LLM Reinforcement Learning

    Oct 23, 2026Yijing Ke, Tianle Cai, Ruosong WangRL for Language ModelsRL Stability

    ByteDance Seed · Peking University

  3. Sharpness-aware Orthogonalized Momentum Algorithms

    Oct 23, 2026Geetika, Shivani Singh Chauhan, Gauranshi Gupta +3Deep Learning OptimizationSharpness-Aware Minimization

    Department of Computer Science and Engineering Indraprastha Institute of Information Technology, Delhi, India. · Department of Mathematics Indraprastha Institute of Information Technology, Delhi, India.

  4. RSI-Opt: Recursive Self-Improvement Optimizer Agents For Model Training

    Oct 23, 2026Binghang Lu, Yige Sun, Guang LinLarge Language Model-Guided OptimizationLearning to Optimize

    Purdue University · Independent Researcher

  5. Hybrid Momentum Methods for Optimizing Deep Learning Models

    Oct 20, 2026Haifeng Xu, Yijin Ren, Chenyu Zhou +3Deep Learning OptimizationMomentum Methods

    Shanghai University of Finance and Economics · Columbia University · Shanghai Jiao Tong University +2

  6. How Does Muon Work in Private Deep Learning? Scalable Differentially Private Training via Orthogonalization

    Oct 17, 2026Alexander Gaponov, Grigory Malinovsky, Egor Shulgin +1Muon OptimizerDeep Learning Optimization

    King Abdullah University of Science and Technology (KAUST) KAUST Center of Excellence for Generative AI Thuwal, Saudi Arabia

  7. Large Diffusion Models Need More Than EMA

    Oct 16, 2026Chuyu Hu, Huanran Chen, Shitong Shao +1Deep Learning OptimizationDiffusion Models

    Tsinghua University · Shanghai Qi Zhi Institute · The Hong Kong University of Science and Technology

  8. A Refresh Interval Does Not Fully Specify a Refresh Schedule: Learning-Rate-Dependent Placement in Kronecker-Preconditioned Language Model Training

    Oct 13, 2026Xinyu Wang, Zhiwei Zhai, Yu Gu +6Deep Learning OptimizationEfficient Language Model Training

    McGill University · The Chinese University of Hong Kong · University of Manitoba +1

  9. Which Optimizer Leaks Most? Optimizer Implicit Bias as a Training-Data Privacy Controller

    Oct 13, 2026Sagi Habani, Yehonathan Refael, Ofir LindenbaumTraining Data Reconstruction AttacksModel Inversion Attacks

    Ben-Gurion University · Tel Aviv University · Bar Ilan University

  10. Same Function, Different Trajectory: No Standard Optimizer Is Equivariant to Transformer Reparameterization

    Oct 13, 2026Bum Jun KimDeep Learning OptimizationTransformer

    The University of Tokyo

  11. ATTUNE: Agentic Throughput Tuning for LLM Training

    Oct 13, 2026Leo Fan, Hongyang Wang, Wenbo Chen +10Efficient Language Model TrainingDeep Learning Optimization

    Amazon

  12. An FPGA-Based Spiking Neural Network for Spatiotemporal Pattern Learning and Classification

    Oct 12, 2026Jingang Jin, Yue Ma, Zhenhang Zhang +1Spiking Neural NetworksFPGA-Based Neural Network Acceleration

    Department of Electrical Engineering and Computer Science Syracuse University, Syracuse, New York, USA

  13. Deeply unrolled alternating projections for phase retrieval

    Oct 12, 2026Jiayi Yao, Vasileios Charisopoulos, Rebecca WillettImage Inverse ProblemsDeep Learning Optimization

    Electrical & Computer Engineering, University of Washington · NSF-Simons National Institute for Theory and Mathematics in Biology · Department of Computer Science, University of Chicago +3

  14. Which Geometry on Which Layer? Measured Sign-Spectral Assignment in Mixed-Optimizer Training

    Oct 12, 2026Hiroki Naganuma, Laura Gomezjurado, Mahdi Ghaznavi +5Spectral OptimizationDeep Learning Optimization

    Mila · Université de Montréal · Stanford University +4

  15. Efficient Preconditioned Optimization Methods with Frequent Directions

    Oct 12, 2026Sifan Yang, Yuanyu Wan, Yibo Wang +1Adaptive Gradient MethodsOnline Convex Optimization

    State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · School of Software Technology, Zhejiang University, Ningbo, China +1

  16. On the Depth Transfer of Neural Optimizers via L-Smoothness

    Oct 9, 2026Maximilian Wong, Chenyu Xu, Jiajin Li +1Deep Learning OptimizationHyperparameter Transfer

    University of Chicago · Peking University · Sauder School of Business University of British Columbia +1

  17. The RNN Homunculus: Learning to Gate Long-Term Memory

    Oct 9, 2026Zeyu Fan, Alexandru-Dragos Manolache, Jan van GemertLong-Range Dependency ModelingRecurrent Neural Networks

    Delft University of Technology, Mekelweg 5, 2628 CD Delft, The Netherlands

  18. Towards Easier-to-Tune Training Algorithms: New Results from AlgoPerf

    Oct 9, 2026Priya Kasimbeg, Ahmed Khaled, Aahlad Chandrabhatta +9Deep Learning OptimizationBenchmark Design

    Alice · Goodfire · University of Tübingen +1

  19. Optimizing Large Language Models with Chained LMOs

    Oct 7, 2026Sungyoon Kim, Kaan Ozkara, Youngsuk ParkLMO MethodsDeep Learning Optimization

    Department of Electrical Engineering, Stanford University · Amazon Annapurna Labs

  20. Spectrally Targeted Muon

    Oct 7, 2026Vishrut Goyal, Rohan RamkumarSpectral OptimizationDeep Learning Optimization

    Department of Computer Science, Stanford University · Department of Electrical Engineering, Stanford University.

  21. Eigenvalues of the Hessian in Deep Learning: The Origin of Symmetry and Its Breaking

    Oct 7, 2026Yossi ArjevaniSymmetry BreakingDeep Learning Optimization

    Yossi Arjevani

  22. Decoupled Optimization for Teacher-Student Semi-Supervised Learning via a Pioneer Student

    Oct 7, 2026Haorong Han, Jidong Yuan, Chixuan Wei +1Semi-Supervised LearningTeacher-Student Learning

    School of Computer Science and Technology Beijing Jiaotong University · School of Data Science and Intelligent Media Communication University of China

  23. ORACLE: Optimizer-Relative Alignment for Constrained LEarning

    Oct 6, 2026Utkarsh Grover, Wyatt Mackey, Kaixun Hua +2Constrained OptimizationDeep Learning Optimization

    University of South Florida · DEVCOM Army Research Lab

  24. The Best Optimizer Depends on Batch Size

    Oct 6, 2026Xingyu Dang, Kaiyue Wen, Sadhika MalladiDeep Learning OptimizationHyperparameter Transfer

    Princeton University · Stanford University · University of California, San Diego

  25. DeltaTTT: Layerwise Optimization for Nonlinear Recurrent Memory

    Oct 6, 2026Yining Li, Dongchen Han, Jie Fu +1Deep Learning OptimizationNeural Network Optimization

    Qiuzhen College, Tsinghua University · LeapLab, Tsinghua University · IQuest Research

  26. How Bregman Divergences Shape Shampoo

    Oct 6, 2026Bing Liu, Wenjie Zhou, Chengcheng Zhao +4Deep Learning OptimizationFunctional Bregman Divergences

    Zhejiang University · University of the Chinese Academy of Sciences · Peking University +2

  27. Does Muon Need Fine-Grained Spectral Shaping?

    Oct 5, 2026Meher Chaitanya, Tianyi Zhou, Aristides GionisDeep Learning OptimizationMuon Optimizer

    KTH Royal Institute of Technology Stockholm, Sweden

  28. Learning Pareto Stationary Fronts via Single-Pass Backpropagation

    Oct 5, 2026Elina Rojin Celik, Marcos Medeiros Raimundo, Isabel ValeraDeep Learning OptimizationMulti-Objective Optimization

    Department of Computer Science, Saarland University, Germany · Instituto de Computação, Universidade Estadual de Campinas, Brazil

  29. Mind the Drift: Diagonal Linear Networks Under Large Learning Rates

    Oct 5, 2026Aniket Sanyal, Tom Jacobs, Rebekka BurkholzEdge of StabilityDeep Learning Optimization

    CISPA Helmholtz Center for Information Security · Technical University of Munich · Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA)

  30. ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training

    Oct 5, 2026Yuanshi Liu, Boyuan Jiang, Liang Hou +4Spectral RegularizationDeep Learning Optimization

    School of Intelligence Science and Technology, Peking University · Kling Team