Deep Learning Optimization

Overview

It studies how parameter updates, learning-rate and batch schedules, regularization, and model structure shape neural-network training. Common approaches range from SGD and AdamW to adaptive preconditioners such as Shampoo, curvature-aware and sharpness-aware methods, and matrix or spectral optimizers such as Muon; theory examines convergence, stability, and geometry, while systems work reduces memory, communication, and step time. Evaluation spans vision and tabular models, language-model pretraining, diffusion, and continual learning, measuring accuracy and loss alongside time to target and memory use. Optimizer rankings depend on architecture, batch size, training horizon, and compute budget, leaving a tension between per-step progress, wall-clock efficiency, generalization, and reliable tuning.

Latest papers

CardsList

217 papers13 papers in the last four weeks, down 65% on the four weeks before. 0.1% of all new papers.

  1. Error Analysis of Deep Learning

    Nov 15, 2026Yuling Jiao, Peiying Wu, Jerry Zhijian Yang +1Statistical Learning TheoryDeep Learning Optimization

  2. Delayed Effects of Temporary Weight Decay in Muon Pretraining

    Nov 13, 2026Md Rifat Arefin, Ravid Shwartz-Ziv, Ernie Chang +6Deep Learning OptimizationWeight Decay

  3. Learning Rate Transfer and Feature Learning Across Depth for Constrained Spectral Optimizers: Complete Scion

    Nov 11, 2026Eline Pot, Laurent Lefebvre, Mathieu Segond +1Deep Learning OptimizationHyperparameter Transfer

  4. On the Few-Shot Adaptation of Vision-Language Models: A Few Gradient Signs Are All You Need

    Nov 5, 2026Nairouz Mrabah, Ismail Ben Ayed, Mahdi S. Hosseini +1Few-Shot VLM AdaptationVLM Adaptation

  5. Architecture-Aware Mixed Precision for Replica-Exchange Training of Geometry-Aware Neural Operators with Uncertainty Quantification

    Nov 2, 2026Ujin Jeon, Michael Penwarden, Wenqian Chen +1Mixed-Precision TrainingGeometry-Aware Neural Operators

  6. Adaptive Optimizers Can Get a Policy Ranking Wrong: Momentum, Dependent Data, and a Two-Run Remedy

    Oct 31, 2026Sunyoung An, Xiaoming HuoTemporal-Difference LearningPolicy Evaluation

  7. FARO: Function-Space Adaptive Return-to-Risk Optimization via Cross-Block Equal-Risk-Contribution Budget Allocation

    Oct 29, 2026Xiaojun Wu, Cehao Yang, Xuhui Jiang +3Deep Learning OptimizationLLM Training

  8. UTA: Universal Training Adaptive Learning Rate Decay

    Oct 26, 2026Anda Tang, Junjie Wang, Xuanlin Yang +2Learning Rate SchedulingDeep Learning Optimization

  9. Sharpness-aware Orthogonalized Momentum Algorithms

    Oct 23, 2026Geetika, Shivani Singh Chauhan, Gauranshi Gupta +3Deep Learning OptimizationSharpness-Aware Minimization

  10. RSI-Opt: Recursive Self-Improvement Optimizer Agents For Model Training

    Oct 23, 2026Binghang Lu, Yige Sun, Guang LinLarge Language Model-Guided OptimizationLearning to Optimize

  11. Hybrid Momentum Methods for Optimizing Deep Learning Models

    Oct 20, 2026Haifeng Xu, Yijin Ren, Chenyu Zhou +3Deep Learning OptimizationMomentum Methods

  12. How Does Muon Work in Private Deep Learning? Scalable Differentially Private Training via Orthogonalization

    Oct 17, 2026Alexander Gaponov, Grigory Malinovsky, Egor Shulgin +1Muon OptimizerDeep Learning Optimization

  13. Large Diffusion Models Need More Than EMA

    Oct 16, 2026Chuyu Hu, Huanran Chen, Shitong Shao +1Deep Learning OptimizationDiffusion Models

  14. Which Optimizer Leaks Most? Optimizer Implicit Bias as a Training-Data Privacy Controller

    Oct 13, 2026Sagi Habani, Yehonathan Refael, Ofir LindenbaumTraining Data Reconstruction AttacksModel Inversion Attacks

  15. An FPGA-Based Spiking Neural Network for Spatiotemporal Pattern Learning and Classification

    Oct 12, 2026Jingang Jin, Yue Ma, Zhenhang Zhang +1Mixed-Precision TrainingSpiking Neural Networks

  16. Deeply unrolled alternating projections for phase retrieval

    Oct 12, 2026Jiayi Yao, Vasileios Charisopoulos, Rebecca WillettImage Inverse ProblemsDeep Learning Optimization

  17. Which Geometry on Which Layer? Measured Sign-Spectral Assignment in Mixed-Optimizer Training

    Oct 12, 2026Hiroki Naganuma, Laura Gomezjurado, Mahdi Ghaznavi +5Spectral OptimizationDeep Learning Optimization

  18. Efficient Preconditioned Optimization Methods with Frequent Directions

    Oct 12, 2026Sifan Yang, Yuanyu Wan, Yibo Wang +1Adaptive Gradient MethodsOnline Convex Optimization

  19. On the Depth Transfer of Neural Optimizers via L-Smoothness

    Oct 9, 2026Maximilian Wong, Chenyu Xu, Jiajin Li +1Deep Learning OptimizationHyperparameter Transfer

  20. The RNN Homunculus: Learning to Gate Long-Term Memory

    Oct 9, 2026Zeyu Fan, Alexandru-Dragos Manolache, Jan van GemertLong-Range Dependency ModelingRecurrent Neural Networks

  21. Towards Easier-to-Tune Training Algorithms: New Results from AlgoPerf

    Oct 9, 2026Priya Kasimbeg, Ahmed Khaled, Aahlad Chandrabhatta +9Deep Learning OptimizationBenchmark Design

  22. Optimizing Large Language Models with Chained LMOs

    Oct 7, 2026Sungyoon Kim, Kaan Ozkara, Youngsuk ParkLMO MethodsDeep Learning Optimization

  23. Spectrally Targeted Muon

    Oct 7, 2026Vishrut Goyal, Rohan RamkumarSpectral OptimizationDeep Learning Optimization

  24. Decoupled Optimization for Teacher-Student Semi-Supervised Learning via a Pioneer Student

    Oct 7, 2026Haorong Han, Jidong Yuan, Chixuan Wei +1Semi-Supervised LearningTeacher-Student Learning