Deep Learning Optimization

Ask about this topic

Momentum

7 papers in the last four weeks, up 133% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 132

All topics
CardsList

  1. Eigenvalues of the Hessian in Deep Learning: The Origin of Symmetry and Its Breaking

    Oct 7, 2026Yossi ArjevaniSymmetry BreakingDeep Learning Optimization

    Yossi Arjevani

  2. Decoupled Optimization for Teacher-Student Semi-Supervised Learning via a Pioneer Student

    Oct 7, 2026Haorong Han, Jidong Yuan, Chixuan Wei +1Semi-Supervised LearningTeacher-Student Learning

    School of Computer Science and Technology Beijing Jiaotong University · School of Data Science and Intelligent Media Communication University of China

  3. ORACLE: Optimizer-Relative Alignment for Constrained LEarning

    Oct 6, 2026Utkarsh Grover, Wyatt Mackey, Kaixun Hua +2Constrained OptimizationDeep Learning Optimization

    University of South Florida · DEVCOM Army Research Lab

  4. The Best Optimizer Depends on Batch Size

    Oct 6, 2026Xingyu Dang, Kaiyue Wen, Sadhika MalladiDeep Learning OptimizationHyperparameter Transfer

    Princeton University · Stanford University · University of California, San Diego

  5. DeltaTTT: Layerwise Optimization for Nonlinear Recurrent Memory

    Oct 6, 2026Yining Li, Dongchen Han, Jie Fu +1Deep Learning OptimizationNeural Network Optimization

    Qiuzhen College, Tsinghua University · LeapLab, Tsinghua University · IQuest Research

  6. How Bregman Divergences Shape Shampoo

    Oct 6, 2026Bing Liu, Wenjie Zhou, Chengcheng Zhao +4Deep Learning OptimizationFunctional Bregman Divergences

    Zhejiang University · University of the Chinese Academy of Sciences · Peking University +2

  7. Does Muon Need Fine-Grained Spectral Shaping?

    Oct 5, 2026Meher Chaitanya, Tianyi Zhou, Aristides GionisDeep Learning OptimizationMuon Optimizer

    KTH Royal Institute of Technology Stockholm, Sweden

  8. Learning Pareto Stationary Fronts via Single-Pass Backpropagation

    Oct 5, 2026Elina Rojin Celik, Marcos Medeiros Raimundo, Isabel ValeraDeep Learning OptimizationMulti-Objective Optimization

    Department of Computer Science Saarland University, Germany · Instituto de Computação Universidade Estadual de Campinas, Brazil

  9. Mind the Drift: Diagonal Linear Networks Under Large Learning Rates

    Oct 5, 2026Aniket Sanyal, Tom Jacobs, Rebekka BurkholzEdge of StabilityDeep Learning Optimization

    CISPA Helmholtz Center for Information Security · Technical University of Munich · Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA)

  10. ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training

    Oct 5, 2026Yuanshi Liu, Boyuan Jiang, Liang Hou +4Spectral RegularizationDeep Learning Optimization

    School of Intelligence Science and Technology, Peking University · Kling Team

  11. SoftServe: A Scalable Quasi-Newton Method for Deep Learning

    Oct 1, 2026Joohwan Ko, Tetiana Parshakova, Diana Cai +1Deep Learning OptimizationSecond-Order Optimization

    University of Massachusetts Amherst · CCM, Flatiron Institute · Cornell University

  12. Increasing Width Allows Greedy Layer-wise Training to Rival End-to-End Backpropagation in Self-Supervised Learning

    Sep 30, 2026Syon Mansur, Joel ZylberbergDeep Learning OptimizationConvolutional Neural Networks

    Jules Stein Eye Institute University of California, Los Angeles Los Angeles, CA 90095, USA

  13. From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes

    Sep 30, 2026Yichen Wang, Fanghui Liu, Yudong ChenLanguage Model PretrainingDeep Learning Optimization

    Department of Computer Sciences, University of Wisconsin–Madison, USA. · School of Mathematical Sciences, Institute of Natural Sciences and MOE-LSC, Shanghai Jiao Tong University, China.

  14. EvE: An Alternate Optimizer to Adam

    Sep 28, 2026Shashank Raj, Kalyanmoy DebDeep Learning OptimizationDifferential Evolution

    Department of Computer Science The University of Alabama Tuscaloosa, AL 35487, USA

  15. Low-Rank Friction for Memory-Efficient Transformer Pretraining

    Sep 24, 2026Rajit Rajpal, Benedict LeimkuhlerDeep Learning OptimizationLow-Rank Compression

    School of Mathematics University of Edinburgh Edinburgh, UK EH9 3FD

  16. AdamX: Cosine similarity meets gradient descent

    Sep 10, 2026Francisco Caldas, Ruben Belo, Cláudia SoaresDeep Learning OptimizationAdaptive Gradient Methods

    NOVA School of Science of Technology Universidade Nova de Lisboa, Caparica, Portugal

  17. BrachistoneLR: A Brachistochrone-Inspired Learning-Rate Schedule and a Controlled Benchmark of Scheduling Policies

    Sep 8, 2026Md. Sadekur Rahman Roni, Md. Jalal uddin Chowdhury, Moutusi Dash NimiDeep Learning OptimizationLearning Rate Scheduling

    Department of Computer Science and Engineering Leading University Sylhet, Bangladesh · DeepNet Research and Development Lab Sylhet 3100, Bangladesh · Department of Electrical and Electronic Engineering Leading UniversitySep Sylhet, Bangladesh

  18. Beyond Dense Adam States: Adaptive Log-Space Quantization for Memory-Efficient Optimizers

    Aug 23, 2026Yan WangDeep Learning OptimizationMemory-Efficient Optimization

    Independent Researcher

  19. Activation-Keyed Momentum: An Anisotropic Momentum Update via the Delta Rule

    Aug 19, 2026Euijin Hong, Guannan QuDeep Learning OptimizationMomentum Methods

    Electrical and Computer Engineering, Carnegie Mellon University

  20. Momentum as Residual-Driven Multiplier Correction for Deep Learning Optimization

    Aug 13, 2026Zhixin Ren, Yau Lyu, Congrong Li +2Deep Learning OptimizationMomentum Methods

    Department of Mathmatical Science, Tsinghua University · School of Vehicle and Mobility, Tsinghua University · College of AI, Tsinghua University

  21. AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization

    Aug 3, 2026Alok Kumar Pandey, Umang Chaturvedi, Aatish Rana +1Deep Learning OptimizationStochastic Optimization Convergence

  22. Towards joint scaling laws with optimal batch size schedules

    Jul 30, 2026Jiaxiang Li, Zhiqi Bu, Shiyun XuDeep Learning OptimizationLanguage Model Scaling Laws

    Meta · Independent researcher

  23. Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training

    Jul 21, 2026Nuemaan MalikDeep Learning OptimizationMixture-of-Experts Language Models

    Independent Researcher

  24. Learning Faster without Deeper Networks: A*-Inspired Batch Selection for Efficient CNN Training

    Jul 17, 2026Anxhelo Shehu, Enes Stastoli, Arben CelaDeep Learning OptimizationConvolutional Neural Networks

    AI Laboratory, University Metropolitan Tirana, Tirana, Albania. · Laboratory of Images Signals and Intelligent Systems, ESIEE Paris, Paris, France.

  25. Reassessing Muon for Matrix Factorization

    Jul 14, 2026Ali Parviz, Gal Mishne, Alex CloningerDeep Learning OptimizationMuon Optimizer

    †. Halicioğlu Data Science Institute, UC San Diego. · ∗. Department of Mathematics, UC San Diego.

  26. LionVote: Per-Layer Learning Rate Adaptation for Lion

    Jul 10, 2026Kris AtallahDeep Learning OptimizationLayer-Wise Learning Rate Adaptation

    New York University, New York, USA · New York University New York, NY, USA

  27. Beyond Backpropagation: Monte Carlo Method Can Train Deep Neural Networks

    Jul 9, 2026Hong ZhaoDeep Learning Optimization

    Department of Physics, Xiamen University, Xiamen 361005, China · Lanzhou Center for Theoretical Physics, Lanzhou University, Lanzhou 730000, China

  28. No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training

    Jul 7, 2026Noel ThomasDeep Learning OptimizationParameter Identifiability

    Mohamed bin Zayed University of Artificial Intelligence

  29. Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks

    Jul 5, 2026Shokhrukh Ibragimov, Arnulf JentzenDeep Learning OptimizationNeural Network Optimization

    Applied Mathematics: Institute for Analysis and Numerics, University of Münster, Germany · School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China

  30. OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

    Jul 4, 2026Siyuan Li, Jiabao Pan, Yumou Liu +9Deep Learning OptimizationNeural Network Optimization

    Shanghai Artificial Intelligence Laboratory · Westlake University · Shanghai University +4