Deep Learning Optimization

Ask about this topic

Momentum

7 papers in the last four weeks, up 133% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 128

All topics
CardsList

  1. DeltaTTT: Layerwise Optimization for Nonlinear Recurrent Memory

    Oct 6, 2026Yining Li, Dongchen Han, Jie Fu +1Deep Learning OptimizationNeural Network Optimization

    Qiuzhen College, Tsinghua University · LeapLab, Tsinghua University · IQuest Research

  2. How Bregman Divergences Shape Shampoo

    Oct 6, 2026Bing Liu, Wenjie Zhou, Chengcheng Zhao +4Deep Learning OptimizationFunctional Bregman Divergences

    Zhejiang University · University of the Chinese Academy of Sciences · Peking University +2

  3. Does Muon Need Fine-Grained Spectral Shaping?

    Oct 5, 2026Meher Chaitanya, Tianyi Zhou, Aristides GionisDeep Learning OptimizationMuon Optimizer

    KTH Royal Institute of Technology Stockholm, Sweden

  4. Learning Pareto Stationary Fronts via Single-Pass Backpropagation

    Oct 5, 2026Elina Rojin Celik, Marcos Medeiros Raimundo, Isabel ValeraDeep Learning OptimizationMulti-Objective Optimization

    Department of Computer Science Saarland University, Germany · Instituto de Computação Universidade Estadual de Campinas, Brazil

  5. Mind the Drift: Diagonal Linear Networks Under Large Learning Rates

    Oct 5, 2026Aniket Sanyal, Tom Jacobs, Rebekka BurkholzEdge of StabilityDeep Learning Optimization

    CISPA Helmholtz Center for Information Security · Technical University of Munich · Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA)

  6. ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training

    Oct 5, 2026Yuanshi Liu, Boyuan Jiang, Liang Hou +4Spectral RegularizationDeep Learning Optimization

    School of Intelligence Science and Technology, Peking University · Kling Team

  7. SoftServe: A Scalable Quasi-Newton Method for Deep Learning

    Oct 1, 2026Joohwan Ko, Tetiana Parshakova, Diana Cai +1Deep Learning OptimizationSecond-Order Optimization

    University of Massachusetts Amherst · CCM, Flatiron Institute · Cornell University

  8. Increasing Width Allows Greedy Layer-wise Training to Rival End-to-End Backpropagation in Self-Supervised Learning

    Sep 30, 2026Syon Mansur, Joel ZylberbergDeep Learning OptimizationConvolutional Neural Networks

    Jules Stein Eye Institute University of California, Los Angeles Los Angeles, CA 90095, USA

  9. From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes

    Sep 30, 2026Yichen Wang, Fanghui Liu, Yudong ChenLanguage Model PretrainingDeep Learning Optimization

    Department of Computer Sciences, University of Wisconsin–Madison, USA. · School of Mathematical Sciences, Institute of Natural Sciences and MOE-LSC, Shanghai Jiao Tong University, China.

  10. EvE: An Alternate Optimizer to Adam

    Sep 28, 2026Shashank Raj, Kalyanmoy DebDeep Learning OptimizationDifferential Evolution

    Department of Computer Science The University of Alabama Tuscaloosa, AL 35487, USA

  11. Low-Rank Friction for Memory-Efficient Transformer Pretraining

    Sep 24, 2026Rajit Rajpal, Benedict LeimkuhlerDeep Learning OptimizationLow-Rank Compression

    School of Mathematics University of Edinburgh Edinburgh, UK EH9 3FD

  12. AdamX: Cosine similarity meets gradient descent

    Sep 10, 2026Francisco Caldas, Ruben Belo, Cláudia SoaresDeep Learning OptimizationAdaptive Gradient Methods

    NOVA School of Science of Technology Universidade Nova de Lisboa, Caparica, Portugal

  13. BrachistoneLR: A Brachistochrone-Inspired Learning-Rate Schedule and a Controlled Benchmark of Scheduling Policies

    Sep 8, 2026Md. Sadekur Rahman Roni, Md. Jalal uddin Chowdhury, Moutusi Dash NimiDeep Learning OptimizationLearning Rate Scheduling

    Department of Computer Science and Engineering Leading University Sylhet, Bangladesh · DeepNet Research and Development Lab Sylhet 3100, Bangladesh · Department of Electrical and Electronic Engineering Leading UniversitySep Sylhet, Bangladesh

  14. Beyond Dense Adam States: Adaptive Log-Space Quantization for Memory-Efficient Optimizers

    Aug 23, 2026Yan WangDeep Learning OptimizationMemory-Efficient Optimization

    Independent Researcher

  15. Activation-Keyed Momentum: An Anisotropic Momentum Update via the Delta Rule

    Aug 19, 2026Euijin Hong, Guannan QuDeep Learning OptimizationMomentum Methods

    Electrical and Computer Engineering, Carnegie Mellon University

  16. Momentum as Residual-Driven Multiplier Correction for Deep Learning Optimization

    Aug 13, 2026Zhixin Ren, Yau Lyu, Congrong Li +2Deep Learning OptimizationMomentum Methods

    Department of Mathmatical Science, Tsinghua University · School of Vehicle and Mobility, Tsinghua University · College of AI, Tsinghua University

  17. AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization

    Aug 3, 2026Alok Kumar Pandey, Umang Chaturvedi, Aatish Rana +1Deep Learning OptimizationStochastic Optimization Convergence

  18. Towards joint scaling laws with optimal batch size schedules

    Jul 30, 2026Jiaxiang Li, Zhiqi Bu, Shiyun XuDeep Learning OptimizationLanguage Model Scaling Laws

    Meta · Independent researcher

  19. Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training

    Jul 21, 2026Nuemaan MalikDeep Learning OptimizationMixture-of-Experts Language Models

    Independent Researcher

  20. Learning Faster without Deeper Networks: A*-Inspired Batch Selection for Efficient CNN Training

    Jul 17, 2026Anxhelo Shehu, Enes Stastoli, Arben CelaDeep Learning OptimizationConvolutional Neural Networks

    AI Laboratory, University Metropolitan Tirana, Tirana, Albania. · Laboratory of Images Signals and Intelligent Systems, ESIEE Paris, Paris, France.

  21. Reassessing Muon for Matrix Factorization

    Jul 14, 2026Ali Parviz, Gal Mishne, Alex CloningerDeep Learning OptimizationMuon Optimizer

    †. Halicioğlu Data Science Institute, UC San Diego. · ∗. Department of Mathematics, UC San Diego.

  22. LionVote: Per-Layer Learning Rate Adaptation for Lion

    Jul 10, 2026Kris AtallahDeep Learning OptimizationLayer-Wise Learning Rate Adaptation

    New York University, New York, USA · New York University New York, NY, USA

  23. Beyond Backpropagation: Monte Carlo Method Can Train Deep Neural Networks

    Jul 9, 2026Hong ZhaoDeep Learning Optimization

    Department of Physics, Xiamen University, Xiamen 361005, China · Lanzhou Center for Theoretical Physics, Lanzhou University, Lanzhou 730000, China

  24. No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training

    Jul 7, 2026Noel ThomasDeep Learning OptimizationParameter Identifiability

    Mohamed bin Zayed University of Artificial Intelligence

  25. Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks

    Jul 5, 2026Shokhrukh Ibragimov, Arnulf JentzenDeep Learning OptimizationNeural Network Optimization

    Applied Mathematics: Institute for Analysis and Numerics, University of Münster, Germany · School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China

  26. OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

    Jul 4, 2026Siyuan Li, Jiabao Pan, Yumou Liu +9Deep Learning OptimizationNeural Network Optimization

    Shanghai Artificial Intelligence Laboratory · Westlake University · Shanghai University +4

  27. Class-Grouped Normalized Momentum and Faster Hyperparameter Exploration to Tackle Class Imbalance in Federated Learning

    Jul 1, 2026Haemin Park, Diego Klabjan, Martin W. Braun +2Class-Imbalanced LearningDeep Learning Optimization

    Department of Industrial Engineering & Management Sciences, Northwestern University, Evanston, IL, USA · Intel Corporation, Chandler, AZ, USA

  28. From Approximation to Emergence: A Theory of Deep Learning

    Jul 1, 2026Zhilin ZhaoNeural Network GeneralizationDeep Learning Optimization

    Sun Yat-sen University

  29. Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimization

    Jun 29, 2026Haoming Meng, Anton Sugolov, Vardan PapyanDeep Learning OptimizationNeural Network Optimization

    Department of Mathematics, University of Toronto, Toronto, Canada · Vector Institute, Toronto, Canada

  30. Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks

    Jun 28, 2026Tejas Pradeep ShirodkarDeep Learning OptimizationNeural Network Optimization

    IIIT, Hyderabad

  31. DMuon: Efficient Distributed Muon Training with Near-Adam Overhead

    Jun 25, 2026Vincent Chen, Starrick Liu, Regis Cheng +8Deep Learning OptimizationDistributed Optimization

    X SQUARE ROBOT TEAM

  32. Tensorion: A Tensor-Aware Generalization of the Muon Optimizer

    Jun 24, 2026Vladimir Bogachev, Vladimir Aletov, Alexander Molozhavenko +2Deep Learning OptimizationMuon Optimizer

    HSE University · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab)

  33. An Analysis of Posterior Collapse, Parameterization and Initialization in Variational Deep Gaussian Processes

    Jun 24, 2026Francisco Javier Sáez-Maldonado, Juan Maroñas, Daniel Hernández-LobatoDeep Learning OptimizationPosterior Collapse

    Department of Computer Science and Artificial Intelligence Universidad de Granada Granada, Spain · Department of Quantitative Methods CUNEF Universidad Madrid, Spain · Machine Learning Group at Universidad Autónoma de Madrid +1

  34. Training for the Model You Return: Improving Optimization for Iterate-Averaged Language Models

    Jun 23, 2026Kwok Chun Au, Adam BlockDeep Learning OptimizationLLM Training

    Department of Computer Science, Columbia University · Department of Electrical Engineering, Columbia University

  35. Fast and Slow Variational Continual Learning

    Jun 22, 2026Subarnaduti Paul, Yohan Jung, Mohammad Emtiyaz Khan +3Continual Learning for LLMsDeep Learning Optimization

    Faculty of Mathematics and Computer Science, University of Bremen, Bremen, Germany · Department of Computer Science & AI, Jeonbuk National University, Republic of Korea · Department of Computer Science, TU Darmstadt, Darmstadt, Germany +3

  36. Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective

    Jun 19, 2026Tianqi Shen, Jinji Yang, Runze Shi +3Deep Learning OptimizationMuon Optimizer

    Department of Computer Science City University of Hong Kong (CityUHK) · Department of Statistics and Data Science University of Pennsylvania · School of Statistics and Data Science Shanghai University of Finance and Economics

  37. Breaking chains with trees: Deep learning with O(log⁡N)\mathcal{O}(\log N) parallel time complexity

    Jun 19, 2026Neeraj Mohan Sushma, Aditya Nagarsekar, Cabrel Teguemne Fokam +4Deep Learning OptimizationNeural Network Optimization

    Center for Cognitive Interaction Technology, Bielefeld University, Germany · Ruhr Universität Bochum, Germany · Birla Institute of Technology and Science, Pilani, Goa Campus +1

  38. Towards Robust Training in NNGPT AutoML Pipeline: A Loss-Optimizer Pairing Selection Study

    Jun 18, 2026Anton Abramochkin, Radu Timofte, Dmitry IgnatovDeep Learning OptimizationNeural Network Optimization

    Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany

  39. eCNNTO: A Highly Generalizable ConvNet for Accelerating Topology Optimization

    Jun 18, 2026Shengbiao Lu, Xiaodong WeiTopology OptimizationDeep Learning Optimization

    Global college, Shanghai Jiao Tong University, 800 Dongchuan Road, Shanghai, 200240, Shanghai, China

  40. NeuronFabric: A Software Reference Architecture for On-Chip Transformer Training with Local Adam

    Jun 15, 2026Evgeny UkladchikovDeep Learning OptimizationTransformer

    Independent Researcher

  41. Schattor: Schatten-family methods for deep learning optimization

    Jun 14, 2026Bohao Ma, Junyu Zhang, Chuan HeDeep Learning OptimizationStochastic Optimization Convergence

    School of Data Science, The Chinese University of Hong Kong (Shenzhen), Shenzhen, Guangdong, China · Department of Industrial Systems Engineering and Management, National University of Singapore, Singapore · Department of Mathematics, Linköping University, Sweden

  42. When to use what Schatten-pp norm in deep learning?

    Jun 13, 2026Thomas PethickDeep Learning OptimizationMuon Optimizer

  43. Beyond a Single Explanation of the Adam--SGD Gap

    Jun 12, 2026Chenxiang Zhang, Rustem Islamov, Enea Monzio Compagnoni +3Deep Learning OptimizationAdaptive Gradient Methods

    University of Luxembourg · MPI for Intelligent Systems · University of Basel +2

  44. Gefen: Optimized Stochastic Optimizer

    Jun 11, 2026Nadav Benedek, Tomer Koren, Ohad FriedDeep Learning OptimizationMemory-Efficient Optimization

    Reichman University · Tel Aviv University, Google Research

  45. LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold

    Jun 11, 2026Franz Louis Cesista, Katherine Crowson, Cédric Simal +1Deep Learning OptimizationRiemannian Optimization

    Ateneo de Manila University · EleutherAI · NaXys +1

  46. Simplicity Suffices for Parameter Noise Injection in Stochastic Gradient Descent

    Jun 10, 2026Benjamin Leblanc, Louis-Jacob Lebel, Teddy Kana +1Neural Network GeneralizationDeep Learning Optimization

    Universit´e Laval

  47. 5% > 100%: Flatness Preference is All You Need for Multimodal Parameter-Efficient Fine-Tuning

    Jun 9, 2026Yifan Zhu, Can Lin, Hangjie Yuan +4Deep Learning OptimizationFlat Minima

    School of Computer Sciences, Beijing University of Posts and Telecommunications, Beijing 100876, China · College of Computer Science and Technology, Zhejiang University, Zhejiang 310027, China · ETH Zürich, Switzerland

  48. FOGO: Forgetting-aware Orthogonalization Optimizer

    Jun 9, 2026Toan Nguyen, Yang Liu, Trung Le +2Deep Learning OptimizationContinual Learning

    School of Computer Science and Engineering University of New South Wales Sydney, Australia · Department of Data Science & AI Monash University Melbourne, Australia · DEVCOM Army Research Laboratory USA

  49. Principles and Practice of Deep Representation Learning: or a Mathematical Theory of Memory

    Jun 4, 2026Sam Buchanan, Druv Pai, Peng Wang +1Deep Learning OptimizationRepresentation Learning

    UC Berkeley · TTIC · University of Macau +2

  50. Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss

    Jun 4, 2026Thomas T. Zhang, Alok Shah, Yifei Zhang +3Deep Learning Optimization

    University of Pennsylvania · Carnegie Mellon University · Amazon FAR

  51. Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering

    Jun 2, 2026Xianliang Li, Zihan Zhang, Weiyang Liu +1Deep Learning OptimizationMomentum Methods

    1The Institute of Statistical Mathematics · 2The Graduate Institute for Advanced Studies, SOKENDAI · 3National Institute of Informatics +3

  52. FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoo

    Jun 1, 2026Kyunghun Nam, Sumyeong AhnDeep Learning OptimizationOptimization Convergence Analysis

    KENTECH, Naju, Republic of Korea.

  53. Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks

    May 29, 2026Tianyu Pang, Vignesh Kothapalli, Shenyang Deng +3Deep Learning OptimizationNeural Network Training Dynamics

    Dartmouth College · Stanford University · Virginia Tech

  54. Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handling

    May 29, 2026Dmitrii Feoktistov, Timofey Belinsky, Andrey Veprikov +2Stochastic OptimizationSign-Based Optimization

    HSE University, Moscow, Russia · BRAIn Lab, Moscow, Russia · Yandex Research, Moscow, Russia +3

  55. Inconsistency-Aware Minimization: Improving Generalization with Unlabeled Data

    May 29, 2026Hee-Sung Kim, Hyeonseong Kim, Sungyoon LeeNeural Network GeneralizationDeep Learning Optimization

    Department of Computer Science, Hanyang University, Seoul, Korea · Agency for Defense Development, Daejeon, Korea

  56. Is Backpropagation Optimal? When Synthetic Gradients Improve Sample Efficiency

    May 27, 2026Yibo Jacky Zhang, Zeyu Tang, Sanmi KoyejoDeep Learning OptimizationBackpropagation

    Stanford University

  57. The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training

    May 26, 2026Hongtao Zhang, Wenjie Zhou, Chenxi Jia +2Language Model PretrainingDeep Learning Optimization

    State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China +1

  58. Reparametrizing Shampoo and SOAP for Subspace Basis Updates and BFloat16 Storage

    May 25, 2026Alan Milligan, Zikun Xu, Simon Lacoste-Julien +2Deep Learning OptimizationMemory-Efficient Optimization

    Mila & Université de Montréal · Microsoft · Concordia University & Mila +1

  59. BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training

    May 25, 2026Zili Zhang, Chengxu Yang, Shenglong Zhang +8Deep Learning OptimizationMultimodal Large Language Models

    1Peking University · †Work done during internships at Xiaohongshu, Inc. · 2Independent Researcher +2

  60. EMA-Nesterov: Stabilizing Nesterov's Lookahead for Accelerated Deep Learning Optimization

    May 25, 2026Chung-Yiu Yau, Dawei Li, Athanasios Glentis +3Deep Learning OptimizationGradient Descent

    University of Minnesota · 2Amazon AGI · 3The Chinese University of Hong Kong