Deep Learning Optimization

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, up 133% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 132

All topics

Fix the Loss, Not the Radius: Rethinking the Adversarial Perturbation of Sharpness-Aware Minimization
May 11, 2026Jinping Wang, Qinhan Liu, Zhiwu Xie +1Neural Network Generalization Sharpness-Aware Minimization
Wenzhou-Kean University · International Frontier Interdisciplinary Research Institute, Wenzhou-Kean University
OUIDecay: Adaptive Layer-wise Weight Decay for CNNs Using Online Activation Patterns
May 11, 2026Alberto Fernández-Hernández, Jose I. Mestre, Cristian Pérez-Corral +3Deep Learning Optimization Weight Decay
Universitat Politècnica de València, Valencia, Spain · Universitat Jaume I, Castelló de la Plana, Spain · Openchip & Software Technologies S.L., Spain
Refresh-Scaling the Memory of Balanced Adam
May 11, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2Deep Learning Optimization Momentum Methods
Universitat Polit`ecnica de Val`encia, Val`encia, Spain · Universitat Jaume I, Castell´o de la Plana, Spain
LBI: Parallel Scan Backpropagation via Latent Bounded Interfaces
May 9, 2026Shaun Christopher Lee, Sangeetha Abdu JyothiDeep Learning Optimization Backpropagation
University of California, Irvine
Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizers
May 9, 2026Aditya RanganathDeep Learning Optimization Memory-Efficient Optimization
Center for Applied Scientific Computing Lawrence Livermore National Laboratory Livermore, CA, USA
Predicting Plasticity in Deep Continual Learning: A Theoretical Perspective
May 9, 2026Jiuqi Wang, Jayanth Srinivasa, Claire Chen +3Deep Learning Optimization Loss of Plasticity
University of Virginia · Cisco Research · California Institute of Technology +1
Muon Does Not Converge on Convex Lipschitz Functions
May 9, 2026Tetiana Parshakova, Ahmed Khaled, Michael Crawshaw +2Deep Learning Optimization Muon Optimizer
Center for Computational Mathematics, Flatiron Institute, Simons Foundation · Princeton University, Google DeepMind · George Mason University +1
Directional Consistency as a Complementary Optimization Signal: The GONO Framework
May 7, 2026Victor Daniel GeraDeep Learning Optimization Stochastic Optimization Convergence
Department of Artificial Intelligence Anurag University, Hyderabad, India
Pro-KLShampoo: Projected KL-Shampoo with Whitening Recovered by Orthogonalization
May 7, 2026Ruotong Sun, Ermin WeiLanguage Model Pretraining Deep Learning Optimization
†Department of Industrial Engineering & Management Sciences, Northwestern University · ∗Department of Electrical & Computer Engineering, Northwestern University
Mean Mode Screaming: Mean--Variance Split Residuals for 1000-Layer Diffusion Transformers
May 7, 2026Pengqi LuDeep Learning Optimization Representation Collapse
Beijing, China
Revealing Modular Gradient Noise Imbalance in LLMs: Calibrating Adam via Signal-to-Noise Ratio
May 7, 2026Ziqing Wen, Zhouyang Liu, Jiahuan Wang +4Deep Learning Optimization Learning Rate Scheduling
College of Computer Science and Technology, National University of Defense Technology, Changsha, Hunan, China
Budget-aware Auto Optimizer Configurator
May 6, 2026Kang Liu, Wei Peng, Jianchen HuDeep Learning Optimization Memory-Efficient Optimization
School of Future Technology Xi’an Jiaotong University Xi’an, China · School of Automation Science and Engineering Xi’an Jiaotong University Xi’an, China
Sparse Tokens Suffice: Jailbreaking Audio Language Models via Token-Aware Gradient Optimization
May 6, 2026Zheng Fang, Xiaosen Wang, Shenyi Zhang +2Deep Learning Optimization Audio-Language Models
1Wuhan University · 2Huazhong University of Science and Tech · Institute for Math & AI, Wuhan University +2
Critical Windows of Complexity Control: When Transformers Decide to Reason or Memorize
May 6, 2026Sarwan AliDeep Learning Optimization Weight Decay
Columbia University, Irving Medical Center, NY, USA
Layerwise LQR for Geometry-Aware Optimization of Deep Networks
May 5, 2026Simon Dufort-Labbé, Pierre-Luc Bacon, Razvan Pascanu +2Deep Learning Optimization Second-Order Optimization
Mila, Université de Montréal · Samsung – SAIL Montreal
Trust, but Verify: Peeling Low-Bit Transformer Networks for Training Monitoring
May 4, 2026Arian Eamaz, Farhang Yeganegi, Mojtaba SoltanalianDeep Learning Optimization Neural Network Training Dynamics
Department of Electrical and Computer Engineering University of Illinois Chicago Chicago, IL, USA
Artificial Jagged Intelligence as Uneven Optimization Energy Allocation Capability Concentration, Redistribution, and Optimization Governance
May 2, 2026Wesley Shu, Peng WeiDeep Learning Optimization
The Institute of Energetic Paradigm · College of Plant Protection Southwest University, China
Improving Calibration in Test-Time Prompt Tuning for Vision-Language Models via Data-Free Flatness-Aware Prompt Pretraining
Apr 30, 2026Hyeonseo Jang, Jaebyeong Jeon, Joong-Won Hwang +1Deep Learning Optimization Vision-Language Models
1Yonsei University · 2ETRI
Learning Rate Engineering: From Coarse Single Parameter to Layered Evolution
Apr 30, 2026Ming-Hong Yao, Di Wang, Jian Cui +5Deep Learning Optimization Fine-Tuning
School of AI, Jilin International Sciences University, Changchun, China.
Learning to Forget: Continual Learning with Adaptive Weight Decay
Apr 29, 2026Aditya A. Ramesh, Alex Lewandowski, Jürgen SchmidhuberDeep Learning Optimization Weight Decay
The Swiss AI Lab, IDSIA USI-SUPSI, Lugano, Switzerland · University of Alberta, Alberta Machine Intelligence Institute, Edmonton, Canada · Center of Excellence for Generative AI, KAUST, Thuwal, Saudi Arabia
Scalable Hyperparameter-Divergent Ensemble Training with Automatic Learning Rate Exploration for Large Models
Apr 27, 2026Hailing Cheng, Tao Huang, Chen Zhu +1Deep Learning Optimization Learning Rate Scheduling
LinkedIn Inc. Mountain View, CA 94043
Learn&Drop: Fast Learning of CNNs based on Layer Dropping
Apr 25, 2026Giorgio Cruciata, Luca Cruciata, Liliana Lo Presti +2Deep Learning Optimization Convolutional Neural Networks
Engineering Department, University of Palermo, V.le delle Scienze, Ed. 6, Palermo, 90128, Italy. · Computer Vision Lab, Delft University of Technology, Van Mourik Broekmanweg 6,2026 Delft, 2628 XE, The Netherlands.
A Layer Separation Optimization Framework for Cross-Entropy Training in Deep Learning
Apr 25, 2026Yaru Liu, Michael K. Ng, Yiqi GuDeep Learning Optimization Nonconvex Optimization
School of Mathematical Sciences, University of Electronic Science and Technology of China, Sichuan 611731, China · Department of Mathematics, Hong Kong Baptist University, Kowloon Tong, Hong Kong, China
Training Time Prediction for Mixed Precision-based Distributed Training
Apr 17, 2026Minchul Kang, Changyong Shin, Jinwoo Jeong +5Deep Learning Optimization Distributed Training
Korea University · KT Corporation
Natural gradient descent with momentum
Apr 16, 2026Anthony Nouy, Agustín SomacalDeep Learning Optimization Riemannian Optimization
Centrale Nantes, Nantes Universit´e, Laboratoire de Math´ematiques Jean Leray UMR CNRS 6629
Benchmarking Optimizers for MLPs in Tabular Deep Learning
Apr 16, 2026Yury Gorishniy, Ivan Rubachev, Dmitrii Feoktistov +1Multilayer Perceptrons Deep Learning Optimization
α Yandex · β HSE University
CLion: Efficient Cautious Lion Optimizer with Enhanced Generalization
Apr 16, 2026Feihu Huang, Guanyi Zhang, Songcan ChenStochastic Optimization Sign-Based Optimization
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China
VISTA: Validation-Informed Trajectory Adaptation via Self-Distillation
Apr 13, 2026Eli Corn, Daphna WeinshallNeural Network Generalization Deep Learning Optimization
Department of Computer Science, Hebrew University of Jerusalem, Jerusalem, Israel.
Sven: Singular Value Descent as a Computationally Efficient Natural Gradient Method
Apr 1, 2026Samuel Bright-Thonney, Thomas R. Harvey, Andre Lukas +1Deep Learning Optimization Natural Gradient Methods
Department of Physics, Massachusetts Institute of Technology · The NSF Institute for Artificial Intelligence and Fundamental Interactions · Rudolf Peierls Centre for Theoretical Physics, University of Oxford +2
SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning
Feb 2, 2026Qifan Yu, Xinyu Ma, Zhijian Zhuo +7Deep Learning Optimization
State Key Laboratory of General Artificial Intelligence, Peking University · ByteDance Seed · Work done at ByteDance Seed.

May 11, 2026, cs.LGJ/K move · Enter open · S save

Fix the Loss, Not the Radius: Rethinking the Adversarial Perturbation of Sharpness-Aware Minimization

Jinping Wang, Qinhan Liu, Zhiwu Xie, Zhiqiang Gao

Wenzhou-Kean University · International Frontier Interdisciplinary Research Institute, Wenzhou-Kean University

Neural Network Generalization Sharpness-Aware Minimization Deep Learning Optimization Nonconvex Optimization Flat Minima

Open paper AskPDF arXiv

Sharpness-Aware Minimization (SAM) improves generalization by minimizing the worst-case loss within a fixed parameter-space radius neighborhood. SAM and its variants mainly rely on a first-order linearized surrogate, while flat minima are inherently a second-order (curvature) notion.We revisit this mismatch and propose Loss-Equated SAM (LE-SAM), which inverts the traditional SAM mechanism that fixed perturbation radius with a fixed loss-space budget,effectively removing gradient-norm-dominated learning signals and shifting optimization toward curvature-dominated terms. Extensive experiments across diverse benchmarks and tasks demonstrate the strong generalization ability of LESAM that consistently outperforms SAM and even its variants, achieving the state-of-the-art performance.

Momentum

Related topics

Latest papers 132