Momentum Methods

Follow this topic Ask about this topic

Momentum

2 papers in the last four weeks, down 33% on the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 43

All topics

MDN: Parallelizing Stepwise Momentum for Delta Linear Attention
May 7, 2026Yulong Huang, Xiang Liu, Hongxiang Huang +5Momentum Methods Linear Attention
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
On the Emergence of Pendular Structure in Multi-Contact Locomotion
May 7, 2026Lingxue Lyu, Zihui LiuLegged Robotics Trajectory Optimization
School of Engineering and Applied Science, University of Pennsylvania, Philadelphia, PA 19104, USA · Department of Aeronautics & Astronautics, Stanford University, Stanford, CA 94305, USA
Bringing Order to Asynchronous SGD: Towards Optimality under Data-Dependent Delays with Momentum
May 3, 2026Tehila Dahan, Roie Reshef, Sharon Goldstein +1Momentum Methods Distributed Optimization
Technion – Israel Institute of Technology, Haifa, Israel
A unified convergence theory for adaptive first-order methods in the nonconvex case, including AdaNorm, full and diagonal AdaGrad, Shampoo and Muo
Apr 19, 2026S. Gratton, Ph. L. TointMomentum Methods Nonconvex Stochastic Optimization
Université de Toulouse, INP, IRIT, Toulouse, France. · Université de Toulouse, INP, IRIT, Toulouse, France, and NAXYS, University of Namur, Namur, Belgium.
Negative Momentum for Convex-Concave Optimization
Apr 18, 2026Henry Shugart, Shuyi Wang, Jason M. AltschulerMomentum Methods Min-Max Optimization
UPenn · Yale
Natural gradient descent with momentum
Apr 16, 2026Anthony Nouy, Agustín SomacalDeep Learning Optimization Riemannian Optimization
Centrale Nantes, Nantes Universit´e, Laboratoire de Math´ematiques Jean Leray UMR CNRS 6629
Critical Damping as a Momentum Schedule: Multi-Seed Validation, a Hybrid Recipe, and an Exhaustive Negative Result on Surgical Layer Selection
Mar 30, 2026Ivan PasichnykGradient-Based Attribution Momentum Methods
We Label Data Inc.
High dimensional theory of two-phase optimizers
Mar 27, 2026Atish AgarwalaMomentum Methods Distributed Optimization
Google DeepMind
Momentum Guidance: Plug-and-Play Guidance for Flow Models
Feb 23, 2026Runlong Liao, Jian Yu, Baiyu Su +3Momentum Methods Diffusion Guidance
The University of Texas at Austin, Austin, TX 78712, USA
Adaptive Momentum and Nonlinear Damping for Neural Network Training
Jan 30, 2026Aikaterini Karoni, Rajit Rajpal, Benedict Leimkuhler +1Momentum Methods Stochastic Optimization Convergence
School of Mathematics University of Bristol · School of Mathematics University of Edinburgh · CERMICS, CNRS, École des Ponts, Institut Polytechnique de Paris; Inria Paris, France
Why $β_1 = β_2$ Is Dynamically Special in Adam
Jan 29, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2Momentum Methods Adaptive Gradient Methods
Universitat Politècnica de València Valencia, Spain · Universitat Jaume I Castelló de la Plana, Spain
Better Convergence Guarantees for Sign-Based Momentum Methods
Jul 16, 2025Wei Jiang, Dingzhi Yu, Sifan Yang +3Sign-Based Optimization Momentum Methods
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China
Accelerated Stochastic Min-Max Optimization Based on Bias-corrected Momentum
Jun 18, 2024Haoyuan Cai, Sulaiman A. Alghunaim, Ali H. SayedMomentum Methods Nonconvex Stochastic Optimization
Ecole Polytechnique Fédérale de Lausanne, Switzerland · Kuwait University, Kuwait

May 7, 2026, cs.LGJ/K move · Enter open · S save

MDN: Parallelizing Stepwise Momentum for Delta Linear Attention

Yulong Huang, Xiang Liu, Hongxiang Huang, Xiaopeng Lin+4

The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China

Momentum Methods Linear Attention Delta-Rule Linear Attention Linear RNNs

Open paper AskPDF arXiv

Linear Attention (LA) offers a promising paradigm for scaling large language models (LLMs) to long sequences by avoiding the quadratic complexity of self-attention. Recent LA models such as Mamba2 and GDN interpret linear recurrences as closed-form online stochastic gradient descent (SGD), but naive SGD updates suffer from rapid information decay and suboptimal convergence in optimization. While momentum-based optimizers provide a natural remedy, they pose challenges in simultaneously achieving training efficiency and effectiveness. To address this, we develop a chunkwise parallel algorithm for LA with a stepwise momentum rule by geometrically reordering the update coefficients. Further, from a dynamical systems perspective, we analyze the momentum-based recurrence as a second-order system that introduces complex conjugate eigenvalues. This analysis guides the design of stable gating constraints. The resulting model, Momentum DeltaNet (MDN), leverages Triton kernels to achieve comparable training throughput with competitive linear models such as Mamba2 and KDA. Extensive experiments on the 400M and 1.3B parameter models demonstrate consistent performance improvements over strong baselines, including Transformers, Mamba2 and GDN, across diverse downstream evaluation benchmarks. Code: https://github.com/HuuYuLong/MomentumDeltaNet .

Momentum

Related topics

Latest papers 43