Convergence

Follow this topic Ask about this topic

Momentum

51 papers in the last four weeks, up 155% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 390

All topics

Why Do We Need Warm-up? A Theoretical Perspective
Oct 3, 2025Foivos Alimisis, Rustem Islamov, Aurelien LucchiWarm Starts Batch
University of Basel, Switzerland
Finite-Time Convergence of Single-Trajectory Chi-Square Robust Q-Learning With Linear Function Approximation
Oct 2, 2025Saptarshi Mandal, Yashaswini Murthy, R. SrikantQ-Learning Offline Reinforcement Learning
ECE, UIUC · ORIE, UT Austin · ECE, CSL, and the director of NCSA at UIUC
Convergence Analysis of the ProbAbilistic Gradient Estimator Algorithm for Weakly Convex Finite-Sum Optimization
Aug 31, 2025Laurent Condat, Peter RichtárikNonconvex Convergence
King Abdullah University of Science and Technology (KAUST) Thuwal, Kingdom of Saudi Arabia
Convergence of Stochastic Gradient Methods for Wide Two-Layer Physics-Informed Neural Networks for the Poisson Equation
Aug 29, 2025Bangti Jin, Longjun WuParametric Physics-Informed Neural Network Stochastic Gradient Descent
Department of Mathematics, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong, P.R. China
Better Convergence Guarantees for Sign-Based Momentum Methods
Jul 16, 2025Wei Jiang, Dingzhi Yu, Sifan Yang +3Momentum Stochastic Gradient Descent Convergence
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China
On the Effectiveness of the z-Transform Method in Quadratic Optimization
Jul 4, 2025Francis BachMomentum Stochastic Gradient Descent Convergence
SIERRA · Inria - Ecole Normale Sup´erieure · PSL Research University
Continuous Policy and Value Iteration for Stochastic Control Problems and Its Convergence
Jun 9, 2025Qi Feng, Gu WangStochastic Optimal Control Boltzmann Policies
Department of Mathematics, Florida State University, Tallahassee, 32306 · Department of Mathematical Sciences, Worcester Polytechnic Institute, Worcester, MA 01609
Never Skip a Batch: Dense Learning of Temporal GNNs via Adaptive Pseudo-Supervision
May 18, 2025Alexander Panyshev, Dmitry Vinichenko, Oleg Travkin +2Temporal Graph Neural Networks Batch
Risk AI Research Lab Moscow Independent Research Institute of Artificial Intelligence Moscow, Russia · Risk AI Research Lab Moscow, Russia · Risk AI Research Lab Applied AI Institute Moscow, Russia
On the $O(\frac{\sqrt{d}}{K^{1/4}})$ Convergence Rate of AdamW Measured by $\ell_1$ Norm
May 17, 2025Huan Li, Yiming Dong, Zhouchen LinAdam Convergence
Institute of Robotics and Automatic Information Systems, College of Artificial Intelligence, Nankai University, Tianjin, China. · National Key Lab of General AI, School of Intelligence Science and Technology, Peking University, Beijing, China.
Rethinking the Global Convergence of Softmax Policy Gradient with Linear Function Approximation: The Case of Multi-Armed Bandits
May 6, 2025Max Qiushi Lin, Jincheng Mei, Matin Aghaei +6Policy Gradient Convergence
Simon Fraser University · Google DeepMind · Google Research +1
A Provably Convergent Plug-and-Play Framework for Stochastic Bilevel Optimization
May 2, 2025Tianshu Chu, Dachuan Xu, Wei Yao +2Bilevel Optimization Stochastic Optimization
Institute of Operations Research and Information Engineering, Beijing University of Technology · National Center for Applied Mathematics Shenzhen · National Center for Applied Mathematics Shenzhen, and Department of Mathematics, Southern University of Science and Technology +2
Efficient and Stable Multi-Dimensional Kolmogorov-Smirnov Distance
Apr 15, 2025Peter Matthew Jacobs, Foad Namjoo, Jeff M. PhillipsTwo-Sample Testing Optimal Sample Complexity
University of Wisconsin–Madison, USA · University of Utah, USA
Reinforcement Learning in Switching Non-Stationary Markov Decision Processes: Algorithms and Convergence Analysis
Mar 24, 2025Mohsen Amiri, Sindri MagnússonMarkov Decision Processes Value Functions
Department of Computer and System Science, Stockholm University, SE-164 25 Stockholm, Sweden
Low-dimensional adaptation of diffusion models: Convergence in total variation
Jan 22, 2025Jiadong Liang, Zhihan Huang, Yuxin ChenScore-Based Diffusion Model Diffusion Models
Department of Statistics and Data Science, the Wharton School, University of Pennsylvania
How Can Incentives and Cut Layer Selection Influence Data Contribution in Split Federated Learning?
Dec 10, 2024Joohyung Lee, Jungchan Cho, Wonjun Lee +2Stackelberg Equilibrium Incentives
School of Computing, Gachon University, Seongnam 13120, Rep. of Korea · School of Cybersecurity, Korea University, Seoul, Rep. of Korea · Department of Electrical and Computer Engineering, Princeton University, Princeton, NJ 08544 USA
Convergence Rate Analysis of LION
Nov 12, 2024Yiming Dong, Huan Li, Zhouchen LinConvergence Stochastic Convex Optimization
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University, Beijing, 100871, China · Institute of Robotics and Automatic Information Systems, College of Artificial Intelligence, Nankai University, Tianjin, 300071, China · Pazhou Laboratory (Huangpu), Guangzhou, 510555, China
Statistical Inference for Policy Evaluation with Temporal Difference Learning
Oct 21, 2024Weichen Wu, Gen Li, Yuting Wei +1Temporal Difference Value Functions
The Voleon Group, New York, NY, USA · Department of Statistics, The Chinese University of Hong Kong, Hong Kong SAR. · Department of Statistics and Data Science, The Wharton School, University of Pennsylvania, Philadelphia, PA, USA. +1
Improved Finite-Particle Convergence Rates for Stein Variational Gradient Descent
Sep 13, 2024Sayan Banerjee, Krishnakumar Balasubramanian, Promit GhosalWasserstein Distance Convergence
Department of Statistics and Operations Research, University of North Carolina, Chapel Hill. · Department of Statistics, University of California, Davis. · Department of Statistics, University of Chicago.
Adversarial dynamical systems characterize when data-driven learning succeeds or fails
Jul 8, 2024Matthew J. Colbrook, Igor Mezić, Alexei StepanenkoKoopman Operator Spectral Representation Method
DAMTP, University of Cambridge, Cambridge, CB3 0WA, UK. · University of California, Santa Barbara, CA 93106, USA.
Provably Efficient Off-Policy Adversarial Imitation Learning with Convergence Guarantees
May 26, 2024Yilei Chen, Vittorio Giammarino, James Queeney +1Imitation Learning Convergence
Division of Systems Engineering Boston University Boston, MA 02215, USA · Department of Computer Science Purdue University West Lafayette, IN 47907, USA · Amazon Robotics North Reading, MA 01864, USA +1
Convergence Analysis of Sequential Federated Learning on Heterogeneous Data
Nov 6, 2023Yipeng Li, Xinchen LyuFederated Learning Convergence
National Engineering Research Center for Mobile Network Technologies Beijing University of Posts and Telecommunications Beijing, 100876, China
Bridging the Gap between Newton-Raphson Method and Regularized Policy Iteration
Oct 11, 2023Zeyang Li, Chuxiong Hu, Yunan Wang +4Entropy Regularized Reinforcement Learning Soft Actor-Critic
Laboratory for Information and Decision Systems, Massachusetts Institute of Technology, Cambridge, MA 02139, United States · Department of Mechanical Engineering, Tsinghua University, Beijing 100084, China · College of Artificial Intelligence and the School of Vehicle and Mobility, Tsinghua University, Beijing 100084, China
How many labelers do you have? A closer look at gold-standard labels
Jun 24, 2022Chen Cheng, Hilal Asi, John DuchiSummary Statistics Convergence
Department of Statistics, Stanford University · Department of Electrical Engineering, Stanford University · Departments of Statistics and Electrical Engineering, Stanford University
Robust Linear Predictions: Analyses of Uniform Concentration, Fast Rates and Model Misspecification
Jan 6, 2022Saptarshi Chakraborty, Debolina Paul, Swagatam DasOutliers Robustness Verification
Department of Statistics, University of California, Berkeley · Department of Statistics, Stanford University · Electronics and Communication Sciences Unit, Indian Statistical Institute, Kolkata, India
Exponential Convergence of Deep Operator Networks for Elliptic Partial Differential Equations
Dec 15, 2021Carlo Marcati, Christoph SchwabNeural Approximations Coefficients
Seminar for Applied Mathematics, ETH Zurich, Switzerland
Partial GFlowNet: Accelerating Convergence in Large State Spaces via Strategic Partitioning
Date pendingXuan Yu, Xu Wang, Rui Zhu +2Generative Flow Networks State Space
University of Science and Technology of China (USTC), Hefei, China · Suzhou Institute for Advanced Research, USTC, Suzhou, China · State Key Laboratory of Precision and Intelligent Chemistry, USTC, Hefei, China
Silver Rate Is (Almost) Optimal for Gradient Descent
Date pendingYuhan Ye, Kaizhao LiuStep Accuracy Gradient Descent
MIT
Stability and Convergence of Optimistic Exponential Weights with Asymmetric Step Sizes in Bimatrix Games
Date pendingHédi Hadiji, Sarah SachsImperfect-Information Games Equilibrium
Laboratoire des Signaux et Systèmes, CentraleSupélec, Paris, France · School of Mathematics, University of Bristol, Bristol, United Kingdom
Can SGD Select Good Fishermen? Local Convergence under Self-Selection Biases
Date pendingAlkis Kalavasis, Anay Mehrotra, Felix ZhouSelection Bias Optimal Sample Complexity
Yale University
Fast Convergence for High-Order ODE Solvers in Diffusion Probabilistic Models
Date pendingDaniel Zhengyu Huang, Jiaoyang Huang, Zhengjiang LinTime Discretization Ordinary Differential Equations
Beijing International Center for Mathematical Research, Center for Machine Learning Research, Peking University, Beijing, China · Department of Statistics and Data Science, University of Pennsylvania, Philadelphia, PA, USA · Department of Mathematics, Massachusetts Institute of Technology, 77 Massachusetts Ave, 02139 Cambridge MA, USA

Oct 3, 2025, cs.LGJ/K move · Enter open · S save

Why Do We Need Warm-up? A Theoretical Perspective

Foivos Alimisis, Rustem Islamov, Aurelien Lucchi

University of Basel, Switzerland

Warm Starts Batch Singular Learning Theory Local Curvature Convergence Deep Learning

Open paper AskPDF arXiv

Learning rate warm-up -- increasing the learning rate at the beginning of training -- has become a ubiquitous heuristic in modern deep learning, yet its theoretical foundations remain poorly understood. In this work, we provide a principled explanation for why warm-up improves training. We rely on a generalization of the

(L_0, L_1)

-smoothness condition, which bounds local curvature as a linear function of the loss suboptimality and exhibits desirable closure properties. We show -- both theoretically and empirically -- that this condition is satisfied by common neural architectures and accurately captures the curvature of the optimization landscape early in training. Adapting the learning rate in response to this curvature condition naturally induces a warm-up-like schedule, and we show that this choice yields provably faster convergence guarantees than using a fixed learning rate. Experiments on language and vision models show that the resulting one-parameter warm-up schedule can match tuned linear warm-up and improve over no warm-up.

Momentum

Related topics

Latest papers 390