Stochastic Optimization Convergence

Follow this topic Ask about this topic

Momentum

21 papers in the last four weeks, against 2 the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 141

All topics

AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization
Aug 3, 2026Alok Kumar Pandey, Umang Chaturvedi, Aatish Rana +1Deep Learning Optimization Stochastic Optimization Convergence
Adaptivity via a Parallel Architecture for Stochastic Gradient Methods
Jul 31, 2026Bin FuStochastic Optimization Stochastic Optimization Convergence
University of Texas Rio Grande Valley
The Convergence Behavior of Adam under Heavy-Tailed Noise
Jul 29, 2026Yijiang PangNonconvex Stochastic Optimization Heavy-Tailed Noise
Michigan State University, East Lansing, USA
Adaptive Gradient-Based Methods for a Broader Class of Optimization Problems under Performative Prediction
Jul 29, 2026Hiroki Hamaguchi, Yuya Hikima, Hiroshi Sawada +1Stochastic Optimization Convergence Adaptive Gradient Methods
Graduate School of Information Science and Technology, The University of Tokyo · Department of Industrial Engineering and Economics, Institute of Science Tokyo · Communication Science Laboratories, NTT, Inc. +1
Variance-Reduced Conditional Gradient Methods under Markovian Sampling for Nonconvex Composite Optimization
Jul 28, 2026Zhaojun PengStochastic Optimization Nonconvex Stochastic Optimization
School of Computer Science Nanjing University of Information Science and Technology
On the Convergence of Stochastic Low-Rank Adaptation
Jul 24, 2026Ru Wang, Chengchang Liu, John C. S. LuiNonconvex Stochastic Optimization Low-Rank Adaptation
The Chinese University of Hong Kong · Westlake University
RELTA-SGLD: Relative-Growth Localized Taming for Nonconvex Stochastic-Gradient Langevin Learning
Jul 21, 2026Yiwei Zhou, Ziheng ChenStochastic Gradient Langevin Dynamics Langevin Dynamics
School of Mathematics and Statistics, Yunnan University, Kunming, Yunnan 650500, China.
Scaling Limits of Constant-Stepsize SGD at Flat Minima
Jul 17, 2026Jingyi Zhang, Cheng Mao, Debankur MukherjeeStochastic Optimization Convergence Flat Minima
Georgia Institute of Technology
What's in a Smoothness Constant? Tighter Rates for Local SGD with Bounded Second-order Heterogeneity
Jul 16, 2026Kumar Kshitij Patel, Rustem Islamov, Sebastian U Stich +3Optimization Convergence Analysis Stochastic Optimization Convergence
Department of Mathematics and Computer Science University of Basel Basel, Switzerland 4001 · Simons Institute for the Theory of Computing, Berkeley · Department of Data Science New Jersey Institute of Technology Newark, NJ, USA 07102
Information-Theoretic Adaptive Cooling for Deterministic MPPI via Entropy Feedback
Jul 15, 2026Shuqi Wang, Wenrong Sun, Tao Han +2Stochastic Optimization Convergence MPPI Control
School of Automation & Intelligent Sensing, Shanghai Jiao Tong University, Shanghai 200240, China · Department of Physics, The Hong Kong University of Science and Technology, Hong Kong S.A.R., China
Same Loss, Same Noise, Opposite Schedules: Noise Structure and Optimizer Normalization Jointly Determine Whether Learning-Rate Cooldown Helps
Jul 14, 2026Subham Singh, Ashutosh Mishra, Subha RautLearning Rate Scheduling Gradient Descent
Department of Mathematics and Statistics, Mississippi State University · Independent Researcher
Decentralized Gradient Descent: Bottleneck Regimes and Budget Complexity
Jul 13, 2026Nicolò MichelusiDistributed Optimization Stochastic Optimization Convergence
School of Electrical, Computer and Energy Engineering, Arizona State University.
Solving Stochastic Fixed-Point Equations with High Probability
Jul 10, 2026Jelena DiakonikolasStochastic Approximation Stochastic Optimization Convergence
Department of Computer Sciences University of Wisconsin-Madison
Vanilla SGD with Momentum Survives Heavy-Tailed Noise: Convergence Analysis without Gradient Clipping or Normalization
Jul 9, 2026Ryusei Yamada, Naoki Sato, Hideaki IidukaHeavy-Tailed Noise Stochastic Optimization Convergence
Meiji University, Japan
Avoiding unsafe sets when training with Langevin Dynamics
Jul 8, 2026Adam M. ObermanLangevin Dynamics Stochastic Optimization Convergence
Department of Mathematics and Statistics, McGill University, Montreal, QC, Canada; Mila, Quebec AI Institute; and LawZero
Finding a stationary point of a stochastic convex problem
Jul 8, 2026Felipe Areces, John Duchi, Malo SommersStochastic Optimization Convergence Nonsmooth Convex Optimization
Stanford University
A Gradient Flow Perspective on Minimum MMD Estimation
Jul 4, 2026Sophia Seulkee Kang, Louis Sharrock, Xiaoyuan Cheng +2Parameter Estimation Maximum Mean Discrepancy
Independent Researcher · University College London
On the Convergence of Adam, Revisited
Jul 3, 2026Steven Heilman, Sampad MohantyOnline Convex Optimization Stochastic Optimization Convergence
Department of Mathematics, University of Southern California, Los Angeles, CA 90089.
Decentralized Stochastic Subgradient-type Methods with Communication Compression for Nonsmooth Nonconvex Optimization
Jul 2, 2026Siyuan Zhang, Nachuan Xiao, Xin LiuStochastic Optimization Nonconvex Optimization
Random Reshuffling Dominates Stochastic Gradient Descent
Jun 30, 2026Zijian LiuStochastic Optimization Convergence Stochastic Gradient Descent
Stern School of Business, New York University
SGD at the Edge of Stability: Stochastic Stabilization with Large Learning Rates
Jun 29, 2026Konstantinos Emmanouilidis, Lachlan MacDonald, Salma Tarmoun +1Edge of Stability Stochastic Optimization Convergence
Department of CIS1 · IDEAS4 · University of Pennsylvania +2
Curvature-Weighted Gradient Diversity: A Noise Measure for Geometry-Adaptive SGD Schedules
Jun 29, 2026Muhammad Hamza, Ayush GoelLearning Rate Scheduling Stochastic Optimization Convergence
Indian Institute of Technology Kharagpur · Indian Institute of Technology Kharagpur (IIT KGP)
A samplewise backpropagation method for neural networks driven by fractional Brownian motion
Jun 28, 2026Yuecai Han, Jianming XuStochastic Differential Equations Stochastic Optimization Convergence
School of Mathematics, Jilin University, Changchun 130012, Jilin, China
Analysis of Adam Algorithms for Stochastic Dynamic Systems
Jun 27, 2026Xin Zheng, Yifei Jin, Lei GuoStochastic Optimization Stochastic Optimization Convergence
State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing 101408, China
High-Probability PL-SGD with Markovian Noise: Optimal Mixing and Tail Dependence
Jun 24, 2026Dhruv Sarkar, Aprameyo Chakrabartty, Vaneet AggarwalStochastic Optimization Optimization Convergence Analysis
Indian Institute of Technology Kharagpur · Mohamed bin Zayed University of Artificial Intelligence · Purdue University
New Bounds for the Last Iterate of the Stochastic subGradient Method
Jun 23, 2026Guglielmo Beretta, Tommaso Cesari, Roberto Colomboni +1Stochastic Optimization Last-Iterate Convergence
Department of Computer Science, Aarhus University, Aarhus, Denmark · School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada · School of Mathematics, University of Bristol, Bristol, United Kingdom
A Single Stepsize Suffices for Unprojected Linear TD(0): Simultaneous Robust and Fast Rates via Polyak--Ruppert Averaging
Jun 23, 2026Wei-Cheng Lee, Francesco OrabonaStochastic Approximation Temporal-Difference Learning
King Abdullah University of Science and Technology (KAUST) Thuwal, 23955-6900, Kingdom of Saudi Arabia
Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?
Jun 22, 2026Dingzhi Yu, Hongyi Tao, Yuanyu Wan +2Stochastic Optimization Heavy-Tailed Noise
1Nanjing University · 2Zhejiang University · 3Fudan University
Clipping the Price of Adaptivity at the Tail
Jun 21, 2026Itai Kreisler, Yair Carmon, Oliver HinderStochastic Optimization Stochastic Optimization Convergence
Tel Aviv University · University of Pittsburgh
Escaping the Variance Trap: Jacobian-Free Dynamics for Root-Finding Bilevel Optimization
Jun 21, 2026Zhiyu Li, Xi Xuan, Davide CarboneStochastic Approximation Bilevel Optimization
University of Science and Technology of China · City University of Hong Kong · Laboratoire de Physique de l’École Normale Supérieure, Université PSL, CNRS, Sorbonne Université, Université de Paris

Aug 3, 2026, cs.LGJ/K move · Enter open · S save

AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization

Alok Kumar Pandey, Umang Chaturvedi, Aatish Rana, Gopi Krishna Nedanuri

Deep Learning Optimization Stochastic Optimization Convergence Adaptive Gradient Methods Neural Network Training Dynamics

Open paper AskPDF arXiv

Single-optimizer training is a poor fit for the distinct phases of deep network optimization: adaptive methods handle noisy early gradients well but overshoot flat minima, while SGD with momentum generalizes better in the late phase but converges slowly early on. We introduce AOS-R (Adaptive Optimizer Switching, Rule-Based), a lightweight controller that monitors six online gradient-space signals -- gradient noise scale (GNS), Hutchinson curvature trace, loss stagnation, update stability ratio, gradient stability index (GSI), and loss improvement ratio (LIR) -- and switches among AdamW, SGD-M, and Lion as the optimization landscape evolves. State-preserving momentum transfer and a 400-step learning-rate bridge prevent accuracy degradation at every transition point. On CIFAR-100/WRN-28x10, AOS-R reaches 78% top-1 in 81 epochs -- 26% fewer than AdamW (109), 43% fewer than SGD-M (143), and 16% fewer than Lion (96). Across eight model-dataset benchmarks, AOS-R achieves best accuracy on 6 of 8 combinations with a mean +0.4 pp gain and 0.80x convergence speedup over AdamW under a single shared hyperparameter configuration.

Momentum

Related topics

Latest papers 141