Implicit Regularization

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 35

All topics

Zeroth-Order Optimization at the Edge of Stability
Apr 16, 2026Minhak Song, Liang Zhang, Bingcong Li +3Edge of Stability Zeroth-Order Optimization
KAIST · KRAFTON · ETH Zurich +2
Asymptotic Behavior of Multi--Task Learning: Implicit Regularization and Double Descent Effects
Mar 5, 2026Ayed M. Alrashdi, Oussama Dhifallah, Houssem SifaouDouble Descent Neural Network Generalization
Department of Electrical Engineering, College of Engineering, University of Ha’il, P.O. Box 2440, Ha’il, 81441, Saudi Arabia · John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138, USA · Department of Electrical and Electronic Engineering, King’s College London – Strand, London WC2R 2LS – London, UK
Decentralized SGD with Controlled Disagreement Finds Flatter Minima
Feb 2, 2026Zesen Wang, Mikael JohanssonDecentralized Learning Flat Minima
Division of Decision and Control Systems KTH Royal Institute of Technology
Basic Inequalities for First-Order Optimization with Applications to Statistical Risk Analysis
Dec 31, 2025Seunghoon Paik, Kangjie Zhou, Matus Telgarsky +1Early Stopping Gradient Descent
University of California, Berkeley · Columbia University · New York University
Pruning as Regularization: Sensitivity-Aware One-Shot Pruning in ASR
Nov 11, 2025Julian Irigoyen, Arthur Söhler, Andreas Søeborg KirkedalAutomatic Speech Recognition Implicit Regularization
Copenhagen Business School, Copenhagen, Denmark · Jabra (GN Group), Copenhagen, Denmark

Apr 16, 2026, cs.LGJ/K move · Enter open · S save

Zeroth-Order Optimization at the Edge of Stability

Minhak Song, Liang Zhang, Bingcong Li, Niao He+2

KAIST · KRAFTON · ETH Zurich+2

Edge of Stability Zeroth-Order Optimization Neural Network Training Dynamics Implicit Regularization

Open paper AskPDF arXiv

Zeroth-order (ZO) methods are widely used when gradients are unavailable or prohibitively expensive, including black-box learning and memory-efficient fine-tuning of large models, yet their optimization dynamics in deep learning remain underexplored. In this work, we provide an explicit step size condition that exactly captures the (mean-square) linear stability of a family of ZO methods based on the standard two-point estimator. Our characterization reveals a sharp contrast with first-order (FO) methods: whereas FO stability is governed solely by the largest Hessian eigenvalue, mean-square stability of ZO methods depends on the entire Hessian spectrum. Since computing the full Hessian spectrum is infeasible in practical neural network training, we further derive tractable stability bounds that depend only on the largest eigenvalue and the Hessian trace. Empirically, we find that full-batch ZO methods operate at the edge of stability: ZO-GD, ZO-GDM, and ZO-Adam consistently stabilize near the predicted stability boundary across a range of deep learning training problems. Our results highlight an implicit regularization effect specific to ZO methods, where large step sizes primarily regularize the Hessian trace, whereas in FO methods they regularize the top eigenvalue.

Momentum

Related topics

Latest papers 35

Zeroth-Order Optimization at the Edge of Stability

Asymptotic Behavior of Multi--Task Learning: Implicit Regularization and Double Descent Effects

Decentralized SGD with Controlled Disagreement Finds Flatter Minima

Basic Inequalities for First-Order Optimization with Applications to Statistical Risk Analysis

Pruning as Regularization: Sensitivity-Aware One-Shot Pruning in ASR