Gradient Descent Dynamics

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 108

All topics

What Makes a Strong Model? A Unified Spectral Analysis of Knowledge Transfer over High-dimensional Linear Regression
May 31, 2026Wendao Wu, Fangqing Zhang, Haihan Zhang +1Weak-to-Strong Generalization Teacher-Student Learning
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University, China · School of Mathematical Sciences, Peking University, China
Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks
May 29, 2026Tianyu Pang, Vignesh Kothapalli, Shenyang Deng +3Deep Learning Optimization Neural Network Training Dynamics
Dartmouth College · Stanford University · Virginia Tech
Gradient Descent with Large Step Size Restores Symmetry in Deep Linear Networks with Multi-Pathway
May 29, 2026Hee-Sung Kim, Sungyoon LeeEdge of Stability Symmetry Breaking
Department of Computer Science, Hanyang University, Seoul, Korea.
Dynamics of Stochastic Momentum with Sparse Updates in High Dimensions
May 27, 2026Katie Everett, Elliot PaquetteMomentum Methods Gradient Descent Dynamics
Google DeepMind & MIT · McGill University & Mila
Implicit Regularization in Perturbed Deep Matrix Factorization: Spectral Conditions and Stability
May 27, 2026Jingzhe Wang, Hung-Hsu ChouAlgorithmic Stability Low-Rank Matrix Decomposition
Department of Informatics and Networked Systems, University of Pittsburgh · Department of Mathematics, University of Pittsburgh
Gradient-Flow Optimization as Dynamic Random-Effects Inference: Testing and Early Stopping with Applications to Deep Learning
May 27, 2026Minhao Yao, Ruoyu Wang, Xihong Lin +2Early Stopping Neural Network Training Dynamics
Centre for Biomedical Data Science, Duke-NUS Medical School, National University of Singapore · Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA, USA · Institute of Natural Sciences, MOE-LSC, School of Mathematical Sciences, CMA-Shanghai, SJTU-Yale2026 Joint Center of Biostatistics and Data Science, Shanghai Jiao Tong University +1
Asymmetric Scaling Laws from Sparse Features
May 22, 2026John Sous, Michael WinerDouble Descent Activation Sparsity
Department of Applied Physics, Yale University, New Haven, Connecticut 06511, USA · Energy Sciences Institute, Yale University, West Haven, Connecticut 06516, USA · Institute for Advanced Study, Princeton, NJ 08540, USA +1
Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics
May 21, 2026Igor Ignashin, Anna Radovskaya, Andrew Semenov +7Neural Network Training Dynamics Gradient Descent Dynamics
Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · P.N. Lebedev Physical Institute of the Russian Academy of Sciences · Innopolis University
Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks
May 21, 2026Margalit Glasgow, Joan BrunaShallow Neural Networks Mean-Field Theory
Massachusetts Institute of Technology · Courant Institute School of Mathematics, Computing and Data Science, New York University
Thermodynamic Irreversibility of Training Algorithms
May 21, 2026Liu Ziyin, Yuanjie Ren, Adam Levine +1Neural Network Training Dynamics Gradient Descent Dynamics
1Massachusetts Institute of Technology · 2NTT Research
Large-Step Training Dynamics of a Two-Factor Linear Transformer Model
May 20, 2026Krishnakumar BalasubramanianTransformer Phase Transitions
Department of Statistics, University of California, Davis
Understanding Dynamics of Adam in Zero-Sum Games: An ODE Approach
May 19, 2026Yi Feng, Weiming Ou, Xiao WangZero-Sum Games Min-Max Optimization
Aarhus University, Aarhus, Denmark · Shanghai University of Finance and Economics, Shanghai, China · MoE Key Laboratory of Interdisciplinary Research of Computation and Economics, Shanghai University of Finance and Economics, Shanghai, China
Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descent
May 18, 2026Behrad Moniri, Hamed HassaniRepresentation Learning Neural Network Training Dynamics
University of Pennsylvania, Philadelphia, PA, USA
Propagation of Chaos in Contextual Flow Maps
May 16, 2026Shi Chen, Zhengjiang Lin, Kaizhao Liu +1Transformer Gradient Descent Dynamics
Metric-Gradient Projection for Stable Multi-Agent Policy Learning
May 12, 2026Zuyuan Zhang, Sizhe Tang, Mahdi Imani +1Hodge Decomposition Multi-Agent Systems
The George Washington University · Northeastern University
On the global convergence of gradient descent for wide shallow models with bounded nonlinearities
May 11, 2026Romain Petit, Clarice Poon, Gabriel PeyréShallow Neural Networks Gradient Descent
CNRS and ENS, PSL Universit´e · Mathematics Institute, University of Warwick
Why Zeroth-Order Adaptation May Forget Less: A Randomized Shaping Theory
May 11, 2026Yao Shu, Jian Mu, Zhongxiang DaiZeroth-Order Optimization Continual Learning
HKUST(GZ)
Dimension-Free Saddle-Point Escape in Muon
May 10, 2026Yanlin Long, Yufei Gu, Zeke XieMuon Optimizer Nonconvex Optimization
xLeaF Lab, The Hong Kong University of Science and Technology (Guangzhou)
The Global Empirical NTK: Self-Referential Bias and Dimensionality of Gradient Descent Learning
May 9, 2026James Hazelden, Laura Driscoll, Eli Shlizerman +1Representation Learning Neural Tangent Kernel
Applied Mathematics, University of Washington; Allen Institute · Allen Institute · Neurobiology & Biophysics, University of Washington +2
Distributional simplicity bias and effective convexity in Energy Based Models
May 8, 2026Aurélien Decelle, Alfonso de Jesús Navas Gómez, Beatriz SeoaneEnergy-Based Models Gradient Descent Dynamics
Escuela Técnica Superior de Ingenieros Industriales Universidad Politécnica de Madrid Calle de José Gutiérrez Abascal 2, Madrid 28006, Spain · Departamento de Física Teórica Universidad Complutense de Madrid Plaza de las Ciencias 1, Madrid 28040 · Departamento de Física Teórica & IPARCOS Universidad Complutense de Madrid Plaza de las Ciencias 1, Madrid 28040
A Rod Flow Model for Adam at the Edge of Stability
May 7, 2026Eric Regis, Sinho ChewiEdge of Stability Momentum Methods
Yale University
The Weight Gram Matrix Captures Sequential Feature Linearization in Deep Networks
May 7, 2026Taehun Cha, Daniel Beaglehole, Adityanarayanan Radhakrishnan +1Neural Network Training Dynamics Gradient Descent Dynamics
Department of Mathematics Korea University · Computer Science and Engineering UC San Diego · MIT Mathematics Broad Institute of MIT and Harvard
Learning reveals invisible structure in low-rank RNNs
May 5, 2026Yoav Ger, Omri BarakNeural Processes Recurrent Neural Networks
Ruth and Bruce Rappaport Faculty of Medicine and Network Biology Research Laboratory Technion – Israel Institute of Technology, Haifa, Israel
A Theory of Saddle Escape in Deep Nonlinear Networks
May 2, 2026Divit Rawal, Michael R. DeWeeseNeural Network Optimization Neural Network Training Dynamics
Department of Physics University of California, Berkeley Berkeley, CA 94720
High-Probability Convergence in Decentralized Stochastic Optimization with Gradient Tracking
Apr 30, 2026Aleksandar Armacki, Haoyuan Cai, Ali H. SayedConvergence Analysis Stochastic Optimization Convergence
Ecole Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications
Apr 30, 2026Nghia Bui, Lijing WangFine-Tuning Gradient Descent Dynamics
Department of Data Science, New Jersey Institute of Technology, New Jersey, USA.
Attractor FCM
Apr 30, 2026Alexis KafantarisRecurrent Neural Networks Physics-Informed ML
State-Dependent Lyapunov Analysis of Rank-1 Matrix Factorization
Apr 28, 2026Jaehong MoonEdge of Stability Lyapunov Stability
Industrial & Enterprise Systems Engineering University of Illinois at Urbana-Champaign Urbana, IL 61801
Sustained Gradient Alignment Mediates Subliminal Learning in a Multi-Step Setting: Evidence from MNIST Auxiliary Logit Distillation Experiment
Apr 28, 2026Chayanon Kitkana, Shivam AroraSubliminal Learning Teacher-Student Learning
Independent · Equivariant labs
Gradient-Direction Sensitivity Reveals Linear-Centroid Coupling Hidden by Optimizer Trajectories
Apr 28, 2026Yongzhong XuWeight-Space Analysis Grokking

May 31, 2026, cs.LGJ/K move · Enter open · S save

What Makes a Strong Model? A Unified Spectral Analysis of Knowledge Transfer over High-dimensional Linear Regression

Wendao Wu, Fangqing Zhang, Haihan Zhang, Cong Fang

State Key Lab of General AI, School of Intelligence Science and Technology, Peking University, China · School of Mathematical Sciences, Peking University, China

Weak-to-Strong Generalization Teacher-Student Learning Knowledge Distillation Gradient Descent Dynamics

Open paper AskPDF arXiv

Teacher-Student Knowledge Transfer (KT) is ubiquitous in modern machine learning, ranging from classical model compression via Knowledge Distillation (KD) to the emergent phenomenon of Weak-to-Strong (W2S) generalization. While existing studies offer isolated insights, a unified theoretical framework explaining the efficacy of KT across these disparate regimes remains lacking. In this work, we establish a unified spectral analysis of SGD dynamics in high-dimensional linear regression, elucidating the efficiency of KT across seemingly disparate regimes. We characterize KT efficiency through two distinct mechanisms: \emph{Spectral Horizon Expansion} in KD, which enables the capture of statistically inaccessible high-frequency signals, and \emph{Spectral Denoising} in W2S, where the student acts as a filter for optimization noise. Our framework unifies these phenomena, revealing that the efficacy of transfer is governed by the interplay between implicit regularization and heterogeneous spectral learning speeds over the spectrum.

Momentum

Related topics

Latest papers 108