stat.MLNov 18, 2026

Decomposing SGD Dynamics in Neural Networks: Teacher-Induced Spikes and Variance Inflation

Authors: Samet Demir, Zafer Dogan

Organizations: MLIP Research Group, KUIS AI Center, Koç University · Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin · Department of EEE, Koç University

Abstract

We study two-layer neural networks trained by stochastic gradient descent (SGD) on a multi-index teacher model, focusing on how first-layer parameters orthogonal to the teacher subspace affect generalization. We decompose the first-layer weights into a signal component aligned with the teacher and an orthogonal complement, and analyze their joint SGD dynamics together with the second layer in the proportional limit. We show that although SGD updates to the complement are initially isotropic, the training dynamics induce teacher-dependent spectral spikes in the complement Gram matrix. This emergent anisotropy provides a concrete mechanism by which representation variance is amplified, and generalization degrades. To isolate this effect, we introduce Decomposed Dynamics, an analytically tractable surrogate whose asymptotic test-error dynamics coincide with those of SGD. In simulations with a ReLU student, freezing the directions of the orthogonal complement improves generalization relative to training them. Finally, we show that the signal component rapidly aligns with the second-layer weights, revealing strong cross-layer coupling even in this shallow setting. Together, our results provide a dynamical explanation for how feature learning outside the teacher subspace harms generalization.

Figures & tables

Explore similar work

CardsList
  1. A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay

    Sep 7, 2026Yuqing Wang, Ioannis G. Kevrekidis, Mikhail BelkinNeural Network GeneralizationNeural Network Training Dynamics

  2. SGD Provably Prioritizes a Shortcut Spurious Feature in the XOR Model

    Jun 29, 2026Tyler LaBonte, Vidya MuthukumarRepresentation LearningReLU Neural Networks

  3. Spectral phase transitions and trainability in neural network learning dynamics

    Jun 26, 2026Chanju Park, Dario Bocchi, Francesco D'Amico +2Random Matrix TheoryNeural Network Training Dynamics