On the Dynamics of Delayed Generalization in Linear Models

Atreya Vedantam; Nataraj Das; Chandrashekar Lakshminarayanan

cs.LGFeb 9, 2026

On the Dynamics of Delayed Generalization in Linear Models

Authors: Atreya Vedantam, Nataraj Das, Chandrashekar Lakshminarayanan

Organizations: Indian Institute of Technology Madras, Chennai.

Abstract

Delayed generalization, where training accuracy becomes nearly perfect substantially before test accuracy, has often been studied in the context of the depth and compositional structure of deep networks. In this paper, we ask whether delayed generalization can occur in simpler settings. We show that it can arise in a linear model trained with gradient descent and an exponential-tail loss on linearly separable data. We characterize the finite-time hyperplane dynamics into three phases: a population phase, a support vector dominated unlearning phase and a generalization phase. The convergence of the residual beyond the maximum-margin direction is sufficient for the unlearning phase to begin. In the support vector dominated phase, we characterize the hyperplane dynamics. The time scale in these dynamics causes delayed generalization. We validate our theory through controlled experiments varying population and support-vector distributions, and further verify the predicted bias behavior in a deep model with a single output neuron on MNIST. Our results provide a unified characterization of the asymptotic and finite-time dynamics underlying delayed generalization in linear models.

Figures & tables

Panel of Figure 1 — Figure 1: Performance on test data under (a) Distribution Shift (left) and (b) same distribution as train data (right).

Panel of Figure 2 — Figure 2: Support Vector distributions affect convergence time. (a) Bias (left) (b) Distance from origin (middle) (c) train and concentrated test accuracy (right) versus number of epochs under different numbers of positive and negative support vectors, where D(1000,1000) mean 1000 positive class points and 1000 negative class points. Similarly S(350,350) means 350 support vectors for positive class and 350 support vectors for negative class. The three shaded regions refer to phase 1, phase 2, phase 3 respectively in all the plots.

Panel of Figure 3 — Figure 3: Evolution of (a) the bias $b(t)$ (left), (b) the distance of the separating hyperplane from the origin $|b(t)|/\|\mathbf{w}(t)\|$ (middle), and (c) training and concentrated test accuracies (right), as functions of training epochs, for varying degrees of positive and negative class imbalance. Notation is the same as Figure 2 .

Panel of Figure 4 — Figure 4: Evolution of (a) the bias $b(t)$ (left), (b) the distance of the separating hyperplane from the origin, $\lvert b(t)\rvert/\|\mathbf{w}(t)\|$ (middle), and (c) training and test accuracies (right), as functions of training epochs, for the MNIST binary classification tasks 4 vs. 9 and 9 vs. 4. For 4 vs. 9, digit 4 is the positive class with $N^{+}=5743$ and $\lvert\mathcal{S}^{+}\rvert=195$ , while digit 9 is the negative class with $N^{-}=5862$ and $\lvert\mathcal{S}^{-}\rvert=178$ . For 9 vs. 4, digit 9 is the positive class with $N^{+}=5860$ and $\lvert\mathcal{S}^{+}\rvert=173$ , while digit 4 is the negative class with $N^{-}=5742$ and $\lvert\mathcal{S}^{-}\rvert=189$ . The shaded regions correspond to phase 1, phase 2, and phase 3, respectively.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Train and test data distribution

Hyperparameter	Synthetic ( Figures 1 , 2 and 3 )	MNIST ( Figure 4 )
Data
Input dimension $d$	$2$	$784$
Input domain	$[-5,5]^{2}$	$[0,1]^{784}$ (pixels $/255$ )
Ground-truth direction	$\mathbf{w}_{\text{true}}=\frac{1}{\sqrt{2}}[1,1]^{\top}$	–
Margin parameter $\gamma$	$10^{-3}$	–
Sensitivity parameter $\alpha$	$10$	–

Appendix

Table 1: Hyperparameters of all experiments. The step size

\eta

is the step size on the unnormalized loss

\mathcal{L}

used in the theory, obtained from the learning rate

\eta_{\mathrm{lr}}

on the normalized loss

\frac{1}{N}\mathcal{L}

\eta=\eta_{\mathrm{lr}}/N

( Equation 12 ).

Linear Classification

See all 19 papers

The Dynamics of Generalization in Deep Learning
Apr 23, 2025Rubing Yang, Pratik ChaudhariNeural Network Generalization Neural Network Training Dynamics
Department of Applied Mathematics and Computational Sciences, University of Pennsylvania Philadelphia, PA, 19104, USA · Department of Electrical and Systems Engineering, University of Pennsylvania Philadelphia, PA, 19104, USA
A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay
Sep 7, 2026Yuqing Wang, Ioannis G. Kevrekidis, Mikhail BelkinNeural Network Generalization Neural Network Training Dynamics
Johns Hopkins University · University of California San Diego
To Grok Grokking: Provable Grokking in Ridge Regression
Jan 27, 2026Mingyue Xu, Gal Vardi, Itay SafranNeural Network Generalization Ridge Regression
Department of Computer Science, Purdue University, West Lafayette, IN, USA · Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Israel · Stein Faculty of Computer and Information Science, Ben-Gurion University of the Negev, Israel

Apr 23, 2025J/K move · Enter open · S save

The Dynamics of Generalization in Deep Learning

Rubing Yang, Pratik Chaudhari

Department of Applied Mathematics and Computational Sciences, University of Pennsylvania Philadelphia, PA, 19104, USA · Department of Electrical and Systems Engineering, University of Pennsylvania Philadelphia, PA, 19104, USA

Neural Network Generalization Neural Network Training Dynamics Statistical Learning Theory Gradient Descent Dynamics

Open paper AskPDF arXiv

We derive a differential equation that governs the evolution of the generalization gap when a model is trained by gradient descent-based methods. This differential equation is driven by two key quantities, a contraction factor that brings together trajectories corresponding to slightly different datasets, and a perturbation factor that accounts for them training on different datasets. The coupled decay of contraction and perturbation guarantees a controlled accumulation of generalization gap during training. We analyze this differential equation to show that the generalization gap is given by a quadratic form that consists of an ``effective Gram matrix'' that depends upon the training trajectory and a certain residual of the predictor at initialization. Our framework is applicable to general deep networks and smooth loss functions. In numerical experiments on different neural network architectures, datasets and sample sizes, we show that this quadratic form accurately captures the actual generalization gap. We also show how to instantiate our framework in a number of examples via analytical calculations. For example, for high-dimensional linear regression, our framework matches existing calculations of generalization gap in the literature exactly in under-parameterized, over-parameterized and critical regimes.

Abstract

Linear Classification

Explore similar work

The Dynamics of Generalization in Deep Learning

A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay

To Grok Grokking: Provable Grokking in Ridge Regression