A Gaussian Comparison Theorem for Training Dynamics in Machine Learning

Ashkan Panahi

cs.LGMar 10, 2026

A Gaussian Comparison Theorem for Training Dynamics in Machine Learning

Authors: Ashkan Panahi

Organizations: Department of Computer Science and Engineering, Chalmers University, Gothenburg, Sweden

Abstract

We study training algorithms with data following a Gaussian mixture model. For a specific family of such algorithms, we present a non-asymptotic result, connecting the evolution of the model to a surrogate dynamical system, which can be easier to analyze. The proof of our result is based on the celebrated Gordon comparison theorem. Using our theorem, we rigorously prove the validity of the dynamic mean-field (DMF) expressions in the asymptotic scenarios. Moreover, we suggest an iterative refinement scheme to obtain more accurate expressions in non-asymptotic scenarios. We specialize our theory to the analysis of training a perceptron model with a generic first-order (full-batch) algorithm and demonstrate that fluctuation parameters in a non-asymptotic domain emerge in addition to the DMF kernels.

Figures & tables

Panel of Figure 1 — Figure 1: Training error for gradient descent ( $s=0,t=.2$ ) with two classes with $\rho(y)=.5,\|\theta_{0}\|=.1$ , and overlaps: $v(y,y)=1$ and varying coupling $=v(0,1)$ .

Panel of Figure 2 — Figure 2: Training error for momentum gradient descent ( $t=.2$ ) with varying forgetting factor $s$ and two classes with $\rho(y)=.5,\ \gamma=1,\ \|\theta_{0}\|=.1$ , and overlaps: $v(y,y)=1$ and coupling $v(0,1)=-.5$ .

Figure 3: Normalized variance of training a perceptron with soft ReLU function, and two classes with $\rho(y)=.5,\ \gamma=1,\ \|\theta_{0}\|=.1$ , and overlaps: $v(y,y)=1$ and coupling $v(0,1)=-.5$ . The empirical values are calculated by averaging over $10^{5}$ realizations.

Neural Network Training Dynamics

Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.

See all 499 papers

Decoupled Descent: Exact Test Error Tracking Via Approximate Message Passing
Apr 30, 2026Max LovigNeural Network Generalization Gradient Descent
Statistics and Data Science, Yale University
Local linear convergence of gradient methods for overparameterized Gaussian mixtures
May 29, 2026Jingxing Wang, Vasileios Charisopoulos, Maryam FazelGradient Descent Gaussian Mixture Models
Electrical & Computer Engineering, University of Washington, Seattle, WA · National Institute for Theory and Mathematics in Biology, Chicago, IL · Amazon, Inc.
Plateaus, Optima, and Overfitting in Multi-Layer Perceptrons: A Saddle-Saddle-Attractor Scenario
Apr 2, 2026Alex Alì Maleknia, Yuzuru SatoNeural Network Training Dynamics Multilayer Perceptrons
Department of Mathematics, Hokkaido University, Kita 12 Nishi 7, Kita-ku, Sapporo, Hokkaido 060-0812, Japan · Present address: IMAG, IROKO, Univ Montpellier, Inria, CNRS · RIES, Hokkaido University, Kita 12 Nishi 7, Kita-ku, Sapporo, Hokkaido 060-0812, Japan

Apr 30, 2026J/K move · Enter open · S save

Decoupled Descent: Exact Test Error Tracking Via Approximate Message Passing

Max Lovig

Statistics and Data Science, Yale University

Neural Network Generalization Gradient Descent Approximate Message Passing

Open paper AskPDF arXiv

In modern parametric model training, full-batch gradient descent (and its variants) suffers due to progressively stronger biasing towards the exact realization of training data; this drives the systematic ``generalization gap'', where the train error becomes an unreliable proxy for test error. Existing approaches either argue this gap is benign through complex analysis or sacrifice data to a validation set. In contrast, we introduce decoupled descent (DD), a novel theory-based training algorithm that satisfies a train-test identity -- enforcing the train error to asymptotically track the test error for stylized Gaussian mixture models. Within this specific regime, leveraging approximate message passing theory, DD iteratively cancels the biases due to data reuse, rigorously demonstrating the feasibility of zero-cost validation and

100\%

data utilization. Moreover, DD is governed by a low-dimensional state evolution recursion, rendering the dynamics of the algorithm transparent and tractable. We validate DD on XOR classification, yielding superior performance compared to GD; additionally, we implement noisy MNIST and non-linear probing of CIFAR-10, demonstrating that even when our stylized assumptions are relaxed, DD narrows the generalization gap compared to GD.

Abstract

Neural Network Training Dynamics

Explore similar work

Decoupled Descent: Exact Test Error Tracking Via Approximate Message Passing

Local linear convergence of gradient methods for overparameterized Gaussian mixtures

Plateaus, Optima, and Overfitting in Multi-Layer Perceptrons: A Saddle-Saddle-Attractor Scenario