Global Convergence in Deep Networks via the Second Law of Thermodynamics

Matan Tsipory; Ofir Gaash; Itamar Harel; Nathan Srebro; Daniel Soudry

cs.LGOct 17, 2026

Global Convergence in Deep Networks via the Second Law of Thermodynamics

Authors: Matan Tsipory, Ofir Gaash, Itamar Harel, Nathan Srebro, Daniel Soudry

Organizations: Technion - Israel Institute of Technology, The Andrew and Erna Viterbi Faculty of Electrical & Computer Engineering, Technion City, Haifa 3200003, Israel. · Technion - Israel Institute of Technology, Faculty of Data and Decision Sciences, Technion City, Haifa 3200003, Israel. · Toyota Technological Institute at Chicago, Chicago, IL, USA.

Abstract

Existing convergence analyses of finite-width overparameterized deep neural networks under standard parameterization often rely on parameters remaining close to their initialization and typically neglect the role of noise, despite empirical evidence that stochasticity can improve optimization and generalization. To bridge this gap, we study $L_2$ -regularized continuous-time Langevin dynamics (CLD) with isotropic noise, for training feedforward networks with smooth activations. We introduce a change-of-measure inequality, inspired by the second law of thermodynamics, that enables control of key properties of the training dynamics under the time-evolving parameter distribution. Leveraging a sharpened lower bound on the minimum eigenvalue of the Neural Tangent Kernel (NTK) at initialization, we show that injected noise helps preserve NTK stability with high probability without confining the weights to a small neighborhood of their initialization. As a result, we derive a non-asymptotic upper bound on the expected training loss. Our analysis yields explicit conditions on width and noise temperature that guarantee convergence to a finite loss floor. Finally, we experimentally show the network Jacobian can change from initialization, departing the commonly analyzed lazy regime.

Figures & tables

Parameter- ization	Setting	Parameters	$\mathcal{L}_{\infty}$	Requirements for $\exists\mathcal{L}_{g}<1:\mathcal{L}_{\infty}\leq\mathcal{L}_{g}$
Standard	Shallow	$\begin{aligned} \left(s_{1},s_{2}\right)&=\left(1,1\right)\\ \left(a_{1},a_{2}\right)&=\left(d,m\right)\\ \kappa&=1\end{aligned}$	$\begin{aligned} &\asymp\beta\exp\left(-c\frac{m}{\beta n\ln n}\right)\\ &\hskip 9.24994pt+\beta^{-1}n\end{aligned}$	$\begin{aligned} m&=\tilde{\Omega}\left(\beta n\right)\\ \beta&=\Omega\left(\frac{n}{\mathcal{L}_{g}}\right)\end{aligned}$
	Deep	$\begin{aligned} s_{l}&=1,\,l\in\left[{L+1}\right]\\ a_{l}&=m_{l-1},\,l\in\left[{L+1}\right]\\ \kappa&=1\end{aligned}$	$\begin{aligned} &\asymp\beta\exp\left(-c\frac{m}{\beta n\ln n}\right)\\ &\hskip 9.24994pt+\beta^{-1}n^{2}\end{aligned}$	$\begin{aligned} m&=\tilde{\Omega}\left(\beta n\right)\\ \beta&=\Omega\left(\frac{n^{2}}{\mathcal{L}_{g}}\right)\\ \end{aligned}$
NTK	Shallow	$\begin{aligned} \left(s_{1},s_{2}\right)&=\left(\frac{1}{\sqrt{d}},\frac{1}{\sqrt{m}}\right)\\ \left(a_{1},a_{2}\right)&=\left(1,1\right)\\ \kappa&=1\end{aligned}$	$\begin{aligned} &\asymp\beta\exp\left(-c\frac{m}{\beta n\ln n}\right)\\ &\hskip 9.24994pt+\beta^{-1}n^{2}\end{aligned}$	$\begin{aligned} m&=\tilde{\Omega}\left(\beta n\right)\\ \beta&=\Omega\left(\frac{n^{2}}{\mathcal{L}_{g}}\right)\end{aligned}$
	Deep	$\begin{aligned} s_{l}&=\frac{1}{\sqrt{m_{l-1}}},\,l\in\left[{L+1}\right]\\ a_{l}&=1,\,l\in\left[{L+1}\right]\\ \kappa&=1\end{aligned}$	$\begin{aligned} &\asymp\beta\exp\left(-c\frac{m}{\beta n\ln n}\right)\\ &\hskip 9.24994pt+\beta^{-1}n^{2}\end{aligned}$	$\begin{aligned} m&=\tilde{\Omega}\left(\beta n\right)\\ \beta&=\Omega\left(\frac{n^{2}}{\mathcal{L}_{g}}\right)\\ \end{aligned}$

Table 1: Convergence bounds from Theorem 5.1 under standard and NTK parameterizations.

Panel of Figure 1 — Figure 1: Training statistics as a function of the number of updates for $\beta\in\{10n,50n,100n,\infty\}$ . In Figure 1(a) we plot the distance of the network Jacobian from its value at initialization. In Figure 1(b) we plot the training loss. In Figure 1(c) we show the minimal eigenvalue of the feature Gram matrix, plotting each ( $\beta$ , seed) pair separately. For $\beta=\infty$ the seed controls only the dataset.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Work	Depth	Regime	Algo.	Width
Du et al. [40]	Deep	NTK	GD	$m=\tilde{\Omega}(n^{4})$
Oymak and Soltanolkotabi [91]	Shallow	$\Theta(1)$ init., fixed last layer	GD	$m=\tilde{\Omega}(n^{2}/d)$
Song et al. [105]	Shallow	Standard	GD	$m=\tilde{\Omega}(n^{5/3}/d)$
Nguyen and Mondelli [85]	Deep, pyramidal	Standard	GD	$m_{1}=\tilde{\Omega}(n^{2})$ in the wide layer
Montanari and Zhong [79] $\dagger$	Shallow	Large $\kappa$ , $\kappa\gtrsim n/\sqrt{md}$	GF	$m=\tilde{\Omega}(n/d)$
Allen-Zhu et al. [2]	Deep	Standard	SGD	$m=\tilde{\Omega}(n^{24})$

Appendix

Table 2: Convergence analyses of gradient descent and stochastic gradient descent. In every row the empirical risk contracts to zero at a linear rate, the parameters must remain within a neighborhood of initialization, and the randomness is either absent or an obstacle the analysis must absorb; the rows differ in the architecture they cover and in the width they require. Regime : the parameterization the analysis is stated in or, where the guarantee is driven by an explicit output scaling rather than by the scale of the initialization, the scaling regime it requires. Here

n

is the number of samples,

d

the input dimension,

m

the hidden-layer width and

\kappa

the output scaling. Rows marked

\dagger

carry qualifications set out after Table 3 .

Work	Depth	Regime	Algo.	Width	Leaves init.	Objective	Rate	Floor
Randomness as an obstacle the proof must absorb
Lugosi and Nualart [72] $\dagger$	Deep	Non-standard init.	Cont. SGD	Not characterized	$\times$	Population	Linear	None
Oberweis and Cayci [89] $\dagger$	Deep	Large $\kappa$	Cont. SGD	Finite; not characterized	$\times$	Empirical	Linear	$\propto\kappa^{-1/q}$
Chen et al. [23] $\dagger$	Shallow	Generalized NTK	CLD	$m=\infty$	$\checkmark$	Empirical	Linear	$\propto\kappa^{-2}\beta^{-2}$
Randomness as the mechanism of the proof
Raginsky et al. [96] $\dagger$	Any	Any	SGLD	Any	$\checkmark$	Empirical	Exponentially slow in $\beta$	$\propto N_{p}\beta^{-1}$

Appendix

Table 3: Convergence analyses of noisy training dynamics. Rows are grouped by the role the randomness plays in the proof. Regime : as in Table 2 . Leaves init. : whether the parameters may travel far from their initialization (

\checkmark

) or must remain close (

\times

). Objective : the quantity the theorem itself contracts, namely the empirical risk, the population risk, the law of the iterates towards a Gibbs measure, or the entropy-regularized free energy. Floor : the non-vanishing residual left in the empirical risk; where the objective is something else, the qualifications give the conversion. Rate : linear when the objective contracts at an exponential rate set by the data and the architecture; exponentially slow in

\beta

when that rate is instead governed by a Poincaré or log-Sobolev constant, for which the only general estimates degrade like

e^{-\Theta(\beta)}

. Here

N_{p}

is the number of parameters and

\beta

the inverse temperature; the remaining symbols are as in Table 2 . Rows marked

\dagger

carry qualifications set out below.

Assumption	Role in the analysis	Relaxation
Essential to the mechanism
Gibbs stationary law: the dynamics admit a stationary $p_{\infty}$ that is Gibbs w.r.t. a reference $\nu$	Structural requirement enabling the change-of-measure inequality ( Lemma 2.2 ); holds for regularized CLD with any $\mathbf{\Lambda}\succ\mathbf{0}$	None; intrinsic to the mechanism
Control the terms of the change-of-measure inequality
Reference matched to initialization, $\nu=p_{0}$ , by tying $\mathbf{\Lambda}$ to the initialization ( Eq. 6 )	Eliminates the $D_{\infty}(p_{0}\\|\nu)$ term from the right-hand side (see Lemma B.7 , Remark B.8 )	Yes, at the cost of a multiplicative factor in Lemma 2.2 and Corollary 3.1
Bounded activation in the last hidden layer ( Assumption 4.1 )	Yields a subexponential loss at initialization, bounding the exponential-moment prefactor in Corollary 3.1	Yes, via output normalization or soft clipping
Artifacts of the analysis

Appendix

Table 4: Assumptions grouped by their role in the analysis, with where each enters and whether it can be relaxed.

Panel of Figure 2 — Figure 2: The deviation from the linearized dynamics as a function of number of updates for $\beta\in\{10n,50n,100n,\infty\}$ , measured as the parameter distance between the neural network and its linearized model given by the first-order approximation at initialization. In Figure 2(a) we see the cosine distance and in Figure 2(b) we see the $L_{2}$ distance divided by the parameter norm at initialization. In Figure 2(c) we see the training loss of the neural network (solid) and the linearized model (dashed).

Panel of Figure 3 — Figure 3: The distance of the Jacobian from its value at initialization as a function of number of updates for $\beta\in\{10n,50n,100n,\infty\}$ . In Figure 3(a) the distance is the cosine distance; In Figure 3(b) the distance is the Frobenius norm divided by the norm of the Jacobian at initialization.

Panel of Figure 4 — Figure 4: Deep network training statistics as a function of the number of updates for $\beta\in\{10n,50n,100n,\infty\}$ . In Figure 4(a) we plot the (cosine) Jacobian drift throughout training. In Figure 4(b) we plot the training loss. In Figure 4(c) we show the minimal eigenvalue of the feature Gram matrix, plotting each ( $\beta$ , seed) pair separately. For $\beta=\infty$ the seed controls only the dataset.

Neural Network Training Dynamics

Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.

See all 499 papers

Aug 11, 2026

Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness

We establish the first convergence guarantees of gradient descent for general feedforward neural networks of any width or depth, with any initialization or dataset. We only assume that the activation functions are linearly bounded, Lipschitz continuous, and Lipschitz smooth---properties that hold for linear, tanh, softplus, sigmoid, and smoothed ReLU functions---and that the loss function is Lipschitz smooth in the model outputs, a mild condition satisfied by mean squared error and binary cross-entropy loss. By relating the Lipschitz properties of one layer to the next, we obtain a novel generalized Lipschitz smoothness condition for an

L

-layer neural network where the change in gradient is upper bounded by the change in the parameter space, multiplied by a polynomial of the parameter norms at both endpoints of degree

2L - 2

. This yields a descent lemma where the loss decreases as long as the learning rate is small enough with respect to the parameter norms. By ensuring that the leading term of the polynomial grows at a controlled rate, we prove that the minimum squared gradient norm converges to zero in

T

iterations at rate

O(1/T^{1/L})

Siqiao Mu, Diego Klabjan

Gradient Descent Neural Network Optimization Feedforward Neural Networks

Ask PDF

May 8, 2026

Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit

A convergence analysis is developed for the regularized Newton method for training neural networks (NNs) in the overparameterized limit. As the number of hidden units tends to infinity, the NN training dynamics converge in probability to the solution of a deterministic limit equation involving a ``Newton neural tangent kernel'' (NNTK). Explicit rates characterizing this convergence are provided and, in the infinite-width limit, we prove that the NN converges exponentially fast to the target data (i.e., a global minimizer with zero loss). We show that this convergence is uniform across the frequency spectrum, addressing the spectral bias inherent in gradient descent. The eigenvalues of the NTK for gradient descent accumulate at zero, leading to slow convergence for target data with high-frequency components. In contrast, the NNTK has uniformly lower bounded eigenvalues if the regularization parameter is selected appropriately, allowing Newton's method to converge more quickly for data with high-frequency components. Mathematical challenges that need to be addressed in our analysis include the implicit parameter update of the Newton method with a potentially indefinite Hessian matrix and the fact that the dimension of this linear system of equations tends to infinity as the NN width grows. This complicates deriving the training dynamics in the overparameterized limit as well as proving the convergence of the finite-width dynamics thereto. The analysis identifies a scaling formula for selecting the regularization parameter, which we show can vanish at a suitable rate as the number of hidden units becomes larger. We prove that, for sufficiently large numbers of hidden units, the regularized Hessian remains positive definite during training and the Newton updates for individual NN parameters converge to zero, showing that the model behaves as a linearization around the initialization.

Konstantin Riedl, Konstantinos Spiliopoulos, Justin Sirignano

Mathematical Institute University of Oxford · Department of Mathematics & Statistics Boston University

Overparameterized Neural Network Neural Network Optimization Spectral Bias Second-Order Optimization Neural Network Training Dynamics Neural Tangent Kernel

Ask PDF

Mar 16, 2026

Controlled Langevin Dynamics for Sampling of Feedforward Neural Networks Trained with Minibatches

Sampling the parameter space of artificial neural networks according to a Boltzmann distribution provides insight into the geometry of low-loss solutions and offers an alternative to conventional loss minimization for training. However, exact sampling methods such as hybrid Monte Carlo (hMC), while formally correct, become computationally prohibitive for realistic datasets because they require repeated evaluation of full-batch gradients. We introduce a pseudo-Langevin (pL) dynamics that enables efficient Boltzmann sampling of feed-forward neural networks trained with large datasets by using minibatches in a controlled manner. The method exploits the statistical properties of minibatch gradient noise and adjusts fictitious masses and friction coefficients to ensure that the induced stochastic process samples efficiently the desired equilibrium distribution. We validate numerically the approach by comparing its equilibrium statistics with those obtained from exact hMC sampling. Performance benchmarks demonstrate that, while hMC rapidly becomes inefficient as network size increases, the pL scheme maintains high computational diffusion and scales favorably to networks with over one million parameters. Finally, we show that sampling at intermediate temperatures yields optimal generalization performance, comparable to SGD, without requiring a validation set or early stopping procedure. These results establish controlled minibatch Langevin dynamics as a practical and scalable tool for exploring and exploiting the solution space of large neural networks.

Alessandro Zambon, Francesca Caruso, Riccardo Zecchina +1

Department of Physics, Università degli Studi di Milano and INFN, via Celoria 16, 20133 Milano, Italy · Department of Computing Sciences and Bocconi Institute for Data Science and Analytics (BIDSA), Bocconi University, 20136 Milano, Italy

Stochastic Gradient Langevin Dynamics Neural Network Training Dynamics Feedforward Neural Networks Neural Network Generalization

Ask PDF

Abstract

Neural Network Training Dynamics

Explore similar work

Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness

Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit

Controlled Langevin Dynamics for Sampling of Feedforward Neural Networks Trained with Minibatches