Global Convergence in Deep Networks via the Second Law of Thermodynamics

Matan Tsipory; Ofir Gaash; Itamar Harel; Nathan Srebro; Daniel Soudry

cs.LGOct 17, 2026

Global Convergence in Deep Networks via the Second Law of Thermodynamics

Authors: Matan Tsipory, Ofir Gaash, Itamar Harel, Nathan Srebro, Daniel Soudry

Organizations: Technion - Israel Institute of Technology, The Andrew and Erna Viterbi Faculty of Electrical & Computer Engineering, Technion City, Haifa 3200003, Israel. · Technion - Israel Institute of Technology, Faculty of Data and Decision Sciences, Technion City, Haifa 3200003, Israel. · Toyota Technological Institute at Chicago, Chicago, IL, USA.

Abstract

Existing convergence analyses of finite-width overparameterized deep neural networks under standard parameterization often rely on parameters remaining close to their initialization and typically neglect the role of noise, despite empirical evidence that stochasticity can improve optimization and generalization. To bridge this gap, we study $L_2$ -regularized continuous-time Langevin dynamics (CLD) with isotropic noise, for training feedforward networks with smooth activations. We introduce a change-of-measure inequality, inspired by the second law of thermodynamics, that enables control of key properties of the training dynamics under the time-evolving parameter distribution. Leveraging a sharpened lower bound on the minimum eigenvalue of the Neural Tangent Kernel (NTK) at initialization, we show that injected noise helps preserve NTK stability with high probability without confining the weights to a small neighborhood of their initialization. As a result, we derive a non-asymptotic upper bound on the expected training loss. Our analysis yields explicit conditions on width and noise temperature that guarantee convergence to a finite loss floor. Finally, we experimentally show the network Jacobian can change from initialization, departing the commonly analyzed lazy regime.

Figures & tables

Parameter- ization	Setting	Parameters	$\mathcal{L}_{\infty}$	Requirements for $\exists\mathcal{L}_{g}<1:\mathcal{L}_{\infty}\leq\mathcal{L}_{g}$
Standard	Shallow	$\begin{aligned} \left(s_{1},s_{2}\right)&=\left(1,1\right)\\ \left(a_{1},a_{2}\right)&=\left(d,m\right)\\ \kappa&=1\end{aligned}$	$\begin{aligned} &\asymp\beta\exp\left(-c\frac{m}{\beta n\ln n}\right)\\ &\hskip 9.24994pt+\beta^{-1}n\end{aligned}$	$\begin{aligned} m&=\tilde{\Omega}\left(\beta n\right)\\ \beta&=\Omega\left(\frac{n}{\mathcal{L}_{g}}\right)\end{aligned}$
	Deep	$\begin{aligned} s_{l}&=1,\,l\in\left[{L+1}\right]\\ a_{l}&=m_{l-1},\,l\in\left[{L+1}\right]\\ \kappa&=1\end{aligned}$	$\begin{aligned} &\asymp\beta\exp\left(-c\frac{m}{\beta n\ln n}\right)\\ &\hskip 9.24994pt+\beta^{-1}n^{2}\end{aligned}$	$\begin{aligned} m&=\tilde{\Omega}\left(\beta n\right)\\ \beta&=\Omega\left(\frac{n^{2}}{\mathcal{L}_{g}}\right)\\ \end{aligned}$
NTK	Shallow	$\begin{aligned} \left(s_{1},s_{2}\right)&=\left(\frac{1}{\sqrt{d}},\frac{1}{\sqrt{m}}\right)\\ \left(a_{1},a_{2}\right)&=\left(1,1\right)\\ \kappa&=1\end{aligned}$	$\begin{aligned} &\asymp\beta\exp\left(-c\frac{m}{\beta n\ln n}\right)\\ &\hskip 9.24994pt+\beta^{-1}n^{2}\end{aligned}$	$\begin{aligned} m&=\tilde{\Omega}\left(\beta n\right)\\ \beta&=\Omega\left(\frac{n^{2}}{\mathcal{L}_{g}}\right)\end{aligned}$
	Deep	$\begin{aligned} s_{l}&=\frac{1}{\sqrt{m_{l-1}}},\,l\in\left[{L+1}\right]\\ a_{l}&=1,\,l\in\left[{L+1}\right]\\ \kappa&=1\end{aligned}$	$\begin{aligned} &\asymp\beta\exp\left(-c\frac{m}{\beta n\ln n}\right)\\ &\hskip 9.24994pt+\beta^{-1}n^{2}\end{aligned}$	$\begin{aligned} m&=\tilde{\Omega}\left(\beta n\right)\\ \beta&=\Omega\left(\frac{n^{2}}{\mathcal{L}_{g}}\right)\\ \end{aligned}$

Table 1: Convergence bounds from Theorem 5.1 under standard and NTK parameterizations.

Panel of Figure 1 — Figure 1: Training statistics as a function of the number of updates for $\beta\in\{10n,50n,100n,\infty\}$ . In Figure 1(a) we plot the distance of the network Jacobian from its value at initialization. In Figure 1(b) we plot the training loss. In Figure 1(c) we show the minimal eigenvalue of the feature Gram matrix, plotting each ( $\beta$ , seed) pair separately. For $\beta=\infty$ the seed controls only the dataset.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Work	Depth	Regime	Algo.	Width
Du et al. [40]	Deep	NTK	GD	$m=\tilde{\Omega}(n^{4})$
Oymak and Soltanolkotabi [91]	Shallow	$\Theta(1)$ init., fixed last layer	GD	$m=\tilde{\Omega}(n^{2}/d)$
Song et al. [105]	Shallow	Standard	GD	$m=\tilde{\Omega}(n^{5/3}/d)$
Nguyen and Mondelli [85]	Deep, pyramidal	Standard	GD	$m_{1}=\tilde{\Omega}(n^{2})$ in the wide layer
Montanari and Zhong [79] $\dagger$	Shallow	Large $\kappa$ , $\kappa\gtrsim n/\sqrt{md}$	GF	$m=\tilde{\Omega}(n/d)$
Allen-Zhu et al. [2]	Deep	Standard	SGD	$m=\tilde{\Omega}(n^{24})$

Appendix

Table 2: Convergence analyses of gradient descent and stochastic gradient descent. In every row the empirical risk contracts to zero at a linear rate, the parameters must remain within a neighborhood of initialization, and the randomness is either absent or an obstacle the analysis must absorb; the rows differ in the architecture they cover and in the width they require. Regime : the parameterization the analysis is stated in or, where the guarantee is driven by an explicit output scaling rather than by the scale of the initialization, the scaling regime it requires. Here

n

is the number of samples,

d

the input dimension,

m

the hidden-layer width and

\kappa

the output scaling. Rows marked

\dagger

carry qualifications set out after Table 3 .

Work	Depth	Regime	Algo.	Width	Leaves init.	Objective	Rate	Floor
Randomness as an obstacle the proof must absorb
Lugosi and Nualart [72] $\dagger$	Deep	Non-standard init.	Cont. SGD	Not characterized	$\times$	Population	Linear	None
Oberweis and Cayci [89] $\dagger$	Deep	Large $\kappa$	Cont. SGD	Finite; not characterized	$\times$	Empirical	Linear	$\propto\kappa^{-1/q}$
Chen et al. [23] $\dagger$	Shallow	Generalized NTK	CLD	$m=\infty$	$\checkmark$	Empirical	Linear	$\propto\kappa^{-2}\beta^{-2}$
Randomness as the mechanism of the proof
Raginsky et al. [96] $\dagger$	Any	Any	SGLD	Any	$\checkmark$	Empirical	Exponentially slow in $\beta$	$\propto N_{p}\beta^{-1}$

Appendix

Table 3: Convergence analyses of noisy training dynamics. Rows are grouped by the role the randomness plays in the proof. Regime : as in Table 2 . Leaves init. : whether the parameters may travel far from their initialization (

\checkmark

) or must remain close (

\times

). Objective : the quantity the theorem itself contracts, namely the empirical risk, the population risk, the law of the iterates towards a Gibbs measure, or the entropy-regularized free energy. Floor : the non-vanishing residual left in the empirical risk; where the objective is something else, the qualifications give the conversion. Rate : linear when the objective contracts at an exponential rate set by the data and the architecture; exponentially slow in

\beta

when that rate is instead governed by a Poincaré or log-Sobolev constant, for which the only general estimates degrade like

e^{-\Theta(\beta)}

. Here

N_{p}

is the number of parameters and

\beta

the inverse temperature; the remaining symbols are as in Table 2 . Rows marked

\dagger

carry qualifications set out below.

Assumption	Role in the analysis	Relaxation
Essential to the mechanism
Gibbs stationary law: the dynamics admit a stationary $p_{\infty}$ that is Gibbs w.r.t. a reference $\nu$	Structural requirement enabling the change-of-measure inequality ( Lemma 2.2 ); holds for regularized CLD with any $\mathbf{\Lambda}\succ\mathbf{0}$	None; intrinsic to the mechanism
Control the terms of the change-of-measure inequality
Reference matched to initialization, $\nu=p_{0}$ , by tying $\mathbf{\Lambda}$ to the initialization ( Eq. 6 )	Eliminates the $D_{\infty}(p_{0}\\|\nu)$ term from the right-hand side (see Lemma B.7 , Remark B.8 )	Yes, at the cost of a multiplicative factor in Lemma 2.2 and Corollary 3.1
Bounded activation in the last hidden layer ( Assumption 4.1 )	Yields a subexponential loss at initialization, bounding the exponential-moment prefactor in Corollary 3.1	Yes, via output normalization or soft clipping
Artifacts of the analysis

Appendix

Table 4: Assumptions grouped by their role in the analysis, with where each enters and whether it can be relaxed.

Panel of Figure 2 — Figure 2: The deviation from the linearized dynamics as a function of number of updates for $\beta\in\{10n,50n,100n,\infty\}$ , measured as the parameter distance between the neural network and its linearized model given by the first-order approximation at initialization. In Figure 2(a) we see the cosine distance and in Figure 2(b) we see the $L_{2}$ distance divided by the parameter norm at initialization. In Figure 2(c) we see the training loss of the neural network (solid) and the linearized model (dashed).

Panel of Figure 3 — Figure 3: The distance of the Jacobian from its value at initialization as a function of number of updates for $\beta\in\{10n,50n,100n,\infty\}$ . In Figure 3(a) the distance is the cosine distance; In Figure 3(b) the distance is the Frobenius norm divided by the norm of the Jacobian at initialization.

Panel of Figure 4 — Figure 4: Deep network training statistics as a function of the number of updates for $\beta\in\{10n,50n,100n,\infty\}$ . In Figure 4(a) we plot the (cosine) Jacobian drift throughout training. In Figure 4(b) we plot the training loss. In Figure 4(c) we show the minimal eigenvalue of the feature Gram matrix, plotting each ( $\beta$ , seed) pair separately. For $\beta=\infty$ the seed controls only the dataset.

Neural Network Training Dynamics

Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.

See all 499 papers

Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness
Aug 11, 2026Siqiao Mu, Diego KlabjanGradient Descent Neural Network Optimization
Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit
May 8, 2026Konstantin Riedl, Konstantinos Spiliopoulos, Justin SirignanoOverparameterized Neural Network Neural Network Optimization
Mathematical Institute University of Oxford · Department of Mathematics & Statistics Boston University
Controlled Langevin Dynamics for Sampling of Feedforward Neural Networks Trained with Minibatches
Mar 16, 2026Alessandro Zambon, Francesca Caruso, Riccardo Zecchina +1Stochastic Gradient Langevin Dynamics Neural Network Training Dynamics
Department of Physics, Università degli Studi di Milano and INFN, via Celoria 16, 20133 Milano, Italy · Department of Computing Sciences and Bocconi Institute for Data Science and Analytics (BIDSA), Bocconi University, 20136 Milano, Italy

Aug 11, 2026J/K move · Enter open · S save

Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness

Siqiao Mu, Diego Klabjan

Gradient Descent Neural Network Optimization Feedforward Neural Networks

Open paper AskPDF arXiv

We establish the first convergence guarantees of gradient descent for general feedforward neural networks of any width or depth, with any initialization or dataset. We only assume that the activation functions are linearly bounded, Lipschitz continuous, and Lipschitz smooth---properties that hold for linear, tanh, softplus, sigmoid, and smoothed ReLU functions---and that the loss function is Lipschitz smooth in the model outputs, a mild condition satisfied by mean squared error and binary cross-entropy loss. By relating the Lipschitz properties of one layer to the next, we obtain a novel generalized Lipschitz smoothness condition for an

L

-layer neural network where the change in gradient is upper bounded by the change in the parameter space, multiplied by a polynomial of the parameter norms at both endpoints of degree

2L - 2

. This yields a descent lemma where the loss decreases as long as the learning rate is small enough with respect to the parameter norms. By ensuring that the leading term of the polynomial grows at a controlled rate, we prove that the minimum squared gradient norm converges to zero in

T

iterations at rate

O(1/T^{1/L})

Abstract

Neural Network Training Dynamics

Explore similar work

Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness

Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit

Controlled Langevin Dynamics for Sampling of Feedforward Neural Networks Trained with Minibatches