Finite-step statistical mechanics of scale-invariant learning: spectral equations of state and curvature transport

Alexander Itin

cond-mat.stat-mechOct 27, 2026

Finite-step statistical mechanics of scale-invariant learning: spectral equations of state and curvature transport

Authors: Alexander Itin

Organizations: Radboud University, Nijmegen, The Netherlands · Independent researcher, Hamburg, Germany

Abstract

Stationary learning admits an ideal-gas description under restrictive noise assumptions. In neural networks whose gradient noise departs from these assumptions, however, an entropy-scaling relation resembling a Maxwell relation remains approximately valid. We ask which thermodynamic-looking relations follow from scale symmetry rather than equilibrium, and how departures from the ideal-gas description can be quantified, for discrete stochastic gradient dynamics with memory. An exact rescaling of trajectories explains the entropy relation without assuming thermal equilibrium. Within a local harmonic approximation, self-adjusting weight norms yield a spectral equation of state whose virial coefficients are set by moments of the curvature spectrum, and coupled parameter groups can sustain stationary probability currents. Our central result beyond this approximation is an exact curvature-transport identity. Together with a finite-step loss balance, it decomposes corrections to the harmonic fluctuation relation into measurable curvature changes, finite-step remainders, gradient-memory correlations and finite-window drift terms. A related displacement-weighted curvature change provides a diagnostic of the arrow of time; its mean and second moment give a lower bound on the relative entropy between forward and reversed state pairs. We also calculate the activation-boundary contribution to curvature for rectified teacher-student networks with Gaussian inputs. Tests in sphere models, teacher-student networks and convolutional networks on CIFAR-10 and CIFAR-100 support the transport description and identify limits of the harmonic and cubic-remainder approximations. These results separate symmetry-based relations from equilibrium assumptions and provide quantitative diagnostics of nonideal fluctuations in learning dynamics.

Figures & tables

Figure 1: The main results. (a) Norm balance of an invariant group

w_{a}

r=\|w_{a}\|

( Proposition 1 ; drawn to scale). The tangential step adds

\eta^{2}\|g_{B,a}\|^{2}

r^{2}

, and decay removes

(2\gamma-\gamma^{2})r^{2}

. The diagram shows SGD. Gradient-memory dynamics obeys the distinct norm balance ( 8 ). Both balances remain valid for an invariant group coupled to a trained head. Neither balance implies equilibrium. (b) Transport against the update-curvature deficit, ( 19 ) (seed means for the thin ResNet; runs with collapsed channels omitted). Softplus settings lie on the diagonal within 1.3 joint standard errors of the gap. The labelled ReLU settings lie below it. The practical ReLU network at

\gamma=10^{-4}

(open symbol) lies above it. (c) Endpoint ratios of the runs with

\rho_{0}

resolved. Softplus runs lie on

\rho_{H}=6\rho_{1}-12\rho_{0}

, the line of a remainder

af^{3}+bf^{4}

along the step. On CIFAR-10 they lie near the cubic point. ReLU runs sit near the boundary value 2 ( Lemma 14 ), also on the line. However, only Softplus runs approximately satisfy it for individual sampled transitions ( Appendix E ). Error bars: one standard error.

$\eta,\ \lambda,\ \gamma=\eta\lambda$	learning rate, weight decay, decay per step	$G=\mathbb{E}\\|g\\|^{2}$	mean-gradient power
$\beta,\ \kappa,\ q$	memory parameter, ( 3 )	$B_{H}=\mathbb{E}[g^{\top}Hg]$	curvature along $g$
$a_{\mathrm{EoS}}$	stability threshold of $\eta h$	$\Xi_{0}=\mathbb{E}\operatorname{Tr}\Sigma$	noise power
$\theta=(w_{a},\psi)$ ; $r_{a},\ n_{a}$	parameters; norm and direction of group $a$	$\Xi_{H}=\mathbb{E}\operatorname{Tr}(\Sigma H)$	noise curvature
$\eta_{0}=\eta/r^{2}$	angular step	$\mathcal{K}=\mathbb{E}[m_{t+1}^{\top}H(\theta_{t})m_{t+1}]$	update curvature
$h_{k},\ s_{k}$	curvature and angular noise of mode $k$	$\mathcal{K}_{\mathrm{lin}}$ , $U=\mathcal{K}/\mathcal{K}_{\mathrm{lin}}$	its linear value, ratio
$R_{k}=\eta_{0}h_{k}/a_{\mathrm{EoS}}$	edge ratio	$c$	linear-FDR ratio, ( 15 )

Table 1: Notation. Expectations are over the stationary law (or the time marginals of a finite window) and over the noise.

Figure 2: Sphere model (

d=51

). (a) Mode temperature against the edge ratio. SGD, gradient memory and AdamW follow

1/(1-R)

(the continuous-time limit predicts 1). With memory,

T^{q}_{k}

and

R^{q}_{k}

of ( 28 ) are plotted. Open symbols mark runs with intermittent excursions (

R_{\mathrm{top}}>0.75

). (b) Stationary squared radius relative to the square of the ideal-gas radius at fixed angular noise (

r_{0}

for SGD and memory,

r_{\mathrm{A}}

for AdamW). Simulation against theory, no fitted parameters. (c) Approach to the threshold at fixed learning rate. For SGD the temperature of the top mode grows with

\eta\lambda

(concentration in the top mode). With memory the chains leave the harmonic regime (open symbols: large-amplitude states).

$\gamma$	$\sigma$	$R_{\mathrm{top}}$	$\\|g\\|^{2}/$ continuum	$\\|g\\|^{2}/$ linear FDR	stiff var/pred	soft var/pred
$10^{-3}$	0.30	0.142	1.157	$1.0008\pm 0.0003$	1.000	0.997 (1.091)
$10^{-3}$	0.10	0.307	1.419	$1.0005\pm 0.0001$	0.999	0.999 (1.042)
$3\cdot 10^{-3}$	0.10	0.389	1.598	$1.0010\pm 0.0001$	1.002	1.011 (1.114)
$10^{-2}$	0.30	0.316	1.431	$1.0044\pm 0.0001$	1.004	1.034 (1.466)
$10^{-2}$	0.10	0.447	1.743	$1.0018\pm 0.0001$	1.008	1.044 (1.351)
$3\cdot 10^{-2}$	0.30	0.399	1.595	$1.0104\pm 0.0001$	1.015	1.106 (2.188)

Table 2: Teacher–student, SGD (7 of 16 runs;

B=16

, Gaussian label noise of level

\sigma

, equal output weights). Mode variances relative to the surrogate ( 26 ): stiff modes (the two upper Hessian levels) and soft modes (44 of 60). In parentheses, soft modes against the contraction-only form

x_{+}=(1-\gamma-\eta_{0}h)x-\eta_{0}\zeta

Figure 3: Teacher–student network with gradient memory (24 runs; graded output weights,

B=16

\sigma=0.3

, 128 chains, 20,000 measured steps), against the memory parameter

\beta

. (a) Relative residual of the exact finite-decay relation ( 15 ): the measured

c

divided by the right-hand side of ( 15 ), evaluated with the measured

U

\rho_{0}

and

\rho_{1}

, minus one. (b) Curvature-transport contribution: the narrow-to-wide ratio

c(\tau{=}0.1)/c(\tau{=}1)

divided by the ratio of the single-step remainder brackets of ( 15 ). It is compared with the predicted memory term

[1-b_{2}\rho_{H}(1)]/[1-b_{2}\rho_{H}(0.1)]

(bars). (c) The signed arrow-of-time observable

\rho_{H}=\mathbb{E}\mathcal{A}/Q

. It is negative and geometric for the wide activation and positive for the narrow one at

\beta\geq 0.9

. Error bars are one standard error over chains. The full table is in Appendix F ; check N7 in Appendix H .

Figure 4: Campaign v1 (thin SI ResNet-18 unless stated). (a)

\|g\|^{2}

relative to the continuous-time (ideal-gas) value

\frac{\eta}{2}\operatorname{Tr}\Sigma H

(“SDE”). It is above 1 for SGD and below 1 with memory and AdamW. (b) Relative to the linear FDR ( 24 ) with the edge term. SGD comes down to 0.8–0.9 at small

\eta\lambda

. Memory and AdamW are unchanged because their edge term is negligible. (c) Edge ratio of the sharpest mode (curvature re-measured on fresh batches). (d) Variance of the top-3 modes (three SGD models) relative to the continuous-time value, against

1/(1-R)

. (e) Radius of the thin ResNet with SGD, measured and predicted with the edge term, relative to the measured-noise baseline

r^{2}_{\mathrm{SDE}}

. (f) Practical ResNet-18: ratio of the two sides of the norm balance ( 7 ) for 2,928 invariant groups (per channel and joint) in 12 runs.

	SGD				EMA momentum			AdamW
$\eta\lambda$	SDE	Edge	$c$	$R_{\mathrm{top}}$	SDE	$c$	$R_{\mathrm{top}}$	SDE	$c$	$R_{\mathrm{top}}$
$1$ – $2\cdot 10^{-5}$	1.08	17%	0.89	0.28	0.97	0.95	0.02	0.90	0.89	0.03
$5\cdot 10^{-5}$ – $10^{-4}$	1.15	23%	0.88	0.49	0.89	0.88	0.03	0.87	0.86	0.04
$2$ – $5\cdot 10^{-4}$	1.16	26%	0.82	0.54	0.78	0.76	0.04	0.76	0.75	0.05
$10^{-3}$	1.35	36%	0.80	0.79	0.74	0.72	0.07	0.72	0.71	0.06
$2$ – $5\cdot 10^{-3}$	1.50	50%	0.64	1.22	0.64	0.61	0.13	0.63	0.61	0.13

Table 3: Thin SI ResNet-18, medians per

\eta\lambda

bin. “SDE”:

\|g\|^{2}/(\frac{\eta}{2}\operatorname{Tr}\Sigma H)

. “Edge”: share of the edge term

\frac{\eta}{a_{\mathrm{EoS}}}g^{\top}Hg

in the linear FDR.

c

\|g\|^{2}

relative to the linear FDR ( 24 ).

R_{\mathrm{top}}

: edge ratio of the sharpest mode. Runs per bin: SGD 4, 6, 4, 4, 4; EMA 4, 4, 4, 2, 4; AdamW 4, 5, 4, 3, 4.

$R_{k}$	modes	var/theory, median	mean $\pm$ s.e.	var/continuum, median
$[0,0.3)$	46	1.07	$1.09\pm 0.03$	1.32
$[0.3,0.5)$	47	0.98	$1.03\pm 0.03$	1.68
$[0.5,0.7)$	24	0.82	$0.87\pm 0.04$	1.96
$[0.7,0.95)$	11	0.50	$0.50\pm 0.06$	2.39

Table 4: Mode test, SGD,

\eta\lambda\leq 10^{-3}

(thin ResNet, ConvNet and practical ResNet). Variance of the weight snapshots along the top-3 Hessian eigenvectors, relative to the discrete prediction

\eta\sigma_{k}^{2}/(2\mu_{k}(1-R_{k}))

(“theory”) and to the continuous-time value

\eta\sigma_{k}^{2}/(2\mu_{k})

. Here

\sigma_{k}^{2}=v_{k}^{\top}\Sigma v_{k}

Figure 5: Campaign v2. (a) SGD: the remainder prediction

c_{\mathrm{R}}

against the measured loss-balance ratio

c_{\mathrm{E}}

, for all 20 SGD runs (thin and practical ResNet; ReLU and Softplus). (b,c) Remainder profiles

\rho(f)=\mathcal{R}(f)/(\frac{f^{2}}{2}\Delta\theta^{\top}H\Delta\theta)

of the thin ResNet with SGD at

\eta\lambda=10^{-4}

and

10^{-3}

(thin lines: individual runs; thick: means over seeds). ReLU:

\rho

flat in

f

(activation boundaries,

\mathcal{R}\propto f^{2}

). Softplus:

\rho\propto f

(

\mathcal{R}\propto f^{3}

). (d) Memory and AdamW: the linear-FDR ratio lies well below the single-step remainder prediction.

model	$\eta\lambda$	activation	$c_{\mathrm{E}}$	$c_{\mathrm{R}}$	$\rho(1)$	$\rho(\frac{1}{16})$	$R_{\mathrm{top}}$	flips	$\chi$	test acc.
thin	$3\cdot 10^{-5}$	ReLU	0.858	0.914	$-0.086$	$-0.071$	0.36	1.3%	–	0.678
		Softplus 0.5	0.909	0.959	$-0.041$	$-0.004$	0.31	1.5%	0.11	0.714
		Softplus 0.05	0.963	0.971	$-0.029$	$-0.002$	0.31	1.4%	1.02	0.680
	$10^{-4}$	ReLU	0.881	0.895	$-0.105$	$-0.093$	0.48	2.1%	–	0.706
		Softplus 0.5	0.929	0.952	$-0.048$	$-0.003$	0.37	2.5%	0.17	0.731
		Softplus 0.05	1.017	0.977	$-0.023$	$+0.001$	0.33	2.4%	1.71	0.703

Table 5: Campaign v2, SGD, seed means (2 seeds for the thin ResNet, 1 for the practical one).

c_{\mathrm{E}}

: loss-balance ratio.

c_{\mathrm{R}}=1+\rho(1)

: its prediction from the remainder.

\rho(1)

\rho(\frac{1}{16})

: remainder profile. Flips: fraction of units that change sign along the full step.

\chi

: RMS pre-activation change in units of

\tau

Figure 6: Direct endpoint measurements with gradient memory (campaigns v3, CIFAR-10, and v4, CIFAR-100;

\beta=0.9

). One point per setting; seed means for the thin ResNet. Open symbols and faded bars: runs with collapsed channels. (a) Residuals of the finite-decay relation in window form (as in Table 6 ) and of the accounting ( 19 ) in units of

\mathcal{K}_{\mathrm{lin}}

. (b) Accounting of the update-curvature deficit: measured

1-U

, transport

-\delta_{D}

and mixed term

-\delta_{X}

. (c) Signed time asymmetry

\rho_{H}=\mathbb{E}\mathcal{A}/Q

. Error bars: one standard error, as in Table 6 (joint bootstrap). For seed means, the error is the larger of the propagated and the seed-scatter error.

CIFAR-10
setting, $\gamma$	$c$	residual	$1-U$	$-\delta_{D}$	gap	$\rho_{1}/\rho_{0}$	$\rho_{H}/\rho_{0}$	$\rho_{H}$
thin ReLU $10^{-4}$	$0.830(25)$	$-0.035(41)$	$0.032(29)$	$0.024(14)$	$0.008(30)$	$2.02(8)$	$0.1(1)$	$-0.011(7)$
thin ReLU $10^{-3}$	$0.715(21)$	$0.059(33)$	$0.224(20)$	$0.122(18)$	$0.102(27)$	$2.17(6)$	$0.5(1)$	$-0.066(11)$
thin Softpl. $10^{-4}$	$0.925(23)$	$-0.027(33)$	$0.039(23)$	$0.081(16)$	$-0.042(34)$	$2.84(3)$	$5.0(2)$	$-0.039(8)$
thin Softpl. $10^{-3}$	$0.808(18)$	$0.006(33)$	$0.178(22)$	$0.159(23)$	$0.019(34)$	$2.90(2)$	$5.4(2)$	$-0.083(14)$
pract. ReLU $10^{-4}$	$0.791(23)$	$-0.009(54)$	$0.062(41)$	$0.253(88)$	$-0.191(112)$	$2.12(10)$	$0.9(2)$	$-0.126(42)$

Table 6: Direct endpoint measurements with gradient memory (

\beta=0.9

; thin SI ResNet-18: means of two seeds on CIFAR-10, three on CIFAR-100, with the larger of the propagated and the seed-scatter error; practical ResNet-18: one run each; joint bootstrap standard errors in parentheses, in units of the last digit). Residual of ( 15 ) (thin) or ( 14 ) with

\lambda\bar{e}

(practical), relative to its left-hand side; deficit

1-U

, transport

-\delta_{D}

of ( 19 ) and the gap

(1-U)-(-\delta_{D})

it leaves;

\rho_{0}

\rho_{1}

\rho_{H}

of ( 5 ), ratios averaged over runs with

\rho_{0}

resolved (

>3

s.e.; – if the mean

\rho_{0}

is not resolved or changes sign between seeds). Softpl.: Softplus 0.5. ∗ Collapsed channels.

Statement	Level	Evidence
Norm balance per invariant group ( Proposition 1 )	exact (invariant group)	networks: median 1.004; per group 1.000
Scaling symmetry, entropy relation ( Proposition 2 , Corollary 3 )	exact (invariant groups; trained head at fixed hyperparameters)	sphere 4.000/16.007; networks 0.8–4.3%; trained head $10^{-15}$
Loss balance, finite-decay identity ( Proposition 4 , Theorem 5 )	exact (any network; common $\eta,\lambda,\beta$ for Theorem 5 )	SGD networks: RMS 0.038 (20 runs); EMA networks, window form: $\chi^{2}=8.5$ , 12.5 (12 runs each); teacher–student $0.9998\pm 0.0009$ (24 runs); trained head $1.0000$ – $1.0001$
Curvature transport, arrow of time ( Propositions 6 and 7 )	exact (any network; common $\beta$ )	teacher–student: within 0.16%, $\mathbb{E}\mathcal{A}$ at 13–277 s.e.; EMA networks: accounting $\chi^{2}=17$ , 21 (12 runs each), $\rho_{H}<0$ resolved in 16 of 22 runs
Activation-boundary remainder $\propto f^{2}$ ( Lemma 14 )	asymptotic (small steps, smooth pre-activation density)	thin ResNet profiles (all but one small, non-monotone Softplus profile)
Singular curvature of ReLU committees ( Proposition 15 )	exact (Gaussian inputs)	fixed states within $4\cdot 10^{-3}$ ; teacher–student $-2.131\%$ vs $-2.127\%$

Table 7: Main statements, the level at which they hold, and the numerical evidence.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

noise	$\varepsilon$	$\eta h$	$c$	$\mathcal{K}$ /rhs ( 18 )	( 16 )	$\rho_{0}$	$\rho_{1}/\rho_{0}$	$\rho_{H}/\rho_{0}$	( 41 )
Gaussian	0.3	0.10	1.0003	1.0002	1.0002	$+0.0003$	1.99	$-0.06$	1.0037
		0.76	1.0015	0.9998	1.0017	$+0.0020$	1.99	$-0.07$	1.0292
		2.00	1.0042	0.9997	1.0045	$+0.0054$	1.99	$-0.08$	1.0825
		4.00	1.0095	1.0001	1.0094	$+0.0113$	1.98	$-0.09$	1.1886
Gaussian	0	0.76	1.0025	0.9999	1.0026	$+0.0026$	2.00	$-0.01$	1.0774
		4.00	1.0144	1.0001	1.0143	$+0.0149$	2.00	$-0.02$	1.6641

Appendix

Table 8: Curvature-transport toy: measured

c

, the transport identity, the exact

\gamma=0

relation and the single-power form.

$\beta$	$\gamma$	noise	$c_{\tau=1}$	$c_{\tau=0.1}$	$c$ /( 15 )	$\frac{1-\beta}{1-q}$	ratio	pred.	mem. (%)	$10^{3}\delta_{X}^{\rm rad}$
0.5	$10^{-4}$	G	0.9996	1.0040	0.9997/0.9998	0.9999	1.0045	1.0044	+0.10	0.1/0.1 (0.1)
		nG	0.9974	1.0017	0.9977/1.0005		1.0043	1.0015	-0.05	0.1/0.2 (0.1)
0.5	$10^{-3}$	G	0.9989	1.0202	1.0001/1.0001	0.9990	1.0213	1.0217	+0.30	1.3/1.5 (1.3)
		nG	0.9974	1.0153	0.9990/1.0002		1.0180	1.0175	+0.10	1.4/1.5 (1.3)
0.9	$10^{-4}$	G	0.9987	1.0027	0.9994/1.0009	0.9991	1.0040	1.0035	+0.30	1.0/1.0 (0.9)
		nG	0.9963	1.0000	0.9983/0.9999		1.0037	1.0015	+0.12	1.0/1.0 (0.9)

Appendix

Table 9: Teacher–student, gradient memory. Linear-FDR ratio

c

for activation widths

\tau=1

and

0.1

, measured

c

over the exact relation ( 15 ) (

\tau=1

\tau=0.1

), and the per-mode factor of Lemma 11 . The ratio

c_{\tau=0.1}/c_{\tau=1}

, measured and predicted from the remainder brackets and the memory term ( Observation 2 (iii)), and the memory term in percent. Radial part of

\delta_{X}

(

\times 10^{3}

\tau=1

\tau=0.1

), with ( 47 ) in parentheses.

Statistical Physics of Learning

Statistical physics of learning uses tools from physics to explain how data, model structure, and training dynamics shape learning and generalization.

See all 54 papers

Nov 10, 2025

Can Stationary Distributions of Scale-Invariant Neural Networks Be Described by the Thermodynamics of an Ideal Gas?

Understanding the training dynamics of deep neural networks remains a major open problem, with physics-inspired approaches offering promising insights. Building on this perspective, we develop a thermodynamic framework to describe the stationary distributions of stochastic gradient descent (SGD) with weight decay for scale-invariant neural networks, a setting that both reflects practical architectures with normalization layers and permits theoretical analysis. We establish analogies between training hyperparameters (e.g., learning rate, weight decay) and thermodynamic variables such as temperature, pressure, and volume. Starting with a simplified isotropic noise model, we uncover a close correspondence between SGD dynamics and ideal gas behavior, validated through theory and simulation. Extending to training of neural networks, we show that key predictions of the framework, including the behavior of stationary entropy, align closely with experimental observations. This framework provides a principled foundation for interpreting training dynamics and may guide future work on hyperparameter tuning and the design of learning rate schedulers.

Ildus Sadrtdinov, Ekaterina Lobacheva, Ivan Klimov +3

Constructor University · Mila – Quebec AI Institute · Université de Montréal +2

Statistical Physics of Learning Neural Network Training Dynamics Stochastic Gradient Descent

Ask PDF

Sep 9, 2026

A Function-Space Approach to the Statistical Mechanics of Learning Dynamics

In the kernel regime, neural-network learning inherits its preferences from a frozen spectrum. During feature learning, this spectrum evolves, yet networks retain systematic biases toward simple, smooth directions. We develop a function-space statistical framework explaining the origin of these preferences, treating functions and their learning operators as macroscopic variables, with parameterization entering through the multiplicity of parameter configurations realizing each function. For mean-squared loss, error relaxes exactly under the evolving learning operator

M=JJ^\ast

. Training stochasticity induces a Gaussian weight over function-space states, while parameter multiplicity contributes an entropic operator

B

, defined by the curvature of its log multiplicity. A local Laplace expansion yields the fluctuation free energy

Φ_{\mathrm{fluc}}(M;B)=\frac{σ_ξ^2}{2}\log\det(M^{-1}+B)+\mathrm{const}

, analogous to an Occam factor. Under mild statistical conditions, this free energy is rotationally stationary exactly when

[M,B]=0

, is minimized by pairing large eigenvalues of

M

with small eigenvalues of

B

, and generates a local restoring force against mismatch. Learning is therefore biased toward faster relaxation along entropically cheaper directions. This preference strengthens with training noise and vanishes in the deterministic limit, beyond gradient-flow accounts of operator alignment. For ReLU networks, we relate entropic curvature to the minimal rearrangement of activation boundaries required for a functional change and bound this structural cost by directional smoothness. Consequently, smooth directions are preferentially learned faster, in a data-adaptive manner, even as the learning operator evolves.

Yizhou Zhang, Weichen Wu, Lun Du +1

Variational AI · The Voleon Group · Independent Researcher +1

Statistical Physics of Learning Representation Learning Neural Network Training Dynamics Statistical Learning Theory Implicit Regularization

Ask PDF

Jun 16, 2026

A Link between Shock-wave Theory and Symmetry-reduced Stochastic Gradient Descent for Artificial Neural Networks

We develop a mathematically explicit link between shock-wave theory and the symmetry-quotiented learning dynamics of stochastic gradient descent, drawing on differential geometry, Lie group theory, and fluid mechanics. Specifically, after quotienting parameter symmetries and applying local-entropy coarse-graining, the effective dynamics satisfy a viscous Hamilton--Jacobi equation on the quotient manifold. Moreover, under the assumption that the raw parameter dynamics can be summarized by a gradient field on the quotiented space, the gradient of the coarse-grained loss function obeys a Burgers-type equation, and shock formation can be established rigorously. We apply our theory to multilayer perceptrons, convolutional neural networks, Transformers, and mean-field networks, and show that they obey the Hamilton--Jacobi or Burgers-type equations. We conjecture that this framework also yields practical diagnostics for deep learning. In architectures such as Transformers, raw parameter norms are often distorted by symmetry redundancy and may therefore be misleading, whereas symmetry-corrected quotient observables provide a principled basis for monitoring, forecasting, and controlling training-phase transitions.

Taiki Miyagawa

NEC Corporation

Neural Network Training Dynamics Gradient Descent Dynamics Stochastic Gradient Descent

Ask PDF

Abstract

Statistical Physics of Learning

Explore similar work

Can Stationary Distributions of Scale-Invariant Neural Networks Be Described by the Thermodynamics of an Ideal Gas?

A Function-Space Approach to the Statistical Mechanics of Learning Dynamics

A Link between Shock-wave Theory and Symmetry-reduced Stochastic Gradient Descent for Artificial Neural Networks