Toward a Physics of Deep Learning and Brains

Arsham Ghavasieh; Meritxell Vila-Minana; Akanksha Khurd; John Beggs; Gerardo Ortiz; Santo Fortunato

cond-mat.dis-nnSep 26, 2025

Toward a Physics of Deep Learning and Brains

Authors: Arsham Ghavasieh, Meritxell Vila-Minana, Akanksha Khurd, John Beggs, Gerardo Ortiz, Santo Fortunato

Organizations: Center for Complex Networks and Systems Research, Luddy School of Informatics, Computing, and Engineering, Indiana University, Bloomington, Indiana 47408, USA · Department of Physics, Indiana University, Bloomington, Indiana 47405, USA · Institute for Advanced Study, Princeton, NJ 08540, USA

Abstract

Deep neural networks and brains both learn and share superficial similarities: processing nodes are likened to neurons and adjustable weights are likened to modifiable synapses. But can a unified theoretical framework be found to underlie them both? Here we show that the equations used to describe neuronal avalanches in living brains can also be applied to cascades of activity in deep neural networks. These equations are derived from non-equilibrium statistical physics and show that deep neural networks learn best when poised between absorbing and active phases. Because these networks are strongly driven by inputs, however, they do not operate at a true critical point but within a quasi-critical regime -- one that still approximately satisfies crackling noise scaling relations. By training networks with different initializations, we show that maximal susceptibility is a more reliable predictor of learning than proximity to the critical point itself. This provides a blueprint for engineering improved network performance. Finally, using finite-size scaling we identify distinct universality classes, including Barkhausen noise and directed percolation. This theoretical framework demonstrates that universal features are shared by both biological and artificial neural networks.

Figures & tables

Figure 1: Avalanches and criticality. Schematic representation of signal propagation through layers $\ell$ in deep neural networks. Circles are the neurons, columns of circles constitute layers of deep network, the darker they get the more active they are (higher $y^{\ell+1}$ , defined in the text). Signal strength (denoted by $\sqrt{q_{\ell}}$ in the text) characterizes the size of the neural gains ( $z^{\ell}$ ) or, also called pre-activations. Let $\sigma_{w}^{2}$ be a control parameter, with $\sigma_{c}^{2}$ being its critical value. When $\sigma_{w}^{2}>\sigma_{c}^{2}$ signals grow with depth, when $\sigma_{w}^{2}<\sigma_{c}^{2}$ they decay, and when in the thermodynamic limit $\sigma_{w}^{2}=\sigma_{c}^{2}$ , signals remain stable, enabling ideal information propagation. Neural networks can best preserve and process information with initializations of the weight parameters near the critical regime $\sigma_{w}^{2}\approx\sigma_{c}^{2}$ . The initial signal strength defines the avalanche threshold. The cumulative signal strength above the threshold value— the area between red line and purple dashes— gives the avalanche size, $S$ , and the number of layers it penetrates until it crosses the threshold defines the avalanche’s duration $D$ .

Figure 2: Crackling noise statistics in Gaussian initialised deep networks. (a-b) Distributions of avalanche size $S$ and duration $D$ for fixed depth $L=3000$ and varying widths $N=25,50,75,100,200,500,1000$ , with zero bias $\sigma_{b}^{2}=0$ and near-critical weights $\sigma_{w}^{2}\approx 1$ . Networks are re-initialised every $5000$ perturbations until $\sim 4\times 10^{6}$ avalanches are collected (See Text). Maximum-likelihood fits are used to find the exponents. (c) Size–duration scaling: $\langle S\rangle_{D}\sim D^{\gamma}$ for different widths with $\gamma\approx 1.53$ being the slope fitted to $N=1000$ — close to the theoretical prediction $\frac{\tau_{d}-1}{\tau_{s}-1}\approx 1.58$ . (d-e) Finite-size extrapolation of exponents via $\tau(N)=\tau+cN^{-w}$ gives $(\tau_{s},\tau_{d})\to(1.34,1.53)$ for large $N$ . Pink dashes show Barkhausen values with their errorbars as shades $(1.33\pm 0.05,\,1.50\pm 0.07)$ . (f) Shape collapse analysis with optimal rescaling value $\gamma^{\ast}\approx 1.58$ , very close to the prediction $\frac{\tau_{d}-1}{\tau_{s}-1}\approx 1.58$ (See Text and Appendices for more information).

Figure 3: Mean-field predictions and simulations of susceptibility, and learning performance. Color indicates bias variance: all curves in panels (a–e) are color-coded according to $\log_{10}\sigma_{b}^{2}$ as shown in the legend in panel (a), ranging from $\sigma_{b}^{2}=0$ (purple) to $\sigma_{b}^{2}=10^{-2.2}$ (red). (a) Steady-state signal strength $\sqrt{q^{(MF)}_{ss}}$ from mean-field theory. The logarithm bases are 10. (b) Mean-field $\sigma_{w}$ -susceptibility $\chi^{(MF)}_{\sigma_{w}^{2}}$ . (c) Test performance on the MNIST classification task after 10 epochs. (d) Empirical average avalanche size $\bar{S}$ , which is the sum of all recorded avalanche sizes divided by the number of recorded avalanches. (e) Empirical $\sigma_{w}$ -susceptibility (from avalanche statistics $\chi_{\sigma_{w}^{2}}=\frac{d\bar{S}}{d\sigma_{w}^{2}}$ ). Panels (c–e) are measured on networks with width $N=300$ and depth $L=300$ . (f) Epochs needed to reach 97% training accuracy for fully connected deep neural networks with $L=300$ , as a function of network width $N$ and fixed $\sigma_{b}^{2}=0$ . White regions indicate models that did not reach 97% within 50 epochs. The onset of successful learning approaches $1^{+}$ as size increases, which is expected for a finite system approaching its thermodynamic-limit transition.

Figure 4: Input magnitude and quasi-criticality. (a) avalanche size distribution, (b) duration distribution, and (c) $\sigma_{w}$ -susceptibility curves for varying input magnitudes $\sqrt{q_{0}}$ . The maximum in the susceptibility shifts to larger $\sigma_{w}^{2}$ values as the input magnitude increases from $\sqrt{q_{0}}=0.1$ to $\sqrt{q_{0}}=1.7$ at the point where the $\sigma_{w}$ -susceptibility rises sharply (c). Consequently, better power laws of (a) and (b) for input magnitudes $\sqrt{q_{0}}=0.1,1,1.7$ are, respectively, obtained at $\sigma_{w}^{2}\approx 1.005,1.025,1.05$ .

Figure 5: Trainability and susceptibility across initialization parameters. Heat maps show the training accuracy for MNIST (a) and CIFAR-10 (b) across the same $(\sigma_{w}^{2},\sigma_{b}^{2})$ control-parameter space. Circles mark the locations of susceptibility peaks, with circle size proportional to the peak height. In both datasets, the susceptibility peaks track the onset of trainability, linking heightened dynamical responsiveness to the region where optimization becomes successful.

Figure 6: Avalanche statistics in ResNets exhibit crackling-noise scaling (a) The distributions of avalanche size $\tilde{S}$ (See Text and Appendices) and (b) duration $D$ follow power laws. The scaling of average avalanche size and duration (c) for multiple ResNet variants thresholded with $n=2$ , or two standard deviations above the mean activity of the layer (see Appendices), suggests crackling noise relationship. (d) The exponents ( $\tau_{s},\tau_{d}$ ) are closely clustered regardless of the thresholding parameter $n$ . $\gamma\approx\frac{\tau_{d}-1}{\tau_{s}-1}\approx 2$ indicates operation near a critical point, well aligned with the universality class of mean-field directed percolation.

Figure 7: Critical transition in ResNets controlled by the BatchNorm scale parameter. (a) The order parameter $\langle\tilde{S}_{z}\rangle$ (see Eq. 9 ) as a function of $\log_{10}\gamma$ for ResNet18, ResNet50, and ResNet152. Varying the BatchNorm affine scale $\gamma$ drives a sharp change in network activity near $\gamma=1$ (vertical dashed line). (b) The corresponding susceptibility $\chi_{\gamma}=\frac{\Delta\bar{\tilde{S}}_{z}}{\Delta\log_{10}\gamma}$ peaks near $\gamma=1$ , identifying the critical region. Remarkably, the susceptibility peak lies at the onset of the successful training range, while performance degrades for $\gamma>1$ and for sufficiently small $\gamma$ [ 9 ] .

Figure 8: Mean-field Widom-like line and the edge of chaos. The edge of chaos marks parameters where the cross-input correlation depth diverges— with light and dark blue indicating the ordered and chaotic regions, respectively. In contrast, the Widom-like line locates the maximum $\sigma_{w}$ -susceptibility where sensitivity to inputs is largest— the size of the dots encode the height of maximum susceptibility. The two curves meet only at the $(\sigma_{w}^{2},\sigma_{b}^{2})=(1,0)$ . Importantly, all points on the edge of chaos correspond to the divergence of cross-input correlation depth. However, as $\sigma_{b}^{2}$ grows, the susceptibility exhibits weaker peaks until the Widom-like line completely vanishes— well aligned with the learning performance (See Text).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Training dynamics and initialization-dependent trainability on CIFAR-10. (a) Training loss and (b) training accuracy across $100$ epochs for networks initialized with fixed bias variance $\sigma_{b}^{2}=10^{-4}$ and different weight variances $\sigma_{w}^{2}$ , indicated by color. Only a restricted range of $\sigma_{w}^{2}$ produces sustained optimization and increasing accuracy, whereas initialization outside this region leads to poor trainability. (c) Training accuracy after $10$ epochs as a function of $\sigma_{w}^{2}$ for different bias variances $\sigma_{b}^{2}$ . Increasing $\sigma_{b}^{2}$ shifts and progressively narrows the range of trainable weight variances.

Figure 10: Training accuracy for networks of increasing depth. Contour plots show the training accuracy obtained after 10 epochs on the MNIST dataset for fully connected deep neural networks with hidden dimension

N=300

and depths of 200, 300, 400, and 500 layers. Each point corresponds to the average accuracy over 200 mini-batches (128 samples each) for a given initialization pair

(\sigma_{w},\sigma_{b})

. The color scale represents the achieved accuracy. The squared bias variance

\sigma_{b}^{2}

is shown on a logarithmic scale and

\sigma_{w}^{2}

on a linear scale.

Phase Transitions in Neural Networks

See all 25 papers

Correlation flow governs learning at criticality
Aug 8, 2026Andrea Combette, Nelly Pustelnik, Antoine VenailleNeural Network Initialization Neural Network Training Dynamics
CNRS, ENS de Lyon, Laboratoire de Physique UMR5672, Lyon, France
Statistical Properties of Training & Generalization
Jun 18, 2026Itay Lavie, Noam Levi, Yonatan KahnNeural Network Generalization Scientific ML
John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA · Tel Aviv University, Tel Aviv, Israel · Department of Physics, University of Toronto and Vector Institute, Toronto, ON, Canada
Pruning-induced phases in fully-connected neural networks: the eumentia, the dementia, and the amentia
Mar 12, 2026Haining Pan, Nakul Aggarwal, J. H. PixleyPhase Transitions in Neural Networks Neural Network Pruning
Department of Physics and Astronomy, Center for Materials Theory, Rutgers University, Piscataway, NJ 08854 USA · Center for Computational Quantum Physics, Flatiron Institute, New York, New York 10010, USA

Aug 8, 2026J/K move · Enter open · S save

Correlation flow governs learning at criticality

Andrea Combette, Nelly Pustelnik, Antoine Venaille

CNRS, ENS de Lyon, Laboratoire de Physique UMR5672, Lyon, France

Neural Network Initialization Neural Network Training Dynamics Neural Tangent Kernel

Open paper AskPDF arXiv

The initialization of deep neural networks determines whether information and gradients can propagate across depth, yet a unified theory connecting these properties to learning dynamics remains elusive. Combining mean-field theory and random matrix theory, we establish a direct link between correlation propagation and the Neural Tangent Kernel (NTK) that governs learning in the sequential limit of infinitely wide, infinitely deep networks. Correlation propagation to infinite depth is possible only at a single, critical point in the weight-bias variance plane. At this point, we leverage the algebraic decay of the end-to-end Jacobian with depth to prove that the NTK becomes exactly proportional to the output correlation at infinite depth, tying together information propagation and learning dynamics. We further show that orthogonal initialization suppresses the leading finite-size corrections present under Gaussian initialization, clarifying the respective roles of the two initialization ensembles in this limit. These theoretical predictions are validated quantitatively on finite-width, finite-depth networks. Together, these results demonstrate that orthogonal initialization and criticality are required to control the asymptotic dynamics of deep learning.

Abstract

Phase Transitions in Neural Networks

Explore similar work

Correlation flow governs learning at criticality

Statistical Properties of Training & Generalization

Pruning-induced phases in fully-connected neural networks: the eumentia, the dementia, and the amentia