Toward a Physics of Deep Learning and Brains

Arsham Ghavasieh; Meritxell Vila-Minana; Akanksha Khurd; John Beggs; Gerardo Ortiz; Santo Fortunato

cond-mat.dis-nnSep 26, 2025

Toward a Physics of Deep Learning and Brains

Authors: Arsham Ghavasieh, Meritxell Vila-Minana, Akanksha Khurd, John Beggs, Gerardo Ortiz, Santo Fortunato

Organizations: Center for Complex Networks and Systems Research, Luddy School of Informatics, Computing, and Engineering, Indiana University, Bloomington, Indiana 47408, USA · Department of Physics, Indiana University, Bloomington, Indiana 47405, USA · Institute for Advanced Study, Princeton, NJ 08540, USA

Abstract

Deep neural networks and brains both learn and share superficial similarities: processing nodes are likened to neurons and adjustable weights are likened to modifiable synapses. But can a unified theoretical framework be found to underlie them both? Here we show that the equations used to describe neuronal avalanches in living brains can also be applied to cascades of activity in deep neural networks. These equations are derived from non-equilibrium statistical physics and show that deep neural networks learn best when poised between absorbing and active phases. Because these networks are strongly driven by inputs, however, they do not operate at a true critical point but within a quasi-critical regime -- one that still approximately satisfies crackling noise scaling relations. By training networks with different initializations, we show that maximal susceptibility is a more reliable predictor of learning than proximity to the critical point itself. This provides a blueprint for engineering improved network performance. Finally, using finite-size scaling we identify distinct universality classes, including Barkhausen noise and directed percolation. This theoretical framework demonstrates that universal features are shared by both biological and artificial neural networks.

Figures & tables

Figure 1: Avalanches and criticality. Schematic representation of signal propagation through layers $\ell$ in deep neural networks. Circles are the neurons, columns of circles constitute layers of deep network, the darker they get the more active they are (higher $y^{\ell+1}$ , defined in the text). Signal strength (denoted by $\sqrt{q_{\ell}}$ in the text) characterizes the size of the neural gains ( $z^{\ell}$ ) or, also called pre-activations. Let $\sigma_{w}^{2}$ be a control parameter, with $\sigma_{c}^{2}$ being its critical value. When $\sigma_{w}^{2}>\sigma_{c}^{2}$ signals grow with depth, when $\sigma_{w}^{2}<\sigma_{c}^{2}$ they decay, and when in the thermodynamic limit $\sigma_{w}^{2}=\sigma_{c}^{2}$ , signals remain stable, enabling ideal information propagation. Neural networks can best preserve and process information with initializations of the weight parameters near the critical regime $\sigma_{w}^{2}\approx\sigma_{c}^{2}$ . The initial signal strength defines the avalanche threshold. The cumulative signal strength above the threshold value— the area between red line and purple dashes— gives the avalanche size, $S$ , and the number of layers it penetrates until it crosses the threshold defines the avalanche’s duration $D$ .

Figure 2: Crackling noise statistics in Gaussian initialised deep networks. (a-b) Distributions of avalanche size $S$ and duration $D$ for fixed depth $L=3000$ and varying widths $N=25,50,75,100,200,500,1000$ , with zero bias $\sigma_{b}^{2}=0$ and near-critical weights $\sigma_{w}^{2}\approx 1$ . Networks are re-initialised every $5000$ perturbations until $\sim 4\times 10^{6}$ avalanches are collected (See Text). Maximum-likelihood fits are used to find the exponents. (c) Size–duration scaling: $\langle S\rangle_{D}\sim D^{\gamma}$ for different widths with $\gamma\approx 1.53$ being the slope fitted to $N=1000$ — close to the theoretical prediction $\frac{\tau_{d}-1}{\tau_{s}-1}\approx 1.58$ . (d-e) Finite-size extrapolation of exponents via $\tau(N)=\tau+cN^{-w}$ gives $(\tau_{s},\tau_{d})\to(1.34,1.53)$ for large $N$ . Pink dashes show Barkhausen values with their errorbars as shades $(1.33\pm 0.05,\,1.50\pm 0.07)$ . (f) Shape collapse analysis with optimal rescaling value $\gamma^{\ast}\approx 1.58$ , very close to the prediction $\frac{\tau_{d}-1}{\tau_{s}-1}\approx 1.58$ (See Text and Appendices for more information).

Figure 3: Mean-field predictions and simulations of susceptibility, and learning performance. Color indicates bias variance: all curves in panels (a–e) are color-coded according to $\log_{10}\sigma_{b}^{2}$ as shown in the legend in panel (a), ranging from $\sigma_{b}^{2}=0$ (purple) to $\sigma_{b}^{2}=10^{-2.2}$ (red). (a) Steady-state signal strength $\sqrt{q^{(MF)}_{ss}}$ from mean-field theory. The logarithm bases are 10. (b) Mean-field $\sigma_{w}$ -susceptibility $\chi^{(MF)}_{\sigma_{w}^{2}}$ . (c) Test performance on the MNIST classification task after 10 epochs. (d) Empirical average avalanche size $\bar{S}$ , which is the sum of all recorded avalanche sizes divided by the number of recorded avalanches. (e) Empirical $\sigma_{w}$ -susceptibility (from avalanche statistics $\chi_{\sigma_{w}^{2}}=\frac{d\bar{S}}{d\sigma_{w}^{2}}$ ). Panels (c–e) are measured on networks with width $N=300$ and depth $L=300$ . (f) Epochs needed to reach 97% training accuracy for fully connected deep neural networks with $L=300$ , as a function of network width $N$ and fixed $\sigma_{b}^{2}=0$ . White regions indicate models that did not reach 97% within 50 epochs. The onset of successful learning approaches $1^{+}$ as size increases, which is expected for a finite system approaching its thermodynamic-limit transition.

Figure 4: Input magnitude and quasi-criticality. (a) avalanche size distribution, (b) duration distribution, and (c) $\sigma_{w}$ -susceptibility curves for varying input magnitudes $\sqrt{q_{0}}$ . The maximum in the susceptibility shifts to larger $\sigma_{w}^{2}$ values as the input magnitude increases from $\sqrt{q_{0}}=0.1$ to $\sqrt{q_{0}}=1.7$ at the point where the $\sigma_{w}$ -susceptibility rises sharply (c). Consequently, better power laws of (a) and (b) for input magnitudes $\sqrt{q_{0}}=0.1,1,1.7$ are, respectively, obtained at $\sigma_{w}^{2}\approx 1.005,1.025,1.05$ .

Figure 5: Trainability and susceptibility across initialization parameters. Heat maps show the training accuracy for MNIST (a) and CIFAR-10 (b) across the same $(\sigma_{w}^{2},\sigma_{b}^{2})$ control-parameter space. Circles mark the locations of susceptibility peaks, with circle size proportional to the peak height. In both datasets, the susceptibility peaks track the onset of trainability, linking heightened dynamical responsiveness to the region where optimization becomes successful.

Figure 6: Avalanche statistics in ResNets exhibit crackling-noise scaling (a) The distributions of avalanche size $\tilde{S}$ (See Text and Appendices) and (b) duration $D$ follow power laws. The scaling of average avalanche size and duration (c) for multiple ResNet variants thresholded with $n=2$ , or two standard deviations above the mean activity of the layer (see Appendices), suggests crackling noise relationship. (d) The exponents ( $\tau_{s},\tau_{d}$ ) are closely clustered regardless of the thresholding parameter $n$ . $\gamma\approx\frac{\tau_{d}-1}{\tau_{s}-1}\approx 2$ indicates operation near a critical point, well aligned with the universality class of mean-field directed percolation.

Figure 7: Critical transition in ResNets controlled by the BatchNorm scale parameter. (a) The order parameter $\langle\tilde{S}_{z}\rangle$ (see Eq. 9 ) as a function of $\log_{10}\gamma$ for ResNet18, ResNet50, and ResNet152. Varying the BatchNorm affine scale $\gamma$ drives a sharp change in network activity near $\gamma=1$ (vertical dashed line). (b) The corresponding susceptibility $\chi_{\gamma}=\frac{\Delta\bar{\tilde{S}}_{z}}{\Delta\log_{10}\gamma}$ peaks near $\gamma=1$ , identifying the critical region. Remarkably, the susceptibility peak lies at the onset of the successful training range, while performance degrades for $\gamma>1$ and for sufficiently small $\gamma$ [ 9 ] .

Figure 8: Mean-field Widom-like line and the edge of chaos. The edge of chaos marks parameters where the cross-input correlation depth diverges— with light and dark blue indicating the ordered and chaotic regions, respectively. In contrast, the Widom-like line locates the maximum $\sigma_{w}$ -susceptibility where sensitivity to inputs is largest— the size of the dots encode the height of maximum susceptibility. The two curves meet only at the $(\sigma_{w}^{2},\sigma_{b}^{2})=(1,0)$ . Importantly, all points on the edge of chaos correspond to the divergence of cross-input correlation depth. However, as $\sigma_{b}^{2}$ grows, the susceptibility exhibits weaker peaks until the Widom-like line completely vanishes— well aligned with the learning performance (See Text).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Training dynamics and initialization-dependent trainability on CIFAR-10. (a) Training loss and (b) training accuracy across $100$ epochs for networks initialized with fixed bias variance $\sigma_{b}^{2}=10^{-4}$ and different weight variances $\sigma_{w}^{2}$ , indicated by color. Only a restricted range of $\sigma_{w}^{2}$ produces sustained optimization and increasing accuracy, whereas initialization outside this region leads to poor trainability. (c) Training accuracy after $10$ epochs as a function of $\sigma_{w}^{2}$ for different bias variances $\sigma_{b}^{2}$ . Increasing $\sigma_{b}^{2}$ shifts and progressively narrows the range of trainable weight variances.

Figure 10: Training accuracy for networks of increasing depth. Contour plots show the training accuracy obtained after 10 epochs on the MNIST dataset for fully connected deep neural networks with hidden dimension

N=300

and depths of 200, 300, 400, and 500 layers. Each point corresponds to the average accuracy over 200 mini-batches (128 samples each) for a given initialization pair

(\sigma_{w},\sigma_{b})

. The color scale represents the achieved accuracy. The squared bias variance

\sigma_{b}^{2}

is shown on a logarithmic scale and

\sigma_{w}^{2}

on a linear scale.

Phase Transitions in Neural Networks

See all 25 papers

Aug 8, 2026

Correlation flow governs learning at criticality

The initialization of deep neural networks determines whether information and gradients can propagate across depth, yet a unified theory connecting these properties to learning dynamics remains elusive. Combining mean-field theory and random matrix theory, we establish a direct link between correlation propagation and the Neural Tangent Kernel (NTK) that governs learning in the sequential limit of infinitely wide, infinitely deep networks. Correlation propagation to infinite depth is possible only at a single, critical point in the weight-bias variance plane. At this point, we leverage the algebraic decay of the end-to-end Jacobian with depth to prove that the NTK becomes exactly proportional to the output correlation at infinite depth, tying together information propagation and learning dynamics. We further show that orthogonal initialization suppresses the leading finite-size corrections present under Gaussian initialization, clarifying the respective roles of the two initialization ensembles in this limit. These theoretical predictions are validated quantitatively on finite-width, finite-depth networks. Together, these results demonstrate that orthogonal initialization and criticality are required to control the asymptotic dynamics of deep learning.

Andrea Combette, Nelly Pustelnik, Antoine Venaille

CNRS, ENS de Lyon, Laboratoire de Physique UMR5672, Lyon, France

Neural Network Initialization Neural Network Training Dynamics Neural Tangent Kernel

Ask PDF

Jun 18, 2026

Statistical Properties of Training & Generalization

Deep learning has managed to evade numerous intuitions from classical statistics to achieve unprecedented performance on a number of real-world tasks. In this article, we investigate the key features and surprises of deep learning from a physics-informed perspective, taking care to point out and justify where possible the many choices inherent in constructing a deep learning model. In particular, we review the phenomenon of neural scaling laws and discuss their interplay with the constraints and inductive biases which may be present when applying machine learning to problems in physics.

Itay Lavie, Noam Levi, Yonatan Kahn

John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA · Tel Aviv University, Tel Aviv, Israel · Department of Physics, University of Toronto and Vector Institute, Toronto, ON, Canada

Neural Network Generalization Scientific ML Neural Network Training Dynamics Neural Scaling Laws

Ask PDF

Mar 12, 2026

Pruning-induced phases in fully-connected neural networks: the eumentia, the dementia, and the amentia

Modern neural networks are heavily overparameterized, and pruning, which removes redundant neurons or connections, has emerged as a key approach to compressing them without sacrificing performance. However, while practical pruning methods are well developed, whether pruning induces sharp phase transitions in the neural networks and, if so, to what universality class they belong, remain open questions. To address this, we study fully-connected neural networks trained on MNIST, independently varying the dropout (i.e., removing neurons) rate at both the training and evaluation stages to map the phase diagram. We identify three distinct phases: eumentia (the network learns), dementia (the network has forgotten), and amentia (the network cannot learn), sharply distinguished by the power-law scaling of the cross-entropy loss with the training dataset size. {In the eumentia phase, the algebraic decay of the loss, as documented in the machine learning literature as neural scaling laws, is from the perspective of statistical mechanics the hallmark of quasi-long-range order.} We demonstrate that the transition between the eumentia and dementia phases is accompanied by scale invariance, with a diverging length scale that exhibits hallmarks of a Berezinskii-Kosterlitz-Thouless-like transition; the phase structure is robust across different network widths and depths. Our results establish that dropout-induced pruning provides a concrete setting in which neural network behavior can be understood through the lens of statistical mechanics.

Haining Pan, Nakul Aggarwal, J. H. Pixley

Department of Physics and Astronomy, Center for Materials Theory, Rutgers University, Piscataway, NJ 08854 USA · Center for Computational Quantum Physics, Flatiron Institute, New York, New York 10010, USA

Phase Transitions in Neural Networks Neural Network Pruning Statistical Physics of Learning Neural Scaling Laws

Ask PDF

Abstract

Phase Transitions in Neural Networks

Explore similar work

Correlation flow governs learning at criticality

Statistical Properties of Training & Generalization

Pruning-induced phases in fully-connected neural networks: the eumentia, the dementia, and the amentia