Pruning-induced phases in fully-connected neural networks: the eumentia, the dementia, and the amentia

Haining Pan; Nakul Aggarwal; J. H. Pixley

cond-mat.dis-nnMar 12, 2026

Pruning-induced phases in fully-connected neural networks: the eumentia, the dementia, and the amentia

Authors: Haining Pan, Nakul Aggarwal, J. H. Pixley

Organizations: Department of Physics and Astronomy, Center for Materials Theory, Rutgers University, Piscataway, NJ 08854 USA · Center for Computational Quantum Physics, Flatiron Institute, New York, New York 10010, USA

Abstract

Modern neural networks are heavily overparameterized, and pruning, which removes redundant neurons or connections, has emerged as a key approach to compressing them without sacrificing performance. However, while practical pruning methods are well developed, whether pruning induces sharp phase transitions in the neural networks and, if so, to what universality class they belong, remain open questions. To address this, we study fully-connected neural networks trained on MNIST, independently varying the dropout (i.e., removing neurons) rate at both the training and evaluation stages to map the phase diagram. We identify three distinct phases: eumentia (the network learns), dementia (the network has forgotten), and amentia (the network cannot learn), sharply distinguished by the power-law scaling of the cross-entropy loss with the training dataset size. {In the eumentia phase, the algebraic decay of the loss, as documented in the machine learning literature as neural scaling laws, is from the perspective of statistical mechanics the hallmark of quasi-long-range order.} We demonstrate that the transition between the eumentia and dementia phases is accompanied by scale invariance, with a diverging length scale that exhibits hallmarks of a Berezinskii-Kosterlitz-Thouless-like transition; the phase structure is robust across different network widths and depths. Our results establish that dropout-induced pruning provides a concrete setting in which neural network behavior can be understood through the lens of statistical mechanics.

Figures & tables

Figure 1: (a) Schematic of the fully-connected neural network. A $28\times 28$ gray scale image from MNIST dataset is first flattened into a $784$ -dimensional input vector, and then passed through a rectangular architecture of depth $L$ hidden layers and width $n_{h}$ , where each hidden layer contains $n_{h}$ neurons. During training, the weights and biases are optimized on the training set by minimizing the cross-entropy loss using the Adam optimizer. During evaluation, these optimized parameters are frozen, and the network is evaluated to compute performance metrics including the classification accuracy $\langle\mathcal{A}\rangle$ and cross-entropy loss $\langle\mathcal{L}_{\text{CE}}\rangle$ on test dataset. The average values are obtained over $20$ independent training runs with different initializations and mini-batch samplings. (See Sec. II for details.) (b) Schematic of the neuron-dropout model. The network architecture is identical to that in panel (a), except neurons in the hidden layers are randomly masked by setting their post-activation outputs to zero. The neurons are dropped independently with dropout rates $p_{\mathrm{train}}$ and $p_{\mathrm{eval}}$ during training and evaluation stages respectively. For each fixed $p_{\text{eval}}$ , we sample $100$ independent dropout masks and estimate the mean accuracy.

Figure 2: (a) Phase diagram as a function of the training dropout rate $p_{\text{train}}$ and evaluation dropout rate $p_{\text{eval}}$ . The three phases eumentia, dementia, and amentia are characterized by the exponent of power-law decay in the cross-entropy loss as shown in panel (c); (b) Averaged accuracy $\expectationvalue{\mathcal{A}}$ for the training dataset size $N=60000$ ; (c) Exponent of cross-entropy loss $\alpha_{\text{CE}}$ defined in Eq. ( 4 ) for the training dataset size $N=60000$ , where the sign implies the three phases.

Figure 3: (a) Linecuts of the cross-entropy loss at a fixed evaluation dropout rate

p_{\text{eval}}=0.7

as a function of the training dropout rate

p_{\text{train}}

for different training dataset size

N

. The vertical dashed line indicates the phase boundaries. (b-d) Power-law scaling of the averaged cross-entropy loss

\expectationvalue{\mathcal{L}_{\text{CE}}}

as a function of the training dataset size

N

at different training dropout rates

p_{\text{train}}

in the dementia, eumentia, and amentia phases, respectively. The solid lines are power-law fits to the data in the log-log scale following Eq. ( 4 ).

Figure 4: (a) Averaged cross-entropy loss as a function of evaluation dropout rate

p_{\text{eval}}

p_{\text{train}}=0.1

for different training dataset size

N

. Inset: the data collapse of the averaged cross-entropy loss

\expectationvalue{\mathcal{L}_{\text{CE}}}

with the fitted value of

p_{\text{eval},c}=0.60(2)

and

\sigma=0.8(1)

. (b) Goodness of the data collapse

\chi_{\nu}^{2}

as a function of the fitting parameter

\sigma

and

p_{\text{eval},c}

, where

\chi_{\nu,\min}^{2}

is marked by the cross with value of 26.6, and the contour shows

1.3\chi_{\nu,\min}^{2}

. (c) Left axis: the fitted exponent

\sigma

from the BKT-like transition ansatz in Eq. ( 6 ) as a function of the training dropout rate

p_{\text{train}}

; right axis: the corresponding goodness of the data collapse

\chi_{\nu}^{2}

Figure 5: (a) Averaged accuracy

\expectationvalue{\mathcal{A}}

at a fixed training dropout rate

p_{\text{train}}=0.1

as a function of the evaluation dropout rate

p_{\text{eval}}

for different training dataset size

N

. The vertical dashed line marks the critical point

p_{\text{eval},c}

obtained from Fig. 4 for all the panels. The horizontal dashed line marks the accuracy from random guessing. (b) Binder cumulant (see Eq. 7 for the definition) of the same data in panel (a). The horizontal dashed line marks the expected value for a Gaussian variable.

Figure 6: Exponent of cross-entropy loss $\alpha_{\text{CE}}$ for different width $n_{h}$ and depth $L$ for training size $N=60000$ . (a-c) width $n_{h}=50$ and depth $L=2,3,4$ ; (d-f) depth $L=5$ and width $n_{h}=100,150,200$ .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: BKT-like finite-size scaling analysis at

p_{\text{train}}=0.2

. (a) Averaged cross-entropy loss versus

p_{\text{eval}}

for different training dataset size

N

. Inset: data collapse with

p_{\text{eval},c}=0.73(4)

and

\sigma=1.1(4)

, where

\overline{p}_{\text{eval}}\equiv(p_{\text{eval}}-p_{\text{eval},c})[\log(N/N_0)]^{1/\sigma}

is the scaling variable in Eq. ( 6 ). (b)

\chi_{\nu}^{2}(\sigma,p_{\text{eval},c})

landscape with

\chi_{\nu,\min}^{2}=115

(cross) and

1.3\,\chi_{\nu,\min}^{2}

contour.

Figure 8: BKT-like finite-size scaling analysis at

p_{\text{train}}=0.3

. (a) Averaged cross-entropy loss versus

p_{\text{eval}}

for different training dataset size

N

. Inset: data collapse with

p_{\text{eval},c}=0.77(4)

and

\sigma=0.9(3)

. (b)

\chi_{\nu}^{2}(\sigma,p_{\text{eval},c})

landscape with

\chi_{\nu,\min}^{2}=184

(cross) and

1.3\,\chi_{\nu,\min}^{2}

contour.

Figure 9: BKT-like finite-size scaling analysis at

p_{\text{train}}=0.4

. (a) Averaged cross-entropy loss versus

p_{\text{eval}}

for different training dataset size

N

. Inset: data collapse with

p_{\text{eval},c}=0.81(4)

and

\sigma=1.0(3)

. (b)

\chi_{\nu}^{2}(\sigma,p_{\text{eval},c})

landscape with

\chi_{\nu,\min}^{2}=106

(cross) and

1.3\,\chi_{\nu,\min}^{2}

contour.

Figure 10: BKT-like finite-size scaling analysis at

p_{\text{train}}=0.5

. (a) Averaged cross-entropy loss versus

p_{\text{eval}}

for different training dataset size

N

. Inset: data collapse with

p_{\text{eval},c}=0.83(3)

and

\sigma=0.8(1)

. (b)

\chi_{\nu}^{2}(\sigma,p_{\text{eval},c})

landscape with

\chi_{\nu,\min}^{2}=188

(cross) and

1.3\,\chi_{\nu,\min}^{2}

contour.

Figure 11: Conventional algebraic finite-size scaling using Eq. ( 19 ) at

p_{\text{train}}=0.1

. The fitting parameters are

\nu

and

p_{\text{eval},c}

. (a) Averaged cross-entropy loss versus

p_{\text{eval}}

for different training dataset size

N

. Inset: data collapse with

p_{\text{eval},c}=0.61(2)

and

\nu=8.1(7)

, where

\overline{p}_{\text{eval}}\equiv(p_{\text{eval}}-p_{\text{eval},c})\,N^{1/\nu}

is the scaling variable in Eq. ( 19 ). (b)

\chi_{\nu}^{2}(\nu,p_{\text{eval},c})

landscape with

\chi_{\nu,\min}^{2}=27.4

(cross) and

1.3\,\chi_{\nu,\min}^{2}

contour.

Figure 12: Conventional algebraic finite-size scaling using Eq. ( 19 ) at

p_{\text{train}}=0.2

. (a) Averaged cross-entropy loss versus

p_{\text{eval}}

for different training dataset size

N

. Inset: data collapse with

p_{\text{eval},c}=0.73(4)

and

\nu=11(3)

. (b)

\chi_{\nu}^{2}(\nu,p_{\text{eval},c})

landscape with

\chi_{\nu,\min}^{2}=114

(cross) and

1.3\,\chi_{\nu,\min}^{2}

contour.

Figure 13: Conventional algebraic finite-size scaling using Eq. ( 19 ) at

p_{\text{train}}=0.3

. (a) Averaged cross-entropy loss versus

p_{\text{eval}}

for different training dataset size

N

. Inset: data collapse with

p_{\text{eval},c}=0.77(4)

and

\nu=9(2)

. (b)

\chi_{\nu}^{2}(\nu,p_{\text{eval},c})

landscape with

\chi_{\nu,\min}^{2}=185

(cross) and

1.3\,\chi_{\nu,\min}^{2}

contour.

Figure 14: Conventional algebraic finite-size scaling using Eq. ( 19 ) at

p_{\text{train}}=0.4

. (a) Averaged cross-entropy loss versus

p_{\text{eval}}

for different training dataset size

N

. Inset: data collapse with

p_{\text{eval},c}=0.81(4)

and

\nu=10(3)

. (b)

\chi_{\nu}^{2}(\nu,p_{\text{eval},c})

landscape with

\chi_{\nu,\min}^{2}=106

(cross) and

1.3\,\chi_{\nu,\min}^{2}

contour.

Figure 15: Conventional algebraic finite-size scaling using Eq. ( 19 ) at

p_{\text{train}}=0.5

. (a) Averaged cross-entropy loss versus

p_{\text{eval}}

for different training dataset size

N

. Inset: data collapse with

p_{\text{eval},c}=0.82(3)

and

\nu=8(1)

. (b)

\chi_{\nu}^{2}(\nu,p_{\text{eval},c})

landscape with

\chi_{\nu,\min}^{2}=181

(cross) and

1.3\,\chi_{\nu,\min}^{2}

contour.

Phase Transitions in Neural Networks

See all 25 papers

Dec 15, 2025

Dropout Neural Network Training Viewed from a Percolation Perspective

In this work, we investigate the existence and effect of percolation in training deep Neural Networks (NNs) with dropout. Dropout methods are regularisation techniques for training NNs, first introduced by G. Hinton et al. (2012). These methods temporarily remove connections in the NN, randomly at each stage of training, and update the remaining subnetwork with Stochastic Gradient Descent (SGD). The process of removing connections from a network at random is similar to percolation, a paradigm model of statistical physics. If dropout were to remove enough connections such that there is no path between the input and output of the NN, then the NN could not make predictions informed by the data. We study new percolation models that mimic dropout in NNs and characterise the relationship between network topology and this path problem. The theory shows the existence of a percolative effect in dropout. We also show that this percolative effect can cause a breakdown when training NNs without biases with dropout; and we argue heuristically that this breakdown extends to NNs with biases.

Finley Devlin, Jaron Sanders

Second Foundation, Czechia · Eindhoven University of Technology, The Netherlands

Neural Network Training Dynamics

Ask PDF

Sep 11, 2026

Theoretical Guarantees for One-Shot Magnitude Pruning and Compute-Adaptive Early Exit

We study compute reduction in neural networks through a unified partial versus full computation view, captured by one-shot magnitude pruning in the static regime and early exit in the adaptive regime. In an asymptotic single-neuron model, we prove a concentration theorem for one-shot magnitude pruning with explicit rates. We also introduce the conditional perceptron for early exit and show that its excess generalization error decays as a power of the compute gap, with an exponent that grows to infinity as the alignment between partial and full computations tends to one. We then extend the analysis to deep networks, characterizing how pruning-induced distortions accumulate with depth and deriving a corresponding compute-accuracy tradeoff for frozen-backbone early exit under a neural network Gaussian process model. Numerical simulations corroborate the predicted scaling laws.

Erdem Koyuncu

Department of Electrical and Computer Engineering, University of Illinois Chicago, IL, USA.

Neural Network Generalization Early-Exit Neural Networks Neural Network Pruning

Ask PDF

Sep 26, 2025

Toward a Physics of Deep Learning and Brains

Deep neural networks and brains both learn and share superficial similarities: processing nodes are likened to neurons and adjustable weights are likened to modifiable synapses. But can a unified theoretical framework be found to underlie them both? Here we show that the equations used to describe neuronal avalanches in living brains can also be applied to cascades of activity in deep neural networks. These equations are derived from non-equilibrium statistical physics and show that deep neural networks learn best when poised between absorbing and active phases. Because these networks are strongly driven by inputs, however, they do not operate at a true critical point but within a quasi-critical regime -- one that still approximately satisfies crackling noise scaling relations. By training networks with different initializations, we show that maximal susceptibility is a more reliable predictor of learning than proximity to the critical point itself. This provides a blueprint for engineering improved network performance. Finally, using finite-size scaling we identify distinct universality classes, including Barkhausen noise and directed percolation. This theoretical framework demonstrates that universal features are shared by both biological and artificial neural networks.

Arsham Ghavasieh, Meritxell Vila-Minana, Akanksha Khurd +3

Center for Complex Networks and Systems Research, Luddy School of Informatics, Computing, and Engineering, Indiana University, Bloomington, Indiana 47408, USA · Department of Physics, Indiana University, Bloomington, Indiana 47405, USA · Institute for Advanced Study, Princeton, NJ 08540, USA

Phase Transitions in Neural Networks Statistical Physics of Learning Neural Network Training Dynamics

Ask PDF

Abstract

Phase Transitions in Neural Networks

Explore similar work

Dropout Neural Network Training Viewed from a Percolation Perspective

Theoretical Guarantees for One-Shot Magnitude Pruning and Compute-Adaptive Early Exit

Toward a Physics of Deep Learning and Brains