When Sparsity Breaks Connectivity: Functional Reachability and Guided Topology Dynamics in Extreme Sparse Training

Mohammad Pasande; Mateusz Jaworski; Jeroen Rook; Elena Mocanu

cs.LGOct 12, 2026

When Sparsity Breaks Connectivity: Functional Reachability and Guided Topology Dynamics in Extreme Sparse Training

Authors: Mohammad Pasande, Mateusz Jaworski, Jeroen Rook, Elena Mocanu

Organizations: University of Twente, Enschede, Netherlands · Paderborn University, Paderborn, Germany

Abstract

Dynamic Sparse Training (DST) periodically prunes and regrows connections, inducing a discrete dynamical mechanism over network topologies. At extreme sparsity ( $\ge 99\%$ ), this dynamic tends to destroy input-output information pathways, and performance with standard magnitude/random updates degrades sharply. We formalize forward, backward, and functional reachability on layered sparse graphs, derive closed-form expectations for reachable neurons and edges, and show that Monte Carlo realizations match the theory. Guided by this reachability-based structure, we bias pruning toward non-functional edges and regrowth toward functionally consistent candidates. We introduce this reachability-driven modification of topology evolution as Reachability-guided in DST, referred to as guided DST. On CIFAR-10/100 and Tiny ImageNet, guided DST recovers large accuracy gaps versus unguided DST (up to $+14.4$ points at $99.9\%$ sparsity on CIFAR-100), and on MLP-Mixer guided DST remains trainable at $99.99\%$ sparsity while unguided random regrowth degrades sharply. These results cast extreme-sparse training as a connectivity-preserving dynamical process. Our implementation is publicly available at https://github.com/mzjaworski/ea_dst.

Figures & tables

Figure 1: Functional edge reachability under an ER initialization for an extreme sparse MLP. The predicted (solid) vs. realizated (dashed) are near-perfect until

\sim\!99.6

–

99.7\%

sparsity. The full grids can be seen in Appendix D .

Figure 2: Empirical reachability and accuracy after training MLPs with DST on CIFAR-100 (final epoch, five seeds).

Initialization	Dataset	Unguided Mag./Rand.	Guided Mag./Rand.	$\Delta$
ER	CIFAR-10	$88.57{\pm}0.92$	$91.44{\pm}0.04$	$+2.9$
	CIFAR-100	$20.90{\pm}0.84$	$35.27{\pm}0.19$	$+14.4$
	Tiny-IN	$13.53{\pm}0.57$	$16.88{\pm}0.27$	$+3.3$
Uniform	CIFAR-10	$59.13{\pm}2.86$	$62.11{\pm}3.12$	$+3.0$
	CIFAR-100	$7.64{\pm}0.82$	$16.82{\pm}1.04$	$+9.2$
	Tiny-IN	$4.56{\pm}0.46$	$9.74{\pm}0.37$	$+5.2$

Table 1: Top-5 test accuracy (

\%

, mean

\pm

std over five seeds) of

4

-layer MLPs trained at

99.9\%

sparsity with unguided Magnitude/Random versus reachability-guided.

Sparsity	Dense	Mag./Grad.	Mag./Rand.	G-Mag./Rand.
$0\%$	$66.33{\pm}1.11$	–	–	–
$99.00\%$	–	$71.92{\pm}1.70$	$68.73{\pm}0.85$	$68.12{\pm}0.45$
$99.90\%$	–	$61.08{\pm}0.81$	$45.81{\pm}0.72$	$57.48{\pm}0.86$
$99.95\%$	–	$42.31{\pm}2.89$	$31.96{\pm}1.11$	${47.65{\pm}0.76}$
$99.99\%$	–	$13.28{\pm}3.59$	$4.22{\pm}1.21$	${13.56{\pm}1.70}$

Table 2: Top-3 test accuracy (

\%

, mean

\pm

std over five seeds) of an MLP-Mixer trained on CIFAR-100 under ER initialization: dense baseline, Magnitude/Gradient, Magnitude/Random, and reachability-guided Magnitude/Random.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: ER initialization: forward, backward, and functional edge reachability (rows: layers

0

–

3

; columns: forward / backward / functional). Predicted (solid) vs. Monte Carlo realized (dashed,

\pm

std).

Figure 4: Uniform initialization: same layout as Figure 3 . Functional reachability decays earlier and more gradually than under ER.

Figure 5: MLP CIFAR-100, ER init,

99.9\%

sparsity: epoch-wise functional edge fraction (solid) and test accuracy (dashed).

Figure 6: Per-layer functional edge fraction at

99.9\%

sparsity (ER, final epoch, five seeds).

Dataset		$99\%$	$99.5\%$	$99.75\%$	$99.875\%$	$99.9\%$
CIFAR-10	U	$57.91{\pm}0.40$	$55.53{\pm}0.26$	$50.99{\pm}0.28$	$45.15{\pm}0.26$	$42.95{\pm}1.15$
CIFAR-10	G	$57.83{\pm}0.35$	$55.56{\pm}0.14$	$52.44{\pm}0.18$	$50.32{\pm}0.33$	$49.02{\pm}0.06$
CIFAR-100	U	$24.83{\pm}0.18$	$21.26{\pm}0.12$	$17.00{\pm}0.10$	$10.53{\pm}0.52$	$8.24{\pm}0.34$
CIFAR-100	G	$24.93{\pm}0.27$	$21.73{\pm}0.27$	$18.46{\pm}0.28$	$15.89{\pm}0.34$	$14.72{\pm}0.20$
Tiny-IN	U	$10.44{\pm}0.17$	$9.02{\pm}0.18$	$7.27{\pm}0.32$	$5.39{\pm}0.17$	$4.61{\pm}0.13$
Tiny-IN	G	$10.40{\pm}0.23$	$9.19{\pm}0.23$	$7.83{\pm}0.13$	$6.53{\pm}0.16$	$6.01{\pm}0.21$

Appendix

Table 3: DST top-1 accuracy (

\%

) under ER initialization: Magnitude/Random (U) vs. reachability-guided (G).

Dataset		$99\%$	$99.5\%$	$99.75\%$	$99.875\%$	$99.9\%$
CIFAR-10	U	$54.79{\pm}0.38$	$48.50{\pm}3.53$	$30.78{\pm}3.68$	$16.55{\pm}5.09$	$16.08{\pm}1.85$
CIFAR-10	G	$57.43{\pm}0.35$	$55.35{\pm}0.34$	$40.14{\pm}3.72$	$25.93{\pm}5.71$	$22.99{\pm}0.96$
CIFAR-100	U	$22.38{\pm}0.46$	$15.13{\pm}0.64$	$7.83{\pm}0.43$	$4.15{\pm}0.28$	$3.17{\pm}0.73$
CIFAR-100	G	$23.22{\pm}0.13$	$19.98{\pm}0.43$	$14.48{\pm}0.60$	$9.55{\pm}0.22$	$8.03{\pm}0.42$
Tiny-IN	U	$8.17{\pm}0.23$	$6.46{\pm}0.28$	$3.66{\pm}0.63$	$1.94{\pm}0.17$	$1.77{\pm}0.31$
Tiny-IN	G	$8.39{\pm}0.13$	$7.53{\pm}0.27$	$6.05{\pm}0.31$	$4.37{\pm}0.22$	$3.67{\pm}0.21$

Appendix

Table 4: DST top-1 accuracy (

\%

) under uniform initialization: Magnitude/Random (U) vs. reachability-guided (G).

Sparsity / method	Acc@3 (%)	Loss	FLOPs	Params
$0\%$ Dense	$66.33{\pm}1.11$	$1.28{\pm}0.03$	$16300$	$21700$
$99.00\%$ Mag./Grad.	$71.92{\pm}1.70$	$2.39{\pm}0.09$	$18.7$	$216.8$
$99.00\%$ Mag./Rand.	$68.73{\pm}0.85$	$2.52{\pm}0.04$	$18.5$	$217.1$
$99.00\%$ G-Mag./Rand.	$68.12{\pm}0.45$	$2.66{\pm}0.03$	$18.6$	$217.1$
$99.90\%$ Mag./Grad.	$61.08{\pm}0.81$	$3.15{\pm}0.02$	$4.20$	$21.7$
$99.90\%$ Mag./Rand.	$45.81{\pm}0.72$	$3.60{\pm}0.01$	$4.02$	$21.7$

Appendix

Table 5: MLP-Mixer (CIFAR-100, ER): top-3 accuracy, training FLOPs (

\times 10^{12}

), and active parameters (

\times 10^{3}

Dynamic Sparse Training

See all 10 papers

SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Training
May 26, 2026Mohammed Adnan, Rohan Jain, Tom Jacobs +4Neural Network Optimization Dynamic Sparse Training
University of Calgary · 2Vector Institute · 3CISPA Helmholtz +2
Heterogeneous Connectivity in Sparse Networks: Fan-in Profiles, Gradient Hierarchy, and Topological Equilibria
Apr 12, 2026Nikodem TomczakDynamic Sparse Training Sparse Neural Networks
Thulge Labs, Singapore
DADP: Dynamic Activity-Dependent Pruning, A Reverse Hebbian-Inspired Structural Pruning Method
Oct 8, 2026Bhushan DeshpandeDynamic Sparse Training Neural Network Pruning
Independent Researcher, Pune, India

May 26, 2026J/K move · Enter open · S save

SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Training

Mohammed Adnan, Rohan Jain, Tom Jacobs, Ekansh Sharma+3

University of Calgary · 2Vector Institute · 3CISPA Helmholtz+2

Neural Network Optimization Dynamic Sparse Training Neural Network Training Dynamics

Open paper AskPDF arXiv

Dynamic Sparse Training (DST) methods train neural networks by maintaining sparsity while dynamically adapting the network topology. Despite the promise of reduced computation, DST methods converge significantly slower than dense training, often requiring comparable training time to achieve similar accuracy. We demonstrate both analytically and empirically that Batch Normalization (BN) adversely affects sparse training, and propose SparseOpt, a sparsity-aware optimizer, to address this. Experiments on ResNet models across CIFAR-100 and ImageNet demonstrate consistently faster convergence and improved generalization with our proposed method. Our work highlights the limitations of current normalization layers in sparse training and provides the first systematic study of the interaction between Batch Normalization, sparse layers, and DST, taking a significant step toward making DST practically competitive with dense training.

Abstract

Dynamic Sparse Training

Explore similar work

SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Training

Heterogeneous Connectivity in Sparse Networks: Fan-in Profiles, Gradient Hierarchy, and Topological Equilibria

DADP: Dynamic Activity-Dependent Pruning, A Reverse Hebbian-Inspired Structural Pruning Method