Heterogeneous Connectivity in Sparse Networks: Fan-in Profiles, Gradient Hierarchy, and Topological Equilibria

Nikodem Tomczak

cs.LGApr 12, 2026

Heterogeneous Connectivity in Sparse Networks: Fan-in Profiles, Gradient Hierarchy, and Topological Equilibria

Authors: Nikodem Tomczak

Organizations: Thulge Labs, Singapore

Abstract

Profiled Sparse Networks (PSN) replace uniform connectivity with deterministic, heterogeneous fan-in profiles defined by continuous, nonlinear functions, creating neurons with both dense and sparse receptive fields. We benchmark PSN across four classification datasets spanning vision and tabular domains, input dimensions from 54 to 784, and network depths of 2--3 hidden layers. At 90% sparsity, all static profiles, including the uniform random baseline, achieve accuracy within 0.2-0.6% of dense baselines on every dataset, demonstrating that heterogeneous connectivity provides no accuracy advantage when hub placement is arbitrary rather than task-aligned. This result holds across sparsity levels (80-99.9%), profile shapes (eight parametric families, lognormal, and power-law), and fan-in coefficients of variation from 0 to 2.5. Internal gradient analysis reveals that structured profiles create a 2-5x gradient concentration at hub neurons compared to the ~1x uniform distribution in random baselines, with the hierarchy strength predicted by fan-in coefficient of variation ( $r = 0.93$ ). When PSN fan-in distributions are used to initialise RigL dynamic sparse training, lognormal profiles matched to the equilibrium fan-in distribution consistently outperform standard ERK initialisation, with advantages growing on harder tasks, achieving +0.16% on Fashion-MNIST ( $p = 0.036$ , $d = 1.07$ ), +0.43% on EMNIST, and +0.49% on Forest Cover. RigL converges to a characteristic fan-in distribution regardless of initialisation. Starting at this equilibrium allows the optimiser to refine weights rather than rearrange topology. Which neurons become hubs matters more than the degree of connectivity variance, i.e., random hub placement provides no advantage, while optimisation-driven placement does.

Figures & tables

Dataset	Train	Test	Input dim	Classes
MNIST ( LeCun et al., 1998 )	60,000	10,000	784	10
Fashion-MNIST ( Xiao et al., 2017 )	60,000	10,000	784	10
EMNIST-Balanced ( Cohen et al., 2017 )	112,800	18,800	784	47
Forest Cover ( Blackard and Dean, 1999 )	464,809	116,203	54	7

Table 1: Datasets used in this study.

Dataset	Architecture	Batch size
MNIST	$784\!\to\!1024\!\to\!1024\!\to\!10$	128
Fashion-MNIST	$784\!\to\!1024\!\to\!1024\!\to\!10$	128
EMNIST-Balanced	$784\!\to\!1024\!\to\!1024\!\to\!47$	128
Forest Cover	$54\!\to\!1024\!\to\!1024\!\to\!1024\!\to\!7$	256

Table 2: Network architectures and training configurations.

Figure 1: PSN methods overview at 90% sparsity for a 784

\to

1024 layer. (a) Eight parametric fan-in profiles. The exponential profile assigns hub neurons

>

250 connections while the quadratic profile reaches

\sim

120. Inverse profiles mirror the spatial arrangement of hubs and specialists. The dashed line marks the random baseline (uniform fan-in 78). (b) Lognormal profiles at five target CCVs (0.5–2.5). Fan-in values are sampled from lognormal distributions, sorted in descending order, and assigned to neuron indices. Higher CCV produces a heavier tail. (c) Binary connectivity mask for the exponential profile with random spreading. Each row corresponds to one output neuron, each column to one input. Connection density decreases from top (high fan-in) to bottom (low fan-in). (d) Multi-peak profile interpolation at

\beta=50

. Increasing peak count

k

from 1 to 50 smoothly transitions from extreme heterogeneity (CCV = 1.23) to near-uniformity (CCV = 0.00). All profiles share the same global sparsity and mean fan-in.

Min fan-in	784-input datasets	Forest Cover ( $M_{\text{in}}\!=\!54$ )
1	99.87%	98.15%
5	99.36%	90.74%
10	98.72%	81.48%
20	97.45%	62.96%

Table 3: Maximum achievable sparsity (

S_{\max}

) as a function of minimum fan-in and input dimension.

Dataset	80%	90%	95%	98%
MNIST	1.62	2.43	3.48	5.49
Fashion-MNIST	1.68	2.52	3.61	5.71
EMNIST-Balanced	1.68	2.53	3.64	5.69
Forest Cover	1.26	2.10	3.17	5.13

Table 4: Equilibrium fan-in CCV reached by RigL (default hyperparameters, 5 seeds). MNIST, Fashion-MNIST, and EMNIST share the same 784

\to

1024

\to

1024 architecture and converge to nearly identical CCVs. Forest Cover is lower at all sparsities due to the 54-input FC1 bottleneck.

Sparsity	Quadratic	Exponential	Linear	Bell	Random
85%	97.77	97.69	97.79	97.76	97.87
88%	97.88	97.67	97.73	97.78	97.86
90%	97.79	97.78	97.77	97.76	97.84
92%	97.75	97.76	97.76	97.69	97.78
94%	97.61	97.55	97.60	97.55	97.74
96%	97.47	97.47	97.48	97.42	97.38

Table 5: Test accuracy (%) for even-spread PSN profiles and random baseline across sparsity levels. Values are means over 5 seeds. Dense baseline: 97.81%

\pm

0.18%.

Figure 2: Static connectivity structure does not affect accuracy. (a) MNIST accuracy versus sparsity for four PSN profiles and the random baseline (5 seeds each). All profiles cluster within 0.2% of each other up to 98% sparsity. The steep decline at 99.9% reflects capacity exhaustion (mean fan-in collapses to

\sim

1), not profile-specific effects. (b) Multi-peak interpolation at 90% sparsity on MNIST. Peak count

k

varies fan-in CCV continuously from 3.2 (

k=1

) to 0 (

k=50

) for three sharpness values

\beta

. Accuracy is flat across the entire range (total spread 0.65%). The only visible deviation occurs at

\beta=1200

k=1

–

2

, where extreme concentration of fan-in into

\sim

100 densely connected neurons produces a 0.7% drop. (c) Accuracy relative to dense baseline at 90% sparsity across all four datasets. Each marker represents one profile (9 profiles per dataset, 5 seeds). All profiles fall within

\pm

0.5% of the dense baseline except lognormal CV = 2.5 on Forest Cover, where the 54-dimensional input limits useful heterogeneity.

Profile (CV)	MNIST	Fashion	EMNIST	Forest
Dense	98.16	89.03	85.97	95.60
Random ( $\approx$ 0)	98.07	89.20	84.68	95.32
Quadratic ( $\approx$ 0.25)	98.12	89.20	84.74	95.25
Bell ( $\approx$ 0.66)	98.11	89.32	84.64	95.25
Exponential ( $\approx$ 0.82)	98.14	89.33	84.70	95.24
Lognormal (1.0)	98.19	89.15	84.71	95.21

Table 6: Test accuracy (%) at 90% sparsity for static PSN profiles across four datasets. Values are means over 5 seeds. Profiles are listed in order of increasing fan-in CV. “Random” denotes uniform random connectivity (CV

\approx

0).

Figure 3: RigL accuracy versus sparsity by initialisation strategy across four datasets (5 seeds each). Six initialisation strategies are compared at each sparsity level. LN (equil.) denotes lognormal initialisation matched to the measured equilibrium fan-in CCV for each (dataset, sparsity) combination. LN (equil. $\pm$ 0.5) offsets the target CCV by 0.5 above or below equilibrium. (a) On MNIST, all strategies perform within 0.5% of each other. (b) On Fashion-MNIST, LN (equil.) maintains a consistent edge over ERK and uniform across all sparsity levels. (c) On EMNIST-Balanced, separation between strategies increases at higher sparsity. (d) On Forest Cover, ERK collapses at 95–98% sparsity due to the 54-input bottleneck in FC1. Static random (dashed gray) outperforms ERK at all sparsity levels on this dataset.

Dynamic Sparse Training

See all 10 papers

Oct 12, 2026

When Sparsity Breaks Connectivity: Functional Reachability and Guided Topology Dynamics in Extreme Sparse Training

Dynamic Sparse Training (DST) periodically prunes and regrows connections, inducing a discrete dynamical mechanism over network topologies. At extreme sparsity (

\ge 99\%

), this dynamic tends to destroy input-output information pathways, and performance with standard magnitude/random updates degrades sharply. We formalize forward, backward, and functional reachability on layered sparse graphs, derive closed-form expectations for reachable neurons and edges, and show that Monte Carlo realizations match the theory. Guided by this reachability-based structure, we bias pruning toward non-functional edges and regrowth toward functionally consistent candidates. We introduce this reachability-driven modification of topology evolution as Reachability-guided in DST, referred to as guided DST. On CIFAR-10/100 and Tiny ImageNet, guided DST recovers large accuracy gaps versus unguided DST (up to

+14.4

points at

99.9\%

sparsity on CIFAR-100), and on MLP-Mixer guided DST remains trainable at

99.99\%

sparsity while unguided random regrowth degrades sharply. These results cast extreme-sparse training as a connectivity-preserving dynamical process. Our implementation is publicly available at https://github.com/mzjaworski/ea_dst.

Mohammad Pasande, Mateusz Jaworski, Jeroen Rook +1

University of Twente, Enschede, Netherlands · Paderborn University, Paderborn, Germany

Dynamic Sparse Training Sparse Neural Networks Neural Network Pruning Neural Network Training Dynamics

Ask PDF

Mar 5, 2026

Does Sparse Connectivity Improve Generalization? Convolutional Networks Below the Edge of Stability

Gradient descent on overparameterized neural networks typically operates at the Edge of Stability (EoS), where the largest Hessian eigenvalue hovers around a step-size-dependent threshold. We study how sparse connectivity changes generalization below this threshold in two-layer ReLU networks. Prior results have shown that for fully-connected networks (FCNs), generalization guarantees in this regime degrade and become vacuous on high-dimensional spherical inputs. Our analysis reveals that sparse connectivity fundamentally alters this picture. Under sparse connectivity, the network processes a collection of low-dimensional patches rather than the full input vector, so the effective constraint imposed by the stability condition is governed by the geometry of the training patch collection. We prove that when the receptive fields are small relative to the ambient dimension, the effective constraint yields non-vacuous generalization bounds in precisely the spherical regime where FCNs provably fail. The same framework also reveals a contrasting failure mode: if the patch collection lacks geometric structure, the constraint becomes unable to prevent overfitting. We corroborate this theory by analyzing the patch geometry of natural images, showing that standard convolutional designs produce patch multiset with low-dimensional structure that facilitates generalization. This provides a principled explanation for the generalization advantage of convolutional networks. Thus, our analysis yields a unified framework that identifies how architecture, data geometry, and gradient descent jointly govern generalization performance.

Tongtong Liang, Esha Singh, Rahul Parhi +2

UC San Diego

Neural Network Generalization Generalization Bounds Convolutional Neural Networks Neural Network Training Dynamics Sparse Neural Networks Edge of Stability

Ask PDF

Jun 10, 2026

Finding Sparse Subnetworks in One Training Cycle via Progressive Magnitude-Based Pruning

Neural network pruning reduces model size by removing less important parameters while aiming to preserve predictive performance. Although the Lottery Ticket Hypothesis (LTH) shows that sparse subnetworks can match dense networks when trained from suitable initializations, its iterative pruning procedure requires multiple complete training cycles. This work evaluates progressive magnitude-based pruning as a single-cycle alternative. The method gradually increases sparsity during training using a linear schedule and updates pruning masks based on active weight magnitudes. We conduct systematic experiments on CIFAR-10 and MNIST across ResNet, VGG-style, and LeNet architectures, comparing the proposed method with representative iterative and initialization-based pruning baselines, including LTH, SNIP, and GraSP. On CIFAR-10, the method achieves 95.12% accuracy on ResNet-18 at 72.9% sparsity, compared with 90.5% reported for LTH. At extreme sparsity, it achieves 93.13% accuracy on a VGG-like architecture at 97% sparsity, compared with approximately 92.0% for SNIP, and 93.44% accuracy on VGG-19 at 97.97% sparsity, compared with 92.19% for GraSP at 98% sparsity. A sparsity-accuracy analysis on ResNet-18 further shows that accuracy remains within 0.1 percentage points of the dense baseline across 70--85% sparsity. These results indicate that progressive magnitude-based pruning provides an effective single-cycle approach for neural network sparsification under the evaluated settings.

Romana Qureshi, Hafida Benhidour, Said Kerrache +1

Department of Computer Science, King Saud University, Riyadh, Saudi Arabia

Sparse Neural Networks Neural Network Pruning

Ask PDF

Abstract

Dynamic Sparse Training

Explore similar work

When Sparsity Breaks Connectivity: Functional Reachability and Guided Topology Dynamics in Extreme Sparse Training

Does Sparse Connectivity Improve Generalization? Convolutional Networks Below the Edge of Stability

Finding Sparse Subnetworks in One Training Cycle via Progressive Magnitude-Based Pruning