DADP: Dynamic Activity-Dependent Pruning, A Reverse Hebbian-Inspired Structural Pruning Method

Bhushan Deshpande

cs.LGOct 8, 2026

DADP: Dynamic Activity-Dependent Pruning, A Reverse Hebbian-Inspired Structural Pruning Method

Authors: Bhushan Deshpande

Organizations: Independent Researcher, Pune, India

Abstract

Modern neural networks are heavily over-parameterized. This redundancy incurs substantial compute and memory overhead during training and inference. Existing pruning methods rely on post-hoc magnitude thresholds or static initialization heuristics. Consequently, they often require manual per-layer sparsity targets or expensive retraining cycles. We propose Dynamic Activity-Dependent Pruning (DADP), a biologically inspired structural plasticity mechanism. During training, DADP measures connection importance via the accumulated product of pre-synaptic activations and post-synaptic error gradients. Using a single global threshold instead of fixed layer budgets, DADP dynamically allocates sparsity across network depth while naturally inducing neuron- and channel-level pruning. Across MLP, VGG-16, ResNet-18, BiLSTM-CRF, and MiniBERT architectures, DADP matches or outperforms Magnitude, SNIP and RigL, retaining 73.67% accuracy (dense baseline: 76.06%) at 99% sparsity on ResNet-18. Finally, matrix-based Shannon entropy and effective rank measurements confirm that DADP preserves latent feature diversity at extreme sparsities without representation collapse.

Figures & tables

Architecture	Method	Sparsity (%)	Accuracy / F1 (%)	Acc. Change vs. Dense
MLP (MNIST)	Dense Baseline	0.00%	98.39%	–
	Magnitude	80.00%	98.63%	+0.24%
	SNIP	80.00%	97.85%	-0.54%
	RigL	80.00%	97.97%	-0.42%
	DADP ( $\tau=1\text{e-}6$ )	84.90 $\pm$ 0.56%	97.91 $\pm$ 0.12%	-0.48 $\pm$ 0.12%
VGG-16 (CIFAR-10)	Dense Baseline	0.00%	85.21%	–

Table 1: Performance comparison of DADP against baseline pruning methods at different target sparsities. DADP values report mean

\pm

standard deviation across 3 random seeds (

42,512,1729

) from statistical sweeps. Baselines are evaluated at the closest target sparsity matching DADP’s emergent sparsity. Complete results across all sparsities are in Appendix D .

Panel of Figure 1 — Figure 1: (a) Test accuracy versus sparsity for ResNet-18 on CIFAR-10. DADP maintains high accuracy up to $99.23\%$ global sparsity. (b) Self-regulating negative feedback dynamics across three pruning thresholds ( $\tau\in\{1\text{e-}6,1\text{e-}5,5\text{e-}5\}$ ) on MLP (MNIST, 100 epochs), illustrating training loss stabilization alongside decaying parameter pruning rates.

Panel of Figure 2 — Figure 2: Layer-wise sparsity allocation across diverse architectural paradigms. Layer indices on the x-axis ( $1,2,3,\dots,L$ ) correspond to sequential layer depths from input to output (full layer name mapping listed in Appendix Table 13 ). (a) Vision models (ResNet-18) preserve early visual feature extractors ( $0\%$ sparsity) while compressing deep representations. (b) Sequence & Language models (MiniBERT) exhibit inverse allocation, heavily pruning early token projections while preserving dense capacity in deep attention decision heads. VGG-16 layer-wise sparsity profile is provided in Appendix E .

Figure 3: Information-theoretic representation quality across layer depths for ResNet-18 on CIFAR-10. DADP maintains high normalized entropy ( $S_{\text{norm}}>0.85$ ) and matches or exceeds dense baseline effective rank, confirming that activity-dependent pruning eliminates correlated parameter noise without collapsing manifold geometry (VGG-16 plot in Appendix E ).

Figure 4: Global active weight value distributions across all five evaluation methods on ResNet-18 at $99\%$ sparsity displayed in a single row. Classical post-hoc Magnitude Pruning and dynamic regrowth methods (RigL) introduce sharp bimodal exclusion gaps around zero, whereas DADP retains a smooth, continuous Gaussian spectrum.

	ResNet-18 (CIFAR-10)		VGG-16 (CIFAR-10)
Initialization Scheme	Sparsity (%)	Test Acc (%)	Sparsity (%)	Test Acc (%)
Kaiming Normal	$95.41\%$	$75.10\%$	$84.86\%$	$84.80\%$
Kaiming Uniform	$95.35\%$	$76.08\%$	$85.03\%$	$85.30\%$
Xavier Normal	$95.66\%$	$77.05\%$	$86.58\%$	$84.84\%$
Xavier Uniform	$95.54\%$	$76.72\%$	$86.73\%$	$85.38\%$
Orthogonal	$95.75\%$	$76.51\%$	$86.61\%$	$85.63\%$

Table 2: Sensitivity ablation across weight initialization protocols under fixed pruning thresholds

\tau

. DADP achieves tight performance and sparsity invariance across all standard variance-preserving schemes.

	ResNet-18 (CIFAR-10)		VGG-16 (CIFAR-10)		MiniBERT (SST-2)
Prune Interval ( $\Delta t$ )	Sparsity (%)	Test Acc (%)	Sparsity (%)	Test Acc (%)	Sparsity (%)	Test Acc (%)
$\Delta t=100$	$92.11\pm 0.15\%$	$76.12\pm 0.40\%$	$93.30\pm 0.18\%$	$84.82\pm 0.20\%$	$97.11\pm 0.12\%$	$51.91\pm 1.41\%$
$\Delta t=250$	$91.64\pm 0.03\%$	$76.92\pm 0.52\%$	$91.58\pm 0.46\%$	$83.36\pm 0.63\%$	$96.58\pm 0.50\%$	$72.06\pm 13.41\%$
$\Delta t=500$ (Default)	$91.41\pm 0.08\%$	$76.76\pm 0.16\%$	$90.42\pm 0.93\%$	$84.20\pm 0.73\%$	$90.07\pm 2.99\%$	$80.35\pm 0.69\%$
$\Delta t=1000$	$91.13\pm 0.04\%$	$76.21\pm 0.40\%$	$89.62\pm 0.72\%$	$84.02\pm 2.11\%$	$85.67\pm 1.01\%$	$80.70\pm 0.19\%$
$\Delta t=1500$	$90.97\pm 0.05\%$	$76.84\pm 0.29\%$	$89.37\pm 0.80\%$	$84.53\pm 0.35\%$	$84.65\pm 0.73\%$	$80.85\pm 0.41\%$

Table 3: Ablation study on the effect of pruning interval (

\Delta t

) across architectures (mean

\pm

std across 3 random seeds). Vision architectures exhibit strong stability across all frequencies, whereas transformer models require moderate intervals (

\Delta t\geq 500

) to prevent rapid capacity collapse.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	MLP	VGG-16	ResNet-18	BiLSTM-CRF	MiniBERT
Dataset	MNIST	CIFAR-10	CIFAR-10	CoNLL-2003	SST-2
Task Modality	Vision (Digits)	Vision (Objects)	Vision (Objects)	Sequence (NER)	NLP (Sentiment)
Total Parameters	667,146	14,728,266	11,173,962	1,514,841	4,386,178
Optimizer	Adam	Adam	Adam	Adam	Adam
Learning Rate ( $\eta$ )	$1\text{e-}3$	$1\text{e-}3$	$1\text{e-}3$	$1\text{e-}3$	$2\text{e-}5$
Weight Decay	$1\text{e-}4$	$1\text{e-}4$	$1\text{e-}4$	$1\text{e-}4$	$1\text{e-}2$

Appendix

Table 4: Hyperparameter configurations and training specifications across all evaluated architectures.

Method	Configuration / Threshold	Final Sparsity (%)	Final Test Acc (%)	Peak Test Acc (%)
Dense Baseline	Unpruned (0%)	0.00%	98.39%	98.44%
DADP (Ours)	$\tau=1\text{e-}6$	84.90 $\pm$ 0.56%	97.91 $\pm$ 0.12%	98.18 $\pm$ 0.02%
DADP (Ours)	$\tau=5\text{e-}6$	94.60 $\pm$ 0.14%	97.55 $\pm$ 0.09%	98.02 $\pm$ 0.03%
DADP (Ours)	$\tau=1\text{e-}5$	96.19 $\pm$ 0.11%	97.38 $\pm$ 0.15%	97.91 $\pm$ 0.08%
DADP (Ours)	$\tau=5\text{e-}5$	98.35 $\pm$ 0.04%	96.69 $\pm$ 0.04%	96.92 $\pm$ 0.05%
DADP (Ours)	$\tau=1\text{e-}5$ (100 Epoch Limit Test)	98.00%	96.94%	97.69%

Appendix

Table 5: Complete empirical evaluation for MLP on MNIST across pruning methods and thresholds (DADP reports mean

\pm

std across 3 random seeds).

Method	Configuration / Threshold	Final Sparsity (%)	Final Test Acc (%)	Peak Test Acc (%)
Dense Baseline	Unpruned (0%)	0.00%	76.06%	77.44%
DADP (Ours)	$\tau=1\text{e-}6$	84.65 $\pm$ 0.21%	76.72 $\pm$ 0.26%	76.97 $\pm$ 0.25%
DADP (Ours)	$\tau=5\text{e-}6$	89.66 $\pm$ 0.10%	76.64 $\pm$ 0.75%	77.36 $\pm$ 0.17%
DADP (Ours)	$\tau=1\text{e-}5$	91.41 $\pm$ 0.08%	76.76 $\pm$ 0.16%	77.25 $\pm$ 0.24%
DADP (Ours)	$\tau=5\text{e-}5$	95.63 $\pm$ 0.05%	76.15 $\pm$ 0.34%	77.08 $\pm$ 0.16%
DADP (Ours)	$\tau=1\text{e-}4$	96.92 $\pm$ 0.01%	76.03 $\pm$ 0.61%	76.80 $\pm$ 0.21%

Appendix

Table 6: Complete empirical evaluation for ResNet-18 on CIFAR-10 across pruning methods and thresholds (DADP reports mean

\pm

std across 3 random seeds).

Method	Configuration / Threshold	Final Sparsity (%)	Final Test F1 (%)	Peak Test F1 (%)
Dense Baseline	Unpruned (0%)	0.00%	85.16%	85.16%
DADP (Ours)	$\tau=1\text{e-}6$	12.10 $\pm$ 3.16%	93.77 $\pm$ 0.12%	93.94 $\pm$ 0.13%
DADP (Ours)	$\tau=5\text{e-}6$	26.19 $\pm$ 3.80%	93.82 $\pm$ 0.06%	93.96 $\pm$ 0.15%
DADP (Ours)	$\tau=1\text{e-}5$	33.57 $\pm$ 4.06%	93.82 $\pm$ 0.14%	93.95 $\pm$ 0.13%
DADP (Ours)	$\tau=5\text{e-}5$	95.00 $\pm$ 3.60%	84.28 $\pm$ 1.34%	93.82 $\pm$ 0.23%
DADP (Ours)	$\tau=1\text{e-}4$	95.09 $\pm$ 0.98%	87.99 $\pm$ 1.73%	93.25 $\pm$ 0.05%

Appendix

Table 7: Complete empirical evaluation for BiLSTM-CRF on CoNLL-2003 (DADP reports mean

\pm

std across 3 random seeds).

Method	Configuration / Threshold	Final Sparsity (%)	Final Test Acc (%)	Peak Test Acc (%)
Dense Baseline	Unpruned (0%)	0.00%	80.62%	82.68%
DADP (Ours)	$\tau=1\text{e-}6$	71.52 $\pm$ 4.93%	79.01 $\pm$ 0.57%	82.19 $\pm$ 0.39%
DADP (Ours)	$\tau=2\text{e-}6$	91.69 $\pm$ 2.03%	78.67 $\pm$ 0.66%	81.57 $\pm$ 0.30%
DADP (Ours)	$\tau=3\text{e-}6$	96.03 $\pm$ 0.79%	80.85 $\pm$ 1.06%	82.11 $\pm$ 0.37%
DADP (Ours)	$\tau=4\text{e-}6$ (Over-threshold)	96.87 $\pm$ 0.16%	64.18 $\pm$ 12.70%	64.22 $\pm$ 12.76%
DADP (Ours)	$\tau=5\text{e-}6$ (Over-threshold)	96.96%	50.92%	50.92%

Appendix

Table 8: Complete empirical evaluation for MiniBERT on SST-2 across pruning methods and thresholds (DADP reports mean

\pm

std across 3 random seeds).

Method	Configuration / Threshold	Final Sparsity (%)	Final Test Acc (%)	Peak Test Acc (%)
Dense Baseline	Unpruned (0%)	0.00%	85.21%	85.21%
DADP (Ours)	$\tau=1\text{e-}6$	74.26 $\pm$ 1.25%	85.09 $\pm$ 0.17%	85.11 $\pm$ 0.16%
DADP (Ours)	$\tau=5\text{e-}6$	89.77 $\pm$ 0.08%	84.70 $\pm$ 0.17%	85.47 $\pm$ 0.34%
DADP (Ours)	$\tau=6\text{e-}6$	91.65 $\pm$ 0.67%	82.65 $\pm$ 0.65%	84.29 $\pm$ 0.53%
DADP (Ours)	$\tau=6.5\text{e-}6$	91.86 $\pm$ 0.54%	83.22 $\pm$ 0.00%	83.22 $\pm$ 0.00%
DADP (Ours)	$\tau=6.7\text{e-}6$	92.16 $\pm$ 0.28%	83.06 $\pm$ 2.01%	84.02 $\pm$ 1.36%

Appendix

Table 9: Complete empirical evaluation for VGG-16 on CIFAR-10 across pruning methods and thresholds (DADP reports mean

\pm

std across 3 random seeds).

Panel of Figure 5 — Figure 5: Comprehensive visual evaluation of VGG-16 on CIFAR-10. (a) DADP preserves classification accuracy up to $92.16\%$ global sparsity. (b) Layer-wise profile shows early spatial filter lock alongside heavy compression in deep linear projection layers ( classifier.0 ). (c) Latent representation quality across all 16 layers confirms that DADP maintains normalized Shannon entropy ( $S_{\text{norm}}\geq 0.85$ ) on par with dense unpruned features. (d) DADP preserves a smooth, zero-centered parameter distribution, avoiding artificial bimodal gaps around zero.

Layer Name	Original Shape	DADP Compressed Shape	SNIP Compressed Shape	Magnitude Compressed Shape	RigL Compressed Shape
features.0	$[64,3,3,3]$	$[64,3,3,3]$ (0 dead)	$[64,3,3,3]$ (0 dead)	$[64,3,3,3]$ (0 dead)	$[59,3,3,3]$ (5 dead)
features.3	$[64,64,3,3]$	$[64,64,3,3]$ (0 dead)	$[64,64,3,3]$ (0 dead)	$[64,64,3,3]$ (0 dead)	$[64,59,3,3]$ (0 dead)
features.7	$[128,64,3,3]$	$[128,64,3,3]$ (0 dead)	$[128,64,3,3]$ (0 dead)	$[128,64,3,3]$ (0 dead)	$[128,64,3,3]$ (0 dead)
features.10	$[128,128,3,3]$	$[128,128,3,3]$ (0 dead)	$[128,128,3,3]$ (0 dead)	$[128,128,3,3]$ (0 dead)	$[128,128,3,3]$ (0 dead)
features.14	$[256,128,3,3]$	$[256,128,3,3]$ (0 dead)	$[256,128,3,3]$ (0 dead)	$[256,128,3,3]$ (0 dead)	$[256,128,3,3]$ (0 dead)
features.17	$[256,256,3,3]$	$[256,256,3,3]$ (0 dead)	$[256,256,3,3]$ (0 dead)	$[256,256,3,3]$ (0 dead)	$[256,256,3,3]$ (0 dead)

Appendix

Table 10: Layer-by-layer physical structured compression shapes for VGG-16 on CIFAR-10 at

\sim 90\%

global sparsity across pruning methods, with the number of pruned (dead) channels/neurons in parentheses.

Layer Name	Original Shape	DADP Compressed Shape	SNIP Compressed Shape	Magnitude Compressed Shape	RigL Compressed Shape
conv1	$[64,3,3,3]$	$[64,3,3,3]$ (0 dead)	$[64,3,3,3]$ (0 dead)	$[64,3,3,3]$ (0 dead)	$[64,3,3,3]$ (0 dead)
layer1.0.conv1	$[64,64,3,3]$	$[64,64,3,3]$ (0 dead)	$[64,64,3,3]$ (0 dead)	$[64,64,3,3]$ (0 dead)	$[64,64,3,3]$ (0 dead)
layer1.0.conv2	$[64,64,3,3]$	$[64,64,3,3]$ (0 dead)	$[64,64,3,3]$ (0 dead)	$[64,64,3,3]$ (0 dead)	$[64,64,3,3]$ (0 dead)
layer1.1.conv1	$[64,64,3,3]$	$[64,64,3,3]$ (0 dead)	$[64,64,3,3]$ (0 dead)	$[64,64,3,3]$ (0 dead)	$[64,64,3,3]$ (0 dead)
layer1.1.conv2	$[64,64,3,3]$	$[64,64,3,3]$ (0 dead)	$[64,64,3,3]$ (0 dead)	$[64,64,3,3]$ (0 dead)	$[64,64,3,3]$ (0 dead)
layer2.0.conv1	$[128,64,3,3]$	$[128,64,3,3]$ (0 dead)	$[128,64,3,3]$ (0 dead)	$[128,64,3,3]$ (0 dead)	$[128,64,3,3]$ (0 dead)

Appendix

Table 11: Layer-by-layer physical structured compression shapes for ResNet-18 on CIFAR-10 at

\sim 95\%

global sparsity across pruning methods, with the number of pruned (dead) channels/neurons in parentheses.

Model	Initialization Scheme	Pruning Threshold $\tau$	Final Sparsity (%)	Final Test Acc (%)
ResNet-18	Kaiming Normal	$\tau=5\text{e-}5$	95.41%	75.10%
ResNet-18	Kaiming Uniform	$\tau=5\text{e-}5$	95.35%	76.08%
ResNet-18	Xavier Normal	$\tau=5\text{e-}5$	95.66%	77.05%
ResNet-18	Xavier Uniform	$\tau=5\text{e-}5$	95.54%	76.72%
ResNet-18	Orthogonal	$\tau=5\text{e-}5$	95.75%	76.51%
ResNet-18	Normal ( $\sigma=0.02$ )	$\tau=5\text{e-}5$	95.96%	77.26%

Appendix

Table 12: Weight initialization scheme ablation evaluation for VGG-16 and ResNet-18 on CIFAR-10.

Layer Index	ResNet-18 (CIFAR-10)	VGG-16 (CIFAR-10)	MiniBERT (SST-2)
1	conv1	features.0 (conv1_1)	embeddings.word_embeddings
2	layer1.0.conv1	features.3 (conv1_2)	embeddings.position_embeddings
3	layer1.0.conv2	features.7 (conv2_1)	encoder.layer.0.attention.query
4	layer1.1.conv1	features.10 (conv2_2)	encoder.layer.0.attention.key
5	layer1.1.conv2	features.14 (conv3_1)	encoder.layer.0.attention.value
6	layer2.0.conv1	features.17 (conv3_2)	encoder.layer.0.output.dense

Appendix

Table 13: Sequential layer index mapping (

1,2,3,\dots,L

) to full parameter layer names for ResNet-18, VGG-16, and MiniBERT.

Explore similar work

Structured Neuron Pruning in Deep Neural Networks Using Multi-Armed Bandits
May 29, 2026Salem Ameen, Sunil VaderaMulti-Armed Bandits Neural Network Compression
School of Science, Engineering and Environment, University of Salford, Salford, United Kingdom
Finding Sparse Subnetworks in One Training Cycle via Progressive Magnitude-Based Pruning
Jun 10, 2026Romana Qureshi, Hafida Benhidour, Said Kerrache +1Sparse Neural Networks Neural Network Pruning
Department of Computer Science, King Saud University, Riyadh, Saudi Arabia
Hyperflux: Pruning Reveals Importance
Apr 6, 2025Eugen Barbulescu, Antonio Alexoaie, Lucian BusoniuSparse Neural Networks Neural Network Training Dynamics
Department of Computer Science Technical University of Cluj-Napoca · Department of Automation Technical University of Cluj-Napoca

May 29, 2026, cs.LGJ/K move · Enter open · S save

Structured Neuron Pruning in Deep Neural Networks Using Multi-Armed Bandits

Salem Ameen, Sunil Vadera

School of Science, Engineering and Environment, University of Salford, Salford, United Kingdom

Multi-Armed Bandits Neural Network Compression Neural Network Pruning

Open paper AskPDF arXiv

Deep neural networks often contain redundant hidden units. Removing individual weights can reduce parameter count, but unstructured sparsity is not always easy to exploit in standard dense implementations. This paper develops a structured pruning framework in which complete neurons are removed using multi-armed bandit (MAB) algorithms. Each candidate neuron is treated as an arm; pulling an arm temporarily masks that neuron, measures the change in loss on a sampled mini-batch, restores the neuron, and updates an estimate of its safe-removal reward. The framework supports stochastic policies, including Epsilon-Greedy, Softmax, UCB1 and Thompson Sampling, and multiplicative-weight policies, including Hedge-style multiplicative weights and EXP3. We evaluate the method on tabular classification, tabular regression and deep neural-network benchmarks covering image, text and reasoning tasks. Statistical comparisons using the Friedman test followed by the Nemenyi post-hoc test show significant differences between methods. On tabular classification tasks, UCB1 obtains the highest mean rank among pruning policies and improves on the unpruned neural network. On regression tasks, UCB1 obtains the highest mean rank and is statistically competitive with, or superior to, several standard regression models according to R^2. On deep-learning tasks, UCB1 and Thompson Sampling obtain the strongest ranks, and several MAB policies significantly outperform the unpruned model, magnitude-based neuron pruning and greedy activation-variation pruning. The results show that MAB-based neuron pruning is an effective and computationally practical approach for structured model reduction.

Abstract

Explore similar work

Structured Neuron Pruning in Deep Neural Networks Using Multi-Armed Bandits

Finding Sparse Subnetworks in One Training Cycle via Progressive Magnitude-Based Pruning

Hyperflux: Pruning Reveals Importance