Growing Networks with Autonomous Pruning

Charles De Lambilly; Stefan Duffner

cs.CVMar 20, 2026

Growing Networks with Autonomous Pruning

Authors: Charles De Lambilly, Stefan Duffner

Organizations: INSA Lyon, CNRS, Ecole Centrale de Lyon, Universite Claude Bernard Lyon 1, Université Lumière Lyon 2, LIRIS, UMR5205, , Villeurbanne, 69621, France

Abstract

This paper introduces Growing Networks with Autonomous Pruning (GNAP) for image classification. Unlike traditional convolutional neural networks, GNAP change their size, as well as the number of parameters they are using, during training, in order to best fit the data while trying to use as few parameters as possible. This is achieved through two complementary mechanisms: growth and pruning. GNAP start with few parameters, but their size is expanded periodically during training to add more expressive power each time the network has converged to a saturation point. Between these growing phases, model parameters are trained for classification and pruned simultaneously, with complete autonomy by gradient descent. Growing phases allow GNAP to improve their classification performance, while autonomous pruning allows them to keep as few parameters as possible. Experimental results on several image classification benchmarks show that our approach can train extremely sparse neural networks with high accuracy. For example, on MNIST, we achieved 99.44% accuracy with as few as 6.2k parameters, while on CIFAR10, we achieved 92.2\ accuracy with 157.8k parameters.

Figures & tables

Figure 1: The overall network architecture of GNAP ( conv : convolution layer, pool : pooling layer, cat : feature map concatenation, fc : fully-connected layer). Each layer may be (sparsely) connected to all preceding layers. Each block may contain one or several convolution layers with the same feature map size. During training, structured (kernel) pruning or unstructured pruning is performed and layers are grown and added repeatedly.

Model	block 1	block 2	block 3	FC
GNAP (MNIST)	10	10		10
GNAP (CF10)	64, 64, 64	128, 128, 128	256, 256, 256	10
GNAP (CF100)	64, 64, 64	128, 128, 128	256, 256, 256	100
NAP (m1) (MNIST)	10	10		10
NAP (m2) (MNIST)	100	100		10
NAP (m3) (CF10)	64	128		10

Table 1: Initial architectures of the different tested models obtaining different accuracy-complexity trade-offs. The numbers indicate the size of the layers (i.e. the number of neurons/channels).

Method	Accuracy		Links		Weights
NAP-U (m1)	0.9875	$\pm$ 0.0008	5762	$\pm$ 437	5762	$\pm$ 437
NAP-U (m2)	0.9938	$\pm$ 0.0008	9264	$\pm$ 2202	9264	$\pm$ 2202
GNAP-U (rand)	0.9943	$\pm$ 0.0004	6154	$\pm$ 291	6154	$\pm$ 291
GNAP-U (ortho)	0.9944	$\pm$ 0.0004	6234	$\pm$ 203	6234	$\pm$ 203
NAP-S (m1)	0.9823	$\pm$ 0.0023	5409	$\pm$ 1388	5928	$\pm$ 1482
NAP-S (m2)	0.9923	$\pm$ 0.0012	21003	$\pm$ 10068	28533	$\pm$ 11678

Table 2: Performance on MNIST.

Method	Accuracy		Links		Weights
NAP-U (m4)	0.8763	$\pm$ 0.0055	235655	$\pm$ 67951	235655	$\pm$ 67951
NAP-U (m5)	0.8849	$\pm$ 0.0107	245168	$\pm$ 68530	245168	$\pm$ 68530
NAP-U (m6)	0.9162	$\pm$ 0.0024	696837	$\pm$ 116179	696837	$\pm$ 116179
GNAP-U (rand)	0.9245	$\pm$ 0.0046	179979	$\pm$ 26074	179979	$\pm$ 26074
GNAP-U (ortho)	0.9222	$\pm$ 0.0041	157765	$\pm$ 18620	157765	$\pm$ 18620
NAP-S (m4)	0.8802	$\pm$ 0.0049	99097	$\pm$ 38844	340501	$\pm$ 65564

Table 3: Performance on CIFAR10.

Method	Accuracy		Links		Weights
NAP-U (m7)	0.5583	$\pm$ 0.0173	393330	$\pm$ 121100	393330	$\pm$ 121100
NAP-U (m8)	0.6408	$\pm$ 0.0158	410827	$\pm$ 118481	410827	$\pm$ 118481
GNAP-U (rand)	0.6692	$\pm$ 0.0106	207289	$\pm$ 59319	207289	$\pm$ 59319
GNAP-U (ortho)	0.6656	$\pm$ 0.0086	179857	$\pm$ 28182	179857	$\pm$ 28182
NAP-S (m7)	0.5085	$\pm$ 0.0124	231668	$\pm$ 3532	247850	$\pm$ 4221
NAP-S (m8)	0.5965	$\pm$ 0.0141	235029	$\pm$ 34575	461149	$\pm$ 142521

Table 4: Performance on CIFAR100.

Method	Accuracy		Weights	Base architecture
SimpleNet [ 7 ]	0.9198		310000
SNIP [ 11 ]	0.9272	$\pm$ 0.0018	186013	ResNet32
SNIP [ 11 ]	0.7612	$\pm$ 0.2196	410968	VGG19
GraSP [ 22 ]	0.9286	$\pm$ 0.0019	186013	ResNet32
GraSP [ 22 ]	0.9216	$\pm$ 0.0014	410968	VGG19
SRatio [ 19 ]	0.9185	$\pm$ 0.0016	93006	ResNet32

Table 5: Comparison with state-of-the-art sparse CNN on CIFAR10. Our approach outperforms most methods combining the criteria of accuracy and number of weights without the dependence on a pre-defined initial dense network.

Method	Accuracy		Weights	Base architecture
SimpleNet [ 7 ]	0.6468		310000
SNIP [ 11 ]	0.6554	$\pm$ 0.0026	93006	ResNet32
SNIP [ 11 ]	0.2534	$\pm$ 0.0916	410968	VGG19
GraSP [ 22 ]	0.6584	$\pm$ 0.0014	93006	ResNet32
GraSP [ 22 ]	0.6584	$\pm$ 0.0036	205484	VGG19
SRatio [ 19 ]	0.6708	$\pm$ 0.0041	93006	ResNet32

Table 6: Comparison with state-of-the-art sparse CNN on CIFAR100. Our approach outperforms most methods combining the criteria of accuracy and number of weights without the dependence on a pre-defined initial dense network.

Figure 2: The evolution of the different losses and number of open gates during training of our GNAP model (here GNAP-U on CIFAR10) (Please note that the scale for the number of open gates is logarithmic).

Figure 3: The evolution of the accuracy vs. number of parameters. Each curve is a snapshot of the model before a given growing phase (e.g. "0" is the model before the first growing phase). The points of a curve represent different gate thresholds

\gamma

at inference.

Figure 4: The evolution of links (open gates) of our GNAP model during training (here GNAP-U on CIFAR10). Numbers indicate training epochs. A pixel at row i and column j represents the link going from neuron j to neuron i. The intensity of a pixel represents the proportion of open gates between two neurons/feature maps.

Dynamic Sparse Training

See all 10 papers

Finding Sparse Subnetworks in One Training Cycle via Progressive Magnitude-Based Pruning
Jun 10, 2026Romana Qureshi, Hafida Benhidour, Said Kerrache +1Sparse Neural Networks Neural Network Pruning
Department of Computer Science, King Saud University, Riyadh, Saudi Arabia
Hyperflux: Pruning Reveals Importance
Apr 6, 2025Eugen Barbulescu, Antonio Alexoaie, Lucian BusoniuSparse Neural Networks Neural Network Training Dynamics
Department of Computer Science Technical University of Cluj-Napoca · Department of Automation Technical University of Cluj-Napoca
Random Cloud: Finding Minimal Neural Architectures Without Training
Apr 29, 2026Javier Gil BlázquezNeural Architecture Search Feedforward Neural Networks

Jun 10, 2026J/K move · Enter open · S save

Finding Sparse Subnetworks in One Training Cycle via Progressive Magnitude-Based Pruning

Romana Qureshi, Hafida Benhidour, Said Kerrache, Nahlah Aljeraisy

Department of Computer Science, King Saud University, Riyadh, Saudi Arabia

Sparse Neural Networks Neural Network Pruning

Open paper AskPDF arXiv

Neural network pruning reduces model size by removing less important parameters while aiming to preserve predictive performance. Although the Lottery Ticket Hypothesis (LTH) shows that sparse subnetworks can match dense networks when trained from suitable initializations, its iterative pruning procedure requires multiple complete training cycles. This work evaluates progressive magnitude-based pruning as a single-cycle alternative. The method gradually increases sparsity during training using a linear schedule and updates pruning masks based on active weight magnitudes. We conduct systematic experiments on CIFAR-10 and MNIST across ResNet, VGG-style, and LeNet architectures, comparing the proposed method with representative iterative and initialization-based pruning baselines, including LTH, SNIP, and GraSP. On CIFAR-10, the method achieves 95.12% accuracy on ResNet-18 at 72.9% sparsity, compared with 90.5% reported for LTH. At extreme sparsity, it achieves 93.13% accuracy on a VGG-like architecture at 97% sparsity, compared with approximately 92.0% for SNIP, and 93.44% accuracy on VGG-19 at 97.97% sparsity, compared with 92.19% for GraSP at 98% sparsity. A sparsity-accuracy analysis on ResNet-18 further shows that accuracy remains within 0.1 percentage points of the dense baseline across 70--85% sparsity. These results indicate that progressive magnitude-based pruning provides an effective single-cycle approach for neural network sparsification under the evaluated settings.

Abstract

Dynamic Sparse Training

Explore similar work

Finding Sparse Subnetworks in One Training Cycle via Progressive Magnitude-Based Pruning

Hyperflux: Pruning Reveals Importance

Random Cloud: Finding Minimal Neural Architectures Without Training