BaCP: Backbone Contrastive Pruning for Preserving Representations in Extremely Sparse Neural Networks

Mohammad Haroon Khawaja; Muhammad Haseeb; Mohammad Fatim Shoaib; Muhammad Tahir

cs.LGOct 1, 2026

BaCP: Backbone Contrastive Pruning for Preserving Representations in Extremely Sparse Neural Networks

Authors: Mohammad Haroon Khawaja, Muhammad Haseeb, Mohammad Fatim Shoaib, Muhammad Tahir

Organizations: Lahore University of Management Sciences Lahore, Pakistan

Abstract

Unstructured pruning at extreme sparsity often suffers from representational collapse, causing sharp drops in accuracy. To address this, we study Backbone Contrastive Pruning (BaCP), which regularizes the sparse network's embedding space by aligning it with pretrained, fine-tuned, and historical snapshot models. Building on the contrastive decomposition of the CAP framework (Xu et al., 2022), we provide a rigorous matched-budget characterization of this approach across multiple pruning criteria. Evaluated across 90 settings, BaCP improves accuracy substantially in extreme sparsity regimes where standard pruning fails, and is close to baseline where representations remain intact.

Figures & tables

Figure 1: BaCP framework. The sparse model embeds one augmented view and each frozen anchor the other. Every anchor contributes one contrastive term over the stacked similarity matrix.

Magnitude ( Han et al., 2015 )
	ResNet-34		ResNet-50		VGG-11		VGG-19
Sparsity	I.P.	$\Delta$	I.P.	$\Delta$	I.P.	$\Delta$	I.P.	$\Delta$
0.95	91.44	+0.36	92.82	+0.10	90.04	+0.32	91.30	+0.24
0.97	91.51	+0.15	92.65	$-$ 0.14	90.31	$-$ 0.10	91.64	+0.40
0.99	90.66	+0.38	91.59	+0.01	90.10	$-$ 0.09	91.18	+0.47
0.999	76.32	+5.33	69.84	+10.76	85.31	$-$ 0.84	84.91	+1.44

Table 1: Test accuracy (%) on held-out CIFAR-10, mean of three seeds. BaCP

=

I.P.

+\ \Delta

; bold marks improvements above

+0.3

. The arms differ in learning rate and augmentation (Section 2.3 ).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Criterion	Sparsity
		0.95	0.97	0.99	0.999	0.9995
ResNet-34	magnitude	$+1.82$	$+1.31$	$-0.33$	$+14.08$	$+11.73$
	SNIP-it	$+1.87$	$+1.62$	$+0.55$	$-1.40$	$+0.07$
	WANDA	$+3.84$	$+7.70$	$+18.85$	$+3.25$	$+0.63$

Appendix

Table 2: ResNet-34 on CIFAR-100 , same protocol. The pattern of Table 1 reproduces on a second dataset and label space: margins near zero while the baseline is intact, large once it is not.

Model	Data	Criterion	$s$	I.P.	BaCP	$\Delta$	surviving
ResNet-34	C-10	magnitude	0.95	91.44 $\pm$ 0.57	91.80 $\pm$ 0.21	$+0.36$	$1\,063\,254$
ResNet-34	C-10	magnitude	0.97	91.51 $\pm$ 0.68	91.66 $\pm$ 0.23	$+0.15$	$637\,952$
ResNet-34	C-10	magnitude	0.99	90.66 $\pm$ 0.42	91.04 $\pm$ 0.53	$+0.38$	$212\,650$
ResNet-34	C-10	magnitude	0.999	76.32 $\pm$ 0.67	81.65 $\pm$ 0.22	$+5.33$	$21\,265$
ResNet-34	C-10	magnitude	0.9995	61.09 $\pm$ 1.06	75.11 $\pm$ 0.14	$+14.02$	$10\,632$
ResNet-34	C-10	SNIP-it	0.95	91.26 $\pm$ 0.65	91.60 $\pm$ 0.14	$+0.34$	$1\,063\,254$

Appendix

Table 3: Every grid cell, both arms. I.P. and BaCP under a matched 50+25-epoch budget, three seeds each.

Model	$s$	Criterion	I.P.	CE-only	BaCP	aug $+$ LR	contrastive	regime
MobileNetV2	0.99	magnitude	85.73 $\pm$ 0.22	79.55 $\pm$ 1.44	80.81 $\pm$ 0.34	$-6.18$	$+1.26$	intact
ResNet-34	0.95	magnitude	91.44 $\pm$ 0.57	90.45 $\pm$ 0.42	91.80 $\pm$ 0.21	$-0.99$	$+1.35$	intact
ResNet-34	0.999	magnitude	76.32 $\pm$ 0.67	80.09 $\pm$ 1.04	81.65 $\pm$ 0.22	$+3.77$	$+1.56$	damaged
ResNet-34	0.999	SNIP-it	79.92 $\pm$ 1.12	77.67 $\pm$ 0.77	79.41 $\pm$ 0.17	$-2.25$	$+1.74$	damaged
ResNet-34	0.999	WANDA	38.25 $\pm$ 1.17	56.25 $\pm$ 3.01	55.13 $\pm$ 0.52	$+18.00$	$-1.12$	damaged
ResNet-50	0.95	magnitude	92.82 $\pm$ 0.42	91.55 $\pm$ 0.00	92.92 $\pm$ 0.32	$-1.27$	$+1.37$	intact

Appendix

Table 4: Objective ablation , all cells, three seeds per arm.

Arm	Model	$s$	acc.	seeds	vs. control
head128	MobileNetV2	0.95	87.95	3	control
head16	MobileNetV2	0.95	87.08	1	$-0.87$
head32	MobileNetV2	0.95	87.56	1	$-0.39$
head64	MobileNetV2	0.95	87.74	1	$-0.21$
headfrozen	MobileNetV2	0.95	87.74	1	$-0.21$
headnone	MobileNetV2	0.95	87.95	1	$0.00$

Appendix

Table 6

Arm	Model	$s$	acc.	vs. control
snapeven2	ResNet-34	0.97	92.32	$+0.28$
snapeven3	ResNet-34	0.97	92.17	$+0.13$
snapeven5	ResNet-34	0.97	92.03	$-0.01$
snapfirst2	ResNet-34	0.97	92.04	$0.00$
snapeven2	ResNet-34	0.999	81.14	$-0.26$
snapeven3	ResNet-34	0.999	81.19	$-0.21$

Appendix

Table 7

Model	$s$	I.P. @ 0.01	BaCP @ 0.05	BaCP @ 0.1	0.1 vs 0.05
VGG-11	0.95	90.04 $\pm$ 0.15	90.36 $\pm$ 0.02	87.29 $\pm$ 1.92	$-3.07$
VGG-11	0.999	85.31 $\pm$ 0.12	84.47 $\pm$ 0.26	83.05 $\pm$ 2.00	$-1.42$

Appendix

Table 8

Model	Data	Criterion	$s$	I.P.	BaCP
MobileNetV2	C-10	magnitude	0.999	10.00	10.00
MobileNetV2	C-10	magnitude	0.9995	10.00	10.00
MobileNetV2	C-10	SNIP-it	0.999	10.00	24.05
MobileNetV2	C-10	SNIP-it	0.9995	10.00	10.00
MobileNetV2	C-10	WANDA	0.99	10.00	10.00
MobileNetV2	C-10	WANDA	0.999	10.00	10.00

Appendix

Table 9

	I.P.	BaCP
Initialisation	ImageNet-pretrained, identical checkpoint
Dense reference	identical, per seed
Criterion	identical (magnitude / SNIP-it / WANDA, global threshold)
Sparsity schedule	cubic, ramp over 80% of steps, $\Delta T=100$
Prunable scope	identical; classifier head pruned, projection head never
Pruning epochs	50	50

Appendix

Table 10

Neural Network Pruning

Neural network pruning removes weights, neurons, channels, heads, or layers to reduce model size and computation while retaining useful behavior.

See all 292 papers

Finding Sparse Subnetworks in One Training Cycle via Progressive Magnitude-Based Pruning
Jun 10, 2026Romana Qureshi, Hafida Benhidour, Said Kerrache +1Sparse Neural Networks Neural Network Pruning
Department of Computer Science, King Saud University, Riyadh, Saudi Arabia
DADP: Dynamic Activity-Dependent Pruning, A Reverse Hebbian-Inspired Structural Pruning Method
Oct 8, 2026Bhushan DeshpandeDynamic Sparse Training Neural Network Pruning
Independent Researcher, Pune, India
Pruning Deep Neural Networks via the Marchenko--Pastur Distribution
May 23, 2026Leonid Berlyand, Theo Bourdais, Houman Owhadi +1Structured Pruning Sparse Neural Networks
Department of Mathematics Pennsylvania State University University Park, PA 16802, USA · Department of Computing and Mathematical Sciences California Institute of Technology Pasadena, CA 91125, USA

Jun 10, 2026J/K move · Enter open · S save

Finding Sparse Subnetworks in One Training Cycle via Progressive Magnitude-Based Pruning

Romana Qureshi, Hafida Benhidour, Said Kerrache, Nahlah Aljeraisy

Department of Computer Science, King Saud University, Riyadh, Saudi Arabia

Sparse Neural Networks Neural Network Pruning

Open paper AskPDF arXiv

Neural network pruning reduces model size by removing less important parameters while aiming to preserve predictive performance. Although the Lottery Ticket Hypothesis (LTH) shows that sparse subnetworks can match dense networks when trained from suitable initializations, its iterative pruning procedure requires multiple complete training cycles. This work evaluates progressive magnitude-based pruning as a single-cycle alternative. The method gradually increases sparsity during training using a linear schedule and updates pruning masks based on active weight magnitudes. We conduct systematic experiments on CIFAR-10 and MNIST across ResNet, VGG-style, and LeNet architectures, comparing the proposed method with representative iterative and initialization-based pruning baselines, including LTH, SNIP, and GraSP. On CIFAR-10, the method achieves 95.12% accuracy on ResNet-18 at 72.9% sparsity, compared with 90.5% reported for LTH. At extreme sparsity, it achieves 93.13% accuracy on a VGG-like architecture at 97% sparsity, compared with approximately 92.0% for SNIP, and 93.44% accuracy on VGG-19 at 97.97% sparsity, compared with 92.19% for GraSP at 98% sparsity. A sparsity-accuracy analysis on ResNet-18 further shows that accuracy remains within 0.1 percentage points of the dense baseline across 70--85% sparsity. These results indicate that progressive magnitude-based pruning provides an effective single-cycle approach for neural network sparsification under the evaluated settings.

Abstract

Neural Network Pruning

Explore similar work

Finding Sparse Subnetworks in One Training Cycle via Progressive Magnitude-Based Pruning

DADP: Dynamic Activity-Dependent Pruning, A Reverse Hebbian-Inspired Structural Pruning Method

Pruning Deep Neural Networks via the Marchenko--Pastur Distribution