A Hierarchical Importance-Guided Multi-objective Evolutionary Framework for Deep Neural Network Pruning

Zak Khan; Azam Asilian Bidgoli

cs.NEApr 1, 2026

A Hierarchical Importance-Guided Multi-objective Evolutionary Framework for Deep Neural Network Pruning

Authors: Zak Khan, Azam Asilian Bidgoli

Organizations: Department of Computer Science and Physics, Wilfrid Laurier University, Waterloo, N2L 3C5, Ontario, Canada

Abstract

The optimization of over-parameterized deep neural networks represents a large-scale, high-dimensional, and strongly non-convex decision problem that challenges existing optimization frameworks. Current evolutionary and gradient-based pruning methods often struggle to scale to such dimensionalities, as they rely on flat search spaces, scalarized objectives, or repeated retraining, leading to premature convergence and prohibitive computational cost. This paper introduces a hierarchical importance-guided evolutionary framework that reformulates convolutional network pruning as a tractable large-scale multi-objective optimization problem. In the first phase, a continuous evolutionary search performs coarse exploration of weight-wise pruning thresholds to shrink the search space and identify promising regions of the Pareto set. The second phase applies a fine-grained binary evolutionary optimization constrained to the surviving weights, where importance-aware sampling and adaptive variation operators refine local search in the sparse region of the Pareto set. This hierarchical design combines global exploration and localized exploitation to achieve a well-distributed Pareto set of networks balancing compactness and accuracy. Empirical results on CIFAR-10 and CIFAR-100 using ResNet-56 and ResNet-110 confirm the method's effectiveness compared to existing evolutionary approaches: pruning achieves up to 51.9% and 38.9% parameter reductions with almost no accuracy loss compared to state-of-the-art evolutionary DNN pruning methods. The proposed method contributes a scalable evolutionary approach for solving very-large-scale multi-objective optimization problems, offering a general paradigm extendable to other domains where the decision space is exponentially large, objective functions are conflicting, and efficient trade-off discovery is essential.

Figures & tables

Figure 1: Two-Phase Evolutionary Pruning Flow Diagram. Phase 1 performs coarse-grained evolutionary pruning in continuous space, while Phase 2 refines the sparse region using binary importance-guided search to produce the final Pareto-optimal models.

Figure 2: Overview of the two-phase pruning framework. Phase 1 constructs the Pareto front $\mathcal{P}_{1}$ via coarse threshold pruning, while Phase 2 performs localized binary refinement between heavy and light anchor models to produce the refined front $\mathcal{P}_{2}$ .

Dataset	Subset	Size	Purpose	Usage Phase
CIFAR-10	Training Set	47,500	Model fine-tuning	Pre-pruning
	Validation Set	1,000	Fitness evaluation	Phase 1 & 2
	Optimization Subset	500	Fast EA evaluation	Phase 1 & 2
	Test Set	10,000	Final model evaluation	Post-optimization
CIFAR-100	Training Set	47,000	Model fine-tuning	Pre-pruning
	Validation Set	2,000	Fitness evaluation	Phase 1 & 2

Table 1: Dataset composition and usage across all experimental stages.

Model	Depth	Params (M)	Variant
ResNet-18	18	11.18	ImageNet
ResNet-50	50	23.53	ImageNet
ResNet-56	56	0.855	CIFAR
ResNet-101	101	42.52	ImageNet
ResNet-110	110	1.731	CIFAR
ResNet-152	152	58.16	ImageNet

Table 2: CNN architectures used in the experiments.

Parameter	Setting
Optimizer	Adam
Learning rate	$1\times 10^{-4}$
Loss function	Cross-Entropy Loss
Epochs	30
Batch size	32
Scheduler	None (constant learning rate)

Table 3: Fine-tuning configuration and hyperparameter settings.

Parameter	Setting
Algorithm	NSGA-II
Population size / Generations	50 / 50
Sampling	Latin Hypercube Sampling (LHS)
Crossover	SBX ( $p=0.9$ , $\eta=15$ )
Mutation	Polynomial ( $p=0.2$ , $\eta=20$ )
Search space	Continuous $(r_{i})$

Table 4: Evolutionary setup for Phase 1.

Parameter	Setting
Algorithm	NSGA-II
Population size / Generations	50 / 50
Sampling	Importance-Guided Probabilistic Sampling
Crossover	Uniform ( $p=0.9$ )
Mutation	Bit-flip ( $p=0.05$ )
Search space	Binary pruning masks $(x_{i,j}\!\in\!\{0,1\})$

Table 5: Evolutionary setup for Phase 2.

Architecture	Test Error CIFAR-10	Test Error CIFAR-100
ResNet18	6.32	23.74
ResNet50	6.03	22.08
ResNet56	7.33	31.88
ResNet101	5.34	20.76
ResNet110	6.82	36.62
ResNet152	5.17	20.56

Table 6: Test error rates (%) of baseline CNN architectures on CIFAR-10 and CIFAR-100.

Baseline (M)
Metric	ResNet18	ResNet50	ResNet56	ResNet101	ResNet110	ResNet152
Baseline Weights (M)	11.18	23.53	0.855	42.52	1.731	58.16
Baseline Accuracy (%)	93.68	93.97	92.67	94.66	93.18	94.83
Phase 1 Results
Best Solution-Non-Zero Weights (M)	6.970	12.02	0.506	17.14	1.575	22.62
Best Solution Accuracy (%)	93.14	93.49	92.18	93.64	92.85	93.35

Table 7: Pruning results for CIFAR-10. Baseline model size and accuracy, Phase 1 pruning outcomes (best, heavy, light anchor solutions), and Phase 2 multi-objective results (number of local and non-dominated Pareto solutions and the combined hypervolume). Results averaged over 10 runs.

Baseline (M)
Metric	ResNet18	ResNet50	ResNet56	ResNet101	ResNet110	ResNet152
Baseline Weights (M)	11.23	23.71	0.862	42.70	1.731	58.35
Baseline Accuracy (%)	76.26	77.92	68.12	79.24	63.38	79.44
Phase 1 Results
Best Solution-Non-Zero Weights (M)	7.810	8.145	0.819	18.06	1.764	24.32
Best Solution Accuracy (%)	76.11	77.31	67.94	79.08	63.02	78.98

Table 8: Pruning results for CIFAR-100. Baseline model size and accuracy, Phase 1 pruning outcomes (best, heavy, light anchor solutions), and Phase 2 multi-objective results (number of local and non-dominated Pareto solutions and the combined hypervolume). Results averaged over 10 runs.

Panel of Figure 3 — Figure 3: Pareto fronts for all ResNet architectures on CIFAR-10 (a) and CIFAR-100 (b). In each dataset, the left column shows Phase 1 solutions, while the right column shows Phase 2 solutions generated by the multi-objective evolutionary search (NSGA-II / MOEA/D). Each plot illustrates the accuracy–sparsity trade-off, highlighting how Phase 2 expands and refines the Pareto front beyond Phase 1.

Method	CNN architecture	Baseline Error (%)	Error after pruning (%)	Difference (%)	Pruned (%)
Greedy-Filter Li et al. (2016)	ResNet56	6.96	6.94	-0.02	27.67
Greedy-Filter Li et al. (2016)	ResNet110	6.47	6.70	-0.23	38.63
Auto-Balanced Filter Ding et al. (2018)	ResNet56	6.96	8.56	1.60	41.51
Auto-Balanced Filter Ding et al. (2018)	ResNet110	–	8.27	–	40.00
DeepPruningES Fernandes Junior and Yen (2020)	ResNet56	6.63	8.12	1.49	21.30
DeepPruningES Fernandes Junior and Yen (2020)	ResNet110	6.20	7.42	1.22	16.74

Table 9: Comparison of pruning on the CIFAR-10 dataset across several evolutionary and structured pruning methods. Baseline Error represents the model’s error before pruning, Error after pruning is the error following pruning, Difference indicates the change in error, and Pruned shows the proportion of parameters removed.

Method	CNN architecture	Baseline Error (%)	Error after pruning (%)	Difference (%)	Pruned (%)
DeepPruningES Fernandes Junior and Yen (2020)	ResNet56	41.94	40.20	-1.74	16.18
DeepPruningES Fernandes Junior and Yen (2020)	ResNet110	50.17	50.98	0.81	17.78
Bi-CNN-Pruning Louati et al. (2024)	ResNet56	41.86	24.83	-17.03	16.38
Bi-CNN-Pruning Louati et al. (2024)	ResNet110	50.47	38.44	-12.03	18.13
Two-Phase Pruning	ResNet56	31.88	32.84	0.86	22.13
Two-Phase Pruning	ResNet110	36.62	37.89	1.27	20.62

Table 10: Comparison of Pruning results obtained on CIFAR100 Dataset.

Neural Network Pruning

Neural network pruning removes weights, neurons, channels, heads, or layers to reduce model size and computation while retaining useful behavior.

See all 278 papers

Aug 26, 2026

Pruning Binarized Neural Networks: A Dedicated Framework and Globally Weighted Algorithms

Extreme compression of deep neural networks, up to full binarization, dramatically reduces memory footprint and arithmetic complexity, facilitating deployment on constrained edge hardware with field-programmable gate arrays (FPGAs) and microcontrollers. Although combining binarization with pruning promises additional efficiency gains, existing pruning strategies are ill-suited to binarized representations and rarely translate into meaningful hardware savings. We introduce a PyTorch-based, research-oriented framework that incorporates freezing and pruning mechanisms for designing and optimizing binarized neural networks. The framework enables rapid and reproducible evaluation of state-of-the-art approaches and the fast prototyping of new ones. Leveraging this framework, we propose a novel pruning method that accounts for the relative importance of learned parameters across abstraction levels. Such a global weighting mechanism consistently achieves a superior trade-off between model accuracy and pruning rate, achieving a 70% pruning rate on VGG11 with constant accuracy, while state-of-the-art results reach only 41% in the binarized setting.

Roan Rubiales, Jean Pierre David

Polytechnique Montreal

Neural Network Compression Neural Network Pruning Binary Neural Networks

Ask PDF

Mar 17, 2026

Mix-and-Match Pruning: Globally Guided Layer-Wise Sparsification of DNNs

Deploying deep neural networks (DNNs) on edge devices requires strong compression with minimal accuracy loss. This paper introduces Mix-and-Match Pruning, a globally guided, layer-wise sparsification framework that leverages sensitivity scores and simple architectural rules to generate diverse, high-quality pruning configurations. The framework addresses a key limitation that different layers and architectures respond differently to pruning, making single-strategy approaches suboptimal. Mix-and-Match derives architecture-aware sparsity ranges, e.g., preserving normalization layers while pruning classifiers more aggressively, and systematically samples these ranges to produce ten strategies per sensitivity signal (magnitude, gradient, or their combination). This eliminates repeated pruning runs while offering deployment-ready accuracy-sparsity trade-offs. Experiments on CNNs and Vision Transformers demonstrate Pareto-optimal results, with Mix-and-Match reducing accuracy degradation on Swin-Tiny by 40% relative to standard single-criterion pruning. These findings show that coordinating existing pruning signals enables more reliable and efficient compressed models than introducing new criteria.

Danial Monachan, Samira Nazari, Mahdi Taheri +4

Brandenburg Technical University, Cottbus, Germany · University of Zanjan, Iran · Tallinn University of Technology, Estonia +1

Neural Network Compression Neural Network Pruning

Ask PDF

Apr 7, 2026

Neural Network Pruning via QUBO Optimization

Neural network pruning can be formulated as a combinatorial optimization problem, yet most existing approaches rely on greedy heuristics that ignore complex interactions between filters. Formal optimization methods such as Quadratic Unconstrained Binary Optimization (QUBO) provide a principled alternative but have so far underperformed due to oversimplified objective formulations based on metrics like the L1-norm. In this work, we propose a unified Hybrid QUBO framework that bridges heuristic importance estimation with global combinatorial optimization. Our formulation integrates gradient-aware sensitivity metrics - specifically first-order Taylor and second-order Fisher information - into the linear term, while utilizing data-driven activation similarity in the quadratic term. This allows the QUBO objective to jointly capture individual filter relevance and inter-filter functional redundancy. We further introduce a dynamic capacity-driven search to strictly enforce target sparsity without distorting the optimization landscape. Finally, we employ a two-stage pipeline featuring a Tensor-Train (TT) Refinement stage - a gradient-free optimizer that fine-tunes the QUBO-derived solution directly against the true evaluation metric. Experiments on the SIDD image denoising dataset demonstrate that the proposed Hybrid QUBO significantly outperforms both greedy Taylor pruning and traditional L1-based QUBO, with TT Refinement providing further consistent gains at appropriate combinatorial scales. This highlights the potential of hybrid combinatorial formulations for robust, scalable, and interpretable neural network compression.

Osama Orabi, Artur Zagitov, Hadi Salloum +3

Department of Computer Science and Engineering, Innopolis University, 420500 Innopolis, Russia · Q Deep, Laboratory of Quantum Computing, 420502 Innopolis, Russia · Research Center of the Artificial Intelligence Institute, Innopolis University, 420500 Innopolis, Russia +2

Neural Network Pruning Quadratic Unconstrained Binary Optimization Neural Combinatorial Optimization

Ask PDF

Abstract

Neural Network Pruning

Explore similar work

Pruning Binarized Neural Networks: A Dedicated Framework and Globally Weighted Algorithms

Mix-and-Match Pruning: Globally Guided Layer-Wise Sparsification of DNNs

Neural Network Pruning via QUBO Optimization