Deep Neural Networks (DNNs) are widely used by engineers to solve difficult problems that require predictive modeling from data. However, these models are often massive, with millions or billions of parameters, and require substantial computational power, RAM, and storage. This becomes a limitation in practical scenarios where strict size and resource constraints must be respected. In this paper, we present a novel concept-based pruning technique for DNNs that guides pruning decisions using human-interpretable concepts, such as features, colors, and classes. This is particularly important in a software engineering context, as DNNs are integrated into systems and must be pruned according to specific system requirements. Our concept-based pruning solution analyzes neuron activations to identify important neurons from a system requirements viewpoint and uses this information to guide the DNN pruning. We assess our solution using the VGG-19 network and a dataset of 26'384 RGB images, focusing on its ability to produce small, effective pruned DNNs and on the computational complexity and performance of these pruned DNNs. We also analyzed the pruning efficiency of our solution and compared alternative configurations. Our results show that concept-based pruning efficiently generates much smaller, effective pruned DNNs. Pruning greatly improves the computational efficiency and performance of DNNs, properties that are particularly useful for practical applications with stringent memory and computational time constraints. Finally, alternative configuration options enable engineers to identify trade-offs adapted to different practical situations.
Figures & tables
Figure 1. Concept-Based Pruning. Flowchart representing the proposed approach: Concept-Based Pruning. The flowchart has two main phases: (i) the Neuron Identifier and (ii) the Pruner. The Neuron Identifier takes as input a Dataset, a set of Concepts, and a Deep Neural Network, and outputs a list of Neurons. The Pruner takes as input the list of Neurons and the DNN and returns the pruned DNN. Finally, there is a feedback loop in which the pruned DNN replaces the original DNN, allowing the pruning process to be repeated iteratively.
Figure 2. Examples images from the RIVAL10 benchmark. The figure contains two examples of images from the \rivalbenchmark. The first image shows a horse's muzzle, taken from a short distance. The horse's head covers the rest of the body. The image is labeled with the Equine class. The second image shows the interior of an airplane taken from a window seat. The image is labeled with the Plane class.
Figure 3. Example of decision tree extracted by FGA. Example of a decision tree extracted by FGA. The decision tree in question is complete, with three layers and four leaf nodes. Each non-leaf node contains the identifier of a neuron, and each branch contains a condition of the activation layer of that neuron. Each leaf node contains a clause specifying whether the concept is present or not, and the number of elements in the dataset that are represented by that leaf node.
Layer
Output Size
# Params
Input
224 × 224 × 3
0
Conv1_1
224 × 224 × 64
1’792
Conv1_2
224 × 224 × 64
36’928
MaxPool
112 × 112 × 64
0
Conv2_1
112 × 112 × 128
73’856
Conv2_2
112 × 112 × 128
147’584
Table 1. Architecture and number of parameters of VGG-19.
Size
Effectiveness
Iteration
FC1
FC2
Params
Size
Accuracy
Precision
Recall
F1-score
(neurons)
(neurons)
(M)
(MB)
(%)
(%)
(%)
(%)
VGG-19
4096
4096
143.67
574.70
84.79
90.74
77.10
83.22
1
2622
2357
94.35
377.42
81.71
90.73
74.31
81.48
10
1241
1088
53.60
214.43
78.34
90.60
71.29
79.32
20
1007
903
47.10
188.44
76.49
90.53
69.62
78.03
Table 2. Size and effectiveness of the pruned DNN generated by CBP.
Active Neurons
Computational Complexity
Inference Efficiency
Iteration
FC1
FC2
MACs FC1
MACs FC2
Total MACs
Latency
Std.
FPS
(neurons)
(neurons)
(M)
(M)
(G)
(ms)
( ± ms)
VGG-19
4096
4096
102.76
16.78
19.668
13.35
0.25
74.91
1
2622
2357
65.78
6.18
19.619
12.27
0.19
81.50
10
1241
1088
31.13
1.35
19.578
11.14
0.17
89.77
20
1007
903
25.26
0.91
19.571
10.96
0.05
91.24
Table 3. Computational cost and runtime analysis on VGG-19’s first two FC layers.
Iteration
Neurons Identifier (s)
Pruner (s)
Total (s)
0
778.25
1.81
780.07
10
383.04
0.92
383.96
20
358.11
0.91
359.01
30
351.61
0.86
352.47
40
346.72
1.01
347.73
50
345.43
0.86
346.29
Table 4. Time required by CBP and its components.
Size
Effectiveness
Iteration
FC1
FC2
Params
Size
Accuracy
Precision
Recall
F1-score
(neurons)
(neurons)
(M)
(MB)
(%)
(%)
(%)
(%)
VGG-19
4096
4096
143.67
574.70
84.79
90.74
77.10
83.22
1
1038
766
47.63
190.55
70.81
90.73
64.28
74.42
2
798
562
41.06
164.26
61.84
90.75
56.04
67.10
3
720
463
38.89
155.57
56.75
90.61
51.40
62.63
Table 5. Size and effectiveness of the pruned DNN generated by CBP including misclassified samples.
Figure 4. Size and effectiveness of the pruned DNN when misclassified inputs are discarded or considered by CBP. The figure presents two plots, one next to the other. In the plot on the left, the x-axis shows the number of iterations (from 0 to 20), and the y-axis shows the number of parameters in millions (from 0 to 150). The plot shows two lines: a blue line for the original \fgpand a red line for the modified version of \fgpwith misclassified images. The plot shows that the original \fgppreserves slightly more parameters than the version with misclassified images for all iterations, with a difference of approx. 10 million parameters in the last iteration. In the plot on the right, the x-axis shows the number of iterations (from 0 to 20), and the y-axis shows the accuracy (from 35% to 90%). As for the other plot, we are comparing the original \fgpapproach (blue) with the alternative version that also uses misclassified inputs (red). The plot shows that the original \fgpsignificantly outperforms the modified version, with an accuracy difference of almost 40% at the last iteration.
Figure 5. Accuracy of different EFGA aggregation policies. The plot shows the performances of different \efgaaggregation policies. The x-axis shows the number of iterations (0-25), and the y-axis shows the accuracy (0-90%). The plot shows ten different lines: nine for the different \efgaaggregation policies (TOP(1), TOP(3), TOP(5), TOP(10), REC(80), REC(85), REC(90), REC(95), and AVG) and one for the original \fgpapproach. For most aggregation approaches, the accuracy degrades with only a few iterations, reaching 0%: TOP(1), TOP(3), TOP(5), TOP(10), and AVG. For REC(80), REC(85), and REC(90), the accuracy never reaches 0%, but they perform significantly worse than the original approach. Only REC(95) shows a similar behaviour to \fgp, and it even outperforms it slightly in some iterations.
Size
Effectiveness
Iteration
FC1
FC2
Params
Size
Accuracy
Precision
Recall
F1-score
(neurons)
(neurons)
(M)
(MB)
(%)
(%)
(%)
(%)
VGG-19
4096
4096
143.67
574.70
84.79
90.74
77.10
83.22
1
1349
1070
56.38
225.57
80.70
90.38
73.43
80.64
5
789
628
40.94
163.81
77.62
90.11
70.65
78.38
10
636
527
36.84
147.41
78.11
89.87
71.10
78.45
Table 6. Size and effectiveness of the pruned DNN generated by CBP with the criterion REC(95).
Figure 6. Size and effectiveness of the pruned DNN for the CBP default configuration and CBP with EFGA and REC(95). The figure presents two plots, one next to the other. In the plot on the left, the x-axis shows the number of iterations (from 0 to 25), and the y-axis shows the number of parameters in millions (from 0 to 150). The plot shows two lines: a blue line for the original \fgpand a red line for \efgawith REC(95). The plot shows that the original \fgppreserves slightly more parameters than the REC(95) version at all iterations, with a difference of approx. 10 million parameters in the last iteration. In the plot on the right, the x-axis shows the number of iterations (0-25), and the y-axis shows the accuracy (70-90%). As for the other plot, we are comparing the original \fgpapproach (blue) with \efgawith REC(95) (red). The plot shows that the behaviour of the two approaches is very similar, with \fgpslightly outperforming REC(95) for most iterations.
The optimization of over-parameterized deep neural networks represents a large-scale, high-dimensional, and strongly non-convex decision problem that challenges existing optimization frameworks. Current evolutionary and gradient-based pruning methods often struggle to scale to such dimensionalities, as they rely on flat search spaces, scalarized objectives, or repeated retraining, leading to premature convergence and prohibitive computational cost. This paper introduces a hierarchical importance-guided evolutionary framework that reformulates convolutional network pruning as a tractable large-scale multi-objective optimization problem. In the first phase, a continuous evolutionary search performs coarse exploration of weight-wise pruning thresholds to shrink the search space and identify promising regions of the Pareto set. The second phase applies a fine-grained binary evolutionary optimization constrained to the surviving weights, where importance-aware sampling and adaptive variation operators refine local search in the sparse region of the Pareto set. This hierarchical design combines global exploration and localized exploitation to achieve a well-distributed Pareto set of networks balancing compactness and accuracy. Empirical results on CIFAR-10 and CIFAR-100 using ResNet-56 and ResNet-110 confirm the method's effectiveness compared to existing evolutionary approaches: pruning achieves up to 51.9% and 38.9% parameter reductions with almost no accuracy loss compared to state-of-the-art evolutionary DNN pruning methods. The proposed method contributes a scalable evolutionary approach for solving very-large-scale multi-objective optimization problems, offering a general paradigm extendable to other domains where the decision space is exponentially large, objective functions are conflicting, and efficient trade-off discovery is essential.
Zak Khan, Azam Asilian Bidgoli
Department of Computer Science and Physics, Wilfrid Laurier University, Waterloo, N2L 3C5, Ontario, Canada
Deploying deep neural networks (DNNs) on edge devices requires strong compression with minimal accuracy loss. This paper introduces Mix-and-Match Pruning, a globally guided, layer-wise sparsification framework that leverages sensitivity scores and simple architectural rules to generate diverse, high-quality pruning configurations. The framework addresses a key limitation that different layers and architectures respond differently to pruning, making single-strategy approaches suboptimal. Mix-and-Match derives architecture-aware sparsity ranges, e.g., preserving normalization layers while pruning classifiers more aggressively, and systematically samples these ranges to produce ten strategies per sensitivity signal (magnitude, gradient, or their combination). This eliminates repeated pruning runs while offering deployment-ready accuracy-sparsity trade-offs. Experiments on CNNs and Vision Transformers demonstrate Pareto-optimal results, with Mix-and-Match reducing accuracy degradation on Swin-Tiny by 40% relative to standard single-criterion pruning. These findings show that coordinating existing pruning signals enables more reliable and efficient compressed models than introducing new criteria.
Danial Monachan, Samira Nazari, Mahdi Taheri +4
Brandenburg Technical University, Cottbus, Germany · University of Zanjan, Iran · Tallinn University of Technology, Estonia +1
This study focuses on the scientific problem of understanding internal factors that govern the diverse performance degradation of deep neural networks (DNNs) when different parameters are pruned. In order to explain why pruning certain parameters leads to significant performance degradation but pruning other parameters does not, we examine how the pruning operation affects the interaction patterns encoded by the DNN. We find that when we progressively increase the pruning ratio, the interaction patterns encoded by DNNs exhibit a distinct three-phase dynamics, i.e., model performance is not largely affected until the pruning operation begins to remove low-order interactions, and low-order interactions exhibit strong generalizability. Moreover, we find that the high sensitivity of DNN performance to the pruning of certain modules is attributed to whether the pruning operation removes generalizable low-order interaction patterns.