Deep Neural Networks (DNNs) are widely used by engineers to solve difficult problems that require predictive modeling from data. However, these models are often massive, with millions or billions of parameters, and require substantial computational power, RAM, and storage. This becomes a limitation in practical scenarios where strict size and resource constraints must be respected. In this paper, we present a novel concept-based pruning technique for DNNs that guides pruning decisions using human-interpretable concepts, such as features, colors, and classes. This is particularly important in a software engineering context, as DNNs are integrated into systems and must be pruned according to specific system requirements. Our concept-based pruning solution analyzes neuron activations to identify important neurons from a system requirements viewpoint and uses this information to guide the DNN pruning. We assess our solution using the VGG-19 network and a dataset of 26'384 RGB images, focusing on its ability to produce small, effective pruned DNNs and on the computational complexity and performance of these pruned DNNs. We also analyzed the pruning efficiency of our solution and compared alternative configurations. Our results show that concept-based pruning efficiently generates much smaller, effective pruned DNNs. Pruning greatly improves the computational efficiency and performance of DNNs, properties that are particularly useful for practical applications with stringent memory and computational time constraints. Finally, alternative configuration options enable engineers to identify trade-offs adapted to different practical situations.
Figures & tables
Figure 1. Concept-Based Pruning. Flowchart representing the proposed approach: Concept-Based Pruning. The flowchart has two main phases: (i) the Neuron Identifier and (ii) the Pruner. The Neuron Identifier takes as input a Dataset, a set of Concepts, and a Deep Neural Network, and outputs a list of Neurons. The Pruner takes as input the list of Neurons and the DNN and returns the pruned DNN. Finally, there is a feedback loop in which the pruned DNN replaces the original DNN, allowing the pruning process to be repeated iteratively.
Figure 2. Examples images from the RIVAL10 benchmark. The figure contains two examples of images from the \rivalbenchmark. The first image shows a horse's muzzle, taken from a short distance. The horse's head covers the rest of the body. The image is labeled with the Equine class. The second image shows the interior of an airplane taken from a window seat. The image is labeled with the Plane class.
Figure 3. Example of decision tree extracted by FGA. Example of a decision tree extracted by FGA. The decision tree in question is complete, with three layers and four leaf nodes. Each non-leaf node contains the identifier of a neuron, and each branch contains a condition of the activation layer of that neuron. Each leaf node contains a clause specifying whether the concept is present or not, and the number of elements in the dataset that are represented by that leaf node.
Layer
Output Size
# Params
Input
224 × 224 × 3
0
Conv1_1
224 × 224 × 64
1’792
Conv1_2
224 × 224 × 64
36’928
MaxPool
112 × 112 × 64
0
Conv2_1
112 × 112 × 128
73’856
Conv2_2
112 × 112 × 128
147’584
Table 1. Architecture and number of parameters of VGG-19.
Size
Effectiveness
Iteration
FC1
FC2
Params
Size
Accuracy
Precision
Recall
F1-score
(neurons)
(neurons)
(M)
(MB)
(%)
(%)
(%)
(%)
VGG-19
4096
4096
143.67
574.70
84.79
90.74
77.10
83.22
1
2622
2357
94.35
377.42
81.71
90.73
74.31
81.48
10
1241
1088
53.60
214.43
78.34
90.60
71.29
79.32
20
1007
903
47.10
188.44
76.49
90.53
69.62
78.03
Table 2. Size and effectiveness of the pruned DNN generated by CBP.
Active Neurons
Computational Complexity
Inference Efficiency
Iteration
FC1
FC2
MACs FC1
MACs FC2
Total MACs
Latency
Std.
FPS
(neurons)
(neurons)
(M)
(M)
(G)
(ms)
( ± ms)
VGG-19
4096
4096
102.76
16.78
19.668
13.35
0.25
74.91
1
2622
2357
65.78
6.18
19.619
12.27
0.19
81.50
10
1241
1088
31.13
1.35
19.578
11.14
0.17
89.77
20
1007
903
25.26
0.91
19.571
10.96
0.05
91.24
Table 3. Computational cost and runtime analysis on VGG-19’s first two FC layers.
Iteration
Neurons Identifier (s)
Pruner (s)
Total (s)
0
778.25
1.81
780.07
10
383.04
0.92
383.96
20
358.11
0.91
359.01
30
351.61
0.86
352.47
40
346.72
1.01
347.73
50
345.43
0.86
346.29
Table 4. Time required by CBP and its components.
Size
Effectiveness
Iteration
FC1
FC2
Params
Size
Accuracy
Precision
Recall
F1-score
(neurons)
(neurons)
(M)
(MB)
(%)
(%)
(%)
(%)
VGG-19
4096
4096
143.67
574.70
84.79
90.74
77.10
83.22
1
1038
766
47.63
190.55
70.81
90.73
64.28
74.42
2
798
562
41.06
164.26
61.84
90.75
56.04
67.10
3
720
463
38.89
155.57
56.75
90.61
51.40
62.63
Table 5. Size and effectiveness of the pruned DNN generated by CBP including misclassified samples.
Figure 4. Size and effectiveness of the pruned DNN when misclassified inputs are discarded or considered by CBP. The figure presents two plots, one next to the other. In the plot on the left, the x-axis shows the number of iterations (from 0 to 20), and the y-axis shows the number of parameters in millions (from 0 to 150). The plot shows two lines: a blue line for the original \fgpand a red line for the modified version of \fgpwith misclassified images. The plot shows that the original \fgppreserves slightly more parameters than the version with misclassified images for all iterations, with a difference of approx. 10 million parameters in the last iteration. In the plot on the right, the x-axis shows the number of iterations (from 0 to 20), and the y-axis shows the accuracy (from 35% to 90%). As for the other plot, we are comparing the original \fgpapproach (blue) with the alternative version that also uses misclassified inputs (red). The plot shows that the original \fgpsignificantly outperforms the modified version, with an accuracy difference of almost 40% at the last iteration.
Figure 5. Accuracy of different EFGA aggregation policies. The plot shows the performances of different \efgaaggregation policies. The x-axis shows the number of iterations (0-25), and the y-axis shows the accuracy (0-90%). The plot shows ten different lines: nine for the different \efgaaggregation policies (TOP(1), TOP(3), TOP(5), TOP(10), REC(80), REC(85), REC(90), REC(95), and AVG) and one for the original \fgpapproach. For most aggregation approaches, the accuracy degrades with only a few iterations, reaching 0%: TOP(1), TOP(3), TOP(5), TOP(10), and AVG. For REC(80), REC(85), and REC(90), the accuracy never reaches 0%, but they perform significantly worse than the original approach. Only REC(95) shows a similar behaviour to \fgp, and it even outperforms it slightly in some iterations.
Size
Effectiveness
Iteration
FC1
FC2
Params
Size
Accuracy
Precision
Recall
F1-score
(neurons)
(neurons)
(M)
(MB)
(%)
(%)
(%)
(%)
VGG-19
4096
4096
143.67
574.70
84.79
90.74
77.10
83.22
1
1349
1070
56.38
225.57
80.70
90.38
73.43
80.64
5
789
628
40.94
163.81
77.62
90.11
70.65
78.38
10
636
527
36.84
147.41
78.11
89.87
71.10
78.45
Table 6. Size and effectiveness of the pruned DNN generated by CBP with the criterion REC(95).
Figure 6. Size and effectiveness of the pruned DNN for the CBP default configuration and CBP with EFGA and REC(95). The figure presents two plots, one next to the other. In the plot on the left, the x-axis shows the number of iterations (from 0 to 25), and the y-axis shows the number of parameters in millions (from 0 to 150). The plot shows two lines: a blue line for the original \fgpand a red line for \efgawith REC(95). The plot shows that the original \fgppreserves slightly more parameters than the REC(95) version at all iterations, with a difference of approx. 10 million parameters in the last iteration. In the plot on the right, the x-axis shows the number of iterations (0-25), and the y-axis shows the accuracy (70-90%). As for the other plot, we are comparing the original \fgpapproach (blue) with \efgawith REC(95) (red). The plot shows that the behaviour of the two approaches is very similar, with \fgpslightly outperforming REC(95) for most iterations.