As the need for more accurate and powerful Convolutional Neural Networks (CNNs) increases, so too does the size, execution time, memory footprint, and power consumption. To overcome this, solutions such as pruning have been proposed with their own metrics and methodologies, or criteria, for how weights should be removed. These solutions do not share a common implementation and are difficult to implement and compare. In this work, we introduce Combine, a criterion- based pruning solution and demonstrate that it is fast and effective framework for iterative pruning, demonstrate that criterion have differing effects on different models, create a standard language for comparing criterion functions, and propose a few novel criterion functions. We show the capacity of these criterion functions and the framework on VGG inspired models, pruning up to 79% of filters while retaining or improving accuracy, and reducing the computations needed by the network by up to 68%.
Figures & tables
Figure 1: Criterion performance in Combine procedure on the Balanced network trained on MNIST Data
Model
MNIST
CIFAR-10
Convolutional Heavy (A)
99.04%
77.14%
Balanced (B)
99.07%
73.26%
Dense Heavy (C)
98.00%
49.72%
Table 1: Model top-1 accuracy of various models on CIFAR-10 and MNIST
Figure 2: Criterion performance in Combine procedure on the Balanced network trained on CIFAR-10 Data
Figure 3: Computation requirement improvements on Balanced model on MNIST data with pruning only dense layers, convolution layers, or both.
Figure 4: VGG inspired model architectures
Pruned
Model
SD
Max Abs
Mean Abs
Abs Range
Both
A
0.860*
0.245
0.855
0.219
B
0.824*
0.244
0.805
0.324
C
0.806*
0.781
0.776
0.778
Conv
A
0.879*
0.232
0.870
0.230
B
0.801
0.230
0.789
0.182
C
0.045
0.036
0.061*
0.042
Table 2: Combine Performance across criterion frunction on MNIST trained models. * denotes best criterion function for the model and layer selection, and bold denotes the best criterion function and layer selection for the model.
Pruned
Model
SD
Max Abs
Mean Abs
Abs Range
Both
A
0.293
0.192
0.343*
0.177
B
0.431
0.239
0.474*
0.242
C
0.166
0.155
0.167*
0.150
Conv
A
0.225
0.156
0.275*
0.158
B
0.363
0.185
0.425*
0.225
C
0.044
0.031*
0.043
0.038
Table 3: Combine Performance across criterion frunction on CIFAR-10 trained models. * denotes best criterion function for the model and layer selection, and bold denotes the best criterion function and layer selection for the model.
Model
Original
Post-prune/retrain
% Pruned
A
99.04%
99.33%
79.00%
B
99.07%
98.73%
76.52%
C
98.00%
97.90%
70.83%
Table 4: Combine top-1 accuracy across models trained on MNIST after selecting a criterion function, pruning, and retraining the pruned model
Model
Original
Post-prune/retrain
% Pruned
A
77.14%
78.36%
16.55%
B
73.26%
79.11%
38.76%
C
49.72%
50.32%
35.72%
Table 5: Combine top-1 accuracy across models trained on CIFAR-10 after selecting a criterion function, pruning, and retraining the pruned model
Model
Original
Post-prune
% Reduction
A
105,595,549
32,973,437
68.77%
B
107,757,094
37,178,977
65.50%
C
752,919
330,418
56.12%
Table 6: Combine Performance across models trained on MNIST after selecting a criterion function, pruning, and retraining the pruned model
Model
Original
Post-prune
% Reduction
A
158,355,901
120,584,938
23.84%
B
166,591,782
98,209,450
41.05%
C
953,727
772,011
19.05%
Table 7: Combine Performance across models trained on CIFAR-10 after selecting a criterion function, pruning, and retraining the pruned model
Department of Mathematics Pennsylvania State University University Park, PA 16802, USA · Department of Computing and Mathematical Sciences California Institute of Technology Pasadena, CA 91125, USA