Structured Channel Network Pruning with Bi-Sparse Matrix Factorization
Authors: Jingjing Liu, Ziyang Wang, Xianchao Xiu, Wanquan Liu
Organizations: School of Microelectronics, Shanghai Key Laboratory of Chips and Systems for Intelligent Connected Vehicle, Shanghai University, Shanghai 200444, China · State Key Laboratory of Integrated Chips and Systems, Fudan University, Shanghai 201203, China · School of Mechatronic Engineering and Automation, Shanghai University, Shanghai 200444, China · School of Intelligent Systems Engineering, Sun Yat-sen University, Shenzhen 510275, China
Structured channel pruning for convolutional neural networks (CNNs) plays an important role in pattern recognition and deep learning. Nevertheless, existing methods usually calculate channel importance scores based on the magnitudes of individual filters, making it difficult to distinguish shared redundancy from individual channel contributions. To tackle this limitation, we propose a novel structured channel pruning method termed bi-sparse matrix factorization (BSMF). Specifically, BSMF characterizes the convolutional weight matrix as output-channel coefficients over a shared basis with row-wise ℓ2,p-norm and column-wise ℓ2,q-norm regularization terms, respectively. Note that p,q∈[0,1), which, to our best knowledge, has not been explored in the literature. To solve the resulting non-convex optimization problem, we develop an efficient update scheme built upon the alternating direction method of multipliers (ADMM), in which all subproblems admit closed-form solutions. After that, a compact CNN is obtained by reconstructing the convolutional filters from the learned factors and discarding the redundant channels. Extensive experiments on the CIFAR-10 and ImageNet-1K datasets demonstrate that our proposed BSMF achieves significant network compression while maintaining high reconstruction accuracy. The source code is publicly available at https://github.com/xianchaoxiu/BSMF
Figures & tables
Figure 1: Comparison between our BSMF and CSMF [ 10 ] , where BSMF scores channels using the row-wise norms of ZA , while CSMF co-synthesizes filters using fused gradients and individual weight increments.
Model
Method
Top-1 Accuracy
FLOPs (PR)
Params (PR)
ResNet-56
Baseline
93.26%
127.09M (0.0%)
0.85M (0.0%)
ℓp [ 5 ]
93.87%
75.62M (40%)
0.56M (34%)
CORING [ 18 ]
93.79%
76.12M (40%)
0.47M (45%)
APIB [ 26 ]
93.92%
76.35M (40%)
0.53M (37%)
BSMF (Ours)
94.18%
75.92M (40%)
0.44M (48%)
HRank [ 24 ]
92.49%
62.72M (50%)
0.49M (42%)
Table 1: Pruning results for ResNet-56 and ResNet-110 on the CIFAR-10 dataset, where the best and second-best pruned values are shown in bold and underlined, respectively.
Model
Method
Top-1 Accuracy
FLOPs (PR)
Params (PR)
VGG-16
Baseline
93.96%
313.73M (0%)
14.98M (0%)
LeanFlex-GKP [ 31 ]
94.15%
178.17M (43%)
8.29M (44%)
GAL-0.1 [ 30 ]
93.42%
171.89M (45%)
2.67M (82%)
HRank [ 24 ]
93.43%
145.61M (54%)
2.51M (83%)
DFPV [ 19 ]
93.76%
139.30M (56%)
3.25M (78%)
IPPKD [ 29 ]
94.24%
139.36M (56%)
6.89M (54%)
Table 2: Pruning results for VGG-16 on the CIFAR-10 dataset, where the best and second-best pruned values are shown in bold and underlined, respectively.
Model
Method
Top-1 Accuracy
Top-5 Accuracy
FLOPs (PR)
Params (PR)
ResNet-50
Baseline [ 38 ]
76.15%
92.87%
4.12B (0.0%)
25.50M (0.0%)
LeanFlex-GKP [ 31 ]
75.62%
N/A
2.76B (33%)
17.80M (30%)
GAL-0.5 [ 30 ]
71.95%
90.94%
2.33B (43%)
21.20M (17%)
ABP [ 28 ]
74.80%
92.36%
2.36B (43%)
13.01M (49%)
HRank-1 [ 24 ]
74.98%
92.33%
2.30B (44%)
16.15M (37%)
ℓ0 Pruner [ 20 ]
75.18%
92.39%
2.27B (45%)
13.39M (48%)
Table 3: Pruning results for ResNet-50 on the ImageNet-1K dataset, where the best and second-best pruned values are shown in bold and underlined, respectively.
Method
Factor.
Sparse opt.
Channel score
Top-1 Accuracy (%)
FLOPs Red.
Params
Baseline
No
No
None
93.26
0%
0.85M
BSMF without pruning
Yes
Yes
None
93.09±0.02
0%
0.85M
ℓ2 pruning
No
No
filter ℓ2 -norm
93.65±0.23
40%
0.55M
BSMF + filter ℓ2
Yes
Yes
filter ℓ2 -norm
93.74±0.10
40%
0.52M
BSMF + filter ℓ1
Yes
Yes
filter ℓ1 -norm
93.91±0.09
40%
0.51M
Factorization-only pruning
Yes
No
A -row norm
93.83±0.18
40%
0.44M
Table 4: Component ablation results for ResNet-56 on the CIFAR-10 dataset, where the best and second-best pruned values are shown in bold and underlined, respectively.
Figure 2: Channel-selection heatmaps for ResNet-56, where (a)-(c) show baseline filter ℓ2 -norms, BSMF A -row scores, and retention masks, respectively.
Figure 3: Pairwise Jaccard similarity of retained-channel sets on the CIFAR-10 dataset, where the top and bottom rows show ResNet-56 and ResNet-110, respectively.
Figure 4: Convergence curves of BSMF on different models and datasets, where (a)-(d) show the objective function values.
Figure 5: Sensitivity to p and q on the CIFAR-10 dataset, where (a,b) show mean best-checkpoint Top-1 accuracy over three seeds for ResNet-56 and ResNet-110, respectively.
Figure 6: Channel counts before and after pruning, where counts are shown per VGG-16 layer and summed over the main-branch convolutions of each ResNet block.
Figure 7: Weight distributions, where (a)-(d) show original weights and (e)-(h) show effective weights constructed using normalized A -row scores.
Inria, Paris, France · ´Ecole polytechnique, Institut Polytechnique de Paris, Palaiseau, France · CISPA Helmholtz Center for Information Security, Saarbrücken, Germany +3