Organizations: School of Computer, Jiangsu University of Science and Technology, Zhenjiang 212003, China · Department of Electrical and Computer Engineering, University of Nevada, Las Vegas, 89115, USA
We propose Spectral Complex Autoencoder Pruning (SCAP), a reconstruction-based criterion that measures functional redundancy at the level of individual output channels. For each convolutional layer, we construct a complex interaction field by pairing the full multi-channel input activation as the real part with a single output-channel activation (spatially aligned and broadcast across input channels) as the imaginary part. We transform this complex field to the frequency domain and train a low-capacity autoencoder to reconstruct normalized spectra. Channels whose spectra are reconstructed with high fidelity are interpreted as lying close to a low-dimensional manifold captured by the autoencoder and are therefore more compressible; conversely, channels with low fidelity are retained as they encode information that cannot be compactly represented by the learned manifold. This yields an importance score (optionally fused with the filter L1 norm) that supports simple threshold-based pruning and produces a structurally consistent pruned network. On VGG16 trained on CIFAR-10, at a fixed threshold of 0.6, we obtain 90.11% FLOP reduction and 96.30% parameter reduction with an absolute Top-1 accuracy drop of 1.67% from a 93.44% baseline after fine-tuning, demonstrating that spectral reconstruction fidelity of complex interaction fields is an effective proxy for channel-level redundancy under aggressive compression.
Figures & tables
Fig. 1: Overview of SCAP. For each convolutional layer and output channel, SCAP constructs a complex interaction descriptor from the input activation and the resized-and-broadcast output response. The descriptor is first transformed into the frequency domain and then normalized in the spectral domain, with the real and imaginary components processed separately. A layer-specific low-capacity spectral reconstructor, implemented as two component-specific tiny MLP branches shared across input channels, is trained to reconstruct the normalized real and imaginary spectral components. During pruning, SCAP computes the absolute-cosine fidelity and derives the fidelity-based importance term Ifid=1−Fid as a scale-invariant reconstructability cue. This term is fused with a layer-normalized set- ℓ1 magnitude term to obtain the final channel-importance score, which is then used for threshold-based structured pruning followed by fine-tuning.
Dataset
Model
Top-1/5 Acc (%)
FLOPs (M)
Params (M)
CIFAR-10
VGGNet-16
93.44 / –
314.57
14.99
ResNet-56
93.66 / –
130.02
0.88
ResNet-110
92.63 / –
259.49
1.75
DenseNet-40
93.72 / –
292.50
1.06
CIFAR-100
VGGNet-16
71.77 / –
341.62
15.04
ResNet-56
70.82 / –
130.03
0.88
TABLE I: Baseline accuracy, FLOPs, and parameter counts of the reference models. FLOPs are reported as multiply-add operations in millions (M); Params are in millions. For CIFAR-10/100, only Top-1 accuracy is reported. For ImageNet-1k, both Top-1 and Top-5 accuracies are reported.
Model
Dataset
Method
Baseline Acc. (%)
Pruned Acc. (%)
Δ Acc ↓ (%)
FR (%)
PR (%)
VGGNet-16
CIFAR-10
SSS [ 10 ]
93.96
93.02
0.94
41.62
73.77
GAL-0.05 [ 45 ]
93.96
92.03
1.93
39.60
77.60
CP [ 24 ]
93.59
93.18
0.41
39.10
73.30
Zhao et al. [ 46 ]
93.83
93.18
0.65
39.40
73.30
GAL-0.1 [ 45 ]
93.96
90.73
3.23
45.20
82.20
CSHE [ 47 ]
93.96
92.00
1.96
69.00
82.10
TABLE II: The experimental results of VGG-16 on CIFAR-10/100.
Model
Dataset
Method
Baseline Acc. (%)
Pruned Acc. (%)
Δ Acc ↓ (%)
FR (%)
PR (%)
ResNet-56
CIFAR-10
CHIP [ 53 ]
93.26
94.16
-0.90
47.40
42.80
DTP [ 54 ]
93.36
92.46
0.90
72.10
–
DepGraph [ 55 ]
93.53
93.77
-0.24
52.40
–
Li et al. [ 52 ]
93.45
93.07
0.38
41.96
18.93
CCM [ 56 ]
93.75
92.74
1.01
69.90
70.30
CP [ 24 ]
93.26
90.80
2.46
50.60
–
TABLE III: The experimental results of ResNet-56 on CIFAR-10/100.
Model
Dataset
Method
Baseline Acc. (%)
Pruned Acc. (%)
Δ Acc ↓ (%)
FR (%)
PR (%)
ResNet-110
CIFAR-10
l1 -norm [ 1 ]
93.53
93.30
0.23
38.70
32.40
NISP [ 6 ]
93.50
93.32
0.18
43.78
43.25
HRank [ 5 ]
93.50
94.23
-0.73
41.20
39.40
CHIP [ 53 ]
93.50
94.44
-0.94
52.10
48.30
GAL-0.5 [ 45 ]
93.50
92.74
0.76
48.50
44.80
FPGM [ 3 ]
93.68
93.74
-0.06
52.30
–
TABLE IV: The experimental results of ResNet-110 on CIFAR-10/100
Model
Dataset
Method
Baseline Acc. (%)
Pruned Acc. (%)
Δ Acc ↓ (%)
FR (%)
PR (%)
DenseNet-40
CIFAR-10
GAL-0.01 [ 45 ]
94.81
94.29
0.52
35.13
35.58
Li et al. [ 52 ]
94.19
93.00
1.19
43.09
24.03
Zhao et al. [ 46 ]
94.11
93.16
0.95
44.78
59.67
HRank [ 5 ]
94.81
93.68
1.13
60.94
53.85
DCT [ 59 ]
94.81
94.32
0.49
38.50
40.40
Chen et al. [ 17 ]
94.22
93.56
0.66
44.40
60.08
TABLE V: The experimental results of DenseNet-40 on CIFAR-10/100.
Model
Method
Baseline Acc. (%)
Pruned Acc. (%)
Δ Top-1 Acc ↓ (%)
Δ Top-5 Acc ↓ (%)
FR (%)
PR (%)
ResNet-50
ThiNet [ 23 ]
72.88/91.14
72.04/90.67
0.84
0.47
36.79
33.72
SSS [ 10 ]
76.00/92.90
74.03/91.94
1.97
0.96
31.05
27.06
GAL-0.5 [ 45 ]
76.15/92.87
71.95/90.94
4.20
1.93
43.03
16.86
FPGM [ 3 ]
76.15/92.87
75.59/92.63
0.56
0.24
42.20
37.50
HRank [ 5 ]
76.15/92.87
74.76/92.33
1.39
0.54
43.70
36.60
SRR-GR [ 60 ]
76.13/92.86
75.76/92.67
0.37
0.19
44.10
–
TABLE VI: The experimental results of ResNet-50 on ImageNet-1k. Baseline and pruned accuracy are reported as Top-1/Top-5. FR and PR denote FLOP reduction and parameter reduction, respectively.
Network
Method
Pruned Acc. (%)
Δ Acc ↓ (%)
FR / PR (%)
VGG-16 71.77%
ℓ1
63.45
8.32
90.30 / 92.33
FPGM
63.28
8.49
Taylor
61.79
9.98
GAS
61.79
9.98
SCAP
63.98
7.79
DenseNet-40 72.95%
ℓ1
64.28
8.67
83.96 / 83.70
TABLE VII: Budget-matched comparison under aggressive compression on CIFAR-100. Lower Δ Acc indicates better accuracy preservation under the same compression level.
Network
Method
τ
Baseline Acc. (%)
Pruned Acc. (%)
Δ Acc ↓ (%)
FR (%)
PR (%)
ResNet-56
Fidelity-only
0.5
70.82
44.47
26.35
86.07
93.47
SCAP
0.5
70.82
65.36
5.46
70.49
74.70
Fidelity-only
0.6
70.82
37.94
32.88
91.33
96.07
SCAP
0.6
70.82
58.83
11.99
82.47
86.83
ResNet-110
Fidelity-only
0.5
71.26
41.67
29.59
93.47
96.04
SCAP
0.5
71.26
65.71
5.55
80.06
80.01
TABLE VIII: Ablation study on CIFAR-100: fidelity-only scoring versus full SCAP scoring. FR and PR denote FLOP reduction and parameter reduction, respectively. The comparison is conducted under the same pruning threshold τ and fine-tuning protocol.
Fig. 2: Sensitivity of SCAP to the fusion weight α under threshold τ=0.5 on CIFAR-100. Each subfigure reports Top-1 accuracy, FLOP reduction (FR), and parameter reduction (PR) as α varies from 0 to 1 .
Criterion
Model fwd.
Criterion bwd.
Aux. training
Prep. time
ℓ1 magnitude
No
No
No
0.64 s
FPGM
No
No
No
4.60 s
Activation statistics
Yes
No
No
8.49 s
HRank
Yes
No
No
9.69 s
Gradient/Taylor
Yes
Yes
No
8.27 s
SCAP
Yes
Yes
Per-layer AE
6618.36 s
TABLE IX: Preparation overhead comparison on CIFAR-100 under a unified calibration protocol. We report the average preparation time across four backbones: VGG-16, ResNet-56, ResNet-110, and DenseNet-40. Fine-tuning time is excluded. “Model fwd.” indicates whether calibration data must be passed through the pretrained network, and “Criterion bwd.” indicates whether the criterion itself requires back-propagation.
Structured channel pruning for convolutional neural networks (CNNs) plays an important role in pattern recognition and deep learning. Nevertheless, existing methods usually calculate channel importance scores based on the magnitudes of individual filters, making it difficult to distinguish shared redundancy from individual channel contributions. To tackle this limitation, we propose a novel structured channel pruning method termed bi-sparse matrix factorization (BSMF). Specifically, BSMF characterizes the convolutional weight matrix as output-channel coefficients over a shared basis with row-wise ℓ2,p-norm and column-wise ℓ2,q-norm regularization terms, respectively. Note that p,q∈[0,1), which, to our best knowledge, has not been explored in the literature. To solve the resulting non-convex optimization problem, we develop an efficient update scheme built upon the alternating direction method of multipliers (ADMM), in which all subproblems admit closed-form solutions. After that, a compact CNN is obtained by reconstructing the convolutional filters from the learned factors and discarding the redundant channels. Extensive experiments on the CIFAR-10 and ImageNet-1K datasets demonstrate that our proposed BSMF achieves significant network compression while maintaining high reconstruction accuracy. The source code is publicly available at https://github.com/xianchaoxiu/BSMF
Jingjing Liu, Ziyang Wang, Xianchao Xiu +1
School of Microelectronics, Shanghai Key Laboratory of Chips and Systems for Intelligent Connected Vehicle, Shanghai University, Shanghai 200444, China · State Key Laboratory of Integrated Chips and Systems, Fudan University, Shanghai 201203, China · School of Mechatronic Engineering and Automation, Shanghai University, Shanghai 200444, China +1
Overparameterized neural networks often contain many removable neurons, yet what makes a neuron redundant remains poorly understood. Existing pruning criteria commonly rely on local quantities such as weight magnitude, activation strength, or gradient sensitivity, but these measures provide limited insight into the structural role of a neuron in the transformation performed by a layer. Here we show that neuronal redundancy can be characterized by weak participation in the spectral structural distortion induced by layer-wise representation transformations. For each hidden layer of a trained network, we record pre-activation and post-activation hidden states, model neurons as graph nodes, and construct input-side and output-side graphs that describe neuron-level relational structure before and after the layer transformation. We then define a spectral structural importance score that measures the contribution of each neuron to the dominant graph-spectral distortion between these two relational structures. Low-participation neurons are treated as structurally redundant and removed through an iterative pruning process in which scores are recomputed after each structural change. No parameter updates are performed during intermediate pruning rounds; after the target parameter reduction is reached, a single recovery fine-tuning stage is applied to the compact model. Direct ablation analysis and experiments across conventional neural networks, encoder-only Transformers, and decoder-only language models show that this graph-spectral criterion identifies removable neurons and Transformer units while preserving task performance after compression. These results suggest that neural redundancy is not merely a consequence of small weights or weak activations, but can be understood through weak participation in the spectral distortion of layer-wise relational structure.
One-shot magnitude pruning can cause severe accuracy collapse in the high-sparsity regime, even when the pruning mask preserves the largest weights. We argue that this failure reflects a granularity mismatch in post-pruning repair. Under global magnitude pruning, nearly collapsed channels can coexist with channels that retain informative activation variance within the same layer. Existing layer-wise activation repair methods apply a single correction to the whole layer, and can therefore over-amplify damaged channels while trying to restore the layer-level signal. We propose Adaptive Signal Resuscitation (ASR), a training-free channel-wise repair method that matches the granularity of repair to the granularity of damage. ASR estimates a variance-matching correction for each output channel and stabilizes it with a data-driven shrinkage rule, suppressing unreliable corrections for channels with weak post-pruning signal while preserving corrections for healthier channels. Applied before BatchNorm recalibration, ASR requires only forward passes on a small calibration set and no retraining. Across three datasets, four convolutional architectures, and both unstructured and structured sparsity settings, ASR generally improves over layer-wise repair, with the clearest gains in high-sparsity regimes. On ResNet-50 at 90% sparsity, ASR recovers 55.6% top-1 accuracy on CIFAR-10, compared with 41.0% for layer-wise repair and 28.0% for BatchNorm-only recalibration. Ablations show that naive channel-wise variance matching is insufficient, and that shrinkage stabilizes post-pruning repair.
Qishi Zhan, Ziheng Chen, Minxuan Hu
Department of Mathematical and Statistical Sciences, Marquette University, USA · The University of Texas at Austin, USA · Cornell Ann S. Bowers College of Computing and Information Science, Cornell University, USA