Big2Small: A Unifying Neural Network Framework for Model Compression
Authors: Jing-Xiao Liao, Haoran Wang, Tao Li, Daoming Lyu, Yi Zhang, Chengjun Cai, Feng-Lei Fan
Organizations: Frontier of Artificial Networks (FAN) Lab, Department of Data Science, City University of Hong Kong, Kowloon, Hong Kong SAR, China · Hong Kong Applied Science and Technology Research Institute (ASTRI) · Sichuan Engineering Research Center for Big Data Visual Analytics, Sichuan University of Science and Engineering, Zigong, China · School of Cyber Science and Engineering, Sichuan University, Chengdu, China · Key Laboratory of Data Protection and Intelligent Management, Ministry of Education, Sichuan University, Chengdu, China · Computer Science and Information Technology Center, City University of Hong Kong (Dongguan), Dongguan, China
With the development of foundational models, model compression has become a critical requirement. Various model compression approaches have been proposed such as low-rank decomposition, pruning, quantization, ergodic dynamic systems, and knowledge distillation, which are based on different heuristics. To elevate the field from fragmentation to a principled discipline, we construct a unifying mathematical framework for model compression grounded in measure theory. We further demonstrate that each model compression technique is mathematically equivalent to a neural network subject to a regularization. Building upon this mathematical and structural equivalence, we propose an experimentally-verified data-free model compression framework, termed Big2Small, which translates Implicit Neural Representations (INRs) from data domain to the domain of network parameters. Big2Small trains compact INRs to encode the weights of larger models and reconstruct the weights during inference. To enhance reconstruction fidelity, we introduce Outlier-Aware Preprocessing to handle extreme weight values and a Frequency-Aware Loss function to preserve high-frequency details. Experiments on image classification and segmentation demonstrate that Big2Small achieves competitive accuracy and compression ratios compared to state-of-the-art baselines.
Figures & tables
Feature
Low-Rank Decomposition
Quantization
Pruning
Ergodic Dynamic Systems
Knowledge Distillation
Primary Goal
Matrix Factorization
Precision Reduction
Redundancy Removal
Trajectory Representation
Behavior Mimicry
Sparsity Type
Structured
Dense
Unstructured or Structured
Continuous
Dense
Core Operations
SVD, CP, Tucker
Integer Mapping
Sparsity Induction
Ergodic Hyperfunctions
Soft Target Matching
Data Reliance
Low (Often Data-free)
Moderate (Calibration)
High (Fine-tuning)
Low (Direct Encoding)
Very High (Teacher Output)
TABLE I: Properties of model compression methods.
Fig. 1: The theoretical framework of Theorem 1.
Fig. 2: The framework of Theorem 2.
Fig. 3: An overview of the proposed Big2Small framework. It uses a “Compression–Decompression” architecture that encodes discrete weight parameter tensors with lightweight INRs and reconstructs the original weights at inference time.
Fig. 4: The structure of INR.
Size (MB)
W/A Bits
Compression Ratio
Top1 Acc (%)
ResNet18 [ 63 ]
44.6
32/32
-
71.47
DSG [ 68 ]
8.4
6/6
5.3
70.46
Squant [ 69 ]
8.4
6/6
5.3
70.74
UDFC [ 30 ]
8.4
6/6
5.3
72.76
RieM [ 56 ]
8.4
8/16
5.3
71.80
Big2Small-32bit
30.4
32/32
1.5
71.85
TABLE II: Comparison of different compression methods for three notable image recognition networks on the ImageNet dataset. W/A means the numbers of weight/activation bits.
Fig. 5: Visualizing segmentation results of the original UNet models and the compressed ones.
Size (MB)
W/A bits
mACC(%)
mIOU(%)
UNet [ 62 ]
51.5
32/32
99.52
95.87
RieM [ 56 ]
25.8
16/16
98.52
94.67
Big2Small-32bit
22.9
32/32
99.07
95.32
Big2Small-16bit
11.5
16/16
98.75
94.86
UNet++ [ 73 ]
9.2
32/32
99.01
95.62
RieM [ 56 ]
4.6
16/16
97.15
90.35
TABLE III: Comparison of different compression methods on the Carvana dataset. Where W/A means the numbers of weight/activation bits.
Fig. 6: Comparison of weight reconstruction qualities on the convolutional layer of ResNet50.
Fig. 7: Reconstruction error vs compression ratios on the convolutional layer of ResNet50.
Layer
Strategy
Rel MAE
Cosine
PSNR
w/o pre
0.0832
0.953
29.32
pre
0.0085
1.000
54.21
MSE
0.0672
0.996
34.29
MSE+Grad
0.0421
0.996
35.21
MSE+Freq
0.0351
0.996
35.52
Convolution
Ours
0.0085
1.000
54.21
TABLE IV: Ablation study of the Big2Small using ResNet50. “w/o pr” denotes without using our preprocessing strategy; “Rel MAE” is relative MAE; “Cosine” represents cosine similarity.
Fig. 8: The Q-Q plot of using different loss functions to reconstruct the weight of the linear layer.
Size (MB)
Compression Ratio
Top1 Acc (%)
ResNet18
44.59
-
71.47
Big2Small
30.4
1.5
71.85
Big2Small+SVD
10.32
2.9
70.32
Big2Small+P (60%)
9.12
4.9
69.82
Big2Small+Q (6bit)
7.59
5.9
71.24
TABLE V: Compressing Big2Small on the ImageNet dataset. Where P denotes pruning and Q denotes quantization.
Fig. 9: The throughput of raw and Big2Small models on the ImageNet test set.
Fig. 10: Comparison of different encoders for compressing a convolution weight of ResNet50.
School of Information Science and Engineering, Lanzhou University, Lanzhou, Gansu Province, China · Department of Data Science, City University of Hong Kong, Hong Kong, China SAR · Department of Mathematics, City University of Hong Kong, Hong Kong, China SAR +1