IGDA-ICMH: an Importance Guided Feature and Network Dual Adaptation Framework towards Image Compression for Machine and Human Vision
Authors: Yanbo Gao, Xinyu Zhao, Shuai Li, Xun Cai, Mao Ye, Li Li
Organizations: School of Software, Shandong University, Jinan 250100, China, and also with Shandong University-WeiHai Research Institute of Industrial Technology, Weihai 264209, China. · School of Control Science and Engineering, Shandong University, Jinan 250100, China. · University of Electronic Science and Technology of China, Sichuan, China. · University of Science and Technology of China (USTC), Hefei, Anhui, China.
With the surge of machine vision applications, image compression for both machine and human vision (ICMH) has become an important research area. Existing studies mainly focus on task-specific adaptation or multi-task joint optimization, yet they often overlook the unequal contributions of different features and network layers to downstream machine vision tasks. To address this issue, we first quantitatively investigate the impact of different feature and network layers on machine vision performance, providing a principled understanding of their importance. Based on this analysis, an Importance Guided feature and network Dual Adaptation (IGDA) framework is developed for ICMH. It adopts implicit neural representation as the basic architecture and designs an efficient adaptation structure with both feature and network weight adaptation. An Importance-Guided Network Adaptation (IGNA) module is developed with a low-rank adaptation strategy to selectively adjust key network layers according to their importance in different tasks. An Importance-Guided Feature Adaptation (IGFA) module is developed with a low-rank and Kronecker product-based transform to explore the non-local property of the features in order to reduce the number of adaptation parameters. Extensive experiments on representative machine vision tasks, including image classification and object detection, demonstrate that our IGDA-ICMH framework achieves significantly better performance than the existing implicit neural representation-based methods for ICMH.
Figures & tables
Fig. 1: Illustration of total accuracy improvement per layer and average accuracy improvement per parameter, indicating different importance of the layers and features to the final performance.
Fig. 2: Overview of the proposed IGDA-ICMH framework, with IGFA to adapt the feature grids of different resolutions and IGNA to adapt different network layers. Importance of different features and network layers are estimated and then used to guide the dual adaptation.
Fig. 3: Rate-Accuracy performance comparison under different machine vision tasks including image classification and object detection on different datasets.
Fig. 4: Rate-distortion curves on (a) Oxford 102 Flowers and (b) Animals with Attributes 2, respectively.
Method
BD-rate(%)
BD-mAP(%)
COOL-CHIC [ 7 ]
0
0
CD-ICMH [ 75 ]
11.97
-0.84
ICMH-net [ 75 ]
-2.39
0.14
TransTIC [ 53 ]
-43.31
2.85
Adapt-ICMH [ 6 ]
-52.97
2.97
Ours
-65.50
3.20
TABLE I: Performance comparison in terms of BD-rate and BD-mAP on COCO2017 dataset.
Method
Decoding Speed
COOL-CHIC [ 7 ]
149.48
Adapt-ICMH [ 6 ]
37.97
MPA [ 76 ]
95.97
Ours
94.52
TABLE II: Comparison of decoding speed in terms of FPS on Oxford 102 Flowers dataset.
Method
MAC/pixel
Encoding time(s)
Param. count
Trainable param
COOL-CHIC [ 7 ]
1435
202.77
88713
88713
Ours
2340
86.07
89686
973 (1.1%)
TABLE III: Comparison of encoding complexity on the Oxford 102 Flowers. Param. count denotes number of total parameters.
Method
Trainable param
Bpp
Accuracy (%)
baseline
-
0.202
67.74
L2 (r=2)
102
0.209
89.29
L3 (r=2)
108
0.209
87.74
L4 (r=2)
108
0.209
89.41
SL(r=2)
110
0.209
90.58
SL(r=4)
220
0.213
91.61
TABLE IV: Ablation results of the proposed IGNA.
Method
Trainable param
Bpp
Accuracy (%)
baseline
-
0.202
67.74
L1 (r=1)
512
0.221
91.09
L2 (r=2)
512
0.221
90.96
L3 (r=2)
256
0.214
91.09
L5 (r=2)
64
0.205
67.74
L6 (r=2)
32
0.203
67.74
TABLE V: Ablation results of the proposed IGFA based on Kronecker product.
Fig. 5: Visualization of features for image classification. For each example, the left image is the ground-truth image, where the red and orange boxes indicate two selected regions. The right images present the corresponding feature visualizations produced by COOL-CHIC and our method, respectively, where the background features are simplified and important features are preserved.
Method
Trainable param
Bpp
Accuracy (%)
baseline
-
0.375
77.29
IGNA
1020
0.406
96.64
IGFA
1168
0.410
96.90
IGFA+IGNA
1151
0.410
96.64
IGDA
973
0.402
97.80
TABLE VI: Results of the IGDA, in comparison with only using IGFA, IGNA, and simply combining them as IGFA+IGNA.
School of Electronics and Communication Engineering, Shenzhen Campus, Sun Yat-Sen University, Shenzhen 518107, China. · School of Information Engineering, Guangdong University of Technology, Guangzhou 510006, China. · School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China. +2