IGDA-ICMH: an Importance Guided Feature and Network Dual Adaptation Framework towards Image Compression for Machine and Human Vision
Authors: Yanbo Gao, Xinyu Zhao, Shuai Li, Xun Cai, Mao Ye, Li Li
Organizations: School of Software, Shandong University, Jinan 250100, China, and also with Shandong University-WeiHai Research Institute of Industrial Technology, Weihai 264209, China. · School of Control Science and Engineering, Shandong University, Jinan 250100, China. · University of Electronic Science and Technology of China, Sichuan, China. · University of Science and Technology of China (USTC), Hefei, Anhui, China.
With the surge of machine vision applications, image compression for both machine and human vision (ICMH) has become an important research area. Existing studies mainly focus on task-specific adaptation or multi-task joint optimization, yet they often overlook the unequal contributions of different features and network layers to downstream machine vision tasks. To address this issue, we first quantitatively investigate the impact of different feature and network layers on machine vision performance, providing a principled understanding of their importance. Based on this analysis, an Importance Guided feature and network Dual Adaptation (IGDA) framework is developed for ICMH. It adopts implicit neural representation as the basic architecture and designs an efficient adaptation structure with both feature and network weight adaptation. An Importance-Guided Network Adaptation (IGNA) module is developed with a low-rank adaptation strategy to selectively adjust key network layers according to their importance in different tasks. An Importance-Guided Feature Adaptation (IGFA) module is developed with a low-rank and Kronecker product-based transform to explore the non-local property of the features in order to reduce the number of adaptation parameters. Extensive experiments on representative machine vision tasks, including image classification and object detection, demonstrate that our IGDA-ICMH framework achieves significantly better performance than the existing implicit neural representation-based methods for ICMH.
Figures & tables
Fig. 1: Illustration of total accuracy improvement per layer and average accuracy improvement per parameter, indicating different importance of the layers and features to the final performance.
Fig. 2: Overview of the proposed IGDA-ICMH framework, with IGFA to adapt the feature grids of different resolutions and IGNA to adapt different network layers. Importance of different features and network layers are estimated and then used to guide the dual adaptation.
Fig. 3: Rate-Accuracy performance comparison under different machine vision tasks including image classification and object detection on different datasets.
Fig. 4: Rate-distortion curves on (a) Oxford 102 Flowers and (b) Animals with Attributes 2, respectively.
Method
BD-rate(%)
BD-mAP(%)
COOL-CHIC [ 7 ]
0
0
CD-ICMH [ 75 ]
11.97
-0.84
ICMH-net [ 75 ]
-2.39
0.14
TransTIC [ 53 ]
-43.31
2.85
Adapt-ICMH [ 6 ]
-52.97
2.97
Ours
-65.50
3.20
TABLE I: Performance comparison in terms of BD-rate and BD-mAP on COCO2017 dataset.
Method
Decoding Speed
COOL-CHIC [ 7 ]
149.48
Adapt-ICMH [ 6 ]
37.97
MPA [ 76 ]
95.97
Ours
94.52
TABLE II: Comparison of decoding speed in terms of FPS on Oxford 102 Flowers dataset.
Method
MAC/pixel
Encoding time(s)
Param. count
Trainable param
COOL-CHIC [ 7 ]
1435
202.77
88713
88713
Ours
2340
86.07
89686
973 (1.1%)
TABLE III: Comparison of encoding complexity on the Oxford 102 Flowers. Param. count denotes number of total parameters.
Method
Trainable param
Bpp
Accuracy (%)
baseline
-
0.202
67.74
L2 (r=2)
102
0.209
89.29
L3 (r=2)
108
0.209
87.74
L4 (r=2)
108
0.209
89.41
SL(r=2)
110
0.209
90.58
SL(r=4)
220
0.213
91.61
TABLE IV: Ablation results of the proposed IGNA.
Method
Trainable param
Bpp
Accuracy (%)
baseline
-
0.202
67.74
L1 (r=1)
512
0.221
91.09
L2 (r=2)
512
0.221
90.96
L3 (r=2)
256
0.214
91.09
L5 (r=2)
64
0.205
67.74
L6 (r=2)
32
0.203
67.74
TABLE V: Ablation results of the proposed IGFA based on Kronecker product.
Fig. 5: Visualization of features for image classification. For each example, the left image is the ground-truth image, where the red and orange boxes indicate two selected regions. The right images present the corresponding feature visualizations produced by COOL-CHIC and our method, respectively, where the background features are simplified and important features are preserved.
Method
Trainable param
Bpp
Accuracy (%)
baseline
-
0.375
77.29
IGNA
1020
0.406
96.64
IGFA
1168
0.410
96.90
IGFA+IGNA
1151
0.410
96.64
IGDA
973
0.402
97.80
TABLE VI: Results of the IGDA, in comparison with only using IGFA, IGNA, and simply combining them as IGFA+IGNA.
Traditional human vision-centric image compression methods are suboptimal for machine vision centric compression due to different visual properties and feature characteristics. To address this problem, we propose a Channel Importance-driven learned Image Coding for Machines (CI-ICM), aiming to maximize the performance of machine vision tasks at a given bitrate constraint. First, we propose a Channel Importance Generation (CIG) module to quantify channel importance in machine vision and develop a channel order loss to rank channels in descending order. Second, to properly allocate bitrate among feature channels, we propose a Feature Channel Grouping and Scaling (FCGS) module that non-uniformly groups the feature channels based on their importance and adjusts the dynamic range of each group. Based on FCGS, we further propose a Channel Importance-based Context (CI-CTX) module to allocate bits among feature groups and to preserve higher fidelity in critical channels. Third, to adapt to multiple machine tasks, we propose a Task-Specific Channel Adaptation (TSCA) module to adaptively enhance features for multiple downstream machine tasks. Experimental results on the COCO2017 dataset show that the proposed CI-ICM achieves BD-mAP@50:95 gains of 16.25% in object detection and 13.72% in instance segmentation over the established baseline codec. Ablation studies validate the effectiveness of each contribution, and computation complexity analysis reveals the practicability of the CI-ICM. This work establishes feature channel optimization for machine vision-centric compression, bridging the gap between image coding and machine perception.
Yun Zhang, Junle Liu, Huan Zhang +3
School of Electronics and Communication Engineering, Shenzhen Campus, Sun Yat-Sen University, Shenzhen 518107, China. · School of Information Engineering, Guangdong University of Technology, Guangzhou 510006, China. · School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China. +2
To reduce deployment cost and retraining overhead, adapting pretrained learned image compression (LIC) models to downstream machine vision tasks has attracted growing attention. However, existing methods typically insert fine-tuning modules independently into frozen backbones, lacking explicit mechanisms for cross-layer coordination. To address this limitation, we propose a novel framework named CrossMambaTuning, which integrates State Space Models with cross-layer interaction mechanisms for parameter-efficient fine-tuning. Specifically, we design an efficient Mamba adapter equipped with task-specific prompts and multi-scale branching to precisely capture both local features and global dependencies. Furthermore, we introduce a Scale-Invariant Cross-Layer Adapter (SICA) utilizing a parameter-sharing strategy to fuse task information across different scales and reduce redundancy. Extensive experiments demonstrate that CrossMambaTuning achieves state-of-the-art (SOTA) performance on multiple machine vision tasks, reducing parameter overhead by 72% compared to SOTA methods. Code is available at https://github.com/rsr1123/CrossMambaTuning.
Haobo Xiong, Shaobo Liu, Kai Liu +1
School of Computer Science and Technology, Xidian University Xi’an, China
Parameter-efficient fine-tuning of pre-trained codecs is a promising direction in image compression for human and machine vision. While most existing works have primarily focused on tuning the feature structure within the encoder-decoder backbones, the adaptation of the statistical semantics within the entropy model has received limited attention despite its function of predicting the probability distribution of latent features. Our analysis reveals that naive adapter insertion into the entropy model can lead to suboptimal outcomes, underscoring that the effectiveness of adapter-based tuning depends critically on the coordination between adapter type and placement across the compression pipeline. Therefore, we introduce Structure-Semantics Co-Tuning (S2-CoT), a novel framework that achieves this coordination via two specialized, synergistic adapters: the Structural Fidelity Adapter (SFA) and the Semantic Context Adapter (SCA). SFA is integrated into the encoder-decoder to preserve high-fidelity representations by dynamically fusing spatial and frequency information; meanwhile, the SCA adapts the entropy model to align with SFA-tuned features by refining the channel context for more efficient statistical coding. Through joint optimization, S2-CoT turns potential performance degradation into synergistic gains, achieving state-of-the-art results across four diverse base codecs with only a small fraction of trainable parameters, closely matching full fine-tuning performance. Code is available at https://github.com/Brock-bit4/S2-CoT.