Organizations: School of Electronics and Communication Engineering, Shenzhen Campus, Sun Yat-Sen University, Shenzhen 518107, China. · School of Information Engineering, Guangdong University of Technology, Guangzhou 510006, China. · School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China. · Faculty of Electrical Engineering and Computer Science, Ningbo University, Ningbo 315211, China. · College of Computing and Data Science, Nanyang Technological University, Singapore
Traditional human vision-centric image compression methods are suboptimal for machine vision centric compression due to different visual properties and feature characteristics. To address this problem, we propose a Channel Importance-driven learned Image Coding for Machines (CI-ICM), aiming to maximize the performance of machine vision tasks at a given bitrate constraint. First, we propose a Channel Importance Generation (CIG) module to quantify channel importance in machine vision and develop a channel order loss to rank channels in descending order. Second, to properly allocate bitrate among feature channels, we propose a Feature Channel Grouping and Scaling (FCGS) module that non-uniformly groups the feature channels based on their importance and adjusts the dynamic range of each group. Based on FCGS, we further propose a Channel Importance-based Context (CI-CTX) module to allocate bits among feature groups and to preserve higher fidelity in critical channels. Third, to adapt to multiple machine tasks, we propose a Task-Specific Channel Adaptation (TSCA) module to adaptively enhance features for multiple downstream machine tasks. Experimental results on the COCO2017 dataset show that the proposed CI-ICM achieves BD-mAP@50:95 gains of 16.25% in object detection and 13.72% in instance segmentation over the established baseline codec. Ablation studies validate the effectiveness of each contribution, and computation complexity analysis reveals the practicability of the CI-ICM. This work establishes feature channel optimization for machine vision-centric compression, bridging the gap between image coding and machine perception.
Figures & tables
Fig. 1: Feature channel importance analysis by adding distortions, where mAP@50:95 was used as precision metric for object detection. (a) zero out for individual channel, (b) zero out for a group of channels, (c) distortions added to individual channel, where “Intensity 1” and “Intensity 2” indicated the magnitudes of the added random noise ranges [−0.5,0.5] and [−1,1] , respectively, (d) distortions added to a group of channels.
Fig. 2: Framework of the proposed CI-ICM. (a) Network architecture of the proposed CI-ICM, (b) The reordering, grouping, and scaling feature flow of y , where darker green y blocks are features with higher importance for machine vision.
Fig. 3: Relationship between the number of removed channels and the normalized object detection task accuracy. (a) Normalized mAP@50:95, (b) Normalized mAP@75.
Fig. 4: Decoding process of the CI-CTX module.
Fig. 5: MSE(Φch-orgi,Φchi) with different si . Note that si , i∈[1,2,3] are plotted in 1/si , and s4 are plotted in log10(s4) for better observation. (a) s1 , (b) s2 , (c) s3 , (d) s4 .
Fig. 6: Object task precisions using different si , where quadratic functions are used to fit results. Note that si , i∈[1,2,3] are plotted in 1/si , and s4 are plotted in log10(s4) for better observation. (a) s1 , (b) s2 , (c) s3 , (d) s4 .
Fig. 7: Structure of the TSCA module.
Fig. 8: Three stages of training for the proposed CI-ICM, where dark green y blocks are features with higher importance for machine vision, and dark blue Wc blocks are the higher importance weights.
Fig. 9: Trained Wc of an image in COCO2017 before and after training using channel order loss. (a) Before training, (b) After training.
Fig. 10: Rate-accuracy curves of the proposed CI-ICM and benchmark schemes on object detection task, (a) mAP@50:95, (b) mAP@50, (c) mAP@75.
Fig. 11: Visualization of object detection results using different codecs, where bpp values are also presented. (a) and (f): Ground truth, (b) and (g): ELIC, (c) and (h): TransTIC, (d) and (i): AdaptICMH, (e) and (j): the proposed CI-ICM, (a)-(e): image of a person and skis, (f)-(j): image of an oven and a refrigerator.
Coding schemes
BD-mAP@50:95
BD-mAP@50
BD-mAP@75
ELIC [ 36 ]
0
0
0
JPEG [ 42 ]
-6.798
-9.933
-7.729
VTM [ 4 ]
0.171
-0.656
0.511
MLIC [ 38 ]
1.128
1.543
1.418
CF [ 20 ]
3.783
2.914
3.555
SC2 [ 21 ]
8.745
10.348
8.697
TABLE I: BD-mAPs( % ) of the proposed CI-ICM and benchmark schemes for object detection task, where ELIC [ 36 ] is the baseline.
Fig. 12: Rate-accuracy curves of the proposed CI-ICM and baseline schemes on instance segmentation task, (a) mAP@50:95, (b) mAP@50, (c) mAP@75.
Fig. 13: Visualization of instance segmentation results from coded images, where bpp values also presented. (a) and (f): Ground truth, (b) and (g): ELIC, (c) and (h): TransTIC, (d) and (i): AdaptICMH, (e) and (j): the proposed CI-ICM, (a)-(e): image of person and bicycles, (f)-(j): image of a girl under an umbrella.
Coding schemes
BD-mAP@50:95
BD-mAP@50
BD-mAP@75
ELIC [ 36 ]
0
0
0
JPEG [ 42 ]
-5.969
-9.150
-6.786
VTM [ 4 ]
0.542
0.165
0.562
MLIC [ 38 ]
1.233
1.654
1.299
TransTIC [ 33 ]
10.208
13.376
11.535
AdaptICMH [ 34 ]
11.282
15.221
12.591
TABLE II: BD-mAPs( % ) of the proposed CI-ICM and benchmark schemes as comparing with the ELIC [ 36 ] for instance segmentation task.
Fig. 14: Rate-accuracy curves of ablation studies on object detection task and instance segmentation task, where the task accuracy is measured with mAP@50:95. (a)-(b) analysis on scaling process in FCGS, CI-CTX, CIG, and channel order loss, (c)-(d) analysis on TSCA modes. (a)(c) object detection, and (b)(d) instance segmentation.
Tasks
Coding schemes
BD-mAP@50:95
BD-mAP@50
BD-mAP@75
Object detection
w/o Scaling
14.566
18.834
16.494
w/o CI-CTX
11.838
15.882
13.723
w/o CO
12.742
16.824
14.619
CI-ICM
16.249
20.924
18.491
Instance segmentation
w/o Scaling
12.650
17.464
14.101
w/o CI-CTX
10.251
13.797
11.545
TABLE III: BD-mAPs( % ) of the ablation studies of the CI-ICM for object detection and instance segmentation, where ELIC [ 36 ] is the anchor.
Tasks
Mode
BD-mAP@50:95
BD-mAP@50
BD-mAP@75
Object detection
Unmatched
12.672
16.938
14.516
Matched
16.249
20.924
18.491
Instance segmentation
Unmatched
12.778
18.646
13.984
Matched
13.718
19.028
15.244
TABLE IV: BD-mAPs( % ) of using TSCA with “matched” and “unmatched” mode for object detection and instance segmentation, where ELIC [ 36 ] is used as anchor.
Fig. 15: Rate-accuracy curves of generalization studies, where the task accuracy is measured with mAP@50:95. (a) Analysis on COCO 2017 dataset, Faster R-CNN with ResNet-101 backbone for object detection. (b) Analysis on Pascal VOC 2012 dataset, Faster R-CNN with ResNet-50 backbone for object detection. (c) Analysis on COCO 2017 dataset, Mask R-CNN with ResNet-101 backbone for instance segmentation. (d) Analysis on Cityscapes dataset, Mask R-CNN with ResNet-50 backbone for instance segmentation.
Coding schemes
Object detection
Instance segmentation
ResNet-101
Pascal VOC
ResNet-101
Cityscapes
ELIC [ 36 ]
0
0
0
0
JPEG [ 42 ]
-7.130
-11.267
-6.393
-3.934
VTM [ 4 ]
-2.190
-3.910
-2.076
0.686
MLIC [ 38 ]
1.207
2.112
1.025
2.239
TransTIC [ 33 ]
9.277
15.294
7.491
2.507
TABLE V: BD-mAP@50:95s( % ) of the proposed CI-ICM and benchmark schemes for different datasets and feature extraction network, where ELIC [ 36 ] is used as the anchor.
Coding schemes
Coding time (s)
FLOPs (G)
Params. (M)
Encoding
Decoding
ELIC [ 36 ]
2.722
0.124
871.420
31.656
MLIC [ 38 ]
2.133
0.203
1317.803
116.481
CF [ 20 ]
2.746
0.251
180.672
34.755
SC2 [ 21 ]
2.401
0.169
151.540
34.014
TransTIC [ 33 ]
2.311
0.175
529.876
9.028
TABLE VI: FLOPs and number of parameters comparison between the proposed CI-ICM and learning-based methods
Visual data is increasingly consumed by machine-vision systems rather than by human observers. Image Coding for Machines (ICM) compresses images assuming the main observer is a computer vision application and that the human observer needs to inspect or validate the decisions. Inspired by just-noticeable distortion, we cap human-observed quality at a desired level and devote the remaining bits to machine performance. Specifically, joint compression-segmentation training is recast as a constrained optimization problem in which the codec must meet a predefined acceptable target visual quality while a task term consumes the remaining coding capacity. This paper proposes two variants of a penalty function that guides the quality toward the target: an absolute function and a bilinear function, the latter applying a steeper slope once the target visual quality is exceeded. Experimental results show that, under the quality constraint, the proposed method achieves BD-rates of −22.82% and −29.81% relative to an unconstrained joint rate--distortion--task optimization and a simple rate--distortion baseline, respectively, showcasing bitrate reduction with the same task performance. This is achieved while the codec also meets the target visual quality with a reasonable error and without adding any complexity overhead.
Image compression for machines calls for a unified codec that serves multiple downstream vision tasks. Existing approaches either adopt task-specific end-to-end designs, raising parameter and deployment overhead, or rely on transfer-based adaptations that remain externally attached and heuristic task design. A key limitation shared by both lines of work is their largely static computation pattern, which applies similar transformations across tokens despite the fact that different image regions exhibit markedly different semantic importance and complexity for machine perception. We propose MoECodec, a token-aware image compression framework that supports multiple downstream tasks within a single model. MoECodec replaces the FFN layers in transformer-based compression model token-wise Mixture-of-Experts (MoE), enabling dynamic, token-level computation conditioned on the input content and task objective. To make MoE effective in compression model, we introduce a stable routing strategy that combines expert-choice routing with spatial total variation regularization to encourage spatially coherent assignments, and we propose a lightweight expert architecture, Group Shuffle MLP (GShMLP), to control parameter growth. Extensive experiments show consistent improvement against baselines on both conventional image reconstruction and machine tasks.
Parameter-efficient fine-tuning of pre-trained codecs is a promising direction in image compression for human and machine vision. While most existing works have primarily focused on tuning the feature structure within the encoder-decoder backbones, the adaptation of the statistical semantics within the entropy model has received limited attention despite its function of predicting the probability distribution of latent features. Our analysis reveals that naive adapter insertion into the entropy model can lead to suboptimal outcomes, underscoring that the effectiveness of adapter-based tuning depends critically on the coordination between adapter type and placement across the compression pipeline. Therefore, we introduce Structure-Semantics Co-Tuning (S2-CoT), a novel framework that achieves this coordination via two specialized, synergistic adapters: the Structural Fidelity Adapter (SFA) and the Semantic Context Adapter (SCA). SFA is integrated into the encoder-decoder to preserve high-fidelity representations by dynamically fusing spatial and frequency information; meanwhile, the SCA adapts the entropy model to align with SFA-tuned features by refining the channel context for more efficient statistical coding. Through joint optimization, S2-CoT turns potential performance degradation into synergistic gains, achieving state-of-the-art results across four diverse base codecs with only a small fraction of trainable parameters, closely matching full fine-tuning performance. Code is available at https://github.com/Brock-bit4/S2-CoT.