What and Where to Adapt: Structure-Semantics Co-Tuning for Machine Vision Compression via Synergistic Adapters

Shaobo Liu; Haobo Xiong; Kai Liu; Yuna Lin

cs.CVApr 11, 2026

What and Where to Adapt: Structure-Semantics Co-Tuning for Machine Vision Compression via Synergistic Adapters

Authors: Shaobo Liu, Haobo Xiong, Kai Liu, Yuna Lin

Organizations: Xidian University

Abstract

Parameter-efficient fine-tuning of pre-trained codecs is a promising direction in image compression for human and machine vision. While most existing works have primarily focused on tuning the feature structure within the encoder-decoder backbones, the adaptation of the statistical semantics within the entropy model has received limited attention despite its function of predicting the probability distribution of latent features. Our analysis reveals that naive adapter insertion into the entropy model can lead to suboptimal outcomes, underscoring that the effectiveness of adapter-based tuning depends critically on the coordination between adapter type and placement across the compression pipeline. Therefore, we introduce Structure-Semantics Co-Tuning (S2-CoT), a novel framework that achieves this coordination via two specialized, synergistic adapters: the Structural Fidelity Adapter (SFA) and the Semantic Context Adapter (SCA). SFA is integrated into the encoder-decoder to preserve high-fidelity representations by dynamically fusing spatial and frequency information; meanwhile, the SCA adapts the entropy model to align with SFA-tuned features by refining the channel context for more efficient statistical coding. Through joint optimization, S2-CoT turns potential performance degradation into synergistic gains, achieving state-of-the-art results across four diverse base codecs with only a small fraction of trainable parameters, closely matching full fine-tuning performance. Code is available at https://github.com/Brock-bit4/S2-CoT.

Figures & tables

Figure 1: Rate-accuracy performance comparison of PEFT methods on the COCO2017 using the Cheng2020-anchor base codec. Our method sets a new state-of-the-art by simultaneously achieving the highest accuracy gains (BD-mAP

\uparrow

) and the largest bitrate savings (BD-Rate

\downarrow

) across both object detection and instance segmentation tasks, significantly outperforming prior works.

Adapter Placement	Adapter Type	Classification		Params $\downarrow$ (M)
Adapter Placement	Adapter Type	BD-Rate $\downarrow$	BD-Acc $\uparrow$	Params $\downarrow$ (M)
$g_{a}$ , $g_{s}$	Structural	-82.00%	18.71	0.28
$h_{a}$ , $h_{s}$	Structural	-1.56%	0.27	0.26
combined	Structural	-81.27%	18.80	0.55

Table 1: A demonstration of adapter-architecture mismatch. The structure-focused adapter from [ 31 ] proves ineffective when applied solely to the entropy model (

h_{a}

h_{s}

) and yields no discernible benefit when combined with the encoder-decoder (

g_{a}

g_{s}

). This highlights the need for targeted adapter design over naive reuse.

Figure 2: Overview of our proposed S 2 -CoT. It includes the Encoder, Decoder, Entropy Model, and Task Models. The colors of the modules indicate their training status: the base codecs and downstream task models are frozen (indicated in blue), while the trainable SFA and SCA are in red and orange, respectively. Here, $\odot$ denotes the Hadamard product and $\oplus$ denotes element-wise addition.

$g_{a}$	$g_{s}$	$h_{a}$	$h_{s}$	bpp $\downarrow$	mAP $\uparrow$	$\rho\downarrow$
0-7	0-7	0-3	0-3	0.0639	37.63	0.239
1,3,5	2,4,6	\	\	0.0829	37.21	0.289
0-7	0-7	\	\	0.0881	37.32	0.314
1,3,5	2,4,6	0-3	0-3	0.0685	37.42	0.260

Table 2: Analysis of a stage-wise freezing experiment on the Lu2022-TIC [ 41 ] for object detection on the COCO2017 dataset. In this experiment, specific network components were frozen to show the effects of fine-tuning only the encoder-decoder (

g_{a}

g_{s}

) or fine-tuning them along with the entropy model (

h_{a},h_{s}

Figure 3: Comparison of rate-accuracy performance across various tasks and base codecs. (a) Object detection and instance segmentation results on the Lu2022-TIC (top row) and Cheng2020-anchor (bottom row) base codecs. (b) Classification performance on the Lu2022-TIC base codec. (c) Object detection performance on the DCAE base codec. Results for the fourth base codec ELIC are in Sec. K .

Base	Method	Venue	Object Detection		Instance Segmentation		Trainable Params $\downarrow$ (M)
Base	Method	Venue	BD-Rate $\downarrow$	BD-mAP $\uparrow$	BD-Rate $\downarrow$	BD-mAP $\uparrow$	Trainable Params $\downarrow$ (M)
Lu2022-TIC	full fine-tuning	–	-73.943%	4.511	-67.977%	3.755	7.51 (100.00%)
	Channel Selection	ICPR'22	6.849%	-0.550	16.511%	-0.949	0.92 (12.25%)
	TransTIC	ICCV'23	-46.301%	2.768	-46.521%	2.690	1.62 (21.57%)
	ICMH-Net	ACM MM'23	-9.080%	0.625	-10.772%	0.654	3.98 (53.00%)
	Adapt-ICMH	ECCV'24	-55.150%	3.547	-52.407%	3.208	0.29 (3.86%)

Table 3: Quantitative comparison for object detection and instance segmentation tasks. Our S 2 -CoT nears full fine-tuning and outperforms other PEFT methods. Best results are in bold , second-best are underlined . See Sec. F for full Cheng2020-anchor -based results.

Stage	Module	Object Detection		Instance Segmentation		Trainable Params $\downarrow$ (M)
Stage	Module	BD-Rate $\downarrow$	BD-mAP $\uparrow$	BD-Rate $\downarrow$	BD-mAP $\uparrow$	Trainable Params $\downarrow$ (M)
Stage II	$+S$	-54.063%	3.568	-56.038%	3.308	0.18 (2.40%)
	$+F$	-52.667%	3.335	-53.160%	3.079	0.15 (2.00%)
Stage III	SFA w/o Fusion	-55.635%	3.563	-53.591%	3.196	0.26 (3.46%)
	SFA	-58.077%	3.842	-56.042%	3.421	0.40 (5.33%)

Table 4: Ablation study on different components of SFA.

Method	SFA	SCA		Object Detection		Instance Segmentation		Trainable Params $\downarrow$ (M)
Method	SFA	$h_{a}$	$h_{s}$	BD-Rate $\downarrow$	BD-mAP $\uparrow$	BD-Rate $\downarrow$	BD-mAP $\uparrow$	Trainable Params $\downarrow$ (M)
(a)	$\checkmark$			-58.077%	3.842	-56.042%	3.421	0.40 (5.33%)
(b)	$\checkmark$	$\checkmark$		-59.344%	3.891	-58.385%	3.439	0.41 (5.50%)
(c)	$\checkmark$		$\checkmark$	-58.533%	3.931	-61.271%	3.443	0.41 (5.50%)
(d)	$\checkmark$	$\checkmark$	$\checkmark$	-60.824%	4.014	-61.784%	3.480	0.42 (5.59%)
(e)	Adapt-ICMH-128 [ 31 ]	$\checkmark$	$\checkmark$	-58.399%	3.709	-60.936%	3.413	0.64 (8.52%)

Table 5: Ablation study of SFA and SCA synergy.

$C^{\prime}$	$r$	Object Detection		Trainable Params $\downarrow$ (M)
$C^{\prime}$	$r$	BD-Rate $\downarrow$	BD-mAP $\uparrow$	Trainable Params $\downarrow$ (M)
32	16	-57.622%	3.875	0.29 (3.86%)
64	8	-60.824%	4.014	0.42 (5.59%)
96	4	-61.329%	4.063	0.57 (7.59%)

Table 7: Effect of varied dimension

C^{\prime}

and reduction ratio

r

Model	KMACs/pixel		Latency (ms)		Params	BD-
Model	Enc.	Dec.	Enc.	Dec.	$\downarrow$ (M)	Acc $\uparrow$
full fine-tuning	132.6	176.4	94.9	94.8	7.51	17.6
Ours	153.6	197.3	109.5	101.0	0.42	17.4

Table 8: Comparison of computational complexity.

Method	Adapters in Encoder-Decoder	Adapters in Entropy Model	Object Detection		Instance Segmentation		Trainable Params $\downarrow$ (M)
Method	Adapters in Encoder-Decoder	Adapters in Entropy Model	BD-Rate $\downarrow$	BD-mAP $\uparrow$	BD-Rate $\downarrow$	BD-mAP $\uparrow$	Trainable Params $\downarrow$ (M)
(a)	SCA	–	-51.179%	3.022	-48.209%	2.863	0.05 (0.67%)
(b)	SCA	SCA	-50.975%	3.255	-51.417%	2.983	0.07 (0.93%)
(c)	SCA	SFA	-49.972%	3.160	-50.445%	2.953	0.19 (2.53%)
(d)	SFA	–	-58.077%	3.842	-56.042%	3.421	0.40 (5.33%)
(e)	SFA	SFA	-55.415%	3.977	-60.365%	3.464	0.54 (7.19%)

Table 6: Ablation study on the placement sensitivity of the SFA and SCA.

Panel of Figure 4 — Figure 4: Visual illustration of the six module placement strategies (a)–(f).

Figure 5: Qualitative comparison on the COCO2017 dataset.

Tasks	Models	Bit Rate	mAP ( $\uparrow$ )	PSNR ( $\uparrow$ )	PSNR w/o Adapters ( $\uparrow$ )
object detection	base codec	0.056	27.123	30.286	30.286
object detection	S 2 -CoT	0.0814	36.767	27.044	30.286
instance segmentation	base codec	0.056	24.81	30.29	30.29
instance segmentation	S 2 -CoT	0.103	34.64	28.18	30.29

Table 9: Performance comparison on different tasks of ICMH.

Figure 6: Pairwise channel similarities of the latent representation

y

for different fine-tuning strategies. Incorporating SCA markedly reduces the inter-channel correlation.

Figure 7: Spatial correlation of

(y-\mu)/\sigma

for models trained with

\lambda

=0.5. SFA+SCA (right) reduces average spatial correlation compared to SFA-only (middle) and the base codec (left), which benefits subsequent machine vision tasks.

Figure 8: Scaled deviation map of two strategies: SFA-only and SFA+SCA.

Figure 9: Each row corresponds to a different strategy and shows the energy of five channels with the highest entropy.

Figure 10: Object detection performance on the COCO2017 dataset using Lu2022-TIC as the base codec. Incorporating SCA reduces the bitrate without compromising detection accuracy.

Figure 11: ResNet50-based FPN architecture, indicating the features selected for perceptual loss evaluation.

	Classification	Detection	Segmentation
Optimizer	Adam	Adam	Adam
Batch size	16	8	8
Epochs	8	40	40
Base learning rate	1e-4	1e-4	1e-4
Lu2022-TIC
Trade-off term $\lambda$	[2.5, 3.5, 5, 6.7, 13]	[0.5, 0.9, 1.8, 3.2]	[0.35, 0.5, 0.9, 1.8, 3.2]

Table 10: Training hyperparameter settings for the main trials.

Base	Method	Venue	Object Detection		Instance Segmentation		Trainable Params $\downarrow$ (M)
Base	Method	Venue	BD-Rate $\downarrow$	BD-mAP $\uparrow$	BD-Rate $\downarrow$	BD-mAP $\uparrow$	Trainable Params $\downarrow$ (M)
Cheng2020 -anchor	full fine-tuning	–	-59.015%	4.699	-74.052%	3.869	26.60 (100.00%)
	Channel Selection	ICPR'22	-11.520%	0.723	-5.156%	0.230	1.34 (5.04%)
	ICMH-Net	ACM MM'23	-8.365%	0.503	-11.980%	0.737	4.43 (16.65%)
	Adapt-ICMH	ECCV'24	-49.245%	3.121	-60.657%	3.389	0.41 (1.54%)
	SVD-LoRA	CVPR'25	-36.354%	2.331	-38.087%	1.899	0.14 (0.53%)

Table 11: Extended quantitative results for object detection and instance segmentation tasks based on the Cheng2020-anchor base codec. This table provides the full results that are referenced in the main paper. Best results are in bold , second-best are underlined .

Method	SFA	SCA		Object Detection		Instance Segmentation		Trainable Params $\downarrow$ (M)
Method	SFA	$h_{a}$	$h_{s}$	BD-Rate $\downarrow$	BD-mAP $\uparrow$	BD-Rate $\downarrow$	BD-mAP $\uparrow$	Trainable Params $\downarrow$ (M)
(d) Ours	$\checkmark$	$\checkmark$	$\checkmark$	-60.824%	4.014	-61.784%	3.480	0.42 (5.59%)
(e)	Adapt-ICMH-128	$\checkmark$	$\checkmark$	-58.399%	3.709	-60.936%	3.413	0.64(8.52%)
(f)	Adapt-ICMH-96	$\checkmark$	$\checkmark$	-59.811%	3.665	-59.309%	3.341	0.48(6.39%)
(g)	Adapt-ICMH-64	$\checkmark$	$\checkmark$	-57.894%	3.894	-57.851%	3.356	0.30 (3.99%)

Table 12: Ablation study of Adapt-ICMH [ 31 ] and SCA synergy (extension of Tab. 5 in the main paper).

Figure 12: Architecture of the hyperprior entropy model with location markers. The figure illustrates the composition of the hyper-encoder $h_{a}$ and hyper-decoder $h_{s}$ .

Placement in $h_{a},h_{s}$	Core Idea	$\Delta\text{mAP}$ $\uparrow$	Params $\downarrow$
hp1, sp5	reverse bottleneck	+0.459	0.497
hp5, sp1	classic bottleneck	-0.496	0.421
hp3, sp4 or hp3, sp3	symmetrical middle layer	+0.479	0.421
hp2, hp4, sp2, sp4	symmetrical middle high layer	+0.242	0.438
hp3, hp5, sp1, sp3	symmetrical middle high layer	+0.343	0.438
hp3, hp5, sp3, sp5	asymmetric layer	+0.467	0.504

Table 13: Ablation study on the placement of SCA within the entropy model. Gain (

\Delta\text{mAP}

) is measured as the difference in mAP between our method’s result and the value predicted by the Adapt-ICMH Rate-mAP curve at the same bitrate.

$C^{\prime}$	$r$	Detection		Segmentation		Params $\downarrow$ (M)
$C^{\prime}$	$r$	BD-Rate $\downarrow$	BD-mAP $\uparrow$	BD-Rate $\downarrow$	BD-mAP $\uparrow$	Params $\downarrow$ (M)
32	32	-60.276%	3.852	-60.255%	3.386	0.28 (3.73%)
32	16	-57.622%	3.875	-61.991%	3.462	0.29 (3.86%)
64	8	-60.824%	4.014	-61.784%	3.480	0.42 (5.59%)
96	4	-61.329%	4.063	-62.429%	3.529	0.57 (7.59%)
128	4	-62.763%	4.165	-63.657%	3.625	0.72 (9.59%)

Table 14: Comprehensive ablation study on the SFA middle dimension

C^{\prime}

and the SCA reduction ratio

r

	Original Image	LIC		Traditional
	Original Image	Base Codec	Ours	JPEG(Q10)
Compression Ratio	None	219:1	296:1	67:1
mAP $\uparrow$	51.302	45.231	47.760	15.481
Methods	Base Codec		Ours
Methods	bpp $\downarrow$	mAP $\uparrow$	bpp $\downarrow$	mAP $\uparrow$
Faster-RCNN	0.1096	31.606	0.0812	36.601

Table 15: Performance comparison. Top: compression ratio and mAP on original images. Bottom: performance on three detectors.

Model	KMACs/pixel		CPU Average Latency (ms)		GPU Average Latency (ms)		Trainable Params $\downarrow$ (M)	BD- Acc $\uparrow$
Model	Enc.	Dec.	Enc.	Dec.	Enc.	Dec.	Trainable Params $\downarrow$ (M)	BD- Acc $\uparrow$
base codec	132.64	176.42	94.87	94.75	33.22	31.18	-	-
full fine-tuning	132.64	176.42	94.87	94.75	33.22	31.18	7.51	17.7
Adapt-ICMH [ 31 ]	147.39	191.16	111.01	110.14	29.49	25.34	0.29	16.9
Ours	153.57	197.34	109.49	101.03	29.60	28.45	0.42	17.4

Table 16: More Detailed Comparison on computational complexity.

Figure 13: Our SFA and SCA are integrated into the Lu2022-TIC codec [ 41 ] . STB denotes the Swin-Transformer Block. Conv(n, 2↑) denotes a transposed convolution (kernel size=n) with a stride of 2 for upsampling, and Conv(n, 2↓) denotes a convolution (kernel size=n) with a stride of 2 for downsampling.

Figure 14: Our SFA and SCA are integrated into the Cheng2020-anchor codec [ 10 ] . ResBlk denotes a residual block, where ResBlk↓ indicates a downsampling residual block with a stride of 2. Conv represents a standard $3\times 3$ convolution. LReLU denotes Leaky ReLU. Sub Conv(3, 2↑) denotes a sub-pixel convolution (kernel size=3) with a stride of 2 for upsampling.

Figure 15: Our SFA and SCA are integrated into the DCAE codec [ 40 ] . Downsample denotes the ResidualBottleneckBlockWithStride blocks, where STB indicates the SwinBlockWithConvMulti blocks. Conv(5, 2↓) represents a $5\times 5$ convolution with a stride of 2. Upsample denotes the ResidualBottleneckBlockWithUpsample blocks. Conv(5, 2↑) represents a $5\times 5$ convolution with a stride of 2 for upsampling.

Figure 16: Our SFA and SCA are integrated into the ELIC codec [ 21 ] . Conv(5, 2↓) represents a $5\times 5$ convolution with a stride of 2. ResB denotes residual blocks, and AttB denotes attention blocks.. Conv(3, 1) represents a standard $3\times 3$ convolution with a padding of 1. The deConv(5, 2↑) denotes a $5\times 5$ transposed convolution with a stride of 2, serving as a learnable spatial upsampling operation.

Base	Method	Venue	Object Detection		Instance Segmentation		Trainable Params $\downarrow$ (M)
Base	Method	Venue	BD-Rate $\downarrow$	BD-mAP $\uparrow$	BD-Rate $\downarrow$	BD-mAP $\uparrow$	Trainable Params $\downarrow$ (M)
DCAE	full fine-tuning	–	-84.677%	4.002	-81.897%	3.049	119.40 (100.00%)
	Channel Selection	ICPR'22	-24.498%	0.977	-18.661%	0.747	2.68 (2.24%)
	TransTIC	ICCV'23	-49.269%	2.339	-50.120%	2.271	1.13 (0.95%)
	Adapt-ICMH	ECCV'24	-64.608%	2.863	-63.240%	2.404	0.36 (0.30%)
	SVD-LoRA	CVPR'25	-58.166%	2.300	-54.371%	2.229	0.17 (0.14%)

Table 17: Quantitative comparison for object detection and instance segmentation tasks based on the DCAE and ELIC codec. Our method is competitive with full fine-tuning against others. Our method demonstrates superior BD-metric performance and parameter efficiency.

Figure 17: Comparison of rate-accuracy performance across various tasks and base codecs. (a) The instance segmentation results on the DCAE base codec. (b) Object detection (left) and instance segmentation (right) performance on the ELIC base codec.

Method	Classification		Params
Method	BD-Rate $\downarrow$	BD-Acc $\uparrow$	$\downarrow$ (M)
full fine-tuning	-	17.688	7.51 (100%)
Channel Selection	-37.178%	6.278	0.92 (12.25%)
TransTIC	-58.529%	9.956	1.62 (21.57%)
ICMH-Net	-18.759%	3.360	3.98 (53.00%)
Adapt-ICMH	-88.573%	16.901	0.29 (3.86%)

Table 18: Quantitative comparison of PEFT methods on the Lu2022-TIC codec for the classification task. Our method demonstrates superior BD-metric performance and parameter efficiency.

Method	Object Detection		Trainable
Lu2022-TIC	BD-Rate $\downarrow$	BD-mAP $\uparrow$	Params $\downarrow$ (M)
+SFA	-58.077%	3.842	0.40 (5.33%)
+SCA	+8.160%	-0.511	0.02 (0.27%)
+SFA +SCA	-60.824%	4.014	0.42 (5.59%)

Table 19: Isolated SCA reveals the entropy model’s inherent hypersensitivity, where even minimal parameter addition leads to performance collapse, while our S 2 -CoT reverses this trend.

Figure 18: More detection qualitative results.

Figure 19: More detection qualitative results.

Figure 20: More segmentation qualitative results.

Figure 21: More segmentation qualitative results.

Figure 22: More detection qualitative results.

Figure 23: More segmentation qualitative results.

Explore similar work

Adaptive Transform Coding for Semantic Compression
Apr 29, 2026Andriy Enttsel, Vincent CorlayLearned Image Compression Embedding Compression
Mitsubishi Electric R&D Centre Europe, Rennes, 35700, France
CrossMambaTuning: Synergistic Spatial and Cross-Layer Adaptation for Machine Vision Compression
Aug 26, 2026Haobo Xiong, Shaobo Liu, Kai Liu +1Parameter-Efficient Fine-Tuning Learned Image Compression
School of Computer Science and Technology, Xidian University Xi’an, China
PerCoV2: Ultra-Low Bit-Rate Perceptual Image Compression via Query-Based 1D Multimodal Image Tokens
Mar 12, 2025Nikolai Körber, Eduard Kromer, Andreas Siebert +3Learned Image Compression Generative Image Compression
Technical University of Munich · University of Applied Sciences Landshut · TU Wien

Apr 29, 2026J/K move · Enter open · S save

Adaptive Transform Coding for Semantic Compression

Andriy Enttsel, Vincent Corlay

Mitsubishi Electric R&D Centre Europe, Rennes, 35700, France

Learned Image Compression Embedding Compression

Open paper AskPDF arXiv

Visual data compression is shifting from human-centered reconstruction to machine-oriented representation coding. In this setting, an image is often mapped to a compact semantic embedding, which is then compressed and transmitted for downstream inference. We propose an adaptive transform-coding method for semantic-feature compression motivated by the conditional rate-distortion function of a Gaussian mixture model. The scheme uses mode-dependent transforms and quantizers selected according to the inferred source component, enabling more efficient coding of heterogeneous feature distributions. Evaluations on features from widely used vision backbones and foundation models show that the proposed method outperforms or is competitive with state-of-the-art neural compression methods while preserving flexibility and interpretability.

Abstract

Explore similar work

Adaptive Transform Coding for Semantic Compression

CrossMambaTuning: Synergistic Spatial and Cross-Layer Adaptation for Machine Vision Compression

PerCoV2: Ultra-Low Bit-Rate Perceptual Image Compression via Query-Based 1D Multimodal Image Tokens