Distributed Semantic Segmentation With Improved Rate-Distortion Trade-Off

Danish Nazir; Timo Bartels; Thorsten Bagdonat; Tim Fingscheidt

cs.CVAug 26, 2026

Distributed Semantic Segmentation With Improved Rate-Distortion Trade-Off

Authors: Danish Nazir, Timo Bartels, Thorsten Bagdonat, Tim Fingscheidt

Organizations: Group Innovation Volkswagen AG Wolfsburg, Germany · Institute for Communications Technology TU Braunschweig Braunschweig, Germany

Abstract

Distributed deep neural networks (DNNs) for dense perception tasks such as semantic segmentation execute an encoder DNN on edge devices, and a decoder DNN typically on a large-scale cloud platform with a particular constraint on transmission bitrate. Recent works employ source codecs to enable bitrate-efficient transmission between the edge device and the cloud. However, as these approaches are typically bound to a particular type of source codec and alternative network architectures are often not explored, this results in a suboptimal rate-distortion (RD) trade-off in the low-bitrate regime. In this work, we propose two novel source codecs that enable extremely low bitrates, while improving RD performance. We demonstrate the effectiveness of our proposed source codecs by achieving state-of-the-art performance in distributed semantic segmentation at below 0.2 (0.03) bits per pixel, measured using the mean intersection-over-union metric on ADE20K (Cityscapes).

Figures & tables

Panel of Figure 1 — Figure 1: High-level overview of state-of-the-art distributed semantic segmentation [ 38 , 37 ] , applicable also to our proposed methods. Here, $\mathbf{E}$ and $\mathbf{JD}$ are the image encoder and the joint feature and task decoder, respectively. Block $\mathbf{FE}$ represents the feature encoder. We propose improved compression encoders and decoders ( $\mathbf{CE}$ and $\mathbf{CD}$ ), respectively, whose network architectures are detailed in Figs. 3 and 6 .

Panel of Figure 2 — Figure 2: Probability density function (PDF) p( $\overline{{\sigma}^{2}}$ ) of the mean variance $\overline{{\sigma}^{2}}=\overline{{\sigma}^{2}_{i,f}}$ , with pixel index $i$ and feature index $f$ , computed over all images in $\mathcal{D}^{\mathrm{val}}_{\mathrm{ADE20K}}$ , from the latent representation $\hat{\mathbf{r}}$ of the proposed HSM source codec vs. the state-of-the-art HS baseline source codec [ 37 , 38 ] , whose network architectures are detailed in Figs. 3 and 4 .

Panel of Figure 3 — Figure 3: Proposed HSM source codec (with $\bm{\sigma}$ and $\bm{\mu}$ ) and baseline (with $\mathbf{\bm{\mu}}$ ) network architecture with compression encoder $\mathbf{CE}$ and compression decoder $\mathbf{CD}$ , utilized in Fig. 1 . Details of the hyperprior encoder ( $\mathbf{HE}$ ) are shown in Fig. 5 (a), details of the baseline and proposed hyperprior decoder ( $\mathbf{HD}$ ) blocks are shown in Figs. 4 (a) and Figs. 4 (b), respectively.

Panel of Figure 4 — Figure 4: Details of hyperprior decoder ( $\mathbf{HD}$ ) architectures for distributed semantic segmentation. Fig. 4 (a) shows the baseline hyperprior decoder ( $\mathbf{HD}$ ), taken from [ 37 , 38 ] . Fig. 4 (b) illustrates the proposed $\mathbf{HD}$ architecture block, utilized in Fig. 3 ( HSM ) and Fig. 6 ( AR-HSM ). Details of the hyperprior standard deviation decoder ( $\mathbf{HSD}$ ) and the hyperprior mean decoder ( $\mathbf{HMD}$ ) are shown in Fig. 5 (b).

Panel of Figure 5 — Figure 5: Network architecture of the hyperprior encoder $\mathbf{HE}$ , hyperprior standard deviation decoder $\mathbf{HSD}$ (both adopted from [ 1 ] ), and the proposed hyperprior mean decoder $\mathbf{HMD}$ , used in Figs. 3 and 6 .

Panel of Figure 6 — Figure 6: Proposed AR-HSM source codec network architecture with compression encoder $\mathbf{CE}$ and compression decoder $\mathbf{CD}$ , utilized in Fig. 1 . Details of the hyperprior encoder ( $\mathbf{HE}$ ) are shown in Fig. 5 (a), details of the conventional and proposed hyperprior decoder ( $\mathbf{HD}$ ) blocks are shown in Figs. 4 (a) and Figs. 4 (b), respectively. Further details of the standard deviation refinement block $\mathbf{SDR}$ and mean refinement block $\mathbf{MDR}$ are shown in Fig. 7 .

Panel of Figure 7 — Figure 7: Standard deviation and mean refinement blocks ( $\mathbf{SDR}$ , $\mathbf{MDR}$ ) used in Fig. 6 , adopted from [ 27 ] . Swin transformer layer ( $\mathbf{STL}$ ) and convolutional block ( $\mathbf{CB}$ ) are detailed in Figs. 8 (a) and (b), respectively.

Panel of Figure 8 — Figure 8: Swin transformer layer $(\mathbf{STL})$ and convolutional block $(\mathbf{CB})$ employed in Fig. 7 , adopted from [ 27 ] .

Dataset	Official subsets	#images	Symbol
ADE20K [ 52 ]	train	20,210	$\mathcal{D}_{\mathrm{ADE20K}}^{\mathrm{train}}$
	val	2,000	$\mathcal{D}_{\mathrm{ADE20K}}^{\mathrm{val}}$
Cityscapes [ 14 ]	train	2,975	$\mathcal{D}_{\mathrm{CS}}^{\mathrm{train}}$
	val	500	$\mathcal{D}_{\mathrm{CS}}^{\mathrm{val}}$

Table 1: Datasets & splits used in our experiments.

Panel of Figure 9 — Figure 9: Illustration of our mIoU-adapted BD-rate metric , inspired from [ 8 ] , employed in Table 2 for the comparison of our proposed methods vs. various baselines.

Panel of Figure 10 — Figure 10: Performance (mIoU) of the proposed HSM and AR-HSM source codecs vs. the state-of-the-art HS source codec [ 37 , 38 ] , the ablated variant AR-HS ( $\bm{\mu}=\mathbf{0}$ ), reference method AR-HS , and no compression for distributed semantic segmentation on (a) $\mathcal{D}_{\mathrm{ADE20K}}^{\mathrm{val}}$ and on (b) $\mathcal{D}_{\mathrm{CS}}^{\mathrm{val}}$ .

Enc.	Source Codec	ADE20K $\mathcal{D}_{\mathrm{ADE20K}}^{\mathrm{val}}$	Cityscapes $\mathcal{D}_{\mathrm{CS}}^{\mathrm{val}}$
Enc.	Source Codec	BD-rate $\downarrow$ (%)	BD-rate $\downarrow$ (%)
RN-50	HS Baseline [ 37 ]	$0.0$	$0.0$
	HSM (ours)	$-\underline{17.6}$	$-\underline{41.0}$
	AR-HS ( $\bm{\mu}=0$ )	$+706.9$	$+621.9$
	AR-HS (reference)	$-7.4$	$+15.5$
	AR-HSM (ours)	$\mathbf{-74.3}$	$\mathbf{-67.3}$

Table 2: BD-rate savings measured with an input resolution of

512\times 512

for

\mathcal{D}^{\mathrm{val}}_{\mathrm{ADE20K}}

and

2048\times 1024

for

\mathcal{D}^{\mathrm{val}}_{\mathrm{CS}}

for the proposed HSM and AR-HSM vs. the state-of-the-art ( HS ) baseline [ 37 , 38 ] , the ablated variant AR-HS (

\bm{\mu}=0

), and the reference method ( AR-HS ). BD-rate is reported relative to the HS baseline. In each table segment, best results bold, second-best underlined.

Panel of Figure 11 — Figure 11: BD-rate savings of our proposals HSM and AR-HSM in comparison to the HS baseline and the AR-HSM reference, in dependence on total GFLOPs (top two plots) and $\#$ params (lower two plots), measured with an input resolution of $512\times 512$ for $\mathcal{D}^{\mathrm{val}}_{\mathrm{ADE20K}}$ and $2048\times 1024$ for $\mathcal{D}^{\mathrm{val}}_{\mathrm{CS}}$ .

Explore similar work

Feb 16, 2026

On the Rate-Distortion-Complexity Tradeoff for Semantic Communication

Semantic communication is a novel communication paradigm that focuses on conveying the user's intended meaning rather than the bit-wise transmission of source signals. One of the key challenges is to effectively represent and extract the semantic meaning of any given source signals. While deep learning (DL)-based solutions have shown promising results in extracting implicit semantic information from a wide range of sources, existing work often overlooks the high computational complexity inherent in both model training and inference for the DL-based encoder and decoder. To bridge this gap, this paper proposes a rate-distortion-complexity (RDC) framework which extends the classical rate-distortion theory by incorporating the constraints on semantic distance, including both the traditional bit-wise distortion metric and statistical difference-based divergence metric, and complexity measure, adopted from the theory of minimum description length and information bottleneck. We derive the closed-form theoretical results of the minimum achievable rate under given constraints on semantic distance and complexity for both Gaussian and binary semantic sources. Our theoretical results show a fundamental three-way tradeoff among achievable rate, semantic distance, and model complexity. Extensive experiments on real-world image and video datasets validate this tradeoff and further demonstrate that our information-theoretic complexity measure effectively correlates with practical computational costs, guiding efficient system design in resource-constrained scenarios.

Jingxuan Chai, Yong Xiao, Guangming Shi

the School of Artificial Intelligence at the Xidian University, Shaanxi 710071, China · the School of Electronic Information and Communications at the Huazhong University of Science and Technology, Wuhan 430074, China · the Pengcheng Laboratory, Shenzhen, Guangdong 518055, China +2

Semantic Communication Rate-Distortion Theory Information-Theoretic Lower Bounds

Ask PDF

Jun 11, 2026

Dual-Constrained Diffusion Image Compression for Operational Rate-Distortion-Perception Optimization

The rate-distortion-perception (RDP) trade-off extends classical rate--distortion theory by imposing a distributional constraint on reconstructions, providing a unified framework for neural image compression that jointly governs fidelity and perceptual realism. While prior work achieves near-optimal rate--perception trade-offs, practical frameworks explicitly realizing the full RDP surface remain scarce, primarily due to the difficulty of introducing common randomness at the decoder. We propose DCIC (Dual-Constrained Diffusion Image Compression), which integrates a learned codec with a diffusion-based decoder governed by joint distortion and idempotence constraints. The distortion constraint bounds reconstruction fidelity relative to the base codec output; the idempotence constraint -- requiring that re-encoding the restored image recovers the base codec reconstruction -- serves as a tractable surrogate for the distributional perception requirement. Together, they steer the reverse denoising process via iterative optimization with consistent noise injection, realizing common randomness without additional rate overhead. At fixed rate, dual attenuation factors

(K_D, K_P)

jointly navigate the Pareto frontier of the distortion-perception plane, enabling continuously adjustable fidelity-realism trade-offs from a single bitstream. DCIC

_{RD}

(

K_P{=}0

) and DCIC

_{RP}

(

K_D{=}0

) arise as boundary curves, with DCIC

_{RDP}

(

K_D = K_P=1

) realizing the optimal interior operating point. Experiments on CelebA-HQ, CLIC2020, and ImageNet-1K across CNN, Transformer, and hybrid architectures confirm that DCIC

_{RDP}

achieves superior BD-PSNR over all perceptual codecs, while DCIC

_{RP}

matches dedicated perception-oriented methods in BD-FID, validating the practical value of full RDP surface navigation.

Sanxin Jiang, Jiro Katto, Heming Sun

Department of Information Engineering Shanghai University of Electric Power Shanghai, China · Department of Computer Science and Communication Engineering Waseda University Tokyo, Japan · Faculty of Engineering Institute of Science Tokyo Tokyo, Japan

Learned Image Compression Rate-Distortion Optimization Distortion-Perception Tradeoff Generative Image Compression Diffusion Models

Ask PDF

Sep 30, 2026

Semantic-Aware Joint Source-Channel Optimization for Encoder-Agnostic Digital Video Communication

Video semantic communication has attracted increasing attention as a promising approach to improving video transmission efficiency. However, most existing approaches rely on computationally intensive deep learning-based video encoders and decoders, which hinders their deployment in resource-constrained scenarios. To address this issue, we propose a lightweight semantic-aware joint source-channel optimization (SAJSCO) scheme that can be integrated into existing digital video communication systems as a plug-in module. Specifically, we develop a video communication system model in which the transmitter jointly optimizes source and channel coding parameters based on the inter-frame semantic importance of the input video and estimated channel state information. On this basis, we formulate an optimization problem that maximizes semantic importance weighted video reconstruction quality under a maximum bitrate constraint. To solve it, we first quantify inter-frame semantic importance using a cosine similarity-based metric with a shifted window mechanism. We then develop a multi-actor proximal policy optimization (MPPO) algorithm to solve the formulated problem by jointly adapting the source compression rate and channel coding rate. The learned policy can be directly applied to different video encoders without encoder-specific retraining or fine-tuning. SAJSCO achieves Bjøntegaard Delta rate reductions of 34.86% and 18.01% when integrated with H.265, a conventional video encoder, and DCVC-RT, a deep learning-based video encoder. Over-the-air experiments on a hardware testbed further demonstrate a PSNR gain of up to 1.448 dB with H.265 and an LPIPS reduction of up to 0.033 with DCVC-RT compared with the respective best-performing fixed-parameter baselines.

Xiangben Zhu, Caili Guo, Yang Yang +2

Beijing Key Laboratory of Network System Architecture and Convergence, School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing 100876, China · China Mobile (Suzhou) Software Technology Company Limited, Suzhou 215011, China · Department of Engineering, King’s College London, London WC2R 2LS, U.K.

Semantic Communication Rate-Distortion Optimization Video Compression

Ask PDF