Efficient Semantic Image Communication for Traffic Monitoring at the Edge

Damir Assylbek; Nurmukhammed Aitymbetov; Marko Ristin; Dimitrios Zorbas

cs.CVApr 14, 2026

Efficient Semantic Image Communication for Traffic Monitoring at the Edge

Authors: Damir Assylbek, Nurmukhammed Aitymbetov, Marko Ristin, Dimitrios Zorbas

Organizations: Nazarbayev University, School of Engineering & Digital Sciences, Astana, Kazakhstan · Zurich University of Applied Sciences (ZHAW), Winterthur, Switzerland

Abstract

Many visual monitoring systems operate under strict communication constraints, where transmitting full-resolution images is impractical and often unnecessary. In such settings, visual data is often used for object presence, spatial relationships, and scene context rather than exact pixel fidelity. This paper presents two semantic image communication pipelines for traffic monitoring, MMSD and SAMR, that reduce transmission cost while preserving meaningful visual information. MMSD (Multi-Modal Semantic Decomposition) targets very high compression together with data confidentiality, since sensitive pixel content is not transmitted. It replaces the original image with compact semantic representations, namely segmentation maps, edge maps, and textual descriptions, and reconstructs the scene at the receiver using a diffusion-based generative model. SAMR (Semantic-Aware Masking Reconstruction) targets higher visual quality while maintaining strong compression. It selectively suppresses non-critical image regions according to semantic importance before standard JPEG encoding and restores the missing content at the receiver through generative inpainting. Both designs follow an asymmetric sender-receiver architecture, where lightweight processing is performed at the edge and computationally intensive reconstruction is offloaded to the server. On a Raspberry Pi~5, the edge-side processing time is about 15s for MMSD and 9s for SAMR. Experimental results show average transmitted-data reductions of 99% for MMSD and 99.1% for SAMR. In addition, MMSD achieves lower payload size than the recent SPIC baseline while preserving strong semantic consistency, whereas SAMR provides a better quality-compression trade-off than standard JPEG and SQ-GAN under comparable operating conditions.

Figures & tables

Figure 1: Pipeline with semantic decomposition and generative reconstruction. The payload consists of (a) the segmentation map, (b) the edge map, and (c) a short description (caption) of the initial image.

Panel of Figure 2 — Figure 2: Original image (top); Segmentation and edge maps (bottom): The segmentation map alone is not enough to distinguish different car structures; this structural detail is better-represented by edges.

Figure 3: Example of the semantic-aware masking and reconstruction pipeline (the class-of-interest is vehicles ); (a) the segmentation map is generated; (b) the masked image is produced, and (c) it is compressed prior transmission. The reconstructed image is produced at the receiver using the inpainting autoencoder.

Figure 4: Example of Semantic-Aware Masking in SAMR: RoI (i.e., vehicles) are not masked at all or are lightly masked, but non-RoI (i.e., road, buildings, and sky) are heavily masked.

Panel of Figure 5 — Figure 5: Samples of images/frames of the two employed datasets.

Semantic	Config 0	Config 2	Config 4	Config 7
Group	(Conservative)	(Moderate)	(Balanced)	(Aggressive)
Vehicles	0.0	0.2	0.4	0.4
Humans	0.2	0.4	0.5	0.6
Flat surfaces	0.2	0.4	0.5	0.6
Construction	0.5	0.6	0.7	0.8
Objects	0.5	0.6	0.7	0.8

Table 1: Hierarchical masking in SAMR: class-specific masking probabilities for selected configurations.

Component	MMSD		SPIC
	Mean	$\sigma$	Mean	$\sigma$
Segmentation Map	3.96	0.93	12.74	2.58
Edge Map	18.41	6.79	–	–
Caption (BLIP)	57 B	5.02 B	–	–
Caption (Gemma)	260.49 B	26.59 B	–	–
Coarse Image	–	–	10.60	1.67

Table 2: Payload Transfer Size Comparison between MMSD and SPIC (all values in KBytes, except captions).

Metric	Value
Total participants	51
Total reconstructed image triplets	500
Unique image pairs shown	343
Pairs per participant	20
Pairs with at least one response	343
Total responses collected	1,238

Table 3: Human Survey Setup and Coverage (MMSD vs SPIC).

Metric	MMSD	SPIC
Position preservation (Q1)	63.2%	36.8%
Number preservation (Q2)	68.4%	31.6%
Overall win rate	66.0%	34.0%

Table 4: Human survey: Preference against the SPIC baseline.

Panel of Figure 6 — Figure 6: Qualitative comparison between the original image and reconstructed ones (MMSD vs SPIC). Top row: full images; Bottom row: zoomed-in crops ( $859\times 333$ px) highlighting reconstruction details.

Metric	BLIP	Gemma
Win rate over SPIC	64.6%	67.5%

Table 5: Human survey: Win rate of the proposed method over SPIC by caption model (decisive responses only).

Panel of Figure 7 — Figure 7: Qualitative comparison between the original image and MMSD’s reconstructions guided by BLIP and Gemma captions.

Metric	MMSD		SPIC
Metric	BLIP	Gemma	SPIC
Position preservation (Q1)	94.44	97.61	93.65
Number preservation (Q2)	88.89	86.90	80.16
Joint success rate (Q1 & Q2)	86.9	86.06	75.07

Table 6: VLM-based Evaluation Results (Gemini 27B).

Metric	Full pipeline		No edge map		No
Metric	BLIP	Gemma	BLIP	Gemma	caption
Position (Q1)	94.44	97.61	52.38	58.73	76.49
Number (Q2)	88.89	86.90	28.57	34.13	54.58
Joint	86.90	86.06	24.21	28.17	50.8

Table 7: Ablation study of MMSD (Gemini 27B).

Panel of Figure 8 — Figure 8: Qualitative comparison of MMSD reconstruction with (a) the full pipeline (BLIP) and (b-c) combinations of missing transferred elements.

Metric	Segmentation	Edge	Caption	Total
MMSD
Time (s)	8.35	0.02	7.07	15.46
Memory (MB)	$\sim$ 400	$<$ 10	$\sim$ 1,000	$\sim$ 1,400
SPIC
Time (s)	8.35	—	—	8.35
Memory (MB)	$\sim$ 400	—	—	$\sim$ 400

Table 8: Inference time and memory usage on a Raspberry Pi 5.

Panel of Figure 9 — Figure 9: Rate-distortion curves comparing SAMR against standard JPEG and SQ-GAN across three perceptual metrics. The x-axis (BPP) uses a logarithmic scale. Lower BPP indicates higher compression.

Panel of Figure 10 — Figure 10: Visual comparison at matched bitrates ( $\sim$ 0.08 BPP). From left to right: original image, JPEG at $Q$ =5, SQ-GAN at full masking ( $m_{x}=m_{s}=1.0$ ), and SAMR (Config 0, $Q$ =5). At comparable bitrates, JPEG exhibits severe blocking artifacts, SQ-GAN produces plausible but low-resolution output, while SAMR preserves structural detail and semantic content at the original resolution.

Metric	Overall
Position preservation (Q1)	100.00
Number preservation (Q2)	99.80
Joint success rate (Q1 & Q2)	99.80

Table 9: VLM-based (Gemini 27B) semantic preservation evaluation with Config 0 masking.

Method	Wins	Win Rate
SAMR (Config 0, Q5)	325	65.0%
SQ-GAN ( $m_{x}=m_{s}=1.0$ )	175	35.0%
JPEG (Q5)	0	0.0%

Table 10: VLM-based perceptual quality comparison at matched bitrate

\sim

0.08 BPP.

SAMR
Metric	Segme-	Masking	JPEG	Total
Metric	ntation	Masking	Encoding	Total
Time (s)	8.35	0.08	0.16	8.59
Memory (MB)	$\sim$ 400	$\sim$ 23	$\sim$ 3	$\sim$ 426
SQ-GAN
Metric	Segmentation		Encoding	Total

Table 11: Inference Time and Memory Usage on a Raspberry Pi 5.

Semantic Communication

Semantic communication sends meaning- or task-relevant information so receivers can act without reconstructing every transmitted bit or signal.

See all 103 papers

Semantic-Aware Generative Image Transmission for Resource-Constrained Visual IoT Systems
Jun 24, 2026Chenyang Zhang, Changwang Liu, Jinqi Zhu +4Learned Image Compression Semantic Communication
School of Computer and Information Engineering, Tianjin Normal University, Tianjin, China · School of Information Science and Engineering, Linyi University, Linyi 276000, China
Progressive Semantic Communication for Efficient Edge-Cloud Vision-Language Models
Apr 29, 2026Cyril Shih-Huan Hsu, Wig Yuan-Cheng Cheng, Chrysa PapagianniEdge-Cloud Collaborative Inference Efficient VLM Inference
Informatics Institute, University of Amsterdam, The Netherlands. · Open-EP Community, Taoyuan City 330049, Taiwan (R.O.C.).
SEG-JPEG: Simple Visual Semantic Communications for Remote Operation of Automated Vehicles over Unreliable Wireless Networks
Feb 16, 2026Sebastian Donnelly, Ruth Anderson, George Economides +4Semantic Communication Autonomous Driving
Autonomous Driving and Intelligent Transport Group, Oxford Brookes University, Oxford, UK · Oxfordshire County Council, Oxford, UK · Department for Transport, London, UK +2

Jun 24, 2026J/K move · Enter open · S save

Semantic-Aware Generative Image Transmission for Resource-Constrained Visual IoT Systems

Chenyang Zhang, Changwang Liu, Jinqi Zhu, Jiayi Chang+3

School of Computer and Information Engineering, Tianjin Normal University, Tianjin, China · School of Information Science and Engineering, Linyi University, Linyi 276000, China

Learned Image Compression Semantic Communication Internet of Things

Open paper AskPDF arXiv

Resource-constrained visual Internet of Things (IoT) systems, such as edge cameras, unmanned sensing platforms, industrial inspection nodes, and remote monitoring sensors, often need to transmit task-relevant visual evidence over low-rate wireless links to an edge/cloud service. Existing image communication methods usually compress or transmit complete global representations, leaving limited room to exploit receiver-side generative restoration. This paper proposes a semantic-aware generative image transmission framework for edge-assisted visual IoT. The image captured by an IoT visual sensor is encoded into a discrete token grid by a VQ encoder. At the IoT transmitter or nearby gateway, token recoverability, estimated from prediction entropy and local structure complexity, is fused with semantic importance obtained from instance segmentation and category-aware scoring. A spatial dispersal sampler then selects the tokens to be transmitted under a bitrate budget. The transmitter sends only the quantization indices of kept tokens and a binary mask map, while the edge/cloud receiver recovers masked tokens through MaskGIT with Halton sequence scheduling. Experiments on Kodak and VisDrone scenes under AWGN and Rayleigh channels show that the proposed method provides a flexible bitrate-quality tradeoff for narrowband visual IoT links. At 0.074 bpp, it uses 44.6% of the transmitted bits of the 0.167-bpp DeepJSCC/WITT reference while achieving 29.9 dB PSNR. A pseudo-GT downstream detection study on Kodak further shows that semantic-aware masking preserves task-relevant objects better than random masking at both 30% and 50% mask ratios.

Abstract

Semantic Communication

Explore similar work

Semantic-Aware Generative Image Transmission for Resource-Constrained Visual IoT Systems

Progressive Semantic Communication for Efficient Edge-Cloud Vision-Language Models

SEG-JPEG: Simple Visual Semantic Communications for Remote Operation of Automated Vehicles over Unreliable Wireless Networks