Efficient Semantic Image Communication for Traffic Monitoring at the Edge
Authors: Damir Assylbek, Nurmukhammed Aitymbetov, Marko Ristin, Dimitrios Zorbas
Organizations: Nazarbayev University, School of Engineering & Digital Sciences, Astana, Kazakhstan · Zurich University of Applied Sciences (ZHAW), Winterthur, Switzerland
Many visual monitoring systems operate under strict communication constraints, where transmitting full-resolution images is impractical and often unnecessary. In such settings, visual data is often used for object presence, spatial relationships, and scene context rather than exact pixel fidelity. This paper presents two semantic image communication pipelines for traffic monitoring, MMSD and SAMR, that reduce transmission cost while preserving meaningful visual information. MMSD (Multi-Modal Semantic Decomposition) targets very high compression together with data confidentiality, since sensitive pixel content is not transmitted. It replaces the original image with compact semantic representations, namely segmentation maps, edge maps, and textual descriptions, and reconstructs the scene at the receiver using a diffusion-based generative model. SAMR (Semantic-Aware Masking Reconstruction) targets higher visual quality while maintaining strong compression. It selectively suppresses non-critical image regions according to semantic importance before standard JPEG encoding and restores the missing content at the receiver through generative inpainting. Both designs follow an asymmetric sender-receiver architecture, where lightweight processing is performed at the edge and computationally intensive reconstruction is offloaded to the server. On a Raspberry Pi~5, the edge-side processing time is about 15s for MMSD and 9s for SAMR. Experimental results show average transmitted-data reductions of 99% for MMSD and 99.1% for SAMR. In addition, MMSD achieves lower payload size than the recent SPIC baseline while preserving strong semantic consistency, whereas SAMR provides a better quality-compression trade-off than standard JPEG and SQ-GAN under comparable operating conditions.
Figures & tables
Figure 1: Pipeline with semantic decomposition and generative reconstruction. The payload consists of (a) the segmentation map, (b) the edge map, and (c) a short description (caption) of the initial image.
Figure 2: Original image (top); Segmentation and edge maps (bottom): The segmentation map alone is not enough to distinguish different car structures; this structural detail is better-represented by edges.
Figure 3: Example of the semantic-aware masking and reconstruction pipeline (the class-of-interest is vehicles ); (a) the segmentation map is generated; (b) the masked image is produced, and (c) it is compressed prior transmission. The reconstructed image is produced at the receiver using the inpainting autoencoder.
Figure 4: Example of Semantic-Aware Masking in SAMR: RoI (i.e., vehicles) are not masked at all or are lightly masked, but non-RoI (i.e., road, buildings, and sky) are heavily masked.
Figure 5: Samples of images/frames of the two employed datasets.
Semantic
Config 0
Config 2
Config 4
Config 7
Group
(Conservative)
(Moderate)
(Balanced)
(Aggressive)
Vehicles
0.0
0.2
0.4
0.4
Humans
0.2
0.4
0.5
0.6
Flat surfaces
0.2
0.4
0.5
0.6
Construction
0.5
0.6
0.7
0.8
Objects
0.5
0.6
0.7
0.8
Table 1: Hierarchical masking in SAMR: class-specific masking probabilities for selected configurations.
Component
MMSD
SPIC
Mean
σ
Mean
σ
Segmentation Map
3.96
0.93
12.74
2.58
Edge Map
18.41
6.79
–
–
Caption (BLIP)
57 B
5.02 B
–
–
Caption (Gemma)
260.49 B
26.59 B
–
–
Coarse Image
–
–
10.60
1.67
Table 2: Payload Transfer Size Comparison between MMSD and SPIC (all values in KBytes, except captions).
Metric
Value
Total participants
51
Total reconstructed image triplets
500
Unique image pairs shown
343
Pairs per participant
20
Pairs with at least one response
343
Total responses collected
1,238
Table 3: Human Survey Setup and Coverage (MMSD vs SPIC).
Metric
MMSD
SPIC
Position preservation (Q1)
63.2%
36.8%
Number preservation (Q2)
68.4%
31.6%
Overall win rate
66.0%
34.0%
Table 4: Human survey: Preference against the SPIC baseline.
Figure 6: Qualitative comparison between the original image and reconstructed ones (MMSD vs SPIC). Top row: full images; Bottom row: zoomed-in crops ( 859×333 px) highlighting reconstruction details.
Metric
BLIP
Gemma
Win rate over SPIC
64.6%
67.5%
Table 5: Human survey: Win rate of the proposed method over SPIC by caption model (decisive responses only).
Figure 7: Qualitative comparison between the original image and MMSD’s reconstructions guided by BLIP and Gemma captions.
Figure 8: Qualitative comparison of MMSD reconstruction with (a) the full pipeline (BLIP) and (b-c) combinations of missing transferred elements.
Metric
Segmentation
Edge
Caption
Total
MMSD
Time (s)
8.35
0.02
7.07
15.46
Memory (MB)
∼ 400
< 10
∼ 1,000
∼ 1,400
SPIC
Time (s)
8.35
—
—
8.35
Memory (MB)
∼ 400
—
—
∼ 400
Table 8: Inference time and memory usage on a Raspberry Pi 5.
Figure 9: Rate-distortion curves comparing SAMR against standard JPEG and SQ-GAN across three perceptual metrics. The x-axis (BPP) uses a logarithmic scale. Lower BPP indicates higher compression.
Figure 10: Visual comparison at matched bitrates ( ∼ 0.08 BPP). From left to right: original image, JPEG at Q =5, SQ-GAN at full masking ( mx=ms=1.0 ), and SAMR (Config 0, Q =5). At comparable bitrates, JPEG exhibits severe blocking artifacts, SQ-GAN produces plausible but low-resolution output, while SAMR preserves structural detail and semantic content at the original resolution.
Resource-constrained visual Internet of Things (IoT) systems, such as edge cameras, unmanned sensing platforms, industrial inspection nodes, and remote monitoring sensors, often need to transmit task-relevant visual evidence over low-rate wireless links to an edge/cloud service. Existing image communication methods usually compress or transmit complete global representations, leaving limited room to exploit receiver-side generative restoration. This paper proposes a semantic-aware generative image transmission framework for edge-assisted visual IoT. The image captured by an IoT visual sensor is encoded into a discrete token grid by a VQ encoder. At the IoT transmitter or nearby gateway, token recoverability, estimated from prediction entropy and local structure complexity, is fused with semantic importance obtained from instance segmentation and category-aware scoring. A spatial dispersal sampler then selects the tokens to be transmitted under a bitrate budget. The transmitter sends only the quantization indices of kept tokens and a binary mask map, while the edge/cloud receiver recovers masked tokens through MaskGIT with Halton sequence scheduling. Experiments on Kodak and VisDrone scenes under AWGN and Rayleigh channels show that the proposed method provides a flexible bitrate-quality tradeoff for narrowband visual IoT links. At 0.074 bpp, it uses 44.6% of the transmitted bits of the 0.167-bpp DeepJSCC/WITT reference while achieving 29.9 dB PSNR. A pseudo-GT downstream detection study on Kodak further shows that semantic-aware masking preserves task-relevant objects better than random masking at both 30% and 50% mask ratios.
Chenyang Zhang, Changwang Liu, Jinqi Zhu +4
School of Computer and Information Engineering, Tianjin Normal University, Tianjin, China · School of Information Science and Engineering, Linyi University, Linyi 276000, China
Deploying Vision-Language Models (VLMs) on edge devices remains challenging due to their substantial computational and memory demands, which exceed the capabilities of resource-constrained embedded platforms. Conversely, fully offloading inference to the cloud is often impractical in bandwidth-limited environments, where transmitting raw visual data introduces substantial latency overhead. While recent edge-cloud collaborative architectures attempt to partition VLM workloads across devices, they typically rely on transmitting fixed-size representations, lacking adaptability to dynamic network conditions and failing to fully exploit semantic redundancy. In this paper, we propose a progressive semantic communication framework for edge-cloud VLM inference, using a Meta AutoEncoder that compresses visual tokens into adaptive, progressively refinable representations, enabling plug-and-play deployment with off-the-shelf VLMs without additional fine-tuning. This design allows flexible transmission at different information levels, providing a controllable trade-off between communication cost and semantic fidelity. We implement a full end-to-end edge-cloud system comprising an embedded NXP i.MX95 platform and a GPU server, communicating over bandwidth-constrained networks. Experimental results show that, at 1 Mbps uplink, the proposed progressive scheme significantly reduces network latency compared to full-edge and full-cloud solutions, while maintaining high semantic consistency even under high compression. The implementation code will be released upon publication at https://github.com/open-ep/ProSemComVLM.
Remote Operation is touted as being key to the rapid deployment of automated vehicles. Streaming imagery to control connected vehicles remotely currently requires a reliable, high throughput network connection, which can be limited in real-world remote operation deployments relying on public network infrastructure. This paper investigates how the application of computer vision assisted semantic communication can be used to circumvent data loss and corruption associated with traditional image compression techniques. By encoding the segmentations of detected road users into colour coded highlights within low resolution greyscale imagery, the required data rate can be reduced by 50% compared with conventional techniques, while maintaining visual clarity. This enables a median glass-to-glass latency of below 200 ms even when the network data rate is below 500 kbit/s, while clearly outlining salient road users to enhance situational awareness of the remote operator. The approach is demonstrated in an area of variable 4G mobile connectivity using an automated last-mile delivery vehicle. Results indicate that large-scale deployment of remotely operated automated vehicles could be possible even on the often constrained public 4G/5G mobile network, providing the potential to expedite the nationwide roll-out of automated vehicles.
Sebastian Donnelly, Ruth Anderson, George Economides +4
Autonomous Driving and Intelligent Transport Group, Oxford Brookes University, Oxford, UK · Oxfordshire County Council, Oxford, UK · Department for Transport, London, UK +2