Mission-critical UAVs increasingly rely on split vision-language model (VLM) perception under tight onboard-resource and wireless-communication constraints. However, corruption of transmitted intermediate features creates a deployment mismatch for clean-trained split interfaces, while stronger channel-aware codecs can impose substantial onboard cost. We present FreshLatent, a lightweight channel-aware latent adapter that trains a power-normalized encoder-decoder through wireless corruption while keeping the surrounding VLM frozen. We formulate deployment around a mission-conditioned perception requirement and embedded interface cost, linking channel quality and communication budget to the operating conditions under which perception remains usable. At 0 dB and the tightest communication budget, FreshLatent improves gIoU and cIoU over clean split compression by 20.79 and 20.87 points, respectively. At the most adverse evaluated SNR (0 dB), across all three communication budgets, FreshLatent recovers 63.5-69.1% of the gIoU improvement achieved by a much heavier, range-trained feature-JSCC codec. On an NVIDIA Jetson AGX Xavier in 10-W mode, FreshLatent uses 37-40x fewer encoder parameters, 7.7-9.9x lower edge-interface latency, and 8.8-10.0x lower edge-interface energy than the heavier codec. Together, these results show that lightweight channel-aware adaptation can recover a substantial fraction of the robustness of a much larger communication interface while broadening quality-valid operation under constrained wireless conditions.
Figures & tables
Figure 1: Motivation for FreshLatent . (a) Mission-driven UAVs rely on query-conditioned semantic perception. (b) Onboard resource constraints motivate UAV–server split VLM execution. (c) Split inference couples onboard interface cost, communication budget, and wireless channel quality. (d) Channel corruption can degrade downstream perception and render the returned evidence insufficient for the required task quality, motivating channel-aware latent adaptation.
Figure 2: Overview of FreshLatent in the split LISA pipeline. The UAV extracts an intermediate SAM feature F and transmits a compressed latent z over the wireless link. The server reconstructs F~ , which is consumed together with the operator query by the frozen downstream LISA pipeline. FreshLatent adapts only the communication interface spanning 4 – 6 ; the surrounding perception pipeline remains unchanged.
Figure 3: Reasoning-segmentation quality under matched communication budgets. Columns correspond to K=0.262 , 0.524 , and 1.311×106 channel uses per image; the upper and lower rows report gIoU and cIoU [ 13 ] , respectively. HEVC fails to decode below 13 dB (gIoU 0.06 , cIoU 0.12 ), so its curve drops below the plotted range. FreshLatent improves gIoU and cIoU over Clean by 20.79 and 20.87 points at 0 dB and K=0.262 M.
Figure 4: Fraction of the 27 evaluated (K,γ) points satisfying selected gIoU requirements τμ . FL-Fixed and SNR-Specialist are omitted because they serve as an ablation and condition-specific reference, respectively. HEVC is omitted because decoding fails below 13 dB, making its coverage dominated by decoder availability rather than the quality requirement. FreshLatent increases quality-valid coverage over Clean by 3.7–11.1 percentage points across all shown requirements.
Communication in latent space offers an intriguing alternative to symbolic messages for decentralized autonomous Unmanned Aerial Vehicle (UAV) swarms operating over bandwidth-constrained, time-varying wireless links. However, when homogeneous frozen models are prompted with discretized perceptual inputs, their broadcast states collapse toward the shared prompt template. In view of this, we propose AeroLat, a channel-aware latent semantic communication framework that uses evidence injection. The resulting latent states are then passed through an explicit communication model that encompasses bandwidth-limited serialization, additive noise and information staleness, which facilitates a joint assessment of communication fidelity and swarm-level coordination. Across multi-seed simulations, AeroLat provably remains resilient to codec choice, faults and increasing swarm size. It consistently reproduces the latent-swarm anomaly, while no-whitening controls recover the collapse. In particular, AeroLat is capable of reducing false similarity by 97.5%.
Continuous inference over concurrent video streams imposes substantial compute and memory demands on vision-language model (VLM) serving. Streaming inference uses sliding windows to maintain a bounded context of recent video, but processing each window independently repeats visual encoding and large language model (LLM) prefilling for similar and overlapping content. Existing optimizations provide limited coordination across these stages and often rely on model-specific training, profiling, or model-generated signals. We present CodecSight, a streaming VLM serving system that uses codec metadata as shared runtime guidance across visual encoding and LLM prefilling, without model-specific training or offline profiling. Codec-derived change signals guide patch pruning before visual encoding, reducing both visual computation and the number of downstream visual tokens. Codec-defined frame types guide selective key-value (KV) refresh across windows, while positional correction enables reuse of the remaining cached keys. Across three VLMs and four video workloads, our vLLM-based implementation supports up to 3.3× as many concurrent streams and achieves up to a 5.3× speedup in average time-to-first-token relative to the state-of-the-art baselines. It also reduces executed FLOPs by up to 93%, with a maximum task-quality decrease of 4.64 percentage points.
Yulin Zou, Wenyan Chen, Yan Chen +6
NTU Singapore Singapore · Beihang University China · A*STAR IAIC Singapore +1
Cloud-hosted large multimodal models (LMMs) can provide strong open-vocabulary perception for Vehicle-to-Everything systems, but naively transmitting full-resolution frames from edge to cloud causes severe communication overhead and high cloud-side prefill latency. We present CABLE, a cloud-assisted bandwidth-efficient LMM-based encoding framework for edge-cloud perception. CABLE propagates the previous cloud segmentation mask on the edge using ego-motion compensation, refines it with residual-motion cues, and consolidates disconnected regions via a corridor envelope to form a robust region of interest (ROI). Only ROI-masked images are uploaded, while the cloud segmentation output is fed back as the prior for the next frame, forming a mask-to-ROI-to-LMM feedback loop. Experiments on five datasets (nuScenes, WOD-ZB, Waymo, KITTI, and CADC) show consistent communication savings while largely preserving perception, achieving 73--87% ROI pixel-coverage reduction with 5--8× estimated LMM prefill speedup at a modest detection-quality trade-off relative to full-frame inference.
Haohua Que, Zhipeng Bao, Qianyi Wu +1
College of Engineering, University of Georgia, Athens, GA 30602, USA