Leveraging Large Vision Model for Multi-UAV Co-perception in Low-Altitude Wireless Networks
Authors: Yunting Xu, Jiacheng Wang, Ruichen Zhang, Changyuan Zhao, Yinqiu Liu, Dusit Niyato, Liang Yu, Haibo Zhou, Dong In Kim
Organizations: College of Computing and Data Science, Nanyang Technological University, Singapore, 639798 · Alibaba Cloud, Hangzhou, China, 311121 · School of Electronic Science and Engineering, Nanjing University, Nanjing, China, 210023 · Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon 16419, South Korea
Multi-uncrewed aerial vehicle (UAV) cooperative perception has emerged as a promising paradigm for diverse low-altitude economy applications, where complementary multi-view observations are leveraged to enhance perception performance via wireless communications. However, the massive visual data generated by multiple UAVs poses significant challenges in terms of communication latency and resource efficiency. To address these challenges, this paper proposes a communication-efficient cooperative perception framework, termed Base-Station-Helped UAV (BHU), which reduces communication overhead while enhancing perception performance. Specifically, we employ a Top-K selection mechanism to identify the most informative pixels from UAV-captured RGB images, enabling sparsified visual transmission with reduced data volume and latency. The sparsified images are transmitted to a ground server via multi-user MIMO (MU-MIMO), where a Swin-large-based MaskDINO encoder extracts bird's-eye-view (BEV) features and performs cooperative feature fusion for ground vehicle perception. Furthermore, we develop a diffusion model-based deep reinforcement learning (DRL) algorithm to jointly select cooperative UAVs, sparsification ratios, and precoding matrices, achieving a balance between communication efficiency and perception utility. Simulation results on the Air-Co-Pred dataset demonstrate that, compared with traditional CNN-based BEV fusion baselines, the proposed BHU framework improves perception performance by over 5% while reducing communication overhead by 85%, providing an effective solution for multi-UAV cooperative perception under resource-constrained wireless environments.
Figures & tables
Fig. 1: Illustration of multi-UAV cooperative perception scenario. The UAV-captured aerial images are transmitted to a ground server via MU-MIMO communication, where multi-view information is fused to enable cooperative feature learning for vehicle instance and motion detection in low-altitude economy scenarios.
Fig. 2: Illustration of the proposed BHU framework for multi-UAV cooperative perception. The aerial images are first sparsified via a Top-K selection mechanism and transmitted to the ground server through wireless links. A MaskDINO-based encoder extracts multi-scale features, which are projected into BEV representations and cooperatively fused across multi-UAVs. The fused BEV features are employed for downstream perception tasks, including vehicle instance segmentation and motion flow prediction.
Fig. 3: Illustration of the proposed DDIM-based DRL framework. An MLP network is employed to determine the cooperative UAVs and Top-K sparsification ratios, while a DDIM module is employed to generate precoding actions by modeling the conditional distribution of optimal precoding vectors through a reverse denoising process.
Fig. 4: Visualization of 3D object detection results in the BEV representation.
Figure 5
Method
IoU (%) ↑
PQ (%) ↑
Comm. Cost (Bps) ↓
DHD [ 19 ]
Short
Long
Avg.
Short
Long
Avg.
Top-k (25%)
EfficientNet (4 UAVs)
60.58
53.14
56.86
51.80
45.89
48.85
6.14×107
EfficientNet (2 UAVs)
50.59
40.82
45.71
43.43
35.37
39.40
2.05×107
EfficientNet w/o Fusion
41.31
33.11
37.21
35.07
28.29
31.68
0
MaskDINO (ResNet50)
63.99
55.97
59.98
54.24
48.23
51.24
6.14×107
MaskDINO (SwinL)
65.52
57.66
61.59
54.10
50.63
52.37
6.14×107
TABLE I: Performance comparison of EfficientNet- and MaskDINO-based networks with/without BEV fusion.
Method
Metrics (%) ↑
Parameters ↓
SwinL (Backbone)
IoU
Prec.
Total
Training
EfficientNet Baseline
52.89
70.15
39.5 M
39.5 M
BHU w pretrained
57.81
77.57
0.3 B
0.3 B
BHU w/o pretrained
53.27
72.22
0.3 B
0.3 B
DHD + LoRA
55.24
73.40
∼ 0.3 B
37.4 M
BHU + LoRA
55.86
74.38
∼ 0.3 B
37.4 M
TABLE II: Results of backbone networks and LoRA fine-tuning.
Fig. 8: IoU versus Top-K ratios under SwinL and LoRA settings.
Method
IoU (%) ↑
PQ (%) ↑
Comm. Cost (Bps) ↓
BHU (Proposed)
Short
Long
Avg.
Short
Long
Avg.
Top-k (25%)
DHD (EfficientNet) Baseline
60.58
53.14
56.86
51.80
45.89
48.85
6.14 ×107
BHU (EfficientNet)
57.39
52.89
55.14
47.23
45.68
46.46
9.22×106
MaskDINO (ResNet50)
64.00
56.32
60.16
56.05
49.40
52.73
9.22×106
MaskDINO (SwinL)
65.71
57.81
61.76
54.64
51.26
52.95
9.22×106
MaskDINO (SwinL) + LoRA
63.51
55.86
59.69
53.74
48.58
51.16
9.22×106
TABLE III: Comparison of the proposed BHU scheme and baselines in terms of utility and communication cost.
Communication-efficient collaborative perception studies how networked agents share useful sensor information with minimal bandwidth while improving their joint view of the scene.
The rapid development of low-altitude economy has driven the proliferation of Unmanned Aerial Vehicle (UAV) applications, including logistics, inspection, and emergency response. However, transmitting high-volume image data from UAVs to ground stations faces significant challenges due to limited bandwidth and stringent privacy requirements. To address these issues, a Semantic Communication (SC) framework based on Federated Learning (FL) is proposed for efficient and privacy-preserving image transmission. A Swin Transformer-based Semantic Communication (STSC) architecture is designed to extract multi-scale semantic features under constrained bandwidth conditions. Dedicated communication and computing nodes are deployed on UAVs to enhance real-time coverage and flexibility. Meanwhile, a FL mechanism enables global model training across distributed devices without sharing raw data, thus preserving user privacy. Simulation experiments conducted on the CIFAR-10 dataset demonstrate that the proposed STSC framework achieves at least 5.7 dB improvement in Peak Signal-to-Noise Ratio (PSNR) compared to DeepJSCC baselines, while also showing superior convergence and generalization performance. The framework effectively integrates UAV-assisted deployment with SC and privacy protection, offering a practical solution for bandwidth-constrained image transmission in low-altitude networks.
Kexin Zhang, Lixin Li, Yuna Yan +5
School of Electronics and Information, Northwestern Polytechnical University, Xi’an 710129, China · DecoreX Intelligent Technologies Co., Ltd., Xi’an 710075, China · Samsung AI Center, Cambridge CB1 2JH, UK +2
In multi-agent collaborative sensing systems, substantial communication overhead from information exchange significantly limits scalability and real-time performance, especially in bandwidth-constrained environments. This often results in degraded performance and reduced reliability. To address this challenge, we propose WaveComm, a wavelet-based communication framework that drastically reduces transmission loads while preserving sensing performance in low-bandwidth scenarios. The core innovation of WaveComm lies in decomposing feature maps using Discrete Wavelet Transform (DWT), transmitting only compact low-frequency components to minimize communication overhead. High-frequency details are omitted, and their effects are reconstructed at the receiver side using a lightweight generator. A Multi-Scale Distillation (MSD) Loss is employed to optimize the reconstruction quality across pixel, structural, semantic, and distributional levels. Experiments on the OPV2V and DAIR-V2X datasets for LiDAR-based and camera-based perception tasks demonstrate that WaveComm maintains state-of-the-art performance even when the communication volume is reduced to 86.3% and 87.0% of the original, respectively. Compared to existing approaches, WaveComm achieves competitive improvements in both communication efficiency and perception accuracy. Ablation studies further validate the effectiveness of its key components.
Erdemt Bao, Jin Yang
School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074, China. · National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Shaanxi 710049, China.
Modern autonomous vehicle perception systems are often constrained by occlusions, blind spots, and limited sensing range. While existing cooperative perception paradigms, such as Vehicle-to-Vehicle (V2V) and Vehicle-to-Infrastructure (V2I), have demonstrated their effectiveness in mitigating these challenges, they remain limited to ground-level collaboration and cannot fully address large-scale occlusions or long-range perception in complex environments. To advance research in cross-view cooperative perception, we present V2U4Real, the first large-scale real-world multi-modal dataset for Vehicle-to-UAV (V2U) cooperative object perception. V2U4Real is collected by a ground vehicle and a UAV equipped with multi-view LiDARs and RGB cameras. The dataset covers urban streets, university campuses, and rural roads under diverse traffic scenarios, comprising over 56K LiDAR frames, 56K multi-view camera images, and 700K annotated 3D bounding boxes across four classes. To support a wide range of research tasks, we establish benchmarks for single-agent 3D object detection, cooperative 3D object detection, and object tracking. Comprehensive evaluations of several state-of-the-art models demonstrate the effectiveness of V2U cooperation in enhancing perception robustness and long-range awareness. The V2U4Real dataset and codebase is available at https://github.com/VjiaLi/V2U4Real.
Weijia Li, Haoen Xiang, Tianxu Wang +4
Fujian Key Laboratory of Urban Intelligent Sensing and Computing, Xiamen University, China · Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, China