RSSA: Robust Semantic and Spatial Aligner for Collaborative Perception
Organizations: Zhejiang University · Zhejiang Provincial Key Laboratory of Multi-Modal Communication Networks and Intelligent Information Processing · China Jiliang University
Abstract
V2X collaborative perception improves single-vehicle perception by aggregating multi-view features from collaborative agents. However, existing methods typically rely on simple spatial warping to align the collaborative features, which can cause semantic misalignment during fusion. Meanwhile, these deterministic warping-based methods are highly sensitive to the positional noises. In this paper, we propose a plug-in module called Robust Semantic and Spatial Aligner (RSSA), which performs semantic and probabilistic spatial transformations for robust feature alignment. Specifically, a Semantic Transformation (SemT) block modulates feature channels based on the observed relative pose to correct semantic inconsistencies. Subsequently, a Probabilistic Spatial Transformation (P-SpaT) block performs probabilistic warping by sampling from the Gaussian-modeled relative pose space. A Relative Transformation Augmentation (RT-Aug) strategy is further introduced to augment the relative poses and facilitate the training of the RSSA. Extensive experiments on both simulated and real-world datasets demonstrate that RSSA improves the state-of-the-art methods by a large margin under the configuration of both clean and noisy positions.
Figures & tables
| Method | V2XSet | V2X-Real | |||
| BEV [email protected]/0.7 | 3D [email protected]/0.7 | BEV [email protected]/0.7 | 3D [email protected]/0.7 | ||
| Att fuse [ 18 ] | H | 93.20/85.46 | 92.07/75.73 | 74.29/58.83 | 70.35/34.92 |
| B | 84.47/68.42 | 81.13/55.09 | 68.79/53.24 | 61.80/26.04 | |
| B+RSSA | (+7.00/+15.19) 91.47/83.61 | (+8.58/+16.17) 89.71/71.26 | (+5.14/+7.12) 73.93/60.36 | (+4.34/+9.49) 66.14/35.53 | |
| CoBEVT [ 16 ] | H | 93.07/85.85 | 91.98/76.11 | 76.43/61.10 | 72.31/37.39 |
| B | 86.80/69.05 | 82.89/55.62 | 72.85/55.71 | 65.78/31.00 | |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Feature | Center error (m) | Orientation error ( ∘ ) |
| Before SemT | 0.52 | 45.93 |
| After SemT | 0.17 | 10.78 |
| Method | BEV [email protected] | 3D [email protected] |
| Broadcast+RSSA | 83.23 | 71.71 |
| Broadcast+RSSA, w/o position embeddings | 81.69 | 67.87 |
| Method | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 |
| RSSA w/o P-SpaT | 72.74 | 70.52 | 68.26 | 66.33 | 64.41 |
| RSSA w/ P-SpaT | 76.46 | 73.66 | 71.50 | 69.88 | 68.28 |
| Method | 0.0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 |
| Broadcast | 85.84 | 85.24 | 83.48 | 81.71 | 79.94 | 78.20 |
| Broadcast+SemT | 89.09 | 87.48 | 82.48 | 77.75 | 74.46 | 71.75 |
| Broadcast+SemT+P-SpaT | 89.65 | 88.06 | 85.03 | 82.61 | 81.05 | 79.90 |
| Broadcast+RSSA | 89.84 | 88.73 | 86.49 | 84.68 | 82.92 | 81.33 |
| Method | 1.0 m/ ∘ | 2.0 m/ ∘ | 4.0 m/ ∘ |
| Handshake | 50.32 | 50.32 | 50.32 |
| Broadcast | 61.00 | 61.00 | 61.00 |
| Broadcast+RSSA | 73.93 | 73.60 | 73.25 |
| Method | 0.2 m/ ∘ | 0.4 m/ ∘ | 0.6 m/ ∘ |
| Handshake | 48.86 | 45.26 | 41.75 |
| Broadcast | 60.88 | 60.43 | 60.06 |
| Broadcast+RSSA | 73.13 | 72.24 | 70.78 |
| Method | 0.25 | 0.50 | 0.75 |
| Handshake | 77.81 | 74.03 | 69.19 |
| Broadcast | 63.78 | 64.00 | 64.30 |
| Broadcast+RSSA | 76.02 | 72.83 | 68.97 |
| Method | 100 ms | 200 ms | 300 ms | 400 ms |
| Handshake | 68.52 | 56.34 | 53.04 | 52.43 |
| Broadcast | 61.45 | 59.57 | 59.18 | 58.88 |
| Broadcast+RSSA | 74.74 | 69.25 | 63.95 | 60.35 |
| Dataset | Training | 0.0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 |
| V2XSet [ 17 ] | Clean | 83.23 | 62.75 | 41.13 | 33.97 | 32.68 | 34.16 |
| V2XSet | Noisy | 78.65 | 76.46 | 73.66 | 71.50 | 69.88 | 68.28 |
| V2X-Real [ 14 ] | Clean | 61.49 | 51.75 | 41.34 | 38.71 | 39.28 | 41.28 |
| V2X-Real | Noisy | 60.61 | 59.32 | 57.51 | 56.38 | 55.23 | 54.10 |
| Method | Parameters | Flops | Latency | BEV [email protected] | |
| (M) | (G) | (ms) | Clean | Noisy (1.0 m/ ∘ ) | |
| Broadcast | 10.51 | 251.05 | 97.18 | 69.05 | 59.82 |
| Broadcast+RSSA | 11.84 | 281.27 | 111.76 | 83.23 | 68.28 |
| RSSA | 1.33 | 30.22 | 14.58 | +14.18 | +8.46 |
Cooperative Perception
Cooperative perception lets connected vehicles, robots, and aerial agents combine sensor observations to understand scenes beyond a single viewpoint.
See all 121 papers