In autonomous driving, multi-modal perception tasks like 3D object detection typically rely on well-synchronized sensors, both at training and inference. However, despite the use of hardware- or software-based synchronization algorithms, perfect synchrony is rarely guaranteed: Sensors may operate at different frequencies, and real-world factors such as network latency, hardware failures, or processing bottlenecks often introduce time offsets between sensors. Such asynchrony degrades perception performance, especially for dynamic objects. To address this challenge, we propose AsyncBEV, a trainable, lightweight and generic module to improve the robustness of 3D Bird's Eye View (BEV) object detection models against sensor asynchrony. Inspired by scene flow estimation, AsyncBEV predicts a dense 2D flow field directly from asynchronous multi-modal BEV features, taking into account the known time offset between these sensor measurements. The predicted feature flow is then used to warp and spatially align the feature maps, which we show can easily be integrated into different current BEV detector architectures (e.g., BEV grid-based and token-based). Extensive experiments demonstrate AsyncBEV improves robustness against both small and large asynchrony between LiDAR or camera sensors in both the token-based CMT and grid-based UniBEV, especially for dynamic objects. We significantly outperform the ego motion compensated CMT and UniBEV baselines, notably by 16.6 % and 11.9 % NDS on dynamic objects in the worst-case scenario of a 0.5s time offset. Code is available at https://github.com/tudelft-iv/AsyncBEV.
Figures & tables
Figure 1: Multi-modal 3D object detection under sensor asynchrony. a) shows a general concept of asynchronous 3D object detection. b) shows multi-modal 3D object detection under sensor asynchrony of a pretrained CMT ( ? ). Predictions exhibit large translation errors. c) shows our AsyncBEV improves CMT’s robustness against sensor asynchrony and mostly corrects the spatial misalignment of predicted boxes.
Figure 2: Overview of AsyncBEV . a) demonstrates an example pipeline of asynchronous multi-modal 3D object detection. In this setup, images are sampled at the reference timestamp t1 , while LiDAR point clouds were obtained at an earlier timestamp t0 . b) shows the Δ -BEVFlow estimation, the core module of AsyncBEV. It takes asynchronous BEV features as input and predicts the velocity of each BEV cell. The BEV flow is calculated by multiplying the velocity by Δt and supervised by the ground truth dense flow representation. Finally, the predicted flow is used to spatially align the features from t0 to t1 with the detector-specific warper.
Figure 3: Detector-specific Warper. a) Token-based methods rely on 3D coordinates of tokens to provide spatial guidance. Generated Δ -BEVFlow is used to correct the corresponding 3D coordinates of tokens from the asynchronous sensor. b) Grid-based methods project the multi-modal features into predefined BEV grids. Generated Δ -BEVFlow is used to build a look-up table from the reference timestamp to the asynchronous timestamp to warp the asynchronous BEV features to the reference timestamp.
All Objects
Static Objects
Dynamic Objects
Method
0s
0.25s
0.5s
0s
0.25s
0.5s
0s
0.25s
0.5s
FPS ↑
NDS ↑
72.9
49.6
43.2
69.0
48.7
44.2
47.5
32.2
26.1
CMT
mAP ↑
70.3
35.9
24.8
61.0
32.2
22.8
36.8
16.4
9.0
6.7
NDS ↑
72.9
66.8
63.3
69.0
68.3
67.5
47.5
34.5
26.8
CMT + EMC
mAP ↑
70.3
60.6
54.1
61.0
59.8
58.6
36.8
21.1
11.9
6.7
NDS ↑
71.5 ↓ 1.4
70.2 ↑ 3.4
67.6 ↑ 4.3
68.2 ↓ 0.8
67.2 ↓ 1.1
65.4 ↓ 2.1
45.8 ↓ 1.7
44.9 ↑ 10.4
41.5 ↑ 14.7
Table 1: Evaluation on the nuScenes val set of asynchronous multi modal 3D object detection ( Best , Second Best ). LiDAR is asynchronous. AsyncBEV is only trained once with each model and evaluated on different time offsets. FD: freeze the whole detector and only train AsyncBEV. Performance deltas are measured relative to the EMC baseline. Inference speed in Frames Per Second (FPS) is measured with a single A100 GPU.
Figure 4: Performance of different methods with increasing time offsets under LiDAR asynchrony.
Figure 5: Qualitative Results of AsyncBEV. The left two columns demonstrate the 3D object detection performance of CMT and CMT+AsyncBEV under 0.5s LiDAR asynchrony. The right two columns show the output of our predicted Δ -BEVFlow and the ground truth Δ -BEVFlow.
Δ -BEVFlow design
All Objects ( 0s )
All Objects ( 0.5s )
EMC+DA
Motion
Velocity
NDS
mAP
NDS
mAP
✗
✗
✗
66.7
62.0
39.7
22.2
✓
✗
✗
63.1 ↓ 3.6
57.6 ↓ 4.4
61.1 ↑ 21.4
54.9 ↑ 32.7
✓
✓
✗
64.2 ↓ 2.5
58.6 ↓ 3.4
62.5 ↑ 22.8
56.1 ↑ 33.9
✓
✗
✓
65.7 ↓ 1.0
60.8 ↓ 1.2
63.3 ↑ 23.6
57.1 ↑ 34.9
Table 2: Ablation study of Δ -BEVFlow design. This table showcases the core design choice for the new task Δ -BEVFlow. EMC+DA : EMC + asynchronous data augmentation training. Motion : BEV-ME. Velocity : BEV-VE, which is our default design.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
All Objects
Static Objects
Dynamic Objects
Method
0s
0.25s
0.5s
0s
0.25s
0.5s
0s
0.25s
0.5s
NDS ↑
72.9
70.7
67.5
69.0
66.9
63.8
47.5
46.9
45.8
CMT
mAP ↑
70.3
66.7
61.2
61.0
57.9
52.4
36.8
36.0
34.2
NDS ↑
72.9
71.4
68.8
69.0
67.6
65.1
47.5
46.9
45.9
CMT + EMC
mAP ↑
70.3
67.8
63.5
61.0
58.9
54.9
36.8
36.0
34.1
NDS ↑
72.7 ↓ 0.2
72.7 ↑ 1.3
72.3 ↑ 3.5
68.8 ↓ 0.2
68.8 ↑ 1.2
68.4 ↑ 3.3
48.1 ↑ 0.6
48.1 ↑ 1.2
48.0 ↑ 2.1
Appendix
Table III: Evaluation on the nuScenes val set of asynchronous multi modal 3D object detection. ( Best , Second Best ). Camera is asynchronous. Trained with one model and evaluated on different time offsets. Performance deltas are measured relative to the EMC baseline.
All Objects
Static Objects
Dynamic Objects
Method
NDS ↑
mAP ↑
NDS ↑
mAP ↑
NDS ↑
mAP ↑
CMT ( 0.5s L)
43.2
25.8
44.2
22.8
26.1
9.0
CMT_C
44.7 ↑ 1.5
38.3 ↑ 12.5
47.4 ↑ 3.2
29.5 ↑ 6.7
28.3 ↑ 2.2
14.7 ↑ 5.7
CMT+AsyncBEV ( 0.5s L)
70.0 ↑ 25.3
66.4 ↑ 28.1
67.6 ↑ 20.2
59.1 ↑ 29.6
43.4 ↑ 15.1
32.3 ↑ 17.6
UniBEV ( 0.5s L)
39.7
22.2
41.2
19.0
25.3
8.9
UniBEV_C
42.6 ↑ 2.9
35.4 ↑ 13.2
46.1 ↑ 4.9
27.1 ↑ 8.1
28.2 ↑ 2.9
13.1 ↑ 4.2
Appendix
Table IV: Performance of single-modality performance of CMT and UniBEV.
Figure VI: Qualitative Results of AsyncBEV. The left two columns demonstrate the 3D object detection performance of CMT and CMT+ AsyncBEV under LiDAR asynchrony with 0.5s offset. The right two columns show the output of our BEV-VE module and the ground truth flow in the BEV space.
Multimodal sensor fusion has demonstrated remarkable performance improvements over unimodal approaches in 3D object detection for autonomous vehicles. Typically, existing methods transform multimodal data from independent sensors, such as camera and LiDAR, into a unified bird's-eye view (BEV) representation for fusion. Although effective in ideal conditions, this strategy suffers from substantial performance deterioration when camera or LiDAR data are missing, corrupted, or noisy. To address this vulnerability, we develop a framework-agnostic fusion module for camera and LiDAR data that allows for handling cases when one of the two modalities is missing or corrupted. To demonstrate the effectiveness of our module, we instantiate it in BEVFusion [1], a well-established framework to combine camera and LiDAR data for 3D object detection. By means of quantitative experiments on the MultiCorrupt dataset, we demonstrate that our module achieves favorable performance improvements under scenarios of missing and corrupted modalities, substantially outperforming existing unified representation approaches across a wide range of sensor deterioration scenarios and reaching state-of-the-art performance in scenarios of corrupted modality due to extreme weather conditions and sensor failure.
Markus Essl, Marta Moscati, Mubashir Noman +4
Johannes Kepler University Linz, Austria · MBZUAI, UAE · Macquarie University, Sydney, Australia +1
In autonomous driving, 3D object detection is essential for accurate perception and reliable decision-making. However, object motion and ego-motion often induce cross-frame spatiotemporal inconsistencies in BEV-based detectors, leading to temporal BEV feature misalignment and degraded spatiotemporal consistency. To address these challenges, we propose Co-Fusion4D, a unified framework that explicitly preserves cross-frame spatiotemporal consistency and suppresses temporal feature drift. Co-Fusion4D adopts a current-frame-centric strategy, treating the current frame as the primary source of information while selectively incorporating historical frames after spatiotemporal filtering and alignment. This dominant-complementary mechanism effectively mitigates cumulative alignment errors, suppresses noisy feature propagation, and exploits reliable temporal cues for a more consistent BEV representation. In addition, Co-Fusion4D integrates a Dual Attention Fusion (DAF) module to further enhance spatiotemporal feature interaction. DAF jointly leverages intra-frame spatial attention and inter-frame temporal attention to adaptively align and fuse multi-frame features, emphasizing motion-consistent regions while suppressing spurious correlations. By departing from conventional uniform fusion paradigms, this design substantially improves the temporal stability and discriminative capability of BEV representations. Extensive experiments on the nuScenes benchmark demonstrate that Co-Fusion4D achieves state-of-the-art performance, with 74.9% mAP and 75.6% NDS, without relying on test-time augmentation or external data.
Accurate 3D bird's-eye view (BEV) object detection is essential for autonomous driving, and depends strongly on effective multimodal representations from complementary sensors such as cameras and LiDAR. Multimodal masked autoencoders have shown strong potential for learning such representations for downstream 3D BEV object detection. However, existing methods typically apply uniform random masking to camera and LiDAR inputs, treating all regions equally, and learn representations only through masked reconstruction. We propose a semantics-guided multimodal masked autoencoder framework that introduces semantic information during pretraining through two separate components: (i) semantics-guided LiDAR voxel masking, which preserves semantically important LiDAR regions more strongly, and (ii) an auxiliary point-wise LiDAR semantic decoder branch that injects semantic guidance in addition to reconstruction. On BEVFusion 3D object detection, our semantics-guided pretraining strategy improves performance on the nuScenes mini validation set compared to the standard UniM2AE baseline: semantics-guided LiDAR voxel masking yields +1.49% mean Average Precision (mAP) and +1.66% nuScenes Detection Score (NDS), while decoder-side point semantic supervision yields +1.39% mAP and +3.22% NDS over the baseline.
Prabuddhi Wariyapperuma, Rajitha de Silva, Marc Hanheide +2
University of Lincoln, Lincoln Centre for Autonomous Systems, Lincoln, UK. · University of Cambridge, Institute for Manufacturing, Department of Engineering, Cambridge, UK.