Multi-Modal Decouple and Recouple Network for Robust 3D Object Detection
Authors: Rui Ding, Zhaonian Kuang, Yuzhe Ji, Meng Yang, Xinhu Zheng, Gang Hua
Organizations: State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, P.R.China · Intelligent Transportation Thrust of the Systems Hub, The Hong Kong University of Science and Technology (Guangzhou), P.R.China · Multimodal Experiences Research Lab, Dolby Laboratories, Los Angeles, CA, USA
Multi-modal 3D object detection with bird's eye view (BEV) has achieved desired advances on benchmarks. Nonetheless, the accuracy may drop significantly in the real world due to data corruption such as sensor configurations for LiDAR and scene conditions for camera. One design bottleneck of previous models resides in the tightly coupling of multi-modal BEV features during fusion, which may degrade the overall system performance if one modality or both is corrupted. To mitigate, we propose a Multi-Modal Decouple and Recouple Network for robust 3D object detection under data corruption. Different modalities commonly share some high-level invariant features. We observe that these invariant features across modalities do not always fail simultaneously, because different types of data corruption affect each modality in distinct ways.These invariant features can be recovered across modalities for robust fusion under data corruption.To this end, we explicitly decouple Camera/LiDAR BEV features into modality-invariant and modality-specific parts. It allows invariant features to compensate each other while mitigates the negative impact of a corrupted modality on the other.We then recouple these features into three experts to handle different types of data corruption, respectively, i.e., LiDAR, camera, and both.For each expert, we use modality-invariant features as robust information, while modality-specific features serve as a complement.Finally, we adaptively fuse the three experts to exact robust features for 3D object detection. For validation, we collect a benchmark with a large quantity of data corruption for LiDAR, camera, and both based on nuScenes. Our model is trained on clean nuScenes and tested on all types of data corruption. Our model consistently achieves the best accuracy on both corrupted and clean data compared to recent models.
Figures & tables
Fig. 1: Effect of data corruption on decoupled Camera/LiDAR features. (a)-(c) We measure the similarity (with the metric in [ 1 , 2 , 3 ] ) between clean and corrupted features for modality-invariant and modality-specific parts, respectively. The modality-specific features of LiDAR and Camera are affected seriously under different types of data corruption, while the modality-invariant features are affected slightly. Moreover, modality-invariant features always exhibit higher stability under different corruption severities. These modality-invariant features can be used for robust fusion. (d) Compared to BEVFusion [ 4 ] , in which LiDAR/Camera BEV features are tightly coupled, our method achieves significant gains on both corrupted and clean data.
Fig. 2: Overview of our framework . First, the LiDAR and camera encoders extract BEV features from point cloud and multi-view images, respectively. Second, the Modality Decouple Module decouples LiDAR/camera BEV features into modality-invariant and modality-specific parts. Then, the Modality Recouple Module recouples the invariant and specific parts into three experts for LiDAR, camera, and fusion, respectively. Finally, the three experts are adaptively fused to generate robust features for fusion based on corruption level of each modality. The detection heads take the fused features for robust 3D object detection. Our model is trained on clean data, and tested on various types of data corruption in the inference to ensure its robustness in the real world.
Fig. 3: Modality Decouple Module . Three encoders decouple features into modality-invariant and modality-specific parts. The specific features capture information unique to each modality. Modality invariant encoder extracts invariant features across modalities. LSim enforces consistency output features between camera and LiDAR, while LDiff separates them from modality-specific ones. An auxiliary invariant head, used only during training, ensures the invariant features are truly useful for detection.
Fig. 4: Cross-modal recouple in the modality recouple module . The cross-modal recouple consists of one self-attention and two cross-attention. In cross-attention, the sample offsets and weights are learnable, therefore cross-attention can dynamically extract useful information from corrupted and invariant features as supplements.
Fig. 5: Examples of data corruption . We simulate various types of data corruption on cameras, LiDAR and both in real world, and add them to clean nuScenes validation images to generate corrupted data for testing.
mAP
Clean
Beam
FOV
CN
mRR
Test
Val
1
4
8
16
90∘
120∘
180∘
240∘
5
FUTR3D [ 68 ]
69.4
67.4
11.9
44.0
49.0
50.0
19.2
24.1
32.2
40.7
46.8
52.4 %
UVTR [ 69 ]
67.1
65.4
12.7
43.7
52.2
53.3
18.6
23.3
30.8
38.8
57.4
56.2 %
Transfusion [ 25 ]
68.9
67.3
4.7
39.2
49.9
51.5
14.7
21.2
29.2
39.6
61.4
51.4 %
AD _ BEVFusion [ 20 ]
69.2
67.9
8.7
43.0
52.8
54.5
15.4
21.0
31.1
39.6
59.1
53.2 %
BEVFusion [ 4 ]
70.2
68.5
11.3
47.3
55.8
56.9
16.4
22.0
31.0
40.3
64.1
55.9 %
TABLE I: Results on sensor corruptions . ’Clean’ refers to the results in nuScenes benchmark. ’CN’ refers to camera numbers. The best results are in bold. Our model consistently achieves the best results, especially on severe data corruptions such as 1 beam.
NDS/mAP
Beam
Crosstalk
Fog
Incomplete
Motion
Snow
mRR
Missing
Echo
Blur
FUTR3D [ 68 ]
69.1/64.1
63.5/56.4
66.7/60.6
69.1/64.2
59.2/57.3
67.1/61.4
90.0 %
UVTR [ 69 ]
68.6/61.3
64.8/56.9
64.8/56.2
67.5/61.3
60.3/54.7
66.3/59.2
89.0 %
TranFusion [ 25 ]
69.5/64.8
63.2/54.8
66.1/59.4
69.4/65.0
57.1/53.1
67.8/61.5
88.9 %
AD _ BEVFusion [ 20 ]
69.6/65.6
62.6/53.8
65.9/59.9
69.6/65.9
58.6/55.5
67.5/62.0
89.0 %
BEVFusion [ 4 ]
70.2/66.8
65.2/58.4
67.9/63.3
70.3/67.0
59.2/57.6
68.9/64.8
91.9 %
TABLE II: Results on scene corruptions of LiDAR . We show the results of heavy severity levels. Our model consistently achieves the best results on all these types of scene corruptions.
NDS/mAP
Brightness
ColorQuant
Fog
LowLight
Motion
Snow
mRR
Blur
AutoAlignV2* [ 70 ]
60.8/55.0
60.1/54.1
-
57.8/49.5
59.0/51.9
-
-
FUTR3D [ 68 ]
69.6/65.2
66.2/58.6
69.3/64.6
63.7/54.4
59.1/46.2
57.1/43.4
82.1 %
UVTR [ 69 ]
69.3/63.7
66.4/58.8
68.8/62.8
65.6/57.1
65.9/57.7
65.9/57.6
91.1 %
Tranfusion [ 25 ]
68.2/64.5
68.5/63.4
68.3/64.5
66.4/61.4
67.8/63.2
67.6/63.0
94.1 %
AD _ BEVFusion [ 20 ]
69.3/65.6
68.5/64.1
69.3/65.4
67.1/61.6
67.7/63.3
68.2/63.4
94.1 %
TABLE III: Results on scene corruptions of camera . We show the results of heavy severity levels. Our model consistently achieves the best results on all these types of scene corruptions. * refers to results from RoboBEV [ 28 ] .
NDS/mAP
Clean(val)
Fog
Snow
MotionBlur
mRR
FUTR3D [ 68 ]
70.9/67.4
66.0/59.6
50.5/32.5
47.0/34.7
62.7 %
UVTR [ 69 ]
70.2/65.4
65.2/56.1
61.7/49.9
53.9/43.9
76.4 %
TransFusion [ 25 ]
71.2/67.3
64.8/58.0
64.0/55.9
54.5/50.9
81.6 %
AD _ BEVFusion [ 20 ]
71.0/67.9
65.1/58.1
64.9/56.7
55.5/50.6
81.1 %
BEVFusion [ 4 ]
71.4/68.5
65.2/58.9
64.3/56.4
55.6/50.8
81.5 %
MetaBEV [ 23 ]
71.3/68.6
65.6/59.6
64.5/56.5
55.9/51.4
81.4 %
TABLE IV: Results on Multi-modal Corruptions . Our model consistently performs the best when cameras and LiDAR are simultaneously corrupted by heavy fog, snow, and motion blur.
Method
Clean(val)
FOV 180∘
FOV 120∘
Transfusion* [ 25 ]
70.9/66.9
51.4/29.3
45.8/20.3
BEVFusion* [ 4 ]
71.0/67.9
55.8/46.4
50.8/41.5
RobBEV* [ 26 ]
71.4/68.7
57.9/48.2
54.0/43.3
MetaBEV [ 23 ]
71.3/68.6
56.7/47.8
52.6/42.7
Ours
72.0/69.5
59.5/49.6
55.3/45.2
TABLE V: Comparison under known corruptions . * refers to results in RobBEV [ 26 ] .
Method
NDS/mAP
BEVFusion [ 4 ]
71.4/68.5
MetaBEV [ 23 ]
71.3/68.6
RoboFusion [ 49 ]
71.3/69.1
RobBEV [ 26 ]
71.4/68.7
SyNet [ 74 ]
72.2/69.4
DGFusion [ 42 ]
71.9/69.2
TABLE VI: Comparison on the nuScenes validation set .
#
decouple
recouple
NDS/mAP
mRR
a
71.0/68.5
77.4 %
b
✓
71.3/68.6
78.2 %
d
✓
71.8/69.1
78.9 %
e
✓
✓
72.0/69.5
81.7 %
TABLE VII: Effectiveness of decouple and recouple modules. ”decouple” and ”recouple” refer to the modality decouple module and the modality recouple module, respectively, in our network.
#
modality specific encoder
NDS/mAP
mRR
a
none
71.0/68.5
77.4 %
b
two CNN layers
71.4/68.6
77.4 %
c
deformable attention
71.7/69.2
77.9 %
TABLE VIII: Ablation study of modality specific encoder. “None” refers to not using the modality-specific encoder.
#
LSim
LDiff
Aux head
NDS/mAP
mRR
a
71.0/68.5
77.4 %
b
✓
✓
70.3/67.6
74.2 %
c
✓
70.7/68.2
77.0 %
d
✓
✓
✓
71.3/68.6
78.2 %
TABLE IX: Ablation studies of modality invariant encoder.
#
other modality
invariant
NDS/mAP
mRR
a
71.0/68.5
77.4 %
b
✓
71.2/68.5
77.9 %
c
✓
71.5/68.9
78.5 %
d
✓
✓
71.8/69.3
79.4 %
TABLE X: Ablation studies of cross-modal recouple.
#
diff input
soft recouple
entropy loss
NDS/mAP
mRR
a
71.0/68.5
77.4 %
b
✓
71.7/68.6
77.6 %
c
✓
71.7/68.7
78.7 %
d
✓
71.1/68.6
77.5 %
e
✓
✓
✓
71.8/69.1
78.9 %
TABLE XI: Ablation studies of adaptive fusion. “diff input” indicates that each expert receives different modality features as input.
Layers
NDS/mAP
mRR
0
71.0/68.5
77.4 %
1MD+1MR
71.5 / 68.9
78.5 %
1MD+2MR
70.9/68.0
78.9 %
2MD+2MR
71.2/68.5
77.9 %
TABLE XII: Layers of decouple and recouple modules.
Points
NDS/mAP
mRR
2
70.9/68.3
77.6 %
4
71.5 /68.9
78.5 %
8
71.5/68.9
78.2 %
12
71.4/ 69.0
77.7 %
TABLE XIII: Reference points of decouple and recouple modules.
Model
FLOPs↓
FPS↑
NDS/mAP
mRR
BEVFusion [ 4 ]
115 GFLOPs
4.0
71.0/68.5
77.4 %
MetaBEV [ 23 ]
157 GFLOPs
3.7
71.3/68.6
77.9 %
Ours
140 GFLOPs
3.9
72.0/69.5
81.7 %
TABLE XIV: Computational complexity comparison .
pre-trained backbones
training epochs
NDS/mAP
✓
6
72.0/69.5
×
6
67.8/62.1
×
24
71.7/69.2
×
48
72.5/69.8
TABLE XV: Impact of pre-trained backbones.
Fig. 6: Visual results on different types of corrupted data . Our model detects more targets, while other baselines miss a few targets on corrupted data.
Fig. 7: Feature maps visualization . The features of our model keep more consistent on clean data and corrupted data compared to the baselines.
Multimodal sensor fusion has demonstrated remarkable performance improvements over unimodal approaches in 3D object detection for autonomous vehicles. Typically, existing methods transform multimodal data from independent sensors, such as camera and LiDAR, into a unified bird's-eye view (BEV) representation for fusion. Although effective in ideal conditions, this strategy suffers from substantial performance deterioration when camera or LiDAR data are missing, corrupted, or noisy. To address this vulnerability, we develop a framework-agnostic fusion module for camera and LiDAR data that allows for handling cases when one of the two modalities is missing or corrupted. To demonstrate the effectiveness of our module, we instantiate it in BEVFusion [1], a well-established framework to combine camera and LiDAR data for 3D object detection. By means of quantitative experiments on the MultiCorrupt dataset, we demonstrate that our module achieves favorable performance improvements under scenarios of missing and corrupted modalities, substantially outperforming existing unified representation approaches across a wide range of sensor deterioration scenarios and reaching state-of-the-art performance in scenarios of corrupted modality due to extreme weather conditions and sensor failure.
Markus Essl, Marta Moscati, Mubashir Noman +4
Johannes Kepler University Linz, Austria · MBZUAI, UAE · Macquarie University, Sydney, Australia +1
Multi-modal fusion has emerged as a promising paradigm for accurate 3D object detection. However, performance degrades substantially when deployed in target domains different from training. In this work, focusing on dual-branch proposal-level detectors, we identify two factors that limit robust cross-domain generalization: 1) in challenging domains such as rain or nighttime, one modality may undergo severe degradation; 2) the LiDAR branch often dominates the detection process, leading to systematic underutilization of visual cues and vulnerability when point clouds are compromised. To address these challenges, we propose three components. First, Query-Decoupled Loss provides independent supervision for 2D-only, 3D-only, and fused queries, rebalancing gradient flow across modalities. Second, LiDAR-Guided Depth Prior augments 2D queries with instance-aware geometric priors through probabilistic fusion of image-predicted and LiDAR-derived depth distributions, improving their spatial initialization. Third, Complementary Cross-Modal Masking applies complementary spatial masks to the image and point cloud, encouraging queries from both modalities to compete within the fused decoder and thereby promoting adaptive fusion. Extensive experiments demonstrate substantial gains over state-of-the-art baselines while preserving source-domain performance. Code and models are publicly available at https://github.com/IMPL-Lab/CCF.
Camera-LiDAR fusion has become a prevailing paradigm for 3D object detection in autonomous driving. However, existing fusion detectors often establish strong inter-modality dependencies by decoding object queries from tightly coupled multimodal representations. Under corrupted driving conditions, such dependencies make the detector vulnerable to unreliable modalities, where degraded observations may interfere with reliable modality-specific evidence and lead to suboptimal predictions. Moreover, modality reliability can vary across both global driving scenes and individual object queries, requiring adaptive fusion decisions at a finer granularity. To bridge this gap, we reformulate robust camera-LiDAR fusion as a scene-aware branch routing problem and propose SARFusion, a robust 3D object detector. Instead of producing detections from a single fused representation, SARFusion decouples object-query decoding into three parallel reasoning branches: a camera branch, a LiDAR branch, and a camera-LiDAR fusion branch. Guided by a Scene Reliability Prior estimated from the global driving context, SARFusion further incorporates object-level evidence to route each query to the most suitable branch. This query-wise routing strategy alleviates harmful cross-modal interference while preserving the benefits of multimodal fusion when complementary cues are trustworthy. On the nuScenes test set, SARFusion achieves strong performance with 72.5 mAP and 74.4 NDS. Extensive analyses demonstrate its robustness under challenging conditions, including sensor corruptions and environmental changes.
Yuting Zhao, Ziyi Zheng, Shuxiao Li
Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Wuhan College, Wuhan, China