Multi-Modal Decouple and Recouple Network for Robust 3D Object Detection

Rui Ding; Zhaonian Kuang; Yuzhe Ji; Meng Yang; Xinhu Zheng; Gang Hua

cs.CVMar 8, 2026

Multi-Modal Decouple and Recouple Network for Robust 3D Object Detection

Authors: Rui Ding, Zhaonian Kuang, Yuzhe Ji, Meng Yang, Xinhu Zheng, Gang Hua

Organizations: State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, P.R.China · Intelligent Transportation Thrust of the Systems Hub, The Hong Kong University of Science and Technology (Guangzhou), P.R.China · Multimodal Experiences Research Lab, Dolby Laboratories, Los Angeles, CA, USA

Abstract

Multi-modal 3D object detection with bird's eye view (BEV) has achieved desired advances on benchmarks. Nonetheless, the accuracy may drop significantly in the real world due to data corruption such as sensor configurations for LiDAR and scene conditions for camera. One design bottleneck of previous models resides in the tightly coupling of multi-modal BEV features during fusion, which may degrade the overall system performance if one modality or both is corrupted. To mitigate, we propose a Multi-Modal Decouple and Recouple Network for robust 3D object detection under data corruption. Different modalities commonly share some high-level invariant features. We observe that these invariant features across modalities do not always fail simultaneously, because different types of data corruption affect each modality in distinct ways.These invariant features can be recovered across modalities for robust fusion under data corruption.To this end, we explicitly decouple Camera/LiDAR BEV features into modality-invariant and modality-specific parts. It allows invariant features to compensate each other while mitigates the negative impact of a corrupted modality on the other.We then recouple these features into three experts to handle different types of data corruption, respectively, i.e., LiDAR, camera, and both.For each expert, we use modality-invariant features as robust information, while modality-specific features serve as a complement.Finally, we adaptively fuse the three experts to exact robust features for 3D object detection. For validation, we collect a benchmark with a large quantity of data corruption for LiDAR, camera, and both based on nuScenes. Our model is trained on clean nuScenes and tested on all types of data corruption. Our model consistently achieves the best accuracy on both corrupted and clean data compared to recent models.

Figures & tables

Fig. 1: Effect of data corruption on decoupled Camera/LiDAR features. (a)-(c) We measure the similarity (with the metric in [ 1 , 2 , 3 ] ) between clean and corrupted features for modality-invariant and modality-specific parts, respectively. The modality-specific features of LiDAR and Camera are affected seriously under different types of data corruption, while the modality-invariant features are affected slightly. Moreover, modality-invariant features always exhibit higher stability under different corruption severities. These modality-invariant features can be used for robust fusion. (d) Compared to BEVFusion [ 4 ] , in which LiDAR/Camera BEV features are tightly coupled, our method achieves significant gains on both corrupted and clean data.

Fig. 2: Overview of our framework . First, the LiDAR and camera encoders extract BEV features from point cloud and multi-view images, respectively. Second, the Modality Decouple Module decouples LiDAR/camera BEV features into modality-invariant and modality-specific parts. Then, the Modality Recouple Module recouples the invariant and specific parts into three experts for LiDAR, camera, and fusion, respectively. Finally, the three experts are adaptively fused to generate robust features for fusion based on corruption level of each modality. The detection heads take the fused features for robust 3D object detection. Our model is trained on clean data, and tested on various types of data corruption in the inference to ensure its robustness in the real world.

Fig. 3: Modality Decouple Module . Three encoders decouple features into modality-invariant and modality-specific parts. The specific features capture information unique to each modality. Modality invariant encoder extracts invariant features across modalities. $L_{Sim}$ enforces consistency output features between camera and LiDAR, while $L_{Diff}$ separates them from modality-specific ones. An auxiliary invariant head, used only during training, ensures the invariant features are truly useful for detection.

Fig. 4: Cross-modal recouple in the modality recouple module . The cross-modal recouple consists of one self-attention and two cross-attention. In cross-attention, the sample offsets and weights are learnable, therefore cross-attention can dynamically extract useful information from corrupted and invariant features as supplements.

Fig. 5: Examples of data corruption . We simulate various types of data corruption on cameras, LiDAR and both in real world, and add them to clean nuScenes validation images to generate corrupted data for testing.

mAP	Clean		Beam				FOV				CN	mRR
mAP	Test	Val	1	4	8	16	$90^{\circ}$	$120^{\circ}$	$180^{\circ}$	$240^{\circ}$	5	mRR
FUTR3D [ 68 ]	69.4	67.4	11.9	44.0	49.0	50.0	19.2	24.1	32.2	40.7	46.8	52.4 $\%$
UVTR [ 69 ]	67.1	65.4	12.7	43.7	52.2	53.3	18.6	23.3	30.8	38.8	57.4	56.2 $\%$
Transfusion [ 25 ]	68.9	67.3	4.7	39.2	49.9	51.5	14.7	21.2	29.2	39.6	61.4	51.4 $\%$
AD $\_$ BEVFusion [ 20 ]	69.2	67.9	8.7	43.0	52.8	54.5	15.4	21.0	31.1	39.6	59.1	53.2 $\%$
BEVFusion [ 4 ]	70.2	68.5	11.3	47.3	55.8	56.9	16.4	22.0	31.0	40.3	64.1	55.9 $\%$

TABLE I: Results on sensor corruptions . ’Clean’ refers to the results in nuScenes benchmark. ’CN’ refers to camera numbers. The best results are in bold. Our model consistently achieves the best results, especially on severe data corruptions such as 1 beam.

NDS/mAP	Beam	Crosstalk	Fog	Incomplete	Motion	Snow	mRR
NDS/mAP	Missing	Crosstalk	Fog	Echo	Blur	Snow	mRR
FUTR3D [ 68 ]	69.1/64.1	63.5/56.4	66.7/60.6	69.1/64.2	59.2/57.3	67.1/61.4	90.0 $\%$
UVTR [ 69 ]	68.6/61.3	64.8/56.9	64.8/56.2	67.5/61.3	60.3/54.7	66.3/59.2	89.0 $\%$
TranFusion [ 25 ]	69.5/64.8	63.2/54.8	66.1/59.4	69.4/65.0	57.1/53.1	67.8/61.5	88.9 $\%$
AD $\_$ BEVFusion [ 20 ]	69.6/65.6	62.6/53.8	65.9/59.9	69.6/65.9	58.6/55.5	67.5/62.0	89.0 $\%$
BEVFusion [ 4 ]	70.2/66.8	65.2/58.4	67.9/63.3	70.3/67.0	59.2/57.6	68.9/64.8	91.9 $\%$

TABLE II: Results on scene corruptions of LiDAR . We show the results of heavy severity levels. Our model consistently achieves the best results on all these types of scene corruptions.

NDS/mAP	Brightness	ColorQuant	Fog	LowLight	Motion	Snow	mRR
NDS/mAP	Brightness	ColorQuant	Fog	LowLight	Blur	Snow	mRR
AutoAlignV2* [ 70 ]	60.8/55.0	60.1/54.1	-	57.8/49.5	59.0/51.9	-	-
FUTR3D [ 68 ]	69.6/65.2	66.2/58.6	69.3/64.6	63.7/54.4	59.1/46.2	57.1/43.4	82.1 $\%$
UVTR [ 69 ]	69.3/63.7	66.4/58.8	68.8/62.8	65.6/57.1	65.9/57.7	65.9/57.6	91.1 $\%$
Tranfusion [ 25 ]	68.2/64.5	68.5/63.4	68.3/64.5	66.4/61.4	67.8/63.2	67.6/63.0	94.1 $\%$
AD $\_$ BEVFusion [ 20 ]	69.3/65.6	68.5/64.1	69.3/65.4	67.1/61.6	67.7/63.3	68.2/63.4	94.1 $\%$

TABLE III: Results on scene corruptions of camera . We show the results of heavy severity levels. Our model consistently achieves the best results on all these types of scene corruptions. * refers to results from RoboBEV [ 28 ] .

NDS/mAP	Clean(val)	Fog	Snow	MotionBlur	mRR
FUTR3D [ 68 ]	70.9/67.4	66.0/59.6	50.5/32.5	47.0/34.7	62.7 $\%$
UVTR [ 69 ]	70.2/65.4	65.2/56.1	61.7/49.9	53.9/43.9	76.4 $\%$
TransFusion [ 25 ]	71.2/67.3	64.8/58.0	64.0/55.9	54.5/50.9	81.6 $\%$
AD $\_$ BEVFusion [ 20 ]	71.0/67.9	65.1/58.1	64.9/56.7	55.5/50.6	81.1 $\%$
BEVFusion [ 4 ]	71.4/68.5	65.2/58.9	64.3/56.4	55.6/50.8	81.5 $\%$
MetaBEV [ 23 ]	71.3/68.6	65.6/59.6	64.5/56.5	55.9/51.4	81.4 $\%$

TABLE IV: Results on Multi-modal Corruptions . Our model consistently performs the best when cameras and LiDAR are simultaneously corrupted by heavy fog, snow, and motion blur.

Method	Clean(val)	FOV $180^{\circ}$	FOV $120^{\circ}$
Transfusion* [ 25 ]	70.9/66.9	51.4/29.3	45.8/20.3
BEVFusion* [ 4 ]	71.0/67.9	55.8/46.4	50.8/41.5
RobBEV* [ 26 ]	71.4/68.7	57.9/48.2	54.0/43.3
MetaBEV [ 23 ]	71.3/68.6	56.7/47.8	52.6/42.7
Ours	72.0/69.5	59.5/49.6	55.3/45.2

TABLE V: Comparison under known corruptions . * refers to results in RobBEV [ 26 ] .

Method	NDS/mAP
BEVFusion [ 4 ]	71.4/68.5
MetaBEV [ 23 ]	71.3/68.6
RoboFusion [ 49 ]	71.3/69.1
RobBEV [ 26 ]	71.4/68.7
SyNet [ 74 ]	72.2/69.4
DGFusion [ 42 ]	71.9/69.2

TABLE VI: Comparison on the nuScenes validation set .

#	decouple	recouple	NDS/mAP	mRR
a			71.0/68.5	77.4 $\%$
b	✓		71.3/68.6	78.2 $\%$
d		✓	71.8/69.1	78.9 $\%$
e	✓	✓	72.0/69.5	81.7 %

TABLE VII: Effectiveness of decouple and recouple modules. ”decouple” and ”recouple” refer to the modality decouple module and the modality recouple module, respectively, in our network.

#	modality specific encoder	NDS/mAP	mRR
a	none	71.0/68.5	77.4 $\%$
b	two CNN layers	71.4/68.6	77.4 $\%$
c	deformable attention	71.7/69.2	77.9 $\%$

TABLE VIII: Ablation study of modality specific encoder. “None” refers to not using the modality-specific encoder.

#	$L_{Sim}$	$L_{Diff}$	Aux head	NDS/mAP	mRR
a				71.0/68.5	77.4 $\%$
b	✓	✓		70.3/67.6	74.2 $\%$
c			✓	70.7/68.2	77.0 $\%$
d	✓	✓	✓	71.3/68.6	78.2 %

TABLE IX: Ablation studies of modality invariant encoder.

#	other modality	invariant	NDS/mAP	mRR
a			71.0/68.5	77.4 $\%$
b	✓		71.2/68.5	77.9 $\%$
c		✓	71.5/68.9	78.5 $\%$
d	✓	✓	71.8/69.3	79.4 %

TABLE X: Ablation studies of cross-modal recouple.

#	diff input	soft recouple	entropy loss	NDS/mAP	mRR
a				71.0/68.5	77.4 $\%$
b	✓			71.7/68.6	77.6 $\%$
c		✓		71.7/68.7	78.7 $\%$
d			✓	71.1/68.6	77.5 $\%$
e	✓	✓	✓	71.8/69.1	78.9 %

TABLE XI: Ablation studies of adaptive fusion. “diff input” indicates that each expert receives different modality features as input.

Layers	NDS/mAP	mRR
0	71.0/68.5	77.4 $\%$
1MD+1MR	71.5 / 68.9	78.5 $\%$
1MD+2MR	70.9/68.0	78.9 $\%$
2MD+2MR	71.2/68.5	77.9 $\%$

TABLE XII: Layers of decouple and recouple modules.

Points	NDS/mAP	mRR
2	70.9/68.3	77.6 $\%$
4	71.5 /68.9	78.5 $\%$
8	71.5/68.9	78.2 $\%$
12	71.4/ 69.0	77.7 $\%$

TABLE XIII: Reference points of decouple and recouple modules.

Model	FLOPs↓	FPS↑	NDS/mAP	mRR
BEVFusion [ 4 ]	115 GFLOPs	4.0	71.0/68.5	77.4 $\%$
MetaBEV [ 23 ]	157 GFLOPs	3.7	71.3/68.6	77.9 $\%$
Ours	140 GFLOPs	3.9	72.0/69.5	81.7 $\%$

TABLE XIV: Computational complexity comparison .

pre-trained backbones	training epochs	NDS/mAP
✓	6	72.0/69.5
$\times$	6	67.8/62.1
$\times$	24	71.7/69.2
$\times$	48	72.5/69.8

TABLE XV: Impact of pre-trained backbones.

Fig. 6: Visual results on different types of corrupted data . Our model detects more targets, while other baselines miss a few targets on corrupted data.

Fig. 7: Feature maps visualization . The features of our model keep more consistent on clean data and corrupted data compared to the baselines.

Explore similar work

SB-BEVFusion: Enhancing the Robustness against Sensor Malfunction and Corruptions
May 12, 2026Markus Essl, Marta Moscati, Mubashir Noman +4Multimodal Robustness Camera-LiDAR Fusion
Johannes Kepler University Linz, Austria · MBZUAI, UAE · Macquarie University, Sydney, Australia +1
CCF: Complementary Collaborative Fusion for Domain Generalized Multi-Modal 3D Object Detection
Mar 24, 2026Yuchen Wu, Kun Wang, Yining Pan +13D Object Detection Camera-LiDAR Fusion
Singapore University of Technology and Design
SARFusion: Scene-Aware Routing Fusion for Robust Camera-LiDAR 3D Object Detection
Sep 24, 2026Yuting Zhao, Ziyi Zheng, Shuxiao LiCamera-LiDAR Fusion 3D Object Detection
Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Wuhan College, Wuhan, China

May 12, 2026J/K move · Enter open · S save

SB-BEVFusion: Enhancing the Robustness against Sensor Malfunction and Corruptions

Markus Essl, Marta Moscati, Mubashir Noman, Muhammad Zaigham Zaheer+3

Johannes Kepler University Linz, Austria · MBZUAI, UAE · Macquarie University, Sydney, Australia+1

Multimodal Robustness Camera-LiDAR Fusion Autonomous Driving 3D Object Detection

Open paper AskPDF arXiv

Multimodal sensor fusion has demonstrated remarkable performance improvements over unimodal approaches in 3D object detection for autonomous vehicles. Typically, existing methods transform multimodal data from independent sensors, such as camera and LiDAR, into a unified bird's-eye view (BEV) representation for fusion. Although effective in ideal conditions, this strategy suffers from substantial performance deterioration when camera or LiDAR data are missing, corrupted, or noisy. To address this vulnerability, we develop a framework-agnostic fusion module for camera and LiDAR data that allows for handling cases when one of the two modalities is missing or corrupted. To demonstrate the effectiveness of our module, we instantiate it in BEVFusion [1], a well-established framework to combine camera and LiDAR data for 3D object detection. By means of quantitative experiments on the MultiCorrupt dataset, we demonstrate that our module achieves favorable performance improvements under scenarios of missing and corrupted modalities, substantially outperforming existing unified representation approaches across a wide range of sensor deterioration scenarios and reaching state-of-the-art performance in scenarios of corrupted modality due to extreme weather conditions and sensor failure.

Abstract

Explore similar work

SB-BEVFusion: Enhancing the Robustness against Sensor Malfunction and Corruptions

CCF: Complementary Collaborative Fusion for Domain Generalized Multi-Modal 3D Object Detection

SARFusion: Scene-Aware Routing Fusion for Robust Camera-LiDAR 3D Object Detection