UBD-Bench: Benchmarking and Analyzing Unimodal Branch Degradation in Multimodal Perception for Autonomous Driving

Ang Liu; Xiang Gu; Ruikai Li; Qiyue Wang; Yizhe Fan; Hao Shan; Ziyang Yan; Han Jiang

cs.CVOct 18, 2026

UBD-Bench: Benchmarking and Analyzing Unimodal Branch Degradation in Multimodal Perception for Autonomous Driving

Authors: Ang Liu, Xiang Gu, Ruikai Li, Qiyue Wang, Yizhe Fan, Hao Shan, Ziyang Yan, Han Jiang,

Organizations: Beihang University · Xiaomi EV · University of Macau

Abstract

Multimodal perception improves autonomous-driving performance by combining complementary sensor information. However, stronger multimodal predictions do not necessarily imply stronger representations in each modality-specific branch. Such branch-level weaknesses can remain hidden in conventional output-level evaluation. We introduce UBD-Bench, a systematic benchmark for studying unimodal branch degradation (UBD) in multimodal perception. UBD-Bench uses a Paired Frozen-Branch Probe to compare the task-relevant representation quality of the same modality after independent unimodal training and multimodal joint training under matched decoding conditions. Across four autonomous-driving perception tasks and 10 models, we find that Camera branches consistently degrade after multimodal joint training, whereas LiDAR branches exhibit weaker and more model-dependent changes, revealing a pronounced modality asymmetry. Hierarchical probing further shows that Camera-branch degradation extends from the visual backbone through the Camera-to-BEV pathway and is most severe at the pre-fusion Camera BEV representation. Guided by this diagnosis, we introduce unimodal-to-multimodal distillation at the degraded representation. This intervention consistently improves both Camera-branch Probe performance and full-model performance across all four tasks. These results show that multimodal gains can coexist with degraded unimodal representations, highlighting branch-level representation quality as an important dimension for evaluating and improving multimodal perception.

Figures & tables

Figure 1: Prediction Discrepancies Between Camera–LiDAR and Camera-Only Models. Representative examples from 3D object detection, BEV map segmentation, occupancy prediction, and online vectorized map construction show that Camera–LiDAR models produce less accurate predictions than their Camera-only counterparts in the highlighted regions.

Figure 2: Paired Frozen-Branch Probe for Camera and LiDAR representations.

Figure 3: Qualitative comparison of modality-specific pre-fusion BEV representations across four perception tasks. The independently trained Camera and LiDAR representations are compared with the corresponding branches learned through multimodal joint training.

Task	Method	Metric	End-to-end			Camera Probe		LiDAR Probe		Fusion Probe
Task	Method	Metric	Full C	Full L	Full F	C2C	F2C	L2L	F2L	F2F
3D Det.	BEVFusion	mAP $\uparrow$	35.55	64.68	68.55	33.47	11.58 ( -21.89 )	47.69	47.70 (+0.01)	62.56
	BEVFusion	NDS $\uparrow$	41.21	69.22	71.41	36.49	19.02 ( -17.47 )	53.01	53.47 (+0.46)	61.31
	CMT	mAP $\uparrow$	40.59	63.21	70.30	36.43	10.50 ( -25.93 )	61.64	61.88 (+0.24)	53.24
	CMT	NDS $\uparrow$	46.04	68.27	72.94	41.94	25.88 ( -16.06 )	67.23	67.78 (+0.55)	60.06
Vec. Map	MapTR	mAP $\uparrow$	48.57	54.26	63.14	51.09	40.42 ( -10.67 )	53.62	55.88 (+2.26)	62.92

Table 1: Cross-task and cross-model evaluation of unimodal branch degradation.

Figure 4: Cross-task comparison of Camera- and LiDAR-branch Probe performance. The top row shows consistent Camera-branch degradation across tasks and models, whereas the bottom row shows more model-dependent changes in the LiDAR branch. Blue and orange bars denote independently trained unimodal branches and the corresponding multimodal-model branches, respectively.

Metric	Pos.	C2C	F2C ( $\Delta_{C}$ )
mAP $\uparrow$	P0	51.09	40.42 ( -10.67 )
	P1	51.58	43.80 ( -7.78 )
	P2	49.84	45.73 ( -4.11 )
	P3	45.07	42.49 ( -2.58 )
	P4	26.09	26.49 (+0.40)

Table 2: Hierarchical probing along the Camera–BEV generation pathway.

Task	Method	Metric	Full multimodal model		Camera-branch Probe
Task	Method	Metric	Baseline	+KD ( $\Delta$ )	F2C	F2C + KD ( $\Delta$ )
3D Det.	BEVFusion	mAP $\uparrow$	68.27	69.25 (+0.98)	12.06	32.48 (+20.42)
3D Det.	BEVFusion	NDS $\uparrow$	71.29	71.88 (+0.59)	19.06	35.12 (+16.06)
Vec. Map	MapTR	mAP $\uparrow$	61.97	64.49 (+2.52)	40.42	43.54 (+3.12)
BEV Seg.	BEVFusion	mIoU $\uparrow$	59.83	60.45 (+0.62)	49.30	55.46 (+6.16)
Occ.	CONet	IoU $\uparrow$	25.30	29.30 (+4.00)	20.10	21.50 (+1.40)

Table 3: Effects of unimodal-to-multimodal reverse distillation.

Figure 5: Qualitative comparison of pre-fusion Camera BEV features for 3D object detection. From left to right: ground-truth 3D bounding boxes, the C2C reference feature, the F2C feature without distillation, and the F2C feature after unimodal-to-multimodal distillation. The same feature channel and color scale are used for all activation maps.

Autonomous Driving Perception

Autonomous driving perception enables vehicles to detect and interpret road users, drivable space, and hazards using onboard and shared sensors.

See all 379 papers

MATS: A novel multi-modality multi-task learning framework for 3D perception in autonomous driving
Jul 27, 2026Junchen Huo, Wanming Hao, Song Wang +3Mixture of Experts Camera-LiDAR Fusion
School of Electrical and Information Engineering, Zhengzhou University, Zhengzhou 450001, China · Tianping College of Suzhou University of Science and Technology, Suzhou 215009, China · department of computer science, Toronto Metropolitan University, Toronto M5B2K3, Canada
Can BEV Perception Gracefully Degrade under Sensor Failures?
May 29, 2026Haifa Zhang, Yijing Wang, Haoyu Wang +2Missing-Modality Learning Multimodal Robustness
Tianjin Key Laboratory of Intelligent Unmanned Swarm Technology and System, School of Electrical and Information Engineering, Tianjin University, Tianjin, China · Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai, China
Mining Cross-Modal Perception Gaps in Autonomous Driving: When Camera and LiDAR Disagree
Oct 12, 2026Hongsen Cao, Mona Jaber, Shanxin Yuan +1Camera-LiDAR Fusion 3D Object Detection
School of Electronic Engineering and Computer Science, Queen Mary University of London, London, United Kingdom.

Jul 27, 2026J/K move · Enter open · S save

MATS: A novel multi-modality multi-task learning framework for 3D perception in autonomous driving

Junchen Huo, Wanming Hao, Song Wang, Enqing Chen+2

School of Electrical and Information Engineering, Zhengzhou University, Zhengzhou 450001, China · Tianping College of Suzhou University of Science and Technology, Suzhou 215009, China · department of computer science, Toronto Metropolitan University, Toronto M5B2K3, Canada

Mixture of Experts Camera-LiDAR Fusion Autonomous Driving Perception Multi-Task Learning Bird's-Eye-View Representation

Open paper AskPDF arXiv

Multi-modality data from different sensors provides rich complementary information for 3D perception, becoming an essential component in reliable autonomous driving systems. Current research typically designs intricate and complex fusion strategies to integrate information from multimodal data on a unified bird's-eye-view (BEV) feature map for the joint learning of multiple perception tasks. However, such a single feature map hardly carries sufficient information to simultaneously meet the requirements of various perception tasks, leading to a very limited perception performance. To mitigate this limitation, this paper proposes MATS, a novel multi-modality multi-task learning approach with modality-adaptive BEV fusion and task-specific Mixture-of-Experts (MoE) for 3D perception. Specifically, a simple modality-adaptive BEV fusion module is designed to adaptively recalibrate the BEV features by modeling the global cross-modality dependencies, generating diverse BEV feature maps for various perception tasks. For joint multi-task learning, this paper proposes a task-specific MoE module to decouple the tasks and enable the network to automatically choose the appropriate BEV feature candidates for each specific task. To validate the effectiveness of the proposed approach, we conduct extensive experiments on the large-scale benchmark nuScenes. With the camera- and LiDAR-modality input data, the proposed approach outperforms the state-of-the-art (SOTA) by a significant margin. Furthermore, the experimental results on the single tasks show that the proposed approach significantly outperforms the baselines. The code and trained models will be available upon publication.

Abstract

Autonomous Driving Perception

Explore similar work

MATS: A novel multi-modality multi-task learning framework for 3D perception in autonomous driving

Can BEV Perception Gracefully Degrade under Sensor Failures?

Mining Cross-Modal Perception Gaps in Autonomous Driving: When Camera and LiDAR Disagree