CCF: Complementary Collaborative Fusion for Domain Generalized Multi-Modal 3D Object Detection

Yuchen Wu; Kun Wang; Yining Pan; Na Zhao

cs.CVMar 24, 2026

CCF: Complementary Collaborative Fusion for Domain Generalized Multi-Modal 3D Object Detection

Authors: Yuchen Wu, Kun Wang, Yining Pan, Na Zhao

Organizations: Singapore University of Technology and Design

Abstract

Multi-modal fusion has emerged as a promising paradigm for accurate 3D object detection. However, performance degrades substantially when deployed in target domains different from training. In this work, focusing on dual-branch proposal-level detectors, we identify two factors that limit robust cross-domain generalization: 1) in challenging domains such as rain or nighttime, one modality may undergo severe degradation; 2) the LiDAR branch often dominates the detection process, leading to systematic underutilization of visual cues and vulnerability when point clouds are compromised. To address these challenges, we propose three components. First, Query-Decoupled Loss provides independent supervision for 2D-only, 3D-only, and fused queries, rebalancing gradient flow across modalities. Second, LiDAR-Guided Depth Prior augments 2D queries with instance-aware geometric priors through probabilistic fusion of image-predicted and LiDAR-derived depth distributions, improving their spatial initialization. Third, Complementary Cross-Modal Masking applies complementary spatial masks to the image and point cloud, encouraging queries from both modalities to compete within the fused decoder and thereby promoting adaptive fusion. Extensive experiments demonstrate substantial gains over state-of-the-art baselines while preserving source-domain performance. Code and models are publicly available at https://github.com/IMPL-Lab/CCF.

Figures & tables

Figure 1: (a) Qualitative examples show that LiDAR and camera modalities degrade differently under adverse conditions. (b) Quantitative results on a baseline dual-branch detector show that our method substantially improves camera-originated queries and narrows their performance gap to LiDAR-originated queries.

Figure 2: Analysis of 2D proposal quality. We compare the 2D mAP@50 of proposals from the 2D detector (Faster R-CNN) against projected 3D boxes from the 3D detector (ISFusion). The results show that native 2D proposals consistently outperform projected 3D proposals across all domains.

Figure 3: Overview of CCF. CCF addresses modality imbalance with three components. (a) Query Decoupled Loss uses three parallel, weight-shared decoder passes (2D-only, 3D-only, and fused) to provide modality-specific supervision while avoiding shortcut learning. (b) LiDAR-Guided Depth Prior adaptively fuses image-predicted and LiDAR-derived depth distributions to improve 2D query initialization. (c) Complementary Cross-Modal Masking applies complementary spatial masking, encouraging balanced competition between camera- and LiDAR-originated queries. Together, these components improve modality balance and robustness under domain shift.

Figure 4: Illustration of our LiDAR-Guided Depth Prior. For each 2D proposal, we extract a learned depth distribution from image features ( $\mathbf{d}^{2d}$ ) and a geometric prior from LiDAR points ( $\mathbf{d}^{3d}$ ). A confidence network adaptively predicts a fusion weight ( $\lambda$ ) to combine them into a fused distribution ( $\mathbf{d}^{Fused}$ ), which provides robust depth initialization for the 2D query.

Model	Reference	Modality	Source		Rain		Night		Boston		Average
Model	Reference	Modality	mAP	NDS	mAP	NDS	mAP	NDS	mAP	NDS	mAP	NDS
FSDv2 [ 15 ]	TPAMI 24	L	59.6	62.7	23.4	41.6	36.6	42.8	28.2	45.1	29.4	43.2
CMT [ 35 ]	ICCV 23	L+C	61.4	62.3	35.7	46.5	37.8	42.0	42.1	50.8	38.5	46.4
MOAD [ 4 ]	arXiv 24	L+C	64.1	64.4	39.2	49.4	41.1	44.3	43.9	53.0	41.4	48.9
MEFormer [ 4 ]	arXiv 24	L+C	63.4	63.7	40.1	50.1	41.1	44.0	44.3	53.0	41.8	49.0
ISFusion [ 36 ]	CVPR 24	L+C	66.3	65.4	39.8	49.4	41.8	45.1	45.4	53.4	42.3	49.3

Table 1: Main results on nuScenes domain generalization splits. We report mAP (%) and NDS (%) on the source domain (Src) and three target domains (Rain, Night, Boston). Average is computed over Rain/Night/Boston. L and C denote LiDAR and Camera, respectively.

\dagger

represents the model trained with nuScenes full split.

Models	All		Rain		Night
Models	mAP	NDS	mAP	NDS	mAP	NDS
BEVDet [ 10 ]	34.9	41.7	30.1	43.2	12.0	21.7
PETR [ 17 ]	37.0	44.2	41.9	50.6	17.2	24.2
CenterPoint [ 37 ]	59.6	66.8	59.2	-	35.4	-
FSDv2 [ 15 ]	64.7	70.4	61.1	68.8	37.1	43.6
Transfusion-L [ 1 ]	65.1	70.1	64.0	69.9	37.5	43.5

Table 2: Oracle results on nuScenes domain generalization splits.

Figure 5: Examples of 3D object detections on different data splits. We visualize the 3D bounding boxes of car, truck and pedestrian with orange , magenta and blue colors in the multi-view images.

Components			Source		Rain		Night		Boston
DL	DP	CM	mAP	NDS	mAP	NDS	mAP	NDS	mAP	NDS
✗	✗	✗	68.4	65.7	41.9	50.0	42.9	44.5	47.4	53.6
✗	✓	✗	68.6	66.0	42.3	50.6	41.8	44.4	47.5	53.8
✗	✗	✓	68.4	65.8	44.5	51.4	43.4	45.2	49.6	56.4
✗	✓	✓	68.7	66.1	44.7	51.5	44.1	45.4	49.7	56.3
✓	✗	✗	68.4	66.1	42.8	52.4	42.1	44.7	48.1	56.0

Table 3: Ablation studies of the proposed components. We evaluate Query Decoupled Loss (DL), LiDAR-Guided Depth Prior (DP), and Complementary Cross-Modal Masking (CM), and report mAP (%) and NDS (%) on all domains.

Configuration			Rain		Night		Boston
	Variant	Cur.	mAP	NDS	mAP	NDS	mAP	NDS
(a)	Image GridMask	✗	42.8	50.7	42.5	45.3	48.3	54.7
(b)	Modal Mask	✗	42.9	49.7	42.6	44.5	48.4	55.4
(c)	Consistent GridMask	✗	42.8	49.8	42.2	44.3	48.9	55.5
(d)	Complementary GridMask	✗	43.9	50.0	43.1	44.2	49.2	55.1
(e)	Complementary RandomMask	✓	44.1	51.1	44.1	46.0	49.7	55.2

Table 4: Ablation studies on Complementary Cross-Modal Mask variants. “Cur.” indicates whether curriculum learning is applied.

Explore similar work

CAMF-Det: Closure-Aware Multimodal Fusion for LiDAR-Camera 3D Object Detection on UAV Platforms
Jun 8, 2026Yanze Jiang, Yanfeng Gu, Xian LiLidar-Based 3D Object Detection Multimodal Robustness
School of Electronics and Information Engineering, Harbin Institute of Technology, Harbin 150001, China
SARFusion: Scene-Aware Routing Fusion for Robust Camera-LiDAR 3D Object Detection
Sep 24, 2026Yuting Zhao, Ziyi Zheng, Shuxiao LiCamera-LiDAR Fusion 3D Object Detection
Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Wuhan College, Wuhan, China
Multi-Modal Decouple and Recouple Network for Robust 3D Object Detection
Mar 8, 2026Rui Ding, Zhaonian Kuang, Yuzhe Ji +33D Object Detection Multimodal Robustness
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, P.R.China · Intelligent Transportation Thrust of the Systems Hub, The Hong Kong University of Science and Technology (Guangzhou), P.R.China · Multimodal Experiences Research Lab, Dolby Laboratories, Los Angeles, CA, USA

Jun 8, 2026J/K move · Enter open · S save

CAMF-Det: Closure-Aware Multimodal Fusion for LiDAR-Camera 3D Object Detection on UAV Platforms

Yanze Jiang, Yanfeng Gu, Xian Li

School of Electronics and Information Engineering, Harbin Institute of Technology, Harbin 150001, China

Lidar-Based 3D Object Detection Multimodal Robustness Camera-LiDAR Fusion 3D Object Detection Object Detection

Open paper AskPDF arXiv

Multimodal 3D object detection based on LiDAR and cameras has demonstrated excellent performance in ground-vehicle scenarios, but has not been explored for Unmanned Aerial Vehicle (UAV) platforms. In UAV top-down scenes, frequent groundobject occlusion dominated by tree canopies causes spatially varying and modality-dependent information degradation. Existing multimodal fusion frameworks neither explicitly model such ground-object occlusion nor embed occlusion awareness into the detection pipeline, limiting their performance in occluded UAV scenes. To address these challenges, we propose CAMF-Det, a closure-aware multimodal fusion framework for LiDAR-camera 3D object detection on UAV platforms, which derives dual-modal occlusion intensity through physics-inspired modeling and embeds them as priors throughout the detection pipeline. First, a dual-modal closure modeling module explicitly constructs occlusion intensity ground truth for both modalities offline via a Beer-Lambert-inspired formulation and building-mask correction. Second, using these ground-truth maps as supervision, a dual-modal prediction network converts the offline modeling results into online occlusion intensity predictions under single-frame inference. Third, both ground-truth and predicted occlusion intensity are injected into data augmentation, feature encoding, multimodal fusion, and detection head, enabling adaptive detection under spatially varying and modality-dependent information degradation. Experiments on two self-built UAV-based multimodal datasets, SI3D-DI and SI3D-DII, demonstrate that CAMF-Det achieves the best performance across all difficulty levels, with hard-level mAP

_{\mathrm{BEV}}

improvements of 9.43% and 4.88% over the best competing methods, respectively. These results confirm the effectiveness of explicit occlusion prior modeling and exploitation for robust multimodal 3D detection in UAV scenes.

Abstract

Explore similar work

CAMF-Det: Closure-Aware Multimodal Fusion for LiDAR-Camera 3D Object Detection on UAV Platforms

SARFusion: Scene-Aware Routing Fusion for Robust Camera-LiDAR 3D Object Detection

Multi-Modal Decouple and Recouple Network for Robust 3D Object Detection