EIMC: Efficient Instance-aware Multi-modal Collaborative Perception

Kang Yang; Peng Wang; Lantao Li; Tianci Bu; Chen Sun; Deying Li; Yongcai Wang

cs.CVMar 3, 2026

EIMC: Efficient Instance-aware Multi-modal Collaborative Perception

Authors: Kang Yang, Peng Wang, Lantao Li, Tianci Bu, Chen Sun, Deying Li, Yongcai Wang

Organizations: School of Information, Renmin University of China, Beijing, China, 100872 · Sony Research and Development Center China, Beijing, China · National University of Defense Technology, Hunan, China, 410073

Abstract

Multi-modal collaborative perception calls for great attention to enhancing the safety of autonomous driving. However, current multi-modal approaches remain a ``local fusion to communication'' sequence, which fuses multi-modal data locally and needs high bandwidth to transmit an individual's feature data before collaborative fusion. EIMC innovatively proposes an early collaborative paradigm. It injects lightweight collaborative voxels, transmitted by neighbor agents, into the ego's local modality-fusion step, yielding compact yet informative 3D collaborative priors that tighten cross-modal alignment. Next, a heatmap-driven consensus protocol identifies exactly where cooperation is needed by computing per-pixel confidence heatmaps. Only the Top-K instance vectors located in these low-confidence, high-discrepancy regions are queried from peers, then fused via cross-attention for completion. Afterwards, we apply a refinement fusion that involves collecting the top-K most confident instances from each agent and enhancing their features using self-attention. The above instance-centric messaging reduces redundancy while guaranteeing that critical occluded objects are recovered. Evaluated on OPV2V and DAIR-V2X, EIMC attains 73.01% [email protected] while reducing byte bandwidth usage by 87.98% compared with the best published multi-modal collaborative detector. Code publicly released at https://github.com/sidiangongyuan/EIMC.

Figures & tables

Fig. 1: Compared with other intermediate fusion methods, EIMC achieves lower communication volume while still attaining the best performance. The

\text{Ours}^{\dagger}

variant represents the version without the Mix-Voxel module. BM2CP [ 3 ] is the multimodal-based collaborative perception work.

Fig. 2: Framework. Given LiDAR and camera inputs, our method first extracts heterogeneous features through dedicated modality-specific encoders. The Mix-Voxel (MV) module leverages lightweight voxel transmission as priors to build the collaborative voxel and then constructs occupancy-guided voxel-based image representations, which are compressed into BEV features and fused with LiDAR BEV features through Heterogeneous Modality Fusion (HMF). Instance Completion (IC) and Instance Refinement (IR) modules subsequently propagate instance-level messages identified from heatmap priors. The collaboration employs multi-scale feature for final detection, with predicted and ground truth bounding boxes visualized as green and red boxes respectively.

Fig. 3: Mix-Voxel module constructs a local graph of voxels, utilizing self-attention mechanisms to facilitate information exchange.

Fig. 4: Heterogeneous Modality Fusion module . The module consists of two streams. The left part uses an attention-based method to establish the interaction between the two modalities, while the right part employs a basic operation (e.g., concatenation) followed by convolutional layers to fuse the modalities.

Fig. 5: Instance-level communication . The Instance Completion (IC) module prioritizes critical regions by analyzing cross-agent heatmap discrepancies, performing instance completion via cross-attention. The IR module first selects agent-specific instances from heatmaps, then refines them via self-attention. Finally, it aggregates instance-to-scene context by cross-attending BEV features (query) to instance representations (key/value).

	OPV2V			DAIR-V2X			Bandwidth
Method	AP30	AP50	AP70	AP30	AP50	AP70	Comm ( $\log_{2}$ )
No Coll	83.63	63.74	58.32	69.99	65.02	53.82	0.00
Fcooper	93.88	89.03	74.28	76.61	69.29	51.37	24.00
Attn	88.08	86.30	75.32	68.78	63.16	49.17	24.00
V2VNet	93.56	93.13	89.00	77.36	72.22	52.95	25.43
V2X-VIT	95.09	93.66	86.06	77.29	71.87	55.46	24.00

TABLE I: Comparison of mainstream works on the OPV2V and DAIR-V2X dataset. The best performance is highlighted in bold .

\text{Ours}^{\dagger}

indicates the model without the Mix-Voxel module.

Fig. 6: Visualization of predictions from BM2CP and EIMC on OPV2V dataset.

Method	AP30	AP50	AP70
No Coll	$65.26$	$59.88$	$48.52$
Fcooper	$73.10$	$65.36$	$44.52$
Attn	$69.85$	$64.33$	$51.12$
V2VNet	$66.92$	$53.60$	$49.83$
V2X-VIT	$67.14$	$63.30$	$49.34$
CoBEVT	75.07	$67.70$	$47.09$

TABLE II: LiDAR-only performance comparison of different methods on DAIR-V2X dataset.

MV	IC	IR	MS	AP30	AP50	AP70	#Params (M)
-	-	-	-	70.25	67.73	50.91	$\sim$ 25.9
✓	-	-	-	71.02	68.39	51.44	$\sim$ 26.1
✓	✓	-	-	74.12	69.34	54.78	$\sim$ 26.8
✓	-	✓	-	71.77	68.24	52.09	$\sim$ 28.0
-	✓	✓	✓	76.82	71.81	56.76	$\sim$ 42.0
✓	✓	✓	-	74.22	70.54	56.28	$\sim$ 30.9

TABLE III: Ablation Studies of Components on the DAIR-V2X Dataset.

Instance Completion				Instance Refinement
$\mathbf{K}_{\text{IC}}$	AP30	AP50	AP70	$\mathbf{K}_{\text{IR}}$	AP30	AP50	AP70
10	$73.46$	$68.69$	$53.51$	200/100/50	75.93	$71.19$	$56.52$
15	$73.28$	$68.16$	$52.68$	150/100/50	$75.09$	$70.82$	$56.72$
20	$75.01$	73.01	58.37	100/50/25	$75.01$	73.01	58.37
25	77.78	$72.15$	$54.96$	100/100/100	$74.34$	$69.84$	$54.70$
30	$75.71$	$70.68$	$54.57$	50/50/50	$72.39$	$68.96$	$54.12$

TABLE IV: Ablation study of Top-k selection strategies on DAIR-V2X Dataset.

Fig. 7: Average Precision at IoU=0.5 with respect to agent number. Note that all methods were trained using the default agent setting of 4, and the results were obtained through inference with varying agent numbers. With the exception of CoBEVT, which, due to the inherent limitations of the method, requires retraining based on the number of agents.

Fig. 8: Visualization of heatmap and result. (a) Ego agent heatmap; (b) Communication agent heatmap; (c) Target heatmap; (d) Output. The area within the yellow circle highlights regions of interest.

Method	AP50 under Noise Level ( $\sigma_{p}$ / $\sigma_{r}$ )								AP70 under Noise Level ( $\sigma_{p}$ / $\sigma_{r}$ )
Method	OPV2V				DAIR-V2X				OPV2V				DAIR-V2X
	0/0	0.2/0.2	0.4/0.4	0.6/0.6	0/0	0.2/0.2	0.4/0.4	0.6/0.6	0/0	0.2/0.2	0.4/0.4	0.6/0.6	0/0	0.2/0.2	0.4/0.4	0.6/0.6
No Coll	63.74	-	-	-	65.02	-	-	-	58.32	-	-	-	53.82	-	-	-
Fcooper	89.03	80.77	66.53	59.30	69.29	67.43	64.18	62.25	74.28	62.62	51.56	47.29	51.37	50.18	48.46	47.56
Attn	88.08	86.31	80.57	77.98	63.16	57.32	54.19	52.06	75.32	73.33	69.09	60.23	49.17	45.77	43.17	42.75
V2VNet	93.13	91.04	74.16	61.45	72.32	68.76	62.21	58.48	89.00	67.33	37.96	28.55	52.95	46.97	43.00	40.85

TABLE V: Robustness analysis under pose noise on OPV2V and DAIR-V2X. The pose noise follows a Gaussian distribution.

Strategy	AP30	AP50	AP70
Attn	$74.18$	$68.80$	$53.94$
Attn+Concat	$75.01$	73.01	58.37
Attn+Add	75.23	$70.94$	$58.06$

TABLE VI: Ablation studies of HMF module.

Method	AP30	AP50	AP70	Bandwidth( $\log_{2}$ )
M1	$75.01$	$73.01$	$58.37$	$20.16$
M2	$74.22$	$72.35$	$56.96$	$19.67$
M3	76.24	73.55	59.17	$22.46$

TABLE VII: Ablation studies of MV module.

Cooperative Perception

Cooperative perception lets connected vehicles, robots, and aerial agents combine sensor observations to understand scenes beyond a single viewpoint.

See all 111 papers

UECP: Uncertainty-Enhanced Collaborative Perception
Jun 22, 2026Kang Yang, Tianci Bu, Peng Wang +3Uncertainty Quantification Autonomous Driving Perception
Renmin University of China, Beijing, China · College of Systems Engineering, National University of Defense Technology, Changsha, China · Intelligent Shipping Center, China Waterborne Transport Research Institute, Beijing, China
GT-Space: Enhancing Heterogeneous Collaborative Perception with Ground Truth Feature Space
Mar 13, 2026Wentao Wang, Haoran Xu, Guang TanHeterogeneous Collaborative Perception Autonomous Driving Perception
Sun Yat-sen University, Shenzhen Campus, Shenzhen, China · Peng Cheng Laboratory (PCL), Shenzhen, China
INTACT: Ego-Guided Typed Sparse Evidence Retrieval for Heterogeneous Collaborative Perception
Jun 3, 2026Chen Li, Shengrong Yuan, Jialong Zuo +3Communication-Efficient Collaborative Perception Cooperative Perception
National Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology · Zhejiang Normal University

Jun 22, 2026J/K move · Enter open · S save

UECP: Uncertainty-Enhanced Collaborative Perception

Kang Yang, Tianci Bu, Peng Wang, Deying Li+2

Renmin University of China, Beijing, China · College of Systems Engineering, National University of Defense Technology, Changsha, China · Intelligent Shipping Center, China Waterborne Transport Research Institute, Beijing, China

Uncertainty Quantification Autonomous Driving Perception Multi-Scale Feature Fusion Cooperative Perception

Open paper AskPDF arXiv

Collaborative perception serves as a pivotal solution to enhance the perception capability of individual agents in autonomous driving, where a core challenge lies in seeking reliable evidence to quantify and weight the contribution of each participating agent. Existing methods typically rely on a confidence map, which is co-trained with the detection head, but it is inherently correlated with the detection results and thus fails to provide unbiased physical evidence. Furthermore, how to deeply integrate evidence into the cooperative fusion process remains an open question. To address these issues, this paper first proposes an uncertainty map, a physically grounded and unambiguous metric for evaluating perception quality. This map is directly supervised by real-time sensor signals, i.e., LiDAR point density, ensuring decoupling from detection noise and thereby providing physical scenario-aware evidence for weighting agent contribution. Based on this map, we develop the Uncertainty-Enhanced Collaborative Perception (UECP) framework, centered on the Uncertainty-Aware Pyramid Fusion (UAPF) module. UAPF uses a coarse-to-fine strategy, with two key components: Uncertainty-Weighted Downsampling (UWD) for high-fidelity feature preservation, and Uncertainty-Guided Residual Fusion (UGRF) to reinforce ego features, suppressing noise and ensuring robust fusion. Extensive experiments on real-world datasets show UECP outperforms state-of-the-art methods in effectiveness and robustness by embedding the uncertainty map into fusion. Code will be publicly available.

Abstract

Cooperative Perception

Explore similar work

UECP: Uncertainty-Enhanced Collaborative Perception

GT-Space: Enhancing Heterogeneous Collaborative Perception with Ground Truth Feature Space

INTACT: Ego-Guided Typed Sparse Evidence Retrieval for Heterogeneous Collaborative Perception