Boosting Instance Awareness via Cross-View Correlation with 4D Radar and Camera for 3D Object Detection

Xiaokai Bai; Lianqing Zheng; Si-Yuan Cao; Xiaohan Zhang; Zhe Wu; Beinan Yu; Fang Wang; Jie Bai

cs.CVFeb 24, 2026

Boosting Instance Awareness via Cross-View Correlation with 4D Radar and Camera for 3D Object Detection

Authors: Xiaokai Bai, Lianqing Zheng, Si-Yuan Cao, Xiaohan Zhang, Zhe Wu, Beinan Yu, Fang Wang, Jie Bai,

Organizations: College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, China · School of Automotive Studies, Tongji University, Shanghai 201804, China · College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China, and also with the Jinhua Institute of Zhejiang University, Jinhua 321299, China · School of Information and Electrical Engineering, Hangzhou City University, Hangzhou 310015, China, and also with the Hangzhou City University Binjiang Innovation Center, Hangzhou 310052, China

Abstract

4D millimeter-wave radar has emerged as a promising sensing modality for autonomous driving due to its robustness and affordability. However, its sparse and weak geometric cues make reliable instance activation difficult, limiting the effectiveness of existing radar-camera fusion paradigms. BEV-level fusion offers global scene understanding but suffers from weak instance focus, while perspective-level fusion captures instance details but lacks holistic context. To address these limitations, we propose SIFormer, a scene-instance aware transformer for 3D object detection using 4D radar and camera. SIFormer first suppresses background noise during view transformation through segmentation- and depth-guided localization. It then introduces a cross-view activation mechanism that injects 2D instance cues into BEV space, enabling reliable instance awareness under weak radar geometry. Finally, a transformer-based fusion module aggregates complementary image semantics and radar geometry for robust perception. As a result, with the aim of enhancing instance awareness, SIFormer bridges the gap between the two paradigms, combining their complementary strengths to address inherent sparse nature of radar and improve detection accuracy. Experiments demonstrate that SIFormer achieves state-of-the-art performance on View-of-Delft, TJ4DRadSet and NuScenes datasets. Source code is available at github.com/shawnnnkb/SIFormer.

Figures & tables

Fig. 1: Comparison of radar and camera fusion pipelines. Previous radar-camera fusion models typically adopt either (a) BEV-level or (b) perspective-level fusion. Our SIFormer, with the aim of enhancing instance awareness, bridges the gap between the two paradigms while combining their complementary strengths, as shown in (c).

Fig. 2: Visualization comparison of LiDAR and 4D radar on the VoD dataset: the first row shows LiDAR, and the second row shows 4D radar. The first column displays 3D ground truth boxes and the point cloud projection onto the foreground mask, with purple points indicating valid points providing object information. The second and third columns show the point cloud projection onto the perspective View and bird’s-eye view, respectively. Dense LiDAR provides strong geometry, while sparse 4D radar only provides weak geometry.

Fig. 3: Comparison between IS-Fusion and SIFormer. (a) IS-Fusion mines instance features directly from scene features. (b) SIFormer employs cross-view correlation to improve radar-camera fusion, addressing weak radar geometry by activating instance awareness using 2D instance features.

Fig. 4: Architecture of our SIFormer. (a) The feature extractor extracts 4D radar and image feature from raw data. (b) The instance initialization stage filters out irrelevant features during view transformation via segmentation and depth-guided localization to focus on regions of interest introduces, while achieving global scene understanding. (c) The instance awareness enhancement stage leverages cross view correlation (CVC) to bridge perspective view instance feature with bird’s-eye view scene feature, followed by the instance enhance attention (IEA) module for further refinement, producing fused feature across scene and instance levels. (d) The decoder head for 3D object detection.

Fig. 5: The detailed illustration of our instance initialization within scene stage. We employ sparse scene integration (SSI) to update depth and context, then fed then into hybrid view transformation to provide image BEV feature.

Fig. 6: The illustration of our instance awareness enhancement stage. We first employ cross view correlation (CVC) to activate all potential regions of interest within scene feature using a learnable token. To be specific, the instance attended correlation connects the aggregated instances with the RC-BEV through matrix operations, producing a correlation map, which is then used to calculate the cosine similarity with the scene feature. Consequently, the output of CVC serve as improved queries for the subsequent instance enhance attention (IEA), facilitating further aggregation of semantics and geometry.

Fig. 7: The illustration of our instance enhancement attention (IEA), consisted of semantic enhancement module and geometry enhancement module.

Fig. 8: Visualization results on the VoD validation set ((a),(b),(c)) and TJ4DRadSet test set ((d),(e),(f)). Each figure corresponds to a frame. Orange and yellow boxes represent ground-truths in the perspective and bird’s-eye view, respectively. Green and blue boxes indicate predicted results.

Fig. 9: Qualitative presentation of SIFormer. The first row is prediction and the second row denotes the ground truth. White dots represent LiDAR, while colored dots represent 4D radar, with color differences indicating velocity variations. The comparison highlights the effectiveness of our SIFormer.

Approach	Input	Entire Annotated Area (%)				Driving Corridor (%)				FPS
Approach	Input	Car	Ped	Cyc	mAP	Car	Ped	Cyc	mAP	FPS
ImVoxelNet (WACV 2022) [ 49 ]	C	19.35	5.62	17.53	14.17	49.52	9.68	28.97	29.39	11.1
RadarPillarNet (IEEE TIM 2023) [ 12 ]	R	39.30	35.10	63.63	46.01	71.65	42.80	83.14	65.86	98.8
LXL - R (IEEE TIV 2023)	R	32.75	39.65	68.13	46.84	70.26	47.34	87.93	68.51	44.7
SMURF (IEEE TIV 2023) [ 32 ]	R	42.31	39.09	71.50	50.97	71.74	50.54	86.87	69.72	-
FUTR3D (CVPR 2023) [ 43 ]	R+C	46.01	35.11	65.98	49.03	78.66	43.10	86.19	69.32	7.3

TABLE I: Comparison with state-of-the-art approaches on the validation set of VoD [ 6 ] . In the Modality column, R denotes 4D radar and C denotes camera. The results of approaches marked with

\dagger

indicate the use of extra LiDAR data. The best values are in bold, and the second-best values are underlined.

Approach	Input	$\text{AP}_{\text{3D}}$ (%)					$\text{AP}_{\text{BEV}}$ (%)
Approach	Input	Car	Ped	Cyc	Truck	mAP	Car	Ped	Cyc	Truck	mAP
ImVoxelNet (WACV 2022) [ 49 ]	C	22.55	13.73	9.67	13.87	14.96	26.10	14.21	10.99	17.18	17.12
RadarPillarNet (IEEE TIM 2023) [ 12 ]	R	28.45	26.24	51.57	15.20	30.37	45.72	29.19	56.89	25.17	39.24
LXL - R (TIV 2023) [ 11 ]	R	-	-	-	-	30.79	-	-	-	-	38.42
SMURF (IEEE TIV 2023) [ 32 ]	R	28.47	26.22	54.61	22.64	32.99	43.13	29.19	58.81	32.80	40.98
FUTR3D (CVPR 2023) [ 43 ]	R+C	-	-	-	-	32.42	-	-	-	-	37.51

TABLE II: Comparison with state-of-the-art approaches on the test set of TJ4DRadSet [ 7 ] . The other settings are the same as in Table I .

Approach	Input	Backbone	Image Size	NDS $\uparrow$	mAP $\uparrow$
BEVDet [ 23 ]	C	R50	256 × 704	39.2	31.2
BEVDepth [ 25 ]	C	R50	256 × 704	47.5	35.1
FB-BEV [ 47 ]	C	R50	256 × 704	49.8	37.8
CenterFusion [ 17 ]	R+C	DLA34	448 × 800	45.3	33.2
CRAFT [ 15 ]	R+C	DLA34	448 × 800	51.7	41.1
RCBEVDet [ 13 ]	R+C	DLA34	448 × 800	56.3	45.3

TABLE III: Comparison of 3D object detection results on the nuScenes val set. L, C, and R represent LiDAR, Camera, and 3D Radar, respectively.

Baseline	w/ SSI	w/ CVC	w/ IEA	$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$
✓				56.44	73.06
✓	✓			58.69	75.24
✓	✓	✓		59.33	76.14
✓	✓	✓	✓	59.94	76.71

TABLE IV: Ablation study on each component of SIFormer on the VoD dataset. We adopt LXL [ 11 ] supervised by extra LiDAR data as our baseline.

Fig. 10: Visualization of the interactions between key components in the SIFormer: Sparse Scene Integration (SSI), Cross-View Correlation (CVC), and Instance Enhance Attention (IEA). The figure shows how feature maps are progressively fused and processed, resulting in the activation of instances and the final detection output. Notably, the CVC mechanism reduces noise from the 4D radar and mitigates blurriness caused by inaccurate view transformations, thereby enhancing the clarity of instance-related regions. Zoom in for better view.

Method	Modality	4D Radar + Camera			Only Camera			Only 4D Radar
Method	Modality	$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$	$\text{mAP}_{\text{FAR}}$	$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$	$\text{mAP}_{\text{FAR}}$	$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$	$\text{mAP}_{\text{FAR}}$
ImVoxelNet (WACV 2022) [ 49 ]	C	-	-	-	14.17	29.39	-	-	-	-
RadarPillarNet (IEEE TIM 2023) [ 12 ]	R	-	-	-	-	-	-	46.33	67.08	31.87
LXL [ 11 ] (IEEE TIV 2023)	R+C	56.44	73.06	44.08	1.77	3.19	0.38	41.69	63.29	27.48
SGDet3D (IEEE RAL 2024) [ 14 ]	R+C	59.75	77.42	45.05	6.90	8.85	1.05	20.88	35.76	10.21
${\text{SIFormer}^{\dagger}}$ ( Ours )	R+C	63.32	83.06	48.91	17.22	34.88	7.27	46.22	67.42	31.41

TABLE V: Results on robustness setting of modality failure cases on the VoD dataset. FAR denotes the area outside the driving corridor within the entire annotated region. We record the mAP only.

Baseline	SSI		$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$
Baseline	w/ DGW	w/ SGW	$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$
✓			56.44	73.06
✓	✓		57.44	74.40
✓		✓	56.98	74.22
✓	✓	✓	58.69	75.24

TABLE VI: Ablation study on BEV-level fusion components on the VoD dataset. We adopt LXL [ 11 ] supervised by extra LiDAR data as our baseline.

Baseline	CVC		IEA		$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$
Baseline	w/ Det2D	w/ FDL	w/ GEM	w/ SEM	$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$
✓					58.69	75.24
✓	✓				58.97	75.20
✓	✓	✓			59.33	76.14
✓	✓	✓	✓		59.47	76.47
✓	✓	✓	✓	✓	59.94	76.71

TABLE VII: Ablation study on perspective-level fusion on the VoD dataset.

Baseline	Feature Disentanglement Learning				$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$
Baseline	w/ $\mathbf{T_{\text{q}}}$	w/ $\mathcal{L}_{\text{seg}}$	w/ $\mathcal{L}_{\text{neg}}$	w/ back	$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$
✓					58.97	75.36
✓	✓				58.85	75.32
✓	✓	✓			59.28	75.98
✓	✓	✓	✓		59.30	76.05
✓	✓	✓	✓	✓	59.33	76.14

TABLE VIII: Ablation study on cross view correlation (CVC) of the perspective-level fusion on the VoD dataset.

Setting	View-of-Delft Dataset			$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$
Setting	None	Radar	LiDAR	$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$
SGDet3D	✓			54.12	72.89
Ours	✓			56.05	74.34
SGDet3D		✓		56.67	74.03
Ours		✓		58.26	75.11
SGDet3D			✓	58.11	76.02

TABLE IX: Ablation study on depth label used for supervising depth prediction. Experiment is conducted using the complete SIFormer network. Half training epochs.

Baseline	w/ depth	w/ LSS	w/ SSI	$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$
✓				56.44	73.06
✓	✓			56.97	73.43
✓		✓		56.22	73.24
✓			✓	55.53	71.63
✓	✓	✓		57.44	74.40
✓	✓	✓	✓	58.69	75.24

TABLE X: Ablation study on the impact of sparse radar depth and LSS [ 45 ] participation on detection performance. We adopt LXL [ 11 ] supervised by extra LiDAR data as our baseline.

Module	$\text{mAP}_{\text{EAA}}$		$\text{mAP}_{\text{DC}}$
Module	IGF	CVC	IGF	CVC
HSF	54.40	56.88	74.75	76.44
SSI	58.13	59.98	72.59	77.23

TABLE XI: Comparison with IS-Fusion [ 9 ] . We replicated IS-Fusion on a 4D radar-based dataset, simply replacing the detection decoder with our anchor-based head.

Disturbance	$\text{mAP}_{\text{EAA}}$		$\text{mAP}_{\text{DC}}$
Disturbance	LXL	SIFormer	LXL	SIFormer
None	56.42	63.32	73.03	83.06
$\pm\text{2}^{\circ},\pm\text{0.2m}$	56.26	63.07	73.16	83.00
$\pm\text{5}^{\circ},\pm\text{0.5m}$	50.25	56.50	72.07	78.99
$\pm\text{10}^{\circ},\pm\text{1.0m}$	39.47	45.55	61.29	58.23
$\pm\text{20}^{\circ},\pm\text{1.5m}$	31.45	37.45	50.85	51.16

TABLE XII: Comparison with LXL [ 11 ] under the disturbance to the calibration matrix.

Percentage	View-of-Delft Dataset		TJ4DRadSet Dataset
Percentage	$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$	$\text{mAP}_{\text{3D}}$	$\text{mAP}_{\text{BEV}}$
None	58.11	75.22	42.12	45.56
25%	59.94	76.71	42.99	47.01
50%	59.29	76.37	42.81	46.75
100%	58.74	76.73	42.02	46.83

TABLE XIII: Ablation on depth-bin selection with half training epochs.

3D Object Detection

3D object detection identifies objects and estimates their locations, sizes, and orientations from images, point clouds, radar, or fused sensors.

See all 180 papers

RadarXFormer: Robust Object Detection via Cross-Dimension Fusion of 4D Radar Spectra and Images for Autonomous Driving
Mar 16, 2026Yue Sun, Yeqiang Qian, Zhe Wang +3Autonomous Driving Perception 3D Object Detection
Global Institute of Future Technology, Shanghai Jiao Tong University, Shanghai, 200240, China. · School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, 200240, China. · Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai Jiao Tong University, Shanghai, 200240, China. +2
Sparse4D-Radar: An Efficient and Robust Framework for Surround-View 3D Object Detection via 4D Radar-Camera Fusion
Jul 5, 2026Fuyuan Ai, Yuchen Tan, Jiehui Chen +2Camera-Radar Fusion Autonomous Driving Perception
State Key Laboratory of Ocean Sensing and Ocean College, Zhejiang University, Zhoushan 316021, China, and also with the Engineering Research Center of Oceanic Sensing Technology and Equipment, Ministry of Education, Zhejiang University, Zhoushan 316021, China · Donghai Laboratory, Zhoushan 316021, China, also with the Ocean College, Institute of Marine Electronics and Intelligent Systems, Zhejiang University, Zhoushan 316021, China, and also with the Engineering Research Center of Oceanic Sensing Technology and Equipment, Ministry of Education, Zhejiang University, Zhoushan 316021, China
R4Det: 4D Radar-Camera Fusion for High-Performance 3D Object Detection
Mar 12, 2026Zhongyu Xia, Yousen Tang, Yongtao Wang +23D Object Detection Camera-Radar Fusion
Wangxuan Institute of Computer Technology, Peking University · EBTech Co. Ltd

Mar 16, 2026J/K move · Enter open · S save

RadarXFormer: Robust Object Detection via Cross-Dimension Fusion of 4D Radar Spectra and Images for Autonomous Driving

Yue Sun, Yeqiang Qian, Zhe Wang, Tianhui Li+2

Global Institute of Future Technology, Shanghai Jiao Tong University, Shanghai, 200240, China. · School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, 200240, China. · Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai Jiao Tong University, Shanghai, 200240, China.+2

Autonomous Driving Perception 3D Object Detection Object Detection Camera-Radar Fusion Multi-Scale Feature Fusion

Open paper AskPDF arXiv

Reliable perception is essential for autonomous driving systems to operate safely under diverse real-world traffic conditions. However, camera- and LiDAR-based perception systems suffer from performance degradation under adverse weather and lighting conditions, limiting their robustness and large-scale deployment in intelligent transportation systems. Radar-vision fusion provides a promising alternative by combining the environmental robustness and cost efficiency of millimeter-wave (mmWave) radar with the rich semantic information captured by cameras. Nevertheless, conventional 3D radar measurements lack height resolution and remain highly sparse, while emerging 4D mmWave radar introduces elevation information but also brings challenges such as signal noise and large data volume. To address these issues, this paper proposes RadarXFormer, a 3D object detection framework that enables efficient cross-modal fusion between 4D radar spectra and RGB images. Instead of relying on sparse radar point clouds, RadarXFormer directly leverages raw radar spectra and constructs an efficient 3D representation that reduces data volume while preserving complete 3D spatial information. The "X" highlights the proposed cross-dimension (3D-2D) fusion mechanism, in which multi-scale 3D spherical radar feature cubes are fused with complementary 2D image feature maps. Experiments on the K-Radar dataset demonstrate improved detection accuracy and robustness under challenging conditions while maintaining real-time inference capability.

Abstract

3D Object Detection

Explore similar work

RadarXFormer: Robust Object Detection via Cross-Dimension Fusion of 4D Radar Spectra and Images for Autonomous Driving

Sparse4D-Radar: An Efficient and Robust Framework for Surround-View 3D Object Detection via 4D Radar-Camera Fusion

R4Det: 4D Radar-Camera Fusion for High-Performance 3D Object Detection