Boosting Instance Awareness via Cross-View Correlation with 4D Radar and Camera for 3D Object Detection

Xiaokai Bai; Lianqing Zheng; Si-Yuan Cao; Xiaohan Zhang; Zhe Wu; Beinan Yu; Fang Wang; Jie Bai

cs.CVFeb 24, 2026

Boosting Instance Awareness via Cross-View Correlation with 4D Radar and Camera for 3D Object Detection

Authors: Xiaokai Bai, Lianqing Zheng, Si-Yuan Cao, Xiaohan Zhang, Zhe Wu, Beinan Yu, Fang Wang, Jie Bai,

Organizations: College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, China · School of Automotive Studies, Tongji University, Shanghai 201804, China · College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China, and also with the Jinhua Institute of Zhejiang University, Jinhua 321299, China · School of Information and Electrical Engineering, Hangzhou City University, Hangzhou 310015, China, and also with the Hangzhou City University Binjiang Innovation Center, Hangzhou 310052, China

Abstract

4D millimeter-wave radar has emerged as a promising sensing modality for autonomous driving due to its robustness and affordability. However, its sparse and weak geometric cues make reliable instance activation difficult, limiting the effectiveness of existing radar-camera fusion paradigms. BEV-level fusion offers global scene understanding but suffers from weak instance focus, while perspective-level fusion captures instance details but lacks holistic context. To address these limitations, we propose SIFormer, a scene-instance aware transformer for 3D object detection using 4D radar and camera. SIFormer first suppresses background noise during view transformation through segmentation- and depth-guided localization. It then introduces a cross-view activation mechanism that injects 2D instance cues into BEV space, enabling reliable instance awareness under weak radar geometry. Finally, a transformer-based fusion module aggregates complementary image semantics and radar geometry for robust perception. As a result, with the aim of enhancing instance awareness, SIFormer bridges the gap between the two paradigms, combining their complementary strengths to address inherent sparse nature of radar and improve detection accuracy. Experiments demonstrate that SIFormer achieves state-of-the-art performance on View-of-Delft, TJ4DRadSet and NuScenes datasets. Source code is available at github.com/shawnnnkb/SIFormer.

Figures & tables

Fig. 1: Comparison of radar and camera fusion pipelines. Previous radar-camera fusion models typically adopt either (a) BEV-level or (b) perspective-level fusion. Our SIFormer, with the aim of enhancing instance awareness, bridges the gap between the two paradigms while combining their complementary strengths, as shown in (c).

Fig. 2: Visualization comparison of LiDAR and 4D radar on the VoD dataset: the first row shows LiDAR, and the second row shows 4D radar. The first column displays 3D ground truth boxes and the point cloud projection onto the foreground mask, with purple points indicating valid points providing object information. The second and third columns show the point cloud projection onto the perspective View and bird’s-eye view, respectively. Dense LiDAR provides strong geometry, while sparse 4D radar only provides weak geometry.

Fig. 3: Comparison between IS-Fusion and SIFormer. (a) IS-Fusion mines instance features directly from scene features. (b) SIFormer employs cross-view correlation to improve radar-camera fusion, addressing weak radar geometry by activating instance awareness using 2D instance features.

Fig. 4: Architecture of our SIFormer. (a) The feature extractor extracts 4D radar and image feature from raw data. (b) The instance initialization stage filters out irrelevant features during view transformation via segmentation and depth-guided localization to focus on regions of interest introduces, while achieving global scene understanding. (c) The instance awareness enhancement stage leverages cross view correlation (CVC) to bridge perspective view instance feature with bird’s-eye view scene feature, followed by the instance enhance attention (IEA) module for further refinement, producing fused feature across scene and instance levels. (d) The decoder head for 3D object detection.

Fig. 5: The detailed illustration of our instance initialization within scene stage. We employ sparse scene integration (SSI) to update depth and context, then fed then into hybrid view transformation to provide image BEV feature.

Fig. 6: The illustration of our instance awareness enhancement stage. We first employ cross view correlation (CVC) to activate all potential regions of interest within scene feature using a learnable token. To be specific, the instance attended correlation connects the aggregated instances with the RC-BEV through matrix operations, producing a correlation map, which is then used to calculate the cosine similarity with the scene feature. Consequently, the output of CVC serve as improved queries for the subsequent instance enhance attention (IEA), facilitating further aggregation of semantics and geometry.

Fig. 7: The illustration of our instance enhancement attention (IEA), consisted of semantic enhancement module and geometry enhancement module.

Fig. 8: Visualization results on the VoD validation set ((a),(b),(c)) and TJ4DRadSet test set ((d),(e),(f)). Each figure corresponds to a frame. Orange and yellow boxes represent ground-truths in the perspective and bird’s-eye view, respectively. Green and blue boxes indicate predicted results.

Fig. 9: Qualitative presentation of SIFormer. The first row is prediction and the second row denotes the ground truth. White dots represent LiDAR, while colored dots represent 4D radar, with color differences indicating velocity variations. The comparison highlights the effectiveness of our SIFormer.

Approach	Input	Entire Annotated Area (%)				Driving Corridor (%)				FPS
Approach	Input	Car	Ped	Cyc	mAP	Car	Ped	Cyc	mAP	FPS
ImVoxelNet (WACV 2022) [ 49 ]	C	19.35	5.62	17.53	14.17	49.52	9.68	28.97	29.39	11.1
RadarPillarNet (IEEE TIM 2023) [ 12 ]	R	39.30	35.10	63.63	46.01	71.65	42.80	83.14	65.86	98.8
LXL - R (IEEE TIV 2023)	R	32.75	39.65	68.13	46.84	70.26	47.34	87.93	68.51	44.7
SMURF (IEEE TIV 2023) [ 32 ]	R	42.31	39.09	71.50	50.97	71.74	50.54	86.87	69.72	-
FUTR3D (CVPR 2023) [ 43 ]	R+C	46.01	35.11	65.98	49.03	78.66	43.10	86.19	69.32	7.3

TABLE I: Comparison with state-of-the-art approaches on the validation set of VoD [ 6 ] . In the Modality column, R denotes 4D radar and C denotes camera. The results of approaches marked with

\dagger

indicate the use of extra LiDAR data. The best values are in bold, and the second-best values are underlined.

Approach	Input	$\text{AP}_{\text{3D}}$ (%)					$\text{AP}_{\text{BEV}}$ (%)
Approach	Input	Car	Ped	Cyc	Truck	mAP	Car	Ped	Cyc	Truck	mAP
ImVoxelNet (WACV 2022) [ 49 ]	C	22.55	13.73	9.67	13.87	14.96	26.10	14.21	10.99	17.18	17.12
RadarPillarNet (IEEE TIM 2023) [ 12 ]	R	28.45	26.24	51.57	15.20	30.37	45.72	29.19	56.89	25.17	39.24
LXL - R (TIV 2023) [ 11 ]	R	-	-	-	-	30.79	-	-	-	-	38.42
SMURF (IEEE TIV 2023) [ 32 ]	R	28.47	26.22	54.61	22.64	32.99	43.13	29.19	58.81	32.80	40.98
FUTR3D (CVPR 2023) [ 43 ]	R+C	-	-	-	-	32.42	-	-	-	-	37.51

TABLE II: Comparison with state-of-the-art approaches on the test set of TJ4DRadSet [ 7 ] . The other settings are the same as in Table I .

Approach	Input	Backbone	Image Size	NDS $\uparrow$	mAP $\uparrow$
BEVDet [ 23 ]	C	R50	256 × 704	39.2	31.2
BEVDepth [ 25 ]	C	R50	256 × 704	47.5	35.1
FB-BEV [ 47 ]	C	R50	256 × 704	49.8	37.8
CenterFusion [ 17 ]	R+C	DLA34	448 × 800	45.3	33.2
CRAFT [ 15 ]	R+C	DLA34	448 × 800	51.7	41.1
RCBEVDet [ 13 ]	R+C	DLA34	448 × 800	56.3	45.3

TABLE III: Comparison of 3D object detection results on the nuScenes val set. L, C, and R represent LiDAR, Camera, and 3D Radar, respectively.

Baseline	w/ SSI	w/ CVC	w/ IEA	$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$
✓				56.44	73.06
✓	✓			58.69	75.24
✓	✓	✓		59.33	76.14
✓	✓	✓	✓	59.94	76.71

TABLE IV: Ablation study on each component of SIFormer on the VoD dataset. We adopt LXL [ 11 ] supervised by extra LiDAR data as our baseline.

Fig. 10: Visualization of the interactions between key components in the SIFormer: Sparse Scene Integration (SSI), Cross-View Correlation (CVC), and Instance Enhance Attention (IEA). The figure shows how feature maps are progressively fused and processed, resulting in the activation of instances and the final detection output. Notably, the CVC mechanism reduces noise from the 4D radar and mitigates blurriness caused by inaccurate view transformations, thereby enhancing the clarity of instance-related regions. Zoom in for better view.

Method	Modality	4D Radar + Camera			Only Camera			Only 4D Radar
Method	Modality	$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$	$\text{mAP}_{\text{FAR}}$	$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$	$\text{mAP}_{\text{FAR}}$	$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$	$\text{mAP}_{\text{FAR}}$
ImVoxelNet (WACV 2022) [ 49 ]	C	-	-	-	14.17	29.39	-	-	-	-
RadarPillarNet (IEEE TIM 2023) [ 12 ]	R	-	-	-	-	-	-	46.33	67.08	31.87
LXL [ 11 ] (IEEE TIV 2023)	R+C	56.44	73.06	44.08	1.77	3.19	0.38	41.69	63.29	27.48
SGDet3D (IEEE RAL 2024) [ 14 ]	R+C	59.75	77.42	45.05	6.90	8.85	1.05	20.88	35.76	10.21
${\text{SIFormer}^{\dagger}}$ ( Ours )	R+C	63.32	83.06	48.91	17.22	34.88	7.27	46.22	67.42	31.41

TABLE V: Results on robustness setting of modality failure cases on the VoD dataset. FAR denotes the area outside the driving corridor within the entire annotated region. We record the mAP only.

Baseline	SSI		$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$
Baseline	w/ DGW	w/ SGW	$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$
✓			56.44	73.06
✓	✓		57.44	74.40
✓		✓	56.98	74.22
✓	✓	✓	58.69	75.24

TABLE VI: Ablation study on BEV-level fusion components on the VoD dataset. We adopt LXL [ 11 ] supervised by extra LiDAR data as our baseline.

Baseline	CVC		IEA		$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$
Baseline	w/ Det2D	w/ FDL	w/ GEM	w/ SEM	$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$
✓					58.69	75.24
✓	✓				58.97	75.20
✓	✓	✓			59.33	76.14
✓	✓	✓	✓		59.47	76.47
✓	✓	✓	✓	✓	59.94	76.71

TABLE VII: Ablation study on perspective-level fusion on the VoD dataset.

Baseline	Feature Disentanglement Learning				$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$
Baseline	w/ $\mathbf{T_{\text{q}}}$	w/ $\mathcal{L}_{\text{seg}}$	w/ $\mathcal{L}_{\text{neg}}$	w/ back	$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$
✓					58.97	75.36
✓	✓				58.85	75.32
✓	✓	✓			59.28	75.98
✓	✓	✓	✓		59.30	76.05
✓	✓	✓	✓	✓	59.33	76.14

TABLE VIII: Ablation study on cross view correlation (CVC) of the perspective-level fusion on the VoD dataset.

Setting	View-of-Delft Dataset			$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$
Setting	None	Radar	LiDAR	$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$
SGDet3D	✓			54.12	72.89
Ours	✓			56.05	74.34
SGDet3D		✓		56.67	74.03
Ours		✓		58.26	75.11
SGDet3D			✓	58.11	76.02

TABLE IX: Ablation study on depth label used for supervising depth prediction. Experiment is conducted using the complete SIFormer network. Half training epochs.

Baseline	w/ depth	w/ LSS	w/ SSI	$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$
✓				56.44	73.06
✓	✓			56.97	73.43
✓		✓		56.22	73.24
✓			✓	55.53	71.63
✓	✓	✓		57.44	74.40
✓	✓	✓	✓	58.69	75.24

TABLE X: Ablation study on the impact of sparse radar depth and LSS [ 45 ] participation on detection performance. We adopt LXL [ 11 ] supervised by extra LiDAR data as our baseline.

Module	$\text{mAP}_{\text{EAA}}$		$\text{mAP}_{\text{DC}}$
Module	IGF	CVC	IGF	CVC
HSF	54.40	56.88	74.75	76.44
SSI	58.13	59.98	72.59	77.23

TABLE XI: Comparison with IS-Fusion [ 9 ] . We replicated IS-Fusion on a 4D radar-based dataset, simply replacing the detection decoder with our anchor-based head.

Disturbance	$\text{mAP}_{\text{EAA}}$		$\text{mAP}_{\text{DC}}$
Disturbance	LXL	SIFormer	LXL	SIFormer
None	56.42	63.32	73.03	83.06
$\pm\text{2}^{\circ},\pm\text{0.2m}$	56.26	63.07	73.16	83.00
$\pm\text{5}^{\circ},\pm\text{0.5m}$	50.25	56.50	72.07	78.99
$\pm\text{10}^{\circ},\pm\text{1.0m}$	39.47	45.55	61.29	58.23
$\pm\text{20}^{\circ},\pm\text{1.5m}$	31.45	37.45	50.85	51.16

TABLE XII: Comparison with LXL [ 11 ] under the disturbance to the calibration matrix.

Percentage	View-of-Delft Dataset		TJ4DRadSet Dataset
Percentage	$\text{mAP}_{\text{EAA}}$	$\text{mAP}_{\text{DC}}$	$\text{mAP}_{\text{3D}}$	$\text{mAP}_{\text{BEV}}$
None	58.11	75.22	42.12	45.56
25%	59.94	76.71	42.99	47.01
50%	59.29	76.37	42.81	46.75
100%	58.74	76.73	42.02	46.83

TABLE XIII: Ablation on depth-bin selection with half training epochs.

3D Object Detection

3D object detection identifies objects and estimates their locations, sizes, and orientations from images, point clouds, radar, or fused sensors.

See all 181 papers

Mar 16, 2026

RadarXFormer: Robust Object Detection via Cross-Dimension Fusion of 4D Radar Spectra and Images for Autonomous Driving

Reliable perception is essential for autonomous driving systems to operate safely under diverse real-world traffic conditions. However, camera- and LiDAR-based perception systems suffer from performance degradation under adverse weather and lighting conditions, limiting their robustness and large-scale deployment in intelligent transportation systems. Radar-vision fusion provides a promising alternative by combining the environmental robustness and cost efficiency of millimeter-wave (mmWave) radar with the rich semantic information captured by cameras. Nevertheless, conventional 3D radar measurements lack height resolution and remain highly sparse, while emerging 4D mmWave radar introduces elevation information but also brings challenges such as signal noise and large data volume. To address these issues, this paper proposes RadarXFormer, a 3D object detection framework that enables efficient cross-modal fusion between 4D radar spectra and RGB images. Instead of relying on sparse radar point clouds, RadarXFormer directly leverages raw radar spectra and constructs an efficient 3D representation that reduces data volume while preserving complete 3D spatial information. The "X" highlights the proposed cross-dimension (3D-2D) fusion mechanism, in which multi-scale 3D spherical radar feature cubes are fused with complementary 2D image feature maps. Experiments on the K-Radar dataset demonstrate improved detection accuracy and robustness under challenging conditions while maintaining real-time inference capability.

Yue Sun, Yeqiang Qian, Zhe Wang +3

Global Institute of Future Technology, Shanghai Jiao Tong University, Shanghai, 200240, China. · School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, 200240, China. · Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai Jiao Tong University, Shanghai, 200240, China. +2

Autonomous Driving Perception 3D Object Detection Object Detection Camera-Radar Fusion Multi-Scale Feature Fusion

Ask PDF

Jul 5, 2026

Sparse4D-Radar: An Efficient and Robust Framework for Surround-View 3D Object Detection via 4D Radar-Camera Fusion

In recent years, 4D imaging radar has gained wide attention in autonomous driving for its robustness against harsh weather and ability to output target velocity. Nevertheless, mainstream 4D radar-camera fusion methods only support front-view perception, lacking mature solutions for surround-view sensing. Directly expanding these pipelines to full 360° coverage introduces excessive computation cost and limits real-world deployment. To tackle these limitations, this work proposes Sparse4D-Radar, an efficient robust surround-view multi-modal fusion framework. We first design a Deformable Fusion module to embed radar-camera features into sparse queries, constructing the lightweight base version Sparse4D-Radar-Base. Two dedicated modules are further introduced to boost localization accuracy and modality stability: Velocity-Consistency Sampling (VCS) refines features via radar velocity cues for motion awareness, and Adaptive Modality Gating (AMG) dynamically adjusts cross-modal fusion weights according to feature confidence. Combining all components, we build Sparse4D-Radar-Acc for high-precision detection demands. Comprehensive experiments on OmniHD-Scenes verify that our approach achieves state-of-the-art surround-view 3D detection performance. Compared with prior arts, our method obtains over 7% mAP and 10% ODS improvements under complex driving scenes while running at nearly 10 FPS, striking a favorable trade-off among detection accuracy, environmental robustness and inference efficiency. Our open-source code is available at https://github.com/Aiuan/Sparse4D-Radar.

Fuyuan Ai, Yuchen Tan, Jiehui Chen +2

State Key Laboratory of Ocean Sensing and Ocean College, Zhejiang University, Zhoushan 316021, China, and also with the Engineering Research Center of Oceanic Sensing Technology and Equipment, Ministry of Education, Zhejiang University, Zhoushan 316021, China · Donghai Laboratory, Zhoushan 316021, China, also with the Ocean College, Institute of Marine Electronics and Intelligent Systems, Zhejiang University, Zhoushan 316021, China, and also with the Engineering Research Center of Oceanic Sensing Technology and Equipment, Ministry of Education, Zhejiang University, Zhoushan 316021, China

Camera-Radar Fusion Autonomous Driving Perception 3D Object Detection

Ask PDF

Mar 12, 2026

R4Det: 4D Radar-Camera Fusion for High-Performance 3D Object Detection

4D radar-camera sensing configuration has gained increasing importance in autonomous driving. However, existing 3D object detection methods that fuse 4D Radar and camera data confront several challenges. First, their absolute depth estimation module is not robust and accurate enough, leading to inaccurate 3D localization. Second, the performance of their temporal fusion module will degrade dramatically or even fail when the ego vehicle's pose is missing or inaccurate. Third, for some small objects, the sparse radar point clouds may completely fail to reflect from their surfaces. In such cases, detection must rely solely on visual unimodal priors. To address these limitations, we propose R4Det, which enhances depth estimation quality via the Panoramic Depth Fusion module, enabling mutual reinforcement between absolute and relative depth. For temporal fusion, we design a Deformable Gated Temporal Fusion module that does not rely on the ego vehicle's pose. In addition, we built an Instance-Guided Dynamic Refinement module that extracts semantic prototypes from 2D instance guidance. Experiments show that R4Det achieves state-of-the-art 3D object detection results on the TJ4DRadSet and VoD datasets. The source code and models will be released at https://github.com/VDIGPKU/R4Det.

Zhongyu Xia, Yousen Tang, Yongtao Wang +2

Wangxuan Institute of Computer Technology, Peking University · EBTech Co. Ltd

3D Object Detection Camera-Radar Fusion Autonomous Driving Perception Multimodal Sensor Fusion Depth Estimation

Ask PDF

Abstract

3D Object Detection

Explore similar work

RadarXFormer: Robust Object Detection via Cross-Dimension Fusion of 4D Radar Spectra and Images for Autonomous Driving

Sparse4D-Radar: An Efficient and Robust Framework for Surround-View 3D Object Detection via 4D Radar-Camera Fusion

R4Det: 4D Radar-Camera Fusion for High-Performance 3D Object Detection