R4Det: 4D Radar-Camera Fusion for High-Performance 3D Object Detection

Zhongyu Xia; Yousen Tang; Yongtao Wang; Zhifeng Wang; Weijun Qin

cs.CVMar 12, 2026

R4Det: 4D Radar-Camera Fusion for High-Performance 3D Object Detection

Authors: Zhongyu Xia, Yousen Tang, Yongtao Wang, Zhifeng Wang, Weijun Qin

Organizations: Wangxuan Institute of Computer Technology, Peking University · EBTech Co. Ltd

Abstract

4D radar-camera sensing configuration has gained increasing importance in autonomous driving. However, existing 3D object detection methods that fuse 4D Radar and camera data confront several challenges. First, their absolute depth estimation module is not robust and accurate enough, leading to inaccurate 3D localization. Second, the performance of their temporal fusion module will degrade dramatically or even fail when the ego vehicle's pose is missing or inaccurate. Third, for some small objects, the sparse radar point clouds may completely fail to reflect from their surfaces. In such cases, detection must rely solely on visual unimodal priors. To address these limitations, we propose R4Det, which enhances depth estimation quality via the Panoramic Depth Fusion module, enabling mutual reinforcement between absolute and relative depth. For temporal fusion, we design a Deformable Gated Temporal Fusion module that does not rely on the ego vehicle's pose. In addition, we built an Instance-Guided Dynamic Refinement module that extracts semantic prototypes from 2D instance guidance. Experiments show that R4Det achieves state-of-the-art 3D object detection results on the TJ4DRadSet and VoD datasets. The source code and models will be released at https://github.com/VDIGPKU/R4Det.

Figures & tables

Figure 1: Comparison of R4Det with current 4D radar-camera real-time detectors.

Figure 2: Overall architecture of R4Det. Our framework progressively purifies the BEV representation in three stages: i) The Panoramic Depth Fusion (PDF) module generates a geometrically-accurate BEV feature map from multi-modal inputs. ii) The Deformable Gated Temporal Fusion (DGTF) module performs pose-free alignment and integration to create a temporally consistent feature. iii) The Instance-Guided Dynamic Refinement (IGDR) module leverages 2D instance prototypes to purify the final features for 3D detection.

Figure 3: Overview of the Panoramic Depth Fusion (PDF) module.

Figure 4: Architecture of the proposed Deformable Gated Temporal Fusion (DGTF) module. DGTF consists of two specialized branches: motion-aware alignment using deformable convolution and a gated temporal update mechanism.

Figure 5: Overview of Instance-Guided Dynamic Refinement (IGDR) module. IGDR adaptively refines radar-camera BEV features by suppressing instance overlap contamination and cross-modality noise, while preserving reliable distant object representations.

Method	Modality	$\text{AP}_{\text{3D}}$ (%)				$\text{mAP}_{\text{3D}}$	$\text{AP}_{\text{BEV}}$ (%)				$\text{mAP}_{\text{BEV}}$
Method	Modality	Car	Pedestrian	Cyclist	Truck	$\text{mAP}_{\text{3D}}$	Car	Pedestrian	Cyclist	Truck	$\text{mAP}_{\text{BEV}}$
PointPillars [ 9 ]	R	21.26	28.33	52.47	11.18	28.31	38.34	32.26	56.11	18.19	36.23
CenterPoint [ 25 ]	R	22.03	25.02	53.32	15.92	29.07	33.03	27.87	58.74	25.09	36.18
RadarPillarNet [ 28 ]	R	28.45	26.24	51.57	15.20	30.37	45.72	29.19	56.89	25.17	39.24
SMURF [ 19 ]	R	28.47	26.22	54.61	22.64	32.99	43.13	29.19	58.81	32.80	40.98
RCFusion [ 28 ]	R+C	29.72	27.17	54.93	23.56	33.85	40.89	30.95	58.30	28.92	39.76

Table 1: Comparison of 3D object detection results on the test set of TJ4DRadSet [ 27 ] . In the modality column, R denotes 4D radar and C denotes camera. The best values are in bold .

Methods	Modality	Entire Annotated Area ( $\text{AP}_{\text{EAA}}$ , %)				Driving Corridor ( $\text{AP}_{\text{DC}}$ , %)				FPS
Methods	Modality	Car	Pedestrian	Cyclist	mAP	Car	Pedestrian	Cyclist	mAP	FPS
PointPillars [ 9 ]	R	37.06	35.04	63.44	45.18	70.15	47.22	85.07	67.48	113.9
CenterPoint [ 25 ]	R	32.74	38.00	65.51	45.42	62.01	48.18	84.98	65.06	-
RadarPillarNet [ 28 ]	R	39.30	35.10	63.63	46.01	71.65	42.80	83.14	65.86	98.8
SMURF [ 19 ]	R	42.31	39.09	71.50	50.97	71.74	50.54	86.87	69.72	-
RCFusion [ 28 ]	R+C	41.70	38.95	68.31	49.65	71.87	47.50	88.33	69.23	9.0

Table 2: Comparison of 3D object detection results on the val set of VoD [ 17 ] . R and C denote 4D radar and camera modalities, respectively. The best values are in bold .

Method	Car	Pedestrian	Cyclist	$\text{mAP}_{\text{EAA}}$
BEVFusion [ 13 ]	37.85	40.96	68.95	49.25
BEVFusion + Ours	41.98	48.15	76.64	55.59
RCBEVDet [ 14 ]	40.63	38.86	70.48	49.99
RCBEVDet + Ours	41.02	48.85	76.12	55.33

Table 3: More 3D object detection results on the val set of VoD to demonstrate the proposed modules are plug-and-play.

Figure 6: Example visualization results of R4Det and baseline on challenging scenarios ( e.g. , low-light conditions or small objects).

PDF	DGTF	IGDR	$\text{mAP}_{\text{BEV}}$	$\text{mAP}_{\text{3D}}$	Car	Ped	Cyc	Truck
			45.15	39.86	58.67	25.93	47.89	26.24
✓			46.86	41.41	59.76	23.35	55.08	29.25
✓	✓		50.41	44.86	61.94	28.90	57.15	30.48
✓	✓	✓	54.07	47.29	63.60	31.24	62.84	31.46

Table 4: Ablation study on each component of R4Det on the TJ4D Dataset .

Setting	$\lambda_{1}$	$\lambda_{\mathrm{abs}}$	$\lambda_{\mathrm{dense}}$	$\lambda_{3}$	BEV mAP $\uparrow$
A	0.1	0.01	–	–	45.15
B	0.1	0.01	0.03	–	46.08
C	0.1	0.01	0.03	0.05	46.86

Table 5: Ablation of the Panoramic Depth Fusion module. Each setting corresponds to different combinations of loss components.

Figure 7: Qualitative comparison of depth predictions by our Panoramic Depth Fusion and the baseline method.

Variant	Concat	DCN	ConvGRU	SE	BEV mAP	3D mAP
V0					46.86	41.41
V1	✓				47.82	42.01
V2	✓	✓			48.86	43.32
V3	✓	✓	✓	✓	49.12	42.47
V4 (Ours)	✓	✓	✓		50.41	44.86

Table 6: Ablation of the Deformable Gated Temporal Fusion module. V1 (+Concat) concatenates current and previous BEV features; V2 (+DCN) adds modulated deformable convolution (DCNv2) to spatially align features; V4 (DGTF, Ours) further incorporates a GRU-style gating mechanism (ConvGRU) for selective temporal fusion; V3 (+SE) adds channel-wise Squeeze-and-Excitation on top of V4.

ID	Component		BEV mAP (%)
ID	$G_{bg}$	Gen.	Cyclist	Pedestrian	Overall
A		None	57.51	31.66	50.41
B		Softmax-Fusion	58.14	32.00	51.29
C	✓	Static Map	58.62	31.92	51.60
D	✓	MLP	62.13	32.43	52.48
E	✓	Attention	62.21	32.21	52.52

Table 7: Ablation on the Instance-Guided Dynamic Refinement module. Gen. indicates the generator used to produce calibration parameters or replacement maps. (1) None (A): Baseline without refinement. (2) Softmax-Fusion (B): Directly adds the Softmax-weighted instance prototype

E_{BEV}

to the main stream without gating. (3) Static Map (C): Statically replaces the foreground region in

F_{RC}

with

E_{BEV}

under gating. (4) MLP (D): Generates

\gamma,\beta

from

E_{proj}

using MLPs, followed by affine transformation. (5) Attention (E): Employs a masked attention mechanism using

F_{RC}

as Query and

E_{proj}

as Key/Value. (6) Conv (F, Ours): Constructs the spatial prototype

E_{BEV}

and generates

\gamma,\beta

via Conv2d layers, enabling spatially-aware calibration. Our convolutional generator achieves the best overall performance.

Figure 8: DGTF Visualization. Learned offsets ( $\Delta p$ , red arrows) align strictly with vehicle motion, while the mask ( $m$ , heatmap) suppresses background, proving explicit temporal alignment.

Setting	$\lambda_{1}$	$\lambda_{\mathrm{abs}}$	$\lambda_{\mathrm{dense}}$	$\lambda_{3}$	BEV mAP $\uparrow$
A	0.1	0.01	–	–	45.15
B (+ $\lambda_{\mathrm{dense}}$ )	0.1	0.01	0.03	–	46.08
C1 (↓ $\lambda_{1}$ )	0.05	0.01	0.03	0.05	46.64
C2 (↓ $\lambda_{3}$ )	0.1	0.01	0.03	0.02	46.30
C3 (↑ $\lambda_{\text{dense}}$ )	0.1	0.01	0.05	0.05	46.45
C4 (↑ $\lambda_{\text{abs}}$ )	0.1	0.03	0.03	0.05	46.12

Table 8: Ablation of the Panoramic Depth Fusion module. Each setting corresponds to different combinations of loss components.

Fusion Depth	BEV mAP	3D mAP
${t~\&~t-3}$	49.20	42.74
${t~\&~t-2}$	49.87	43.12
$\bm{t~\&~t-1}$	50.41	44.86

Table 9: Sensitivity to historical frame gap. Evaluation of different historical fusion depths. Single-step (

{t-1}

) achieves the best accuracy and stability.

3D Object Detection

3D object detection identifies objects and estimates their locations, sizes, and orientations from images, point clouds, radar, or fused sensors.

See all 181 papers

May 20, 2026

RCGDet3D: Rethinking 4D Radar-Camera Fusion-based 3D Object Detection with Enhanced Radar Feature Encoding

4D automotive radar is indispensable for autonomous driving due to its low cost and robustness, yet its point cloud sparsity challenges 3D object detection. Existing 4D radar-camera fusion methods focus on complex fusion strategies, trading inference speed for marginal gains. This trade-off hinders real-time deployment due to heavy computation on dense feature maps. In contrast, feature extraction from sparse radar points is less time-consuming but remains under-explored. This work uncovers that simply enhancing radar feature extraction can achieve comparable or even higher performance than elaborate fusion modules, while maintaining real-time performance. Based on this finding, we propose RCGDet3D, which centers on radar feature encoding and simplifies multi-modal fusion. Its encoder inherits from the efficient Gaussian Splatting-based Point Gaussian Encoder (PGE) in RadarGaussianDet3D with two key improvements. First, the Ray-centric PGE (R-PGE) predicts Gaussian attributes in ray-aligned coordinate systems before unifying them to Bird's-Eye View (BEV) space, significantly improving geometric consistency and reducing learning difficulty by decoupling the coordinate transformation from representation learning. Second, a Semantic Injection (SI) module incorporates visual cues from images, producing more geometrically accurate and semantically enriched radar features. Experiments on View-of-Delft (VoD) and TJ4DRadSet show that RCGDet3D outperforms state-of-the-art methods in both accuracy and speed, setting a new benchmark for real-time deployment.

Weiyi Xiong, Bing Zhu

School of Automation Science and Electrical Engineering, Beihang University, Beijing, P.R. China

Camera-Radar Fusion 3D Object Detection Object Detection Real-Time Object Detection

Ask PDF

Feb 24, 2026

Boosting Instance Awareness via Cross-View Correlation with 4D Radar and Camera for 3D Object Detection

4D millimeter-wave radar has emerged as a promising sensing modality for autonomous driving due to its robustness and affordability. However, its sparse and weak geometric cues make reliable instance activation difficult, limiting the effectiveness of existing radar-camera fusion paradigms. BEV-level fusion offers global scene understanding but suffers from weak instance focus, while perspective-level fusion captures instance details but lacks holistic context. To address these limitations, we propose SIFormer, a scene-instance aware transformer for 3D object detection using 4D radar and camera. SIFormer first suppresses background noise during view transformation through segmentation- and depth-guided localization. It then introduces a cross-view activation mechanism that injects 2D instance cues into BEV space, enabling reliable instance awareness under weak radar geometry. Finally, a transformer-based fusion module aggregates complementary image semantics and radar geometry for robust perception. As a result, with the aim of enhancing instance awareness, SIFormer bridges the gap between the two paradigms, combining their complementary strengths to address inherent sparse nature of radar and improve detection accuracy. Experiments demonstrate that SIFormer achieves state-of-the-art performance on View-of-Delft, TJ4DRadSet and NuScenes datasets. Source code is available at github.com/shawnnnkb/SIFormer.

Xiaokai Bai, Lianqing Zheng, Si-Yuan Cao +6

College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, China · School of Automotive Studies, Tongji University, Shanghai 201804, China · College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China, and also with the Jinhua Institute of Zhejiang University, Jinhua 321299, China +1

3D Object Detection Camera-Radar Fusion mmWave Radar Sensing 3D ViTs

Ask PDF

Jul 5, 2026

Sparse4D-Radar: An Efficient and Robust Framework for Surround-View 3D Object Detection via 4D Radar-Camera Fusion

In recent years, 4D imaging radar has gained wide attention in autonomous driving for its robustness against harsh weather and ability to output target velocity. Nevertheless, mainstream 4D radar-camera fusion methods only support front-view perception, lacking mature solutions for surround-view sensing. Directly expanding these pipelines to full 360° coverage introduces excessive computation cost and limits real-world deployment. To tackle these limitations, this work proposes Sparse4D-Radar, an efficient robust surround-view multi-modal fusion framework. We first design a Deformable Fusion module to embed radar-camera features into sparse queries, constructing the lightweight base version Sparse4D-Radar-Base. Two dedicated modules are further introduced to boost localization accuracy and modality stability: Velocity-Consistency Sampling (VCS) refines features via radar velocity cues for motion awareness, and Adaptive Modality Gating (AMG) dynamically adjusts cross-modal fusion weights according to feature confidence. Combining all components, we build Sparse4D-Radar-Acc for high-precision detection demands. Comprehensive experiments on OmniHD-Scenes verify that our approach achieves state-of-the-art surround-view 3D detection performance. Compared with prior arts, our method obtains over 7% mAP and 10% ODS improvements under complex driving scenes while running at nearly 10 FPS, striking a favorable trade-off among detection accuracy, environmental robustness and inference efficiency. Our open-source code is available at https://github.com/Aiuan/Sparse4D-Radar.

Fuyuan Ai, Yuchen Tan, Jiehui Chen +2

State Key Laboratory of Ocean Sensing and Ocean College, Zhejiang University, Zhoushan 316021, China, and also with the Engineering Research Center of Oceanic Sensing Technology and Equipment, Ministry of Education, Zhejiang University, Zhoushan 316021, China · Donghai Laboratory, Zhoushan 316021, China, also with the Ocean College, Institute of Marine Electronics and Intelligent Systems, Zhejiang University, Zhoushan 316021, China, and also with the Engineering Research Center of Oceanic Sensing Technology and Equipment, Ministry of Education, Zhejiang University, Zhoushan 316021, China

Camera-Radar Fusion Autonomous Driving Perception 3D Object Detection

Ask PDF

Abstract

3D Object Detection

Explore similar work

RCGDet3D: Rethinking 4D Radar-Camera Fusion-based 3D Object Detection with Enhanced Radar Feature Encoding

Boosting Instance Awareness via Cross-View Correlation with 4D Radar and Camera for 3D Object Detection

Sparse4D-Radar: An Efficient and Robust Framework for Surround-View 3D Object Detection via 4D Radar-Camera Fusion