R4Det: 4D Radar-Camera Fusion for High-Performance 3D Object Detection

Zhongyu Xia; Yousen Tang; Yongtao Wang; Zhifeng Wang; Weijun Qin

cs.CVMar 12, 2026

R4Det: 4D Radar-Camera Fusion for High-Performance 3D Object Detection

Authors: Zhongyu Xia, Yousen Tang, Yongtao Wang, Zhifeng Wang, Weijun Qin

Organizations: Wangxuan Institute of Computer Technology, Peking University · EBTech Co. Ltd

Abstract

4D radar-camera sensing configuration has gained increasing importance in autonomous driving. However, existing 3D object detection methods that fuse 4D Radar and camera data confront several challenges. First, their absolute depth estimation module is not robust and accurate enough, leading to inaccurate 3D localization. Second, the performance of their temporal fusion module will degrade dramatically or even fail when the ego vehicle's pose is missing or inaccurate. Third, for some small objects, the sparse radar point clouds may completely fail to reflect from their surfaces. In such cases, detection must rely solely on visual unimodal priors. To address these limitations, we propose R4Det, which enhances depth estimation quality via the Panoramic Depth Fusion module, enabling mutual reinforcement between absolute and relative depth. For temporal fusion, we design a Deformable Gated Temporal Fusion module that does not rely on the ego vehicle's pose. In addition, we built an Instance-Guided Dynamic Refinement module that extracts semantic prototypes from 2D instance guidance. Experiments show that R4Det achieves state-of-the-art 3D object detection results on the TJ4DRadSet and VoD datasets. The source code and models will be released at https://github.com/VDIGPKU/R4Det.

Figures & tables

Figure 1: Comparison of R4Det with current 4D radar-camera real-time detectors.

Figure 2: Overall architecture of R4Det. Our framework progressively purifies the BEV representation in three stages: i) The Panoramic Depth Fusion (PDF) module generates a geometrically-accurate BEV feature map from multi-modal inputs. ii) The Deformable Gated Temporal Fusion (DGTF) module performs pose-free alignment and integration to create a temporally consistent feature. iii) The Instance-Guided Dynamic Refinement (IGDR) module leverages 2D instance prototypes to purify the final features for 3D detection.

Figure 3: Overview of the Panoramic Depth Fusion (PDF) module.

Figure 4: Architecture of the proposed Deformable Gated Temporal Fusion (DGTF) module. DGTF consists of two specialized branches: motion-aware alignment using deformable convolution and a gated temporal update mechanism.

Figure 5: Overview of Instance-Guided Dynamic Refinement (IGDR) module. IGDR adaptively refines radar-camera BEV features by suppressing instance overlap contamination and cross-modality noise, while preserving reliable distant object representations.

Method	Modality	$\text{AP}_{\text{3D}}$ (%)				$\text{mAP}_{\text{3D}}$	$\text{AP}_{\text{BEV}}$ (%)				$\text{mAP}_{\text{BEV}}$
Method	Modality	Car	Pedestrian	Cyclist	Truck	$\text{mAP}_{\text{3D}}$	Car	Pedestrian	Cyclist	Truck	$\text{mAP}_{\text{BEV}}$
PointPillars [ 9 ]	R	21.26	28.33	52.47	11.18	28.31	38.34	32.26	56.11	18.19	36.23
CenterPoint [ 25 ]	R	22.03	25.02	53.32	15.92	29.07	33.03	27.87	58.74	25.09	36.18
RadarPillarNet [ 28 ]	R	28.45	26.24	51.57	15.20	30.37	45.72	29.19	56.89	25.17	39.24
SMURF [ 19 ]	R	28.47	26.22	54.61	22.64	32.99	43.13	29.19	58.81	32.80	40.98
RCFusion [ 28 ]	R+C	29.72	27.17	54.93	23.56	33.85	40.89	30.95	58.30	28.92	39.76

Table 1: Comparison of 3D object detection results on the test set of TJ4DRadSet [ 27 ] . In the modality column, R denotes 4D radar and C denotes camera. The best values are in bold .

Methods	Modality	Entire Annotated Area ( $\text{AP}_{\text{EAA}}$ , %)				Driving Corridor ( $\text{AP}_{\text{DC}}$ , %)				FPS
Methods	Modality	Car	Pedestrian	Cyclist	mAP	Car	Pedestrian	Cyclist	mAP	FPS
PointPillars [ 9 ]	R	37.06	35.04	63.44	45.18	70.15	47.22	85.07	67.48	113.9
CenterPoint [ 25 ]	R	32.74	38.00	65.51	45.42	62.01	48.18	84.98	65.06	-
RadarPillarNet [ 28 ]	R	39.30	35.10	63.63	46.01	71.65	42.80	83.14	65.86	98.8
SMURF [ 19 ]	R	42.31	39.09	71.50	50.97	71.74	50.54	86.87	69.72	-
RCFusion [ 28 ]	R+C	41.70	38.95	68.31	49.65	71.87	47.50	88.33	69.23	9.0

Table 2: Comparison of 3D object detection results on the val set of VoD [ 17 ] . R and C denote 4D radar and camera modalities, respectively. The best values are in bold .

Method	Car	Pedestrian	Cyclist	$\text{mAP}_{\text{EAA}}$
BEVFusion [ 13 ]	37.85	40.96	68.95	49.25
BEVFusion + Ours	41.98	48.15	76.64	55.59
RCBEVDet [ 14 ]	40.63	38.86	70.48	49.99
RCBEVDet + Ours	41.02	48.85	76.12	55.33

Table 3: More 3D object detection results on the val set of VoD to demonstrate the proposed modules are plug-and-play.

Figure 6: Example visualization results of R4Det and baseline on challenging scenarios ( e.g. , low-light conditions or small objects).

PDF	DGTF	IGDR	$\text{mAP}_{\text{BEV}}$	$\text{mAP}_{\text{3D}}$	Car	Ped	Cyc	Truck
			45.15	39.86	58.67	25.93	47.89	26.24
✓			46.86	41.41	59.76	23.35	55.08	29.25
✓	✓		50.41	44.86	61.94	28.90	57.15	30.48
✓	✓	✓	54.07	47.29	63.60	31.24	62.84	31.46

Table 4: Ablation study on each component of R4Det on the TJ4D Dataset .

Setting	$\lambda_{1}$	$\lambda_{\mathrm{abs}}$	$\lambda_{\mathrm{dense}}$	$\lambda_{3}$	BEV mAP $\uparrow$
A	0.1	0.01	–	–	45.15
B	0.1	0.01	0.03	–	46.08
C	0.1	0.01	0.03	0.05	46.86

Table 5: Ablation of the Panoramic Depth Fusion module. Each setting corresponds to different combinations of loss components.

Figure 7: Qualitative comparison of depth predictions by our Panoramic Depth Fusion and the baseline method.

Variant	Concat	DCN	ConvGRU	SE	BEV mAP	3D mAP
V0					46.86	41.41
V1	✓				47.82	42.01
V2	✓	✓			48.86	43.32
V3	✓	✓	✓	✓	49.12	42.47
V4 (Ours)	✓	✓	✓		50.41	44.86

Table 6: Ablation of the Deformable Gated Temporal Fusion module. V1 (+Concat) concatenates current and previous BEV features; V2 (+DCN) adds modulated deformable convolution (DCNv2) to spatially align features; V4 (DGTF, Ours) further incorporates a GRU-style gating mechanism (ConvGRU) for selective temporal fusion; V3 (+SE) adds channel-wise Squeeze-and-Excitation on top of V4.

ID	Component		BEV mAP (%)
ID	$G_{bg}$	Gen.	Cyclist	Pedestrian	Overall
A		None	57.51	31.66	50.41
B		Softmax-Fusion	58.14	32.00	51.29
C	✓	Static Map	58.62	31.92	51.60
D	✓	MLP	62.13	32.43	52.48
E	✓	Attention	62.21	32.21	52.52

Table 7: Ablation on the Instance-Guided Dynamic Refinement module. Gen. indicates the generator used to produce calibration parameters or replacement maps. (1) None (A): Baseline without refinement. (2) Softmax-Fusion (B): Directly adds the Softmax-weighted instance prototype

E_{BEV}

to the main stream without gating. (3) Static Map (C): Statically replaces the foreground region in

F_{RC}

with

E_{BEV}

under gating. (4) MLP (D): Generates

\gamma,\beta

from

E_{proj}

using MLPs, followed by affine transformation. (5) Attention (E): Employs a masked attention mechanism using

F_{RC}

as Query and

E_{proj}

as Key/Value. (6) Conv (F, Ours): Constructs the spatial prototype

E_{BEV}

and generates

\gamma,\beta

via Conv2d layers, enabling spatially-aware calibration. Our convolutional generator achieves the best overall performance.

Figure 8: DGTF Visualization. Learned offsets ( $\Delta p$ , red arrows) align strictly with vehicle motion, while the mask ( $m$ , heatmap) suppresses background, proving explicit temporal alignment.

Setting	$\lambda_{1}$	$\lambda_{\mathrm{abs}}$	$\lambda_{\mathrm{dense}}$	$\lambda_{3}$	BEV mAP $\uparrow$
A	0.1	0.01	–	–	45.15
B (+ $\lambda_{\mathrm{dense}}$ )	0.1	0.01	0.03	–	46.08
C1 (↓ $\lambda_{1}$ )	0.05	0.01	0.03	0.05	46.64
C2 (↓ $\lambda_{3}$ )	0.1	0.01	0.03	0.02	46.30
C3 (↑ $\lambda_{\text{dense}}$ )	0.1	0.01	0.05	0.05	46.45
C4 (↑ $\lambda_{\text{abs}}$ )	0.1	0.03	0.03	0.05	46.12

Table 8: Ablation of the Panoramic Depth Fusion module. Each setting corresponds to different combinations of loss components.

Fusion Depth	BEV mAP	3D mAP
${t~\&~t-3}$	49.20	42.74
${t~\&~t-2}$	49.87	43.12
$\bm{t~\&~t-1}$	50.41	44.86

Table 9: Sensitivity to historical frame gap. Evaluation of different historical fusion depths. Single-step (

{t-1}

) achieves the best accuracy and stability.

3D Object Detection

3D object detection identifies objects and estimates their locations, sizes, and orientations from images, point clouds, radar, or fused sensors.

See all 180 papers

RCGDet3D: Rethinking 4D Radar-Camera Fusion-based 3D Object Detection with Enhanced Radar Feature Encoding
May 20, 2026Weiyi Xiong, Bing ZhuCamera-Radar Fusion 3D Object Detection
School of Automation Science and Electrical Engineering, Beihang University, Beijing, P.R. China
Boosting Instance Awareness via Cross-View Correlation with 4D Radar and Camera for 3D Object Detection
Feb 24, 2026Xiaokai Bai, Lianqing Zheng, Si-Yuan Cao +63D Object Detection Camera-Radar Fusion
College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, China · School of Automotive Studies, Tongji University, Shanghai 201804, China · College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China, and also with the Jinhua Institute of Zhejiang University, Jinhua 321299, China +1
Sparse4D-Radar: An Efficient and Robust Framework for Surround-View 3D Object Detection via 4D Radar-Camera Fusion
Jul 5, 2026Fuyuan Ai, Yuchen Tan, Jiehui Chen +2Camera-Radar Fusion Autonomous Driving Perception
State Key Laboratory of Ocean Sensing and Ocean College, Zhejiang University, Zhoushan 316021, China, and also with the Engineering Research Center of Oceanic Sensing Technology and Equipment, Ministry of Education, Zhejiang University, Zhoushan 316021, China · Donghai Laboratory, Zhoushan 316021, China, also with the Ocean College, Institute of Marine Electronics and Intelligent Systems, Zhejiang University, Zhoushan 316021, China, and also with the Engineering Research Center of Oceanic Sensing Technology and Equipment, Ministry of Education, Zhejiang University, Zhoushan 316021, China

May 20, 2026J/K move · Enter open · S save

RCGDet3D: Rethinking 4D Radar-Camera Fusion-based 3D Object Detection with Enhanced Radar Feature Encoding

Weiyi Xiong, Bing Zhu

School of Automation Science and Electrical Engineering, Beihang University, Beijing, P.R. China

Camera-Radar Fusion 3D Object Detection Object Detection Real-Time Object Detection

Open paper AskPDF arXiv

4D automotive radar is indispensable for autonomous driving due to its low cost and robustness, yet its point cloud sparsity challenges 3D object detection. Existing 4D radar-camera fusion methods focus on complex fusion strategies, trading inference speed for marginal gains. This trade-off hinders real-time deployment due to heavy computation on dense feature maps. In contrast, feature extraction from sparse radar points is less time-consuming but remains under-explored. This work uncovers that simply enhancing radar feature extraction can achieve comparable or even higher performance than elaborate fusion modules, while maintaining real-time performance. Based on this finding, we propose RCGDet3D, which centers on radar feature encoding and simplifies multi-modal fusion. Its encoder inherits from the efficient Gaussian Splatting-based Point Gaussian Encoder (PGE) in RadarGaussianDet3D with two key improvements. First, the Ray-centric PGE (R-PGE) predicts Gaussian attributes in ray-aligned coordinate systems before unifying them to Bird's-Eye View (BEV) space, significantly improving geometric consistency and reducing learning difficulty by decoupling the coordinate transformation from representation learning. Second, a Semantic Injection (SI) module incorporates visual cues from images, producing more geometrically accurate and semantically enriched radar features. Experiments on View-of-Delft (VoD) and TJ4DRadSet show that RCGDet3D outperforms state-of-the-art methods in both accuracy and speed, setting a new benchmark for real-time deployment.

Abstract

3D Object Detection

Explore similar work

RCGDet3D: Rethinking 4D Radar-Camera Fusion-based 3D Object Detection with Enhanced Radar Feature Encoding

Boosting Instance Awareness via Cross-View Correlation with 4D Radar and Camera for 3D Object Detection

Sparse4D-Radar: An Efficient and Robust Framework for Surround-View 3D Object Detection via 4D Radar-Camera Fusion