Camera Radar LiDAR Fusion via Point-Cloud Projection for Multi-Task Learning
Authors: Kavin Chandrasekaran, Sorin Grigorescu
Organizations: Research and Development Department, Elektrobit Automotive GmbH, Munich, Germany · Department of Robotics, Transilvania University of Brasov, Brasov, Romania
Autonomous vehicles require robust perception systems to operate safely under diverse environmental conditions. While cameras provide high-resolution visual data, their performance significantly degrades in low-light and adverse weather. High Definition (HD) radar offers superior reliability and angular resolution under such conditions but lacks rich visual feature recognition. LiDAR, though expensive, delivers highly precise distance measurements and dense point clouds. Each sensor exhibits unique trade-offs, necessitating the development of advanced sensor fusion algorithms to harness complementary strengths while minimizing computational complexity and maintaining accuracy. In this study, the proposed fusion architecture performs early fusion by combining camera images with perspective-view point cloud data from HD radar and LiDAR sensors. The proposed model is trained to detect vehicles and segment free drivable space simultaneously. Experiments are performed on the RADIal dataset where we demonstrate that fusing point cloud data with camera images in front-view surpasses the performances of the most recent standalone and fusion models.
Figures & tables
Fig. 1 : Architecture Overview: Camera-Radar-LiDAR early fusion by concatenation performing vehicle detection and free space segmentation simultaneously. The predicted free drivable space is in green and the detected vehicles in blue.
Fig. 2 : Deep Seeing Through Sensors: Early fusion is performed using inputs from (a) the camera, (b) a sparse radar point cloud, and (c) a dense LiDAR point cloud. The encoder comprises a pre-encoder followed by four ResNet-50-style blocks, whose output feature maps are denoted as x0, x1, x2, and x3. A thick purple arrow indicates the connection from the encoder to the decoder, through which the encoded features are upsampled to higher spatial resolutions. The dashed lines indicate skip connections that preserve spatial information. The final outputs are produced by a Detection Head and a Segmentation Head, and the corresponding prediction results are illustrated in Fig. 1 and Fig. 3 .
Accuracy
Resource Efficiency
Methods
Modality
AP(%) ↑
AR(%) ↑
F1(%) ↑
mIoU(%) ↑
# ↓
Avg FPS ↑
Model size ↓
GPU cost ↓
FFTRadNet [ 12 ]
RD
93.45
83.35
88.11
74
3.28
67.32
39.2
2.01
TFFTRadNet [ 20 ]
ADC
90.80
88.31
89.54
80.2
9.13
52.41
109.5
2.04
ADCNet† [ 21 ]
ADC
95
89
91.9
78.59
-
-
-
-
SparseRad† [ 29 ]
RD
96.00
91.78
93.84
78.48
-
-
-
-
TABLE I : Quantitative results on the test split. ADC, RD, RT, RPC, LPC, C, C BEV correspond to Analog-To-Digital Converter signal, Range-Doppler, Range-Time, Radar Point Cloud, LiDAR Point Cloud, Camera data in Front and Bird’s-Eye View respectively. The single modality methods are separated from multimodal fusion methods by a dashed line. Best results are in bold and second best are underlined. The values missing are marked by a ”-”, either unreported in the respective works or due to code unavailability†. * Trained using camera only data of the proposed architecture.
Fig. 3 : Qualitative evaluation on the test set: (a) ground truth, (b) camera-only predictions, and (c) camera–radar–LiDAR fusion results. The green regions denote free-space segmentation, while the blue regions indicate detected vehicles.