RAHF-Net: Range-Aware Heterogeneous Fusion for LiDAR-Assisted RGB Semantic Segmentation

R. D. Lin; Pengcheng Weng; Ge Wang; Han Ding; Fei Wang

cs.CVOct 18, 2026

RAHF-Net: Range-Aware Heterogeneous Fusion for LiDAR-Assisted RGB Semantic Segmentation

Authors: R. D. Lin, Pengcheng Weng, Ge Wang, Han Ding, Fei Wang

Organizations: School of Software Engineering, Xi’an Jiaotong University, China · School of Computer Science and Technology, Xi’an Jiaotong University, China

Abstract

This paper studies LiDAR-assisted RGB semantic segmentation, where LiDAR point-cloud features improve dense pixel-wise predictions in the RGB image plane rather than 3D point labels or detections. Existing 2D fusion methods often project point clouds into shallow depth/range maps and apply spatially uniform fusion, discarding rich 3D structure and ignoring the range-dependent reliability of outdoor LiDAR. We propose RAHF-Net, a Range-Aware Heterogeneous Fusion Network. Its Cylindrical-to-Perspective (C2P) module projects pretrained cylindrical voxel features into the image plane, preserving structured 3D semantic priors. A Range-Aware Heterogeneous Fusion (RAHF) module then uses a soft depth-derived range mask to perform LiDAR-guided geometric enhancement in reliable near-field regions and visual-context enhancement in sparse far-field regions. On KITTI-360, RAHF-Net achieves 65.76 mIoU, outperforming representative RGB-only and RGB-LiDAR baselines, including CMX by 1.45 points. Experiments on DELIVER further demonstrate improved robustness under adverse weather, illumination degradation, and sensor corruption.

Figures & tables

Figure 1: Range-heterogeneity of LiDAR point clouds in driving scenes. Near-field points are dense and reliable, while far-field points are sparse and more likely to introduce noise in multi-modal fusion.

Direction	Output space	Matched
RGB-only 2D segmentation [ 17 , 5 , 23 ]	Image labels	Yes
RGB-X / RGB-LiDAR 2D fusion [ 9 , 22 , 25 ]	Image labels	Yes
Image-assisted 3D perception [ 21 , 13 , 11 ]	Points/objects	No
LiDAR-only 3D segmentation [ 29 , 1 , 10 ]	Point labels	No
LiDAR-assisted 2D segmentation ( Ours )	Image labels	Yes

Table 1: Task-scope taxonomy for LiDAR-camera perception. The “Matched” column indicates whether a method predicts labels in our target image space.

Figure 2: Overview of RAHF-Net. The proposed C2P module projects pretrained cylindrical LiDAR features into the image plane, and the RAHF module performs range-aware fusion with near-field geometric refinement and far-field visual context reasoning.

Method	Mod.	mIoU	Road	Building	T. Sign	Ped.
FCN [ 17 ]	R	54.02	91.44	80.29	44.36	51.62
U-Net [ 20 ]	R	54.83	91.73	81.36	44.79	51.48
Mask2Former [ 5 ]	R	56.37	93.52	83.69	45.13	52.93
HRFuser [ 3 ]	R+L	48.74	87.61	78.81	37.24	48.51
TokenFusion [ 22 ]	R+L	54.55	92.18	82.25	43.94	52.47
CMX [ 25 ]	R+L	64.31	94.57	87.43	52.08	58.24

Table 2: Comparison with task-matched methods on KITTI-360. R and L denote RGB and LiDAR, respectively.

Figure 3: Qualitative comparison on KITTI-360. Red boxes highlight improvements in a distant building and a nearby vehicle.

3D feature	mIoU	Road	Building	T. Sign	Ped.
Without 3D feature	62.18	92.72	87.05	47.51	55.74
2D range projection	63.12	94.04	86.78	48.12	56.53
Cartesian voxelization	63.42	93.15	87.91	48.40	56.85
Pretrained cylindrical encoder	65.76	94.92	88.78	52.22	59.96

Table 3: Ablation on LiDAR feature representations.

	$\beta=2$	$\beta=5$	$\beta=8$
$\alpha=5$	64.74	62.81	62.43
$\alpha=10$	64.29	65.76	64.71
$\alpha=20$	63.87	64.82	65.37

Table 4: Ablation on the soft-mask hyperparameters.

Fusion strategy	mIoU	Road	Building	T. Sign	Ped.
2D only	62.18	92.72	87.05	47.51	55.74
Near-field 3D only	63.61	93.56	87.47	48.92	53.89
Far-field context only	62.39	93.18	87.19	48.24	55.20
3D in both near/far fields	63.15	92.90	87.30	48.66	56.77
Gated heterogeneous fusion	65.76	94.92	88.78	52.22	59.96

Table 5: Ablation on the fusion strategy.

Condition	mIoU	Road	Building	T. Sign	Ped.
Cloudy	61.37	98.17	89.75	59.39	62.54
Foggy	58.78	98.39	80.36	65.85	68.17
Night	51.08	97.43	71.25	59.78	63.38
Rainy	60.11	97.86	88.87	37.64	71.98
Sunny	60.25	98.06	88.09	49.35	72.71
Motion blur	57.91	98.04	88.19	61.83	69.87

Table 6: Robustness on DELIVER under challenging conditions.

Camera-LiDAR Fusion

Camera–LiDAR fusion combines image semantics with LiDAR geometry for 3D perception, depth estimation, localization, and mapping.

See all 103 papers

EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentation
Jun 21, 2021Mingkui Tan, Zhuangwei Zhuang, Sitao Chen +4Camera-LiDAR Fusion Autonomous Driving Perception
School of Software Engineering, South China University of Technology · Pazhou Laboratory, Guangzhou, China · School of Electronic and Information Engineering, South China University of Technology +2
Enhancing 3D LiDAR Segmentation by Shaping Dense and Accurate 2D Semantic Predictions
Feb 21, 2026Xiaoyu Dong, Tiankui Xian, Wanshui Gan +1Multimodal Semantic Segmentation 3D Semantic Segmentation
The University of Tokyo, Tokyo, Japan · RIKEN AIP, Tokyo, Japan
Adaptive Augmentation-Aware Latent Learning for Robust LiDAR Semantic Segmentation
Mar 1, 2026Wangkai Li, Zhaoyang Li, Yuwen Pan +33D Point Cloud Segmentation 3D Semantic Segmentation
University of Science and Technology of China · National Key Laboratory of Deep Space Exploration, Deep Space Exploration Laboratory

Jun 21, 2021J/K move · Enter open · S save

EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentation

Mingkui Tan, Zhuangwei Zhuang, Sitao Chen, Rong Li+3

School of Software Engineering, South China University of Technology · Pazhou Laboratory, Guangzhou, China · School of Electronic and Information Engineering, South China University of Technology+2

Camera-LiDAR Fusion Autonomous Driving Perception 3D Semantic Segmentation Multimodal Sensor Fusion

Open paper AskPDF arXiv

We study multi-sensor fusion for 3D semantic segmentation that is important to scene understanding for many applications, such as autonomous driving and robotics. Existing fusion-based methods, however, may not achieve promising performance due to the vast difference between the two modalities. In this work, we investigate a collaborative fusion scheme called perception-aware multi-sensor fusion (PMF) to effectively exploit perceptual information from two modalities, namely, appearance information from RGB images and spatio-depth information from point clouds. To this end, we project point clouds to the camera coordinate using perspective projection, and process both inputs from LiDAR and cameras in 2D space while preventing the information loss of RGB images. Then, we propose a two-stream network to extract features from the two modalities, separately. The extracted features are fused by effective residual-based fusion modules. Moreover, we introduce additional perception-aware losses to measure the perceptual difference between the two modalities. Last, we propose an improved version of PMF, i.e., EPMF, which is more efficient and effective by optimizing data pre-processing and network architecture under perspective projection. Specifically, we propose cross-modal alignment and cropping to obtain tight inputs and reduce unnecessary computational costs. We then explore more efficient contextual modules under perspective projection and fuse the LiDAR features into the camera stream to boost the performance of the two-stream network. Extensive experiments on benchmark data sets show the superiority of our method. For example, on nuScenes test set, our EPMF outperforms the state-of-the-art method, i.e., RangeFormer, by 0.9% in mIoU. Our source code is available at https://github.com/ICEORY/PMF.

Abstract

Camera-LiDAR Fusion

Explore similar work

EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentation

Enhancing 3D LiDAR Segmentation by Shaping Dense and Accurate 2D Semantic Predictions

Adaptive Augmentation-Aware Latent Learning for Robust LiDAR Semantic Segmentation