HiPFuse: Real-Time LiDAR Semantic Segmentation for Robotic Perception on Edge Devices
Authors: Haiyang Wu, George Vosselman, Ville Lehtola
Organizations: Haiyang Wu, George Vosselman, and Ville Lehtola are with the Department of Geoinformatics, Faculty of Geo-Information Science and Earth Observation (ITC), University of Twente, 7522 NB Enschede, The Netherlands.
Real-time robotic perception requires accurate LiDAR semantics under limited resources. Bird's-eye-view (BEV) aggregation is efficient but can mix differently labeled surfaces in columns. We present HiPFuse, a point-BEV fusion framework that exchanges point and height-partitioned BEV features across horizontal scales, combining point-wise detail with 2D spatial context from one LiDAR scan. Fixed height assignments preserve separate vertical aggregation groups, with auxiliary supervision of initial BEV features using targets from the same 3D cells. A dual-scale BEV context module adds broader context from a coarser grid. On SemanticKITTI and MCD, HiPFuse achieves 63.7% and 62.5% mIoU at 14.2 and 22.4 FPS on Jetson AGX Orin, respectively, exceeding the 10 Hz scan rate. Separate training and evaluation show indoor applicability. Replacing only the baseline semantic front end with HiPFuse in fixed robotic systems reduces odometry absolute trajectory error RMSE by 10.6% on average across two outdoor sequences and increases accepted-loop recall by 8.7 percentage points on SemanticKITTI. Qualitative semantic maps show more continuous road labeling. Our code is available at https://github.com/haiyang2022/HiPFuse.
Figures & tables
Fig. 1: SemanticKITTI test accuracy versus measured Jetson AGX Orin throughput (Tables III and V ). Shading indicates throughput meeting the 10 Hz scan rate under the measured settings.
TABLE II: Class-wise IoU and mIoU (%) on the MCD NTU test sequence. Bold: best; underline: second best.
Method
Category
Year
Wall
Floor
Ceiling
Column
Beam
Door
Stair
Table
Chair
Sofa
Cab./shelf
Screen
Light
Vegetation
Board
Other
mIoU (%)
SalsaNext [ 6 ]
Projection-based
2020
83.1
90.7
92.2
43.9
76.9
51.0
47.3
76.9
69.6
52.7
57.9
91.2
90.0
89.8
31.5
72.2
69.8
WaffleIron [ 20 ]
Projection-based
2023
88.4
93.4
93.8
55.7
81.0
60.7
79.6
83.0
80.2
73.8
67.3
93.8
97.2
94.7
44.8
81.1
79.3
HARP-NeXt [ 11 ]
Fusion-based
2025
88.5
92.1
94.4
55.5
80.1
56.1
63.8
79.9
75.6
68.2
43.2
86.7
94.5
91.2
55.6
77.9
75.2
Ours
Fusion-based
2026
86.7
92.6
93.7
53.2
76.7
52.1
37.5
81.7
69.8
71.5
58.0
85.9
95.1
90.1
27.1
73.4
71.6
TABLE IV: Class-wise IoU and mIoU (%) on the Semantic-ITC test set. Bold: best; underline: second best.
Fig. 4: Selected semantic predictions on (a) SemanticKITTI validation, (b) MCD test, and (c) Semantic-ITC test scans. Red boxes identify corresponding regions across methods. Row legends identify the colored classes; gray denotes other valid classes.
Dataset
Method
Category
Params (M)
NVIDIA L40 GPU
Orin
GPU Memory (MiB)
Time/scan (ms)
FPS
GPU Memory (MiB)
Time/scan (ms)
FPS
MCD [ 14 ]
SalsaNext [ 6 ]
Projection-based
6.7
294.5
13.2
75.7
186.7
29.9
33.5
SPVCNN [ 12 ]
Fusion-based
21.8
146.1
77.8
12.8
151.2
61.5
16.3
HARP-NeXt [ 11 ]
Fusion-based
5.4
384.9
20.3
49.1
276.9
45.8
21.8
Ours
Fusion-based
14.3
358.4
18.4
54.3
359.4
44.6
22.4
SemanticKITTI [ 13 ]
SalsaNext [ 6 ]
Projection-based
6.7
360.1
13.7
72.9
1167.0
64.2
15.6
TABLE V: Segmentation throughput and resource usage on NVIDIA L40 and Orin. Bold and underline mark the highest and second-highest FPS per dataset and device.
Variant
Projection
Relative- z encoding
Height-aligned BEV aux.
Dual-scale BEV context
mIoU on MCD (%)
mIoU on SemanticKITTI (%)
Single-layer point–BEV baseline
Single-layer BEV
×
×
×
60.6
64.3
Height-partitioned point–BEV baseline
Height-partitioned BEV
×
×
×
60.5
63.2
w/o dual-scale BEV context
Height-partitioned BEV
√
√
×
60.9
64.6
w/o height-aligned BEV aux.
Height-partitioned BEV
√
×
√
60.4
64.0
w/o relative- z offsets
Height-partitioned BEV
×
√
√
61.2
64.0
Full model
Height-partitioned BEV
√
√
√
62.7±0.2
64.6±0.2
TABLE VI: Component ablation on MCD test and SemanticKITTI validation with fixed training settings per dataset. Full-model results summarize multiple training runs; other variants use seed 0.
Height partitions Z
mIoU on MCD (%)
mIoU on SemanticKITTI (%)
1
61.8
63.9
3
61.3
64.8
6
62.5
63.7
8
62.4
–
TABLE VII: Sensitivity to the number of height partitions Z on MCD test and SemanticKITTI validation using seed 0. Bold marks the highest mIoU per dataset; – denotes an unevaluated setting.
Fig. 5: Planar/nonplanar label grouping for Hybrid-VoxelMap [ 2 ] on a selected SemanticKITTI 08 scan. Blue and gray indicate correct planar and nonplanar assignments; yellow and red indicate false-planar and missed-planar assignments, respectively. Red boxes mark corresponding regions. The reference groups are derived from semantic labels.
Dataset
Sequence
Semantic input
SLAM
Semantic front end
System throughput (Hz) ↑
ATE RMSE (m) ↓
Runtime (ms) ↓
CPU Memory (MiB) ↓
GPU Memory (MiB) ↓
SemanticKITTI [ 13 ]
08
Gaussian-only, no semantics
3.83
70.1
855.2
–
14.1
08
Ground-truth semantics
3.48
75.5
856.4
–
13.1
[3pt/2pt]
08
SalsaNext [ 6 ] semantics
3.77
81.8
852.2
184.4
11.2
08
HARP-NeXt [ 11 ] semantics
3.76
80.5
858.6
804.6
8.0
08
Our semantics
3.71
84.4
852.4
626.9
11.1
TABLE VIII: Outdoor semantic-assisted odometry with Hybrid-VoxelMap [ 2 ] on Orin. Bold: lowest ATE among learned semantic inputs per dataset; –: inapplicable.
Dataset
Sequence
Semantic input
SLAM
Semantic front end
System throughput (Hz) ↑
Endpoint discrepancy (m) ↓
Runtime (ms) ↓
CPU Memory (MiB) ↓
GPU Memory (MiB) ↓
Indoor campus
Campus route
Gaussian-only, no semantics
5.09
52.5
244.6
–
18.8
[3pt/2pt]
Campus route
SalsaNext [ 6 ] semantics
1.94
55.6
236.9
76.4
17.7
Campus route
HARP-NeXt [ 11 ] semantics
0.02
59.1
212.8
222.5
15.3
Campus route
Our semantics
0.01
59.1
217.7
364.0
15.2
TABLE IX: Indoor endpoint position discrepancy with Hybrid-VoxelMap [ 2 ] on Orin. Bold: lowest discrepancy among learned semantic inputs; –: inapplicable.
Semantic input
Candidate Recall@5 (%) ↑
Accepted-loop recall (%) ↑
Semantic front-end service (ms) ↓
Ground-truth semantics
93.2
70.4
–
[3pt/2pt] SalsaNext [ 6 ]
81.6
57.0
92.8
HARP-NeXt [ 11 ]
86.9
56.7
119.8
SegNet4D [ 33 ] †
90.7
63.6
413.9
Ours
91.9
65.4
98.5
TABLE X: Loop retrieval with SGLC [ 3 ] on SemanticKITTI 08, measured on Orin. Bold: best among learned semantic inputs.
Fig. 6: Semantic maps generated with SG-SLAM [ 4 ] on SemanticKITTI 08. Left: global map generated using our semantic input; the compared region is boxed in red. Right: local maps from four semantic inputs, each with its own estimated poses. Ground truth in the panel labels denotes the semantic labels used as input.
Efficient indoor LiDAR perception is challenging because mobile robots must understand cluttered three-dimensional environments under strict latency and memory constraints. Existing point-based and voxel-based methods often incur substantial computational overhead, whereas conventional bird's-eye-view (BEV) representations improve efficiency at the cost of discarding vertical geometric information. We present IndoorBEV, a lightweight LiDAR perception framework that mitigates this tradeoff through a height-aware BEV representation and geometry-conditioned feature fusion. IndoorBEV summarizes the vertical point distribution in each BEV cell using statistical height features and multi-frequency height encoding, allowing informative three-dimensional cues to be processed efficiently by two-dimensional convolutions. A lightweight encoder then integrates complementary geometric features with multi-scale local representations and compact global scene context. Decoupled dense prediction heads jointly produce semantic BEV maps and oriented object bounding boxes. IndoorBEV contains only 0.6M parameters and requires 2.3 MB of model storage. On an NVIDIA AGX Orin, it uses 21.52 MB of GPU memory per inference and achieves a mean latency of 169.6 ms under a 200 ms perception deadline, with a deadline miss ratio of 1.8%. Evaluations on simulated scenes, real-world robot scans, and an open-source indoor point-cloud dataset demonstrate a favorable tradeoff among perception accuracy, latency, and memory consumption. These results indicate that explicitly encoding vertical geometry within a compact BEV representation provides an effective approach to resource-efficient indoor LiDAR perception.
This paper studies LiDAR-assisted RGB semantic segmentation, where LiDAR point-cloud features improve dense pixel-wise predictions in the RGB image plane rather than 3D point labels or detections. Existing 2D fusion methods often project point clouds into shallow depth/range maps and apply spatially uniform fusion, discarding rich 3D structure and ignoring the range-dependent reliability of outdoor LiDAR. We propose RAHF-Net, a Range-Aware Heterogeneous Fusion Network. Its Cylindrical-to-Perspective (C2P) module projects pretrained cylindrical voxel features into the image plane, preserving structured 3D semantic priors. A Range-Aware Heterogeneous Fusion (RAHF) module then uses a soft depth-derived range mask to perform LiDAR-guided geometric enhancement in reliable near-field regions and visual-context enhancement in sparse far-field regions. On KITTI-360, RAHF-Net achieves 65.76 mIoU, outperforming representative RGB-only and RGB-LiDAR baselines, including CMX by 1.45 points. Experiments on DELIVER further demonstrate improved robustness under adverse weather, illumination degradation, and sensor corruption.
R. D. Lin, Pengcheng Weng, Ge Wang +2
School of Software Engineering, Xi’an Jiaotong University, China · School of Computer Science and Technology, Xi’an Jiaotong University, China
Efficient structural perception is essential for mapping and autonomous navigation on resource-constrained robots. Existing 3D methods are computationally prohibitive, while traditional 2D geometric approaches lack robustness. This paper presents a lightweight, real-time framework that projects 3D LiDAR data into 2D Bird's-Eye-View (BEV) images to enable efficient detection of structural elements relevant to mapping and navigation. Within this representation, we systematically evaluate several feature extraction strategies, including classical geometric techniques (Hough Transform, RANSAC, and LSD) and a deep learning detector based on YOLO-OBB. The resulting detections are integrated through a spatiotemporal fusion module that improves stability and robustness across consecutive frames. Experiments conducted on a standard mobile robotic platform highlight clear performance trade-offs. Classical methods such as Hough and LSD provide fast responses but exhibit strong sensitivity to noise, with LSD producing excessive segment fragmentation that leads to system congestion. RANSAC offers improved robustness but fails to meet real-time constraints. In contrast, the YOLO-OBB-based approach achieves the best balance between robustness and computational efficiency, maintaining an end-to-end latency (satisfying 10 Hz operation) while effectively filtering cluttered observations in a low-power single-board computer (SBC) without using GPU acceleration. The main contribution of this work is a computationally efficient BEV-based perception pipeline enabling reliable real-time structural detection from 3D LiDAR on resource-constrained robotic platforms that cannot rely on GPU-intensive processing. The source code and pre-trained models are publicly available.
Guanliang Li, Pedro Espinosa-Angulo, David Perez-Saura +1