HiPFuse: Real-Time LiDAR Semantic Segmentation for Robotic Perception on Edge Devices
Authors: Haiyang Wu, George Vosselman, Ville Lehtola
Organizations: Haiyang Wu, George Vosselman, and Ville Lehtola are with the Department of Geoinformatics, Faculty of Geo-Information Science and Earth Observation (ITC), University of Twente, 7522 NB Enschede, The Netherlands.
Real-time robotic perception requires accurate LiDAR semantics under limited resources. Bird's-eye-view (BEV) aggregation is efficient but can mix differently labeled surfaces in columns. We present HiPFuse, a point-BEV fusion framework that exchanges point and height-partitioned BEV features across horizontal scales, combining point-wise detail with 2D spatial context from one LiDAR scan. Fixed height assignments preserve separate vertical aggregation groups, with auxiliary supervision of initial BEV features using targets from the same 3D cells. A dual-scale BEV context module adds broader context from a coarser grid. On SemanticKITTI and MCD, HiPFuse achieves 63.7% and 62.5% mIoU at 14.2 and 22.4 FPS on Jetson AGX Orin, respectively, exceeding the 10 Hz scan rate. Separate training and evaluation show indoor applicability. Replacing only the baseline semantic front end with HiPFuse in fixed robotic systems reduces odometry absolute trajectory error RMSE by 10.6% on average across two outdoor sequences and increases accepted-loop recall by 8.7 percentage points on SemanticKITTI. Qualitative semantic maps show more continuous road labeling. Our code is available at https://github.com/haiyang2022/HiPFuse.
Figures & tables
Fig. 1: SemanticKITTI test accuracy versus measured Jetson AGX Orin throughput (Tables III and V ). Shading indicates throughput meeting the 10 Hz scan rate under the measured settings.
TABLE II: Class-wise IoU and mIoU (%) on the MCD NTU test sequence. Bold: best; underline: second best.
Method
Category
Year
Wall
Floor
Ceiling
Column
Beam
Door
Stair
Table
Chair
Sofa
Cab./shelf
Screen
Light
Vegetation
Board
Other
mIoU (%)
SalsaNext [ 6 ]
Projection-based
2020
83.1
90.7
92.2
43.9
76.9
51.0
47.3
76.9
69.6
52.7
57.9
91.2
90.0
89.8
31.5
72.2
69.8
WaffleIron [ 20 ]
Projection-based
2023
88.4
93.4
93.8
55.7
81.0
60.7
79.6
83.0
80.2
73.8
67.3
93.8
97.2
94.7
44.8
81.1
79.3
HARP-NeXt [ 11 ]
Fusion-based
2025
88.5
92.1
94.4
55.5
80.1
56.1
63.8
79.9
75.6
68.2
43.2
86.7
94.5
91.2
55.6
77.9
75.2
Ours
Fusion-based
2026
86.7
92.6
93.7
53.2
76.7
52.1
37.5
81.7
69.8
71.5
58.0
85.9
95.1
90.1
27.1
73.4
71.6
TABLE IV: Class-wise IoU and mIoU (%) on the Semantic-ITC test set. Bold: best; underline: second best.
Fig. 4: Selected semantic predictions on (a) SemanticKITTI validation, (b) MCD test, and (c) Semantic-ITC test scans. Red boxes identify corresponding regions across methods. Row legends identify the colored classes; gray denotes other valid classes.
Dataset
Method
Category
Params (M)
NVIDIA L40 GPU
Orin
GPU Memory (MiB)
Time/scan (ms)
FPS
GPU Memory (MiB)
Time/scan (ms)
FPS
MCD [ 14 ]
SalsaNext [ 6 ]
Projection-based
6.7
294.5
13.2
75.7
186.7
29.9
33.5
SPVCNN [ 12 ]
Fusion-based
21.8
146.1
77.8
12.8
151.2
61.5
16.3
HARP-NeXt [ 11 ]
Fusion-based
5.4
384.9
20.3
49.1
276.9
45.8
21.8
Ours
Fusion-based
14.3
358.4
18.4
54.3
359.4
44.6
22.4
SemanticKITTI [ 13 ]
SalsaNext [ 6 ]
Projection-based
6.7
360.1
13.7
72.9
1167.0
64.2
15.6
TABLE V: Segmentation throughput and resource usage on NVIDIA L40 and Orin. Bold and underline mark the highest and second-highest FPS per dataset and device.
Variant
Projection
Relative- z encoding
Height-aligned BEV aux.
Dual-scale BEV context
mIoU on MCD (%)
mIoU on SemanticKITTI (%)
Single-layer point–BEV baseline
Single-layer BEV
×
×
×
60.6
64.3
Height-partitioned point–BEV baseline
Height-partitioned BEV
×
×
×
60.5
63.2
w/o dual-scale BEV context
Height-partitioned BEV
√
√
×
60.9
64.6
w/o height-aligned BEV aux.
Height-partitioned BEV
√
×
√
60.4
64.0
w/o relative- z offsets
Height-partitioned BEV
×
√
√
61.2
64.0
Full model
Height-partitioned BEV
√
√
√
62.7±0.2
64.6±0.2
TABLE VI: Component ablation on MCD test and SemanticKITTI validation with fixed training settings per dataset. Full-model results summarize multiple training runs; other variants use seed 0.
Height partitions Z
mIoU on MCD (%)
mIoU on SemanticKITTI (%)
1
61.8
63.9
3
61.3
64.8
6
62.5
63.7
8
62.4
–
TABLE VII: Sensitivity to the number of height partitions Z on MCD test and SemanticKITTI validation using seed 0. Bold marks the highest mIoU per dataset; – denotes an unevaluated setting.
Fig. 5: Planar/nonplanar label grouping for Hybrid-VoxelMap [ 2 ] on a selected SemanticKITTI 08 scan. Blue and gray indicate correct planar and nonplanar assignments; yellow and red indicate false-planar and missed-planar assignments, respectively. Red boxes mark corresponding regions. The reference groups are derived from semantic labels.
Dataset
Sequence
Semantic input
SLAM
Semantic front end
System throughput (Hz) ↑
ATE RMSE (m) ↓
Runtime (ms) ↓
CPU Memory (MiB) ↓
GPU Memory (MiB) ↓
SemanticKITTI [ 13 ]
08
Gaussian-only, no semantics
3.83
70.1
855.2
–
14.1
08
Ground-truth semantics
3.48
75.5
856.4
–
13.1
[3pt/2pt]
08
SalsaNext [ 6 ] semantics
3.77
81.8
852.2
184.4
11.2
08
HARP-NeXt [ 11 ] semantics
3.76
80.5
858.6
804.6
8.0
08
Our semantics
3.71
84.4
852.4
626.9
11.1
TABLE VIII: Outdoor semantic-assisted odometry with Hybrid-VoxelMap [ 2 ] on Orin. Bold: lowest ATE among learned semantic inputs per dataset; –: inapplicable.
Dataset
Sequence
Semantic input
SLAM
Semantic front end
System throughput (Hz) ↑
Endpoint discrepancy (m) ↓
Runtime (ms) ↓
CPU Memory (MiB) ↓
GPU Memory (MiB) ↓
Indoor campus
Campus route
Gaussian-only, no semantics
5.09
52.5
244.6
–
18.8
[3pt/2pt]
Campus route
SalsaNext [ 6 ] semantics
1.94
55.6
236.9
76.4
17.7
Campus route
HARP-NeXt [ 11 ] semantics
0.02
59.1
212.8
222.5
15.3
Campus route
Our semantics
0.01
59.1
217.7
364.0
15.2
TABLE IX: Indoor endpoint position discrepancy with Hybrid-VoxelMap [ 2 ] on Orin. Bold: lowest discrepancy among learned semantic inputs; –: inapplicable.
Semantic input
Candidate Recall@5 (%) ↑
Accepted-loop recall (%) ↑
Semantic front-end service (ms) ↓
Ground-truth semantics
93.2
70.4
–
[3pt/2pt] SalsaNext [ 6 ]
81.6
57.0
92.8
HARP-NeXt [ 11 ]
86.9
56.7
119.8
SegNet4D [ 33 ] †
90.7
63.6
413.9
Ours
91.9
65.4
98.5
TABLE X: Loop retrieval with SGLC [ 3 ] on SemanticKITTI 08, measured on Orin. Bold: best among learned semantic inputs.
Fig. 6: Semantic maps generated with SG-SLAM [ 4 ] on SemanticKITTI 08. Left: global map generated using our semantic input; the compared region is boxed in red. Right: local maps from four semantic inputs, each with its own estimated poses. Ground truth in the panel labels denotes the semantic labels used as input.