Autonomous mobile robots confront significant chal lenges in recognizing the motion properties of objects in dy namic environments. However, existing methods are hindered by limitations such as boundary ambiguity, occlusion, viewpoint constraints, and training complexity. In this paper, we propose an innovative end-to-end Multi-View Spatio-Temporal Fusion network, named MVSTF, which utilizes sequential point clouds to generate multi-view images and feature differences, and designs a multi-branch network to extract spatio-temporal and dimen sional features in parallel, and realizes high-precision moving object segmentation. First, sequential point clouds are utilized to generate multi-view images and feature differences. Then, an end-to-end multi-branch network including Dual Down-sampling Blocks (DDB), Spatial Attention Blocks (SAB), Multi-Kernel Convolutional Neural Network blocks (MKCNN), Dual Feature Fusion Blocks (DFFB), and the Point Fusion (PF) module, is designed to extract motion features from feature differences and geometric and semantic features from multi-view images. Finally, an auxiliary semantic head is introduced to further improve the algorithm's robustness and the model's interpretability. This work is validated on the SemanticKITTI and KITTI-Road bench marks, achieving state-of-the-art results on the SemanticKITTI MOS validation set and the KITTI-Road validation set, and competitive results on the hidden SemanticKITTI-MOS test set. The code is available at https://github.com/ZhouJJ111/MVSTF.
Figures & tables
Fig. 1: Comparative methods. (a) LMNet [ 1 ] integrates current range images from 3D LiDAR scans with prior residual images, enhancing feature extraction through temporal data integration. (b) MotionSeg3D [ 2 ] employs a dual-branch architecture that separately processes range images and residuals to isolate appearance and motion features. (c) Unlike the single-view focus of LMNet [ 1 ] and MotionSeg3D [ 2 ] , MVSTF extracts multi-view spatio-temporal features from point clouds and utilizes an auxiliary semantic head to accelerate convergence to enhance network performance.
Fig. 2: The Multi-view representations: range view and bird’s-eye view.
Fig. 3: Overview of MVSTF . Multi-view feature encoding: We convert the point cloud into BEV and RV images, perform ego-motion compensation on the historical frames, and calculate BEV feature differences. Multi-Branch Networks: The Fully Convolutional Network (FCN) consists of Dual Down-sampling Blocks (DDB), Spatial Attention Blocks (SAB), Multi-Kernel Convolutional Neural Network blocks (MKCNN), Dual Feature Fusion Blocks (DFFB), and Point Fusion (PF) modules to enable end-to-end network training. In the training phase, the auxiliary semantic head promotes parameter convergence and remarkably improves network performance, and we remove the auxiliary semantic head in the inference phase. The BEV feature difference module takes the current and historical BEV feature maps as input, computes their differences, and outputs the concatenation of the current-frame features and the feature differences, which is then fed into the BEV branch.
Fig. 4: The Multi-Kernel Convolutional Neural Network block (MKCNN).
Fig. 5: The Spatial Attention Block (SAB) and the Dual Feature Fusion Block (DFFB).
Fig. 6: The Point Fusion (PF) module.
Fig. 7: The Point to Grid and Grid to Point.
Fig. 8: Qualitative comparison of the segmentation results of the various methods in the range view. Red represents moving point clouds and gray represents static point clouds. There are three main types of detection errors: incomplete object segmentation, undetected motion objects, and incorrect motion classification.
Fig. 9: Qualitative comparison of MVSTF with MotionSeg3D [ 2 ] , and MotionBEV [ 8 ] on the SemanticKITTI-MOS validation set. Green circles highlight false predictions.
Fig. 10: Qualitative analysis of different methods for LiDAR-MOS on the SemanticKITTI-MOS validation set.
Method
IoUMOS(%)
Validation 08
Test 11-21
LiMoSeg [ 35 ]
52.6
-
LMNet [ 1 ]
67.1
54.5
MotionSeg3D [ 2 ]
71.4
64.9
4DMOS [ 5 ]
77.2
65.2
MotionBEV [ 8 ]
76.5
69.7
TABLE I: Comparison With State-of-the-Art Methods on the SemanticKITTI-MOS Benchmark.
Method
IoUMOS
Precision
Recall
Mean IoU
LMNet [ 1 ]
67.11
87.58
74.17
83.49
MotionSeg3D [ 2 ]
69.76
88.15
76.98
84.82
InsMOS [ 6 ]
70.72
92.95
74.72
85.30
MotionBEV [ 8 ]
77.88
93.52
82.33
88.90
MVSTF
84.51
96.52
87.16
92.23
MVSTF *
85.60
96.21
88.59
92.77
TABLE II: Performance Comparison of Different Methods on Moving Object Segmentation. All Values Are Reported in Percentages.
Method
Close ( < 20m)
Medium ( ≥ 20m, < 50m)
Far ( ≥ 50m)
IoUMOS
R
P
IoUMOS
R
P
IoUMOS
R
P
LMNet [ 1 ]
70.72
76.89
89.80
43.88
54.30
69.56
0.00
0.00
-
MotionSeg3D [ 2 ]
72.26
79.40
88.94
52.11
59.32
81.08
5.30
5.30
100.00
InsMOS [ 6 ]
75.29
88.78
83.21
57.67
66.81
80.84
10.88
10.89
98.63
MotionBEV [ 8 ]
80.85
85.40
93.81
56.35
59.90
90.50
0.00
0.00
-
MF-MOS [ 37 ]
79.31
84.98
92.23
54.67
64.10
78.81
47.97
50.08
91.94
TABLE III: MOS Performance on the SemanticKITTI-MOS Validation Set at Different Ranges. R Denotes Recall and P Denotes Precision.
Temporal Branch
Component
Head Structure
BEV
RV
DDB
SAB
MKCNN
DFFB
One head
Two Head
IoUMOS(%)
(a)
2
-
✓
✓
70.27
(b)
-
2
✓
✓
65.73
(c)
2
1
✓
✓
✓
74.83
(d)
8
1
✓
✓
✓
78.98
(f)
8
1
✓
✓
✓
✓
81.09
TABLE IV: Ablation Study on Each Component in MVSTF on the SemanticKITTI-MOS Validation Set. DDB: Dual Down-sampling Blocks, SAB: Spatial Attention Blocks, MKCNN: Multi-Kernel Convolutional Neural Network Blocks, DFFB: Dual Feature Fusion Blocks.
Input (N=8)
IoUMOS(%)
(a)
Continuous frames
76.37
(b)
Height Residuals
79.09
(c)
Feature Differences
84.51
TABLE V: Ablation Study About Different Input Types of the BEV Branch on the SemanticKITTI-MOS Validation Set.
Frame N
Height Residuals
Feature Differences
IoUMOS(%)
runtime(ms)
IoUMOS(%)
runtime(ms)
2
72.71
68.22
81.60
190.94
4
74.92
76.22
82.76
223.36
6
75.60
81.85
82.96
233.66
8
79.09
94.99
85.60*
238.92
10
74.65
102.92
81.32
269.12
TABLE VI: Ablation Study About the Number of Historical Frames on the SemanticKITTI-MOS Validation Set.
Method
IoUMOS(%)
LMNet [ 1 ]
87.4
MotionBEV [ 8 ]
80.5
InsMOS [ 6 ]
83.9
MF-MOS [ 37 ]
87.9
MambaMOS [ 39 ]
89.4
MVSTF
89.8
TABLE VII: Comparison with state-of-the-art methods on the KITTI-Road dataset.
Configuration
Odometry without loop closure
SLAM with loop closure
trel (%) ↓
rrel ( ∘ /100 m) ↓
trel (%) ↓
rrel ( ∘ /100 m) ↓
SuMa++
1.3709±0.0000
0.3411±0.0000
1.3475±0.0139
0.3148±0.0039
SuMa++ + MVSTF
0.8848±0.0000
0.3272±0.0000
0.9841±0.1913
0.3496±0.0758
TABLE VIII: Trajectory errors before and after integrating MVSTF into SuMa++.
Method
Dynamic-point removal ↑
Static-point retention ↑
Dynamic-voxel residual ↓
Static-voxel retention ↑
LMNet
92.45
99.9788
9.51
99.9674
MotionBEV
85.04
99.9914
12.61
99.9835
MVSTF
95.17
99.9941
4.02
99.9743
TABLE IX: Dynamic filtering and static-structure preservation of different motion segmentation methods (%).
Fig. 11: 3D mapping results for two local regions of sequence 08. Panels (a)–(c) show ROI 1, and panels (d)–(f) show ROI 2. The columns correspond to LMNet, MotionBEV, and MVSTF, respectively. Red denotes dynamic-only reference voxels that remain occupied, while gray denotes other occupied voxels. Blue dashed boxes indicate the regions enlarged in the lower-right insets. The percentage in the upper-right corner of each panel denotes the dynamic-voxel residual rate over the entire corresponding ROI; lower values are better. All methods use the same baseline-estimated poses, voxel grid, and visualization settings.
Haiyang Wu, George Vosselman, and Ville Lehtola are with the Department of Geoinformatics, Faculty of Geo-Information Science and Earth Observation (ITC), University of Twente, 7522 NB Enschede, The Netherlands.
Department of Mechanical and Industrial Engineering, Tallinn University of Technology, Tallinn, Estonia · FinEst Centre for Smart Cities, Tallinn University of Technology, Tallinn, Estonia · Universitas Mercatorum, Rome, Italy