Fore-Mamba3D: Mamba-based Foreground-Enhanced Encoding for 3D Object Detection
Authors: Zhiwei Ning, Xuanang Gao, Jiaxi Cao, Runze Yang, Huiying Xu, Xinzhong Zhu, Jie Yang, Wei Liu
Organizations: School of Automation and Intelligent Sensing, Shanghai Jiao Tong University · Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University · School of Computer Science and Technology, Zhejiang Normal University · Beijing Geekplus Technology Co., Ltd · Institute of Medical Robotics, Shanghai Jiao Tong University
Linear modeling methods like Mamba have been merged as the effective backbone for the 3D object detection task. However, previous Mamba-based methods utilize the bidirectional encoding for the whole non-empty voxel sequence, which contains abundant useless background information in the scenes. Though directly encoding foreground voxels appears to be a plausible solution, it tends to degrade detection performance. We attribute this to the response attenuation and restricted context representation in the linear modeling for fore-only sequences. To address this problem, we propose a novel backbone, termed Fore-Mamba3D, to focus on the foreground enhancement by modifying Mamba-based encoder. The foreground voxels are first sampled according to the predicted scores. Considering the response attenuation existing in the interaction of foreground voxels across different instances, we design a regional-to-global slide window (RGSW) to propagate the information from regional split to the entire sequence. Furthermore, a semantic-assisted and state spatial fusion module (SASFMamba) is proposed to enrich contextual representation by enhancing semantic and geometric awareness within the Mamba model. Our method emphasizes foreground-only encoding and alleviates the distance-based and causal dependencies in the linear autoregression model. The superior performance across various benchmarks demonstrates the effectiveness of Fore-Mamba3D in the 3D object detection task.
Figures & tables
Figure 1: Comparison of previous group-based and group-free Mamba methods with our approach.
Figure 2: The overall framework of Fore-Mamba3D . (a) Framework: the backbone of Fore-Mamba3D consists of four stages and each contains an instance selection block and a downsampling block. (b) Instance Selection Block: we predict the foreground score for each voxel and select the top- k foreground voxels for further linear encoding. (c) RGSW: we utilize a regional-to-global sliding window process for iterative encoding. (d) SASFMamba: semantic-assisted and state spatial fusion modules are designed to enhance the semantic and geometric recognition of state variables.
Method
Present at
mAP
NDS
Car
Truck
Bus
Trailer
C.V.
Ped.
Motor.
Bike
T.C.
Barrier
Performance on the validation dataset
CenterPoint ( Yin et al., 2021 )
CVPR21
59.2
66.5
84.9
57.4
70.7
38.1
16.9
85.1
59.0
42.0
69.8
68.3
TransFusion-L ( Bai et al., 2022 )
CVPR22
65.5
70.1
86.9
60.8
73.1
43.4
25.2
87.5
72.9
57.3
77.2
70.3
VoxelNeXt ( Chen et al., 2023 )
CVPR23
64.5
70.0
84.6
53.0
64.7
55.8
28.7
85.8
73.2
45.7
79.0
74.6
DSVT ( Wang et al., 2023 )
CVPR23
66.4
71.1
87.4
62.6
75.9
42.1
25.3
88.2
74.8
58.7
77.9
71.0
HEDNet ( Zhang et al., 2024b )
NIPS23
66.7
71.4
87.7
60.6
77.8
50.7
28.9
87.1
74.3
56.8
76.3
66.9
Table 1: Performances on the nuScenes validation and test set. ‘C.V.’, ‘Ped.’, and ‘T.C.’ are short for construction vehicle, pedestrian, and traffic cone, respectively. The first and second best results are in bold and underlined , respectively. All the results are conducted without class-balance ground-truth sampling (CBGS).
Methods
BackBone
Car
Pedestrian
Cyclists
Hard
Mod
Easy
Hard
Mod
Easy
Hard
Mod
Easy
PointPillars ( Lang et al., 2019 )
MLP
79.1
75.0
68.3
52.1
43.5
41.5
75.8
59.1
52.9
IA-SSD ( Zhang et al., 2022 )
MLP
88.3
80.1
75.0
46.5
39.0
35.6
78.4
61.9
55.7
VoxelNet ( Zhou and Tuzel, 2018 )
SpCNN
77.5
65.1
57.7
39.5
33.7
31.5
61.2
48.4
44.4
DSVT ( Wang et al., 2023 )
Transformer
87.8
77.8
76.8
66.1
59.7
55.2
83.5
66.7
63.2
DGT-Det ( Ren et al., 2023 )
Transformer
89.6
80.6
78.8
–
–
–
82.1
68.9
61.0
Table 2: Comparison of the performance on the KITTI validation set with an average recall of 11. ‡ indicates that the result is reproduced by us. The first and second best results are in bold and underlined , respectively.
Methods
Vehicle
Pedestrian
Cyclist
mAP
AP
APH
AP
APH
AP
APH
L1
L2
L1
L2
L1
L2
L1
L2
L1
L2
L1
L2
L2
SECOND † ( Yan et al., 2018 )
71.0
62.6
70.3
62.0
65.2
57.2
54.2
47.5
57.1
55.0
55.6
53.5
58.3
PV-RCNN † ( Shi et al., 2020a )
75.4
67.4
74.7
66.8
72.0
63.7
61.2
54.0
65.8
63.4
64.3
61.8
64.8
Part-A2 † ( Shi et al., 2019b )
74.7
65.8
74.1
65.3
71.7
62.5
62.2
54.1
66.5
64.1
65.2
62.8
64.1
CenterPoint † ( Yin et al., 2021 )
71.3
63.2
70.8
62.7
72.1
64.3
65.5
58.2
68.7
66.1
67.4
64.9
64.5
Table 3: Performance on the Waymo dataset (trained on the 20% training dataset and evaluated on the full validation dataset). † denotes that the results of previous work are implemented by OpenPCDet ( OpenPCDet Development Team, 2020 ) . The first and second best results are in bold and underlined , respectively.
Table 6Table 7
Figure 3: The association matrix in (a) vanilla SSM, (b, c) semantic-assisted strategy with different kernel sizes K and (d) the semantic association labels. We define the semantic labels to follow a Gaussian distribution for the distance.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Model Scale
3D
BEV
Easy
Mod
Hard
Easy
Mod
Hard
-T
88.53
80.91
78.46
90.87
87.78
87.27
-B
90.32
82.16
79.54
92.27
89.09
88.11
(+1.80)
(+1.25)
(+1.08)
(+1.40)
(+1.31)
(+0.84)
Appendix
Table 8: Performance on different model scales. ‘-T’ indicates the channel dimension in the backbone is 64, and ‘-B’ denotes the channels dimension is set as 128.
Figure 4: The detailed pipeline in the SAF module, which contains the rearrangement, the 1D convolution, and the reverse process.
Figure 5: The accuracy in foreground scores prediction. We accumulate the scores along the z-axis and visualize the heatmap in the BEV.
Methods
Car
Ped.
Cyc.
3D
BEV
3D
BEV
3D
BEV
PointRCNN ( Shi et al., 2019a )
75.64
87.39
39.37
46.13
58.82
67.24
PointPillars ( Lang et al., 2019 )
74.99
86.10
43.53
50.23
59.07
62.25
VoxelNet ( Zhou and Tuzel, 2018 )
65.11
79.26
33.69
40.74
48.36
54.76
SECOND ( Yan et al., 2018 )
73.66
-
42.56
-
53.85
-
TANet ( Liu et al., 2020 )
75.94
86.54
44.34
51.38
59.44
63.77
Appendix
Table 9: Comparison of the performance on the KITTI test dataset with an average recall of 40. All the results are reported in the moderate difficulty level.
Table 13
Model
NDS
mAP
FLOPs
FPS
RetNet
72.1
67.7
31.12
59
RWKV
71.9
67.1
36.15
47
LSTM
70.8
65.9
23.92
78
Mamba
72.3
68.4
26.04
67
Appendix
Table 12: Comparison of the different sequence modeling, including RetNet, RWKV, LSTM, and Mamba.
noise ratio
NDS
mAP
Car
Truck
C.V.
Bus
Trailer
Barrier
Motor
Bike
Ped.
T.C.
w/o
72.3
68.4
88.4
65.2
28.2
80.3
48.0
71.2
75.7
57.7
89.3
80.0
5%
71.9
67.9
87.9
65.1
30.2
77.4
48.1
71.5
74.3
56.0
88.9
79.2
10%
71.6
67.4
87.4
64.4
30.3
77.0
47.4
72.0
73.2
55.4
88.4
78.1
15%
71.0
66.5
86.7
63.4
30.1
76.4
46.7
72.2
70.8
54.3
87.6
76.9
Appendix
Table 13: Comparison of the different ratios of noise in the sampled foreground voxels.
Figure 6: Visualization of the failure cases on the KITTI dataset. Compared with SECOND ( Yan et al., 2018 ) and LION ( Liu et al., 2024b ) , our method can predict the targets more consistently with the ground truth.
Figure 7: Visualization of the failure cases between our method and the previous method in the nuScenes dataset.
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, 710049