Organizations: Graduate School of Information Science and Electrical Engineering, Kyushu University, Fukuoka, Japan · Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA, USA · Faculty of Information Science and Electrical Engineering, Kyushu University, Fukuoka, Japan · Technical University of Cartagena, Cartagena, Spain
Semantic place categorization, which is one of the essential tasks for autonomous robots and vehicles, allows them to have capabilities of self-decision and navigation in unfamiliar environments. In particular, outdoor places are more difficult targets than indoor ones due to perceptual variations, such as dynamic illuminance over twenty-four hours and occlusions by cars and pedestrians. This paper presents a novel method of categorizing outdoor places using convolutional neural networks (CNNs), which take omnidirectional depth/reflectance images obtained by 3D LiDARs as the inputs. First, we construct a large-scale outdoor place dataset named Multi-modal Panoramic 3D Outdoor (MPO) comprising two types of point clouds captured by two different LiDARs. They are labeled with six outdoor place categories: coast, forest, indoor/outdoor parking, residential area, and urban area. Second, we provide CNNs for LiDAR-based outdoor place categorization and evaluate our approach with the MPO dataset. Our results on the MPO dataset outperform traditional approaches and show the effectiveness in which we use both depth and reflectance modalities. To analyze our trained deep networks we visualize the learned features.
Figures & tables
Figure 1: Samples from our dataset. The panoramic images represent the surrounding geometric and photometric features compactly.
Figure 2: Omnidirectional laser scanner, GPS, and 360 deg. camera
Figure 3: Data acquisition setup
Figure 4: Examples of an RGB (top), depth (middle), and reflectance image (bottom) for each place category.
Figure 5: Scanning locations for Sparse Multi-modal Panoramic 3D Outdoor Dataset
Figure 6: FARO Focus 3D S120
Figure 7: Examples of an RGB (top), depth (middle), and reflectance image (bottom) for each place category.
Figure 8: Scanning locations for Dense Multi-modal Panoramic 3D Outdoor Dataset
Figure 9: Architecture of multi-modal models
Figure 10: Confusion matrix of best models on a depth and reflectance inputs.
Figure 11: Horizontal rotation of an input image vs. accuracy.
Figure 12: Visualization of contributing features given rotated images, by Grad-CAM [ 41 ] .
Figure 13: GradCAM of contributing regions which the CNN models activate to predict correct categories using Grad-CAM [ 41 ] . Each group shows the average Grad-CAM maps on test sets obtained from the baseline VGG (top) and the modified VGG with RWMP and HCC (bottom). The modification works for sharpening of feature selectivity and equilibration of feature extractability between the center and the side area.
Graduate School of Information Science and Electrical Engineering, Kyushu University, Fukuoka 819-0395, Japan · Polytechnic University of Cartagena (UPCT), Spain · Faculty of Information Science and Electrical Engineering, Kyushu University, Fukuoka 819-0395, Japan
Department of Aerospace Engineering, Seoul National University, Seoul, Republic of Korea. · InnoCORE AI-Transformed Aerospace Research Center, KAIST, Daejeon, Republic of Korea. · Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, Republic of Korea.