Organizations: Graduate School of Information Science and Electrical Engineering, Kyushu University, Fukuoka, Japan · Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA, USA · Faculty of Information Science and Electrical Engineering, Kyushu University, Fukuoka, Japan · Technical University of Cartagena, Cartagena, Spain
Semantic place categorization, which is one of the essential tasks for autonomous robots and vehicles, allows them to have capabilities of self-decision and navigation in unfamiliar environments. In particular, outdoor places are more difficult targets than indoor ones due to perceptual variations, such as dynamic illuminance over twenty-four hours and occlusions by cars and pedestrians. This paper presents a novel method of categorizing outdoor places using convolutional neural networks (CNNs), which take omnidirectional depth/reflectance images obtained by 3D LiDARs as the inputs. First, we construct a large-scale outdoor place dataset named Multi-modal Panoramic 3D Outdoor (MPO) comprising two types of point clouds captured by two different LiDARs. They are labeled with six outdoor place categories: coast, forest, indoor/outdoor parking, residential area, and urban area. Second, we provide CNNs for LiDAR-based outdoor place categorization and evaluate our approach with the MPO dataset. Our results on the MPO dataset outperform traditional approaches and show the effectiveness in which we use both depth and reflectance modalities. To analyze our trained deep networks we visualize the learned features.
Figures & tables
Figure 1: Samples from our dataset. The panoramic images represent the surrounding geometric and photometric features compactly.
Figure 2: Omnidirectional laser scanner, GPS, and 360 deg. camera
Figure 3: Data acquisition setup
Figure 4: Examples of an RGB (top), depth (middle), and reflectance image (bottom) for each place category.
Figure 5: Scanning locations for Sparse Multi-modal Panoramic 3D Outdoor Dataset
Figure 6: FARO Focus 3D S120
Figure 7: Examples of an RGB (top), depth (middle), and reflectance image (bottom) for each place category.
Figure 8: Scanning locations for Dense Multi-modal Panoramic 3D Outdoor Dataset
Figure 9: Architecture of multi-modal models
Figure 10: Confusion matrix of best models on a depth and reflectance inputs.
Figure 11: Horizontal rotation of an input image vs. accuracy.
Figure 12: Visualization of contributing features given rotated images, by Grad-CAM [ 41 ] .
Figure 13: GradCAM of contributing regions which the CNN models activate to predict correct categories using Grad-CAM [ 41 ] . Each group shows the average Grad-CAM maps on test sets obtained from the baseline VGG (top) and the modified VGG with RWMP and HCC (bottom). The modification works for sharpening of feature selectivity and equilibration of feature extractability between the center and the side area.
We present two multi-modal panoramic 3D outdoor (MPO) datasets for semantic place categorization with six categories: forest, coast, residential area, urban area and indoor/outdoor parking lot. The first dataset consists of 650 static panoramic scans of dense (9,000,000 points) 3D color and reflectance point clouds obtained using a FARO laser scanner with synchronized color images. The second dataset consists of 34,200 real-time panoramic scans of sparse (70,000 points) 3D reflectance point clouds obtained using a Velodyne laser scanner while driving a car. The datasets were obtained in the city of Fukuoka, Japan and are publicly available in [1], [2]. In addition, we compare several approaches for semantic place categorization with best results of 96.42% (dense) and 89.67% (sparse).
Hojung Jung, Yuki Oto, Oscar M. Mozos +2
Graduate School of Information Science and Electrical Engineering, Kyushu University, Fukuoka 819-0395, Japan · Polytechnic University of Cartagena (UPCT), Spain · Faculty of Information Science and Electrical Engineering, Kyushu University, Fukuoka 819-0395, Japan
We present DXPR, a depth-based cross-modal place recognition (CMPR) framework that uses vision foundation models (VFMs) to match monocular camera queries against a LiDAR map without modality-specific encoders. This enables robots and autonomous vehicles to robustly localize using only cameras within pre-built LiDAR maps, even under severe seasonal, weather, and illumination changes. The key idea is to convert both camera images and LiDAR scans into a unified depth image representation so that a single VFM backbone with an aggregation head can learn modality-invariant global descriptors. To make pairwise metric learning faithful to scene geometry, we introduce a geometry-aware overlap miner: after cross-modal scale alignment of camera and LiDAR depth, we forward-warp measurements between views to compute a pixel-level overlap score. This score relabels ambiguous pairs and adaptively modulates the positive margin in a multi-similarity loss to avoid overfitting on weakly overlapping views. Extensive experiments on KITTI odometry and Boreas demonstrate strong performance and robustness across seasons, weather, and day/night. On KITTI, DXPR achieves near-perfect Recall@1 on most sequences and outperforms prior CMPR baselines. On Boreas, DXPR achieves intra-sequence performance on par with a strong single-modal baseline (DINOv2-SALAD), while showing clear improvements in the more challenging inter-sequence setting. Compared with RangeBEV, our method consistently performs better in both intra- and inter-sequence evaluations, demonstrating robustness under diverse seasonal and illumination changes.
Yungsoo Han, Youngseok Jang, Seungwon Roh +2
Department of Aerospace Engineering, Seoul National University, Seoul, Republic of Korea. · InnoCORE AI-Transformed Aerospace Research Center, KAIST, Daejeon, Republic of Korea. · Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, Republic of Korea.
LiDAR serves as a primary sensing modality for robots operating in outdoor environments. However, the performance of deep learning models in this domain is severely limited by the scarcity of labeled data, a direct result of the high cost of 3D annotation. Self-supervised learning addresses this scarcity by learning general-purpose features from unlabeled data. In this work, we present a multi-modal, multi-teacher distillation framework for self-supervised learning on outdoor LiDAR point clouds. Building upon the Sonata architecture, we introduce Vernata, consisting of three extensions: sparse view augmentation to improve robustness against varying point densities, a memory bank mechanism to stabilize resource-constrained training, and cross-modal distillation utilizing dense, high-resolution 2D image features to enable fine-grained semantic guidance. We evaluate our method on the GrandTour, TartanGround, and Waymo datasets, as well as data collected from our own robotic platforms. Our experiments demonstrate a significant performance improvement over Sonata baselines, yielding mIoU scores of 54.7 on TartanGround (+5.9 points, +12.1%) and 57.1 on Waymo (+7.3 points, +14.7%). Finally, we show that the self-supervised approach maintains strong performance even in reduced-modality settings (lacking color or normals), achieving competitive mIoU scores of 49.4 and 50.2 on the respective datasets.