3D Instance Segmentation

Follow this topic Ask about this topic

3D instance segmentation assigns scene elements to individual objects, often alongside semantic labels, so robots and mapping systems can reason about cluttered or incomplete environments. Common approaches use point-cloud networks with query-based mask decoders or superpoint clustering, while newer pipelines lift 2D foundation-model masks across views or learn object identities in 3D Gaussian scenes. Indoor benchmarks such as ScanNet and ScanNet200, outdoor driving datasets such as SemanticKITTI and nuScenes, and domain-specific forest and biomedical data test these methods in different settings. A central challenge is balancing accurate, consistent object boundaries and open-vocabulary generalization with limited labels, occlusion, and the speed and memory demands of large scenes.

Latest papers

61 papers7 papers in the last four weeks, down 22% on the four weeks before. 0.0% of all new papers.

Voxeland: Probabilistic Instance-Aware Semantic Mapping with Evidence-based Uncertainty Quantification
Nov 13, 2024Jose-Luis Matez-Bandera, Pepe Ojeda, Javier Monroy +23D Semantic Mapping Segmentation Uncertainty Estimation
Machine Perception and Intelligent Robotics (MAPIR) Group, Malaga Institute for Mechatronics Engineering and Cyber-Physical Systems (IMECH.UMA), University of Malaga, Campus de Teatinos, 29071 Malaga, Spain

Nov 13, 2024J/K move · Enter open · S save

Voxeland: Probabilistic Instance-Aware Semantic Mapping with Evidence-based Uncertainty Quantification

Jose-Luis Matez-Bandera, Pepe Ojeda, Javier Monroy, Javier Gonzalez-Jimenez+1

Machine Perception and Intelligent Robotics (MAPIR) Group, Malaga Institute for Mechatronics Engineering and Cyber-Physical Systems (IMECH.UMA), University of Malaga, Campus de Teatinos, 29071 Malaga, Spain

3D Semantic Mapping Segmentation Uncertainty Estimation 3D Instance Segmentation 3D Scene Understanding

Open paper AskPDF arXiv

Robots in human-centered environments require accurate scene understanding to perform high-level tasks effectively. This understanding can be achieved through instance-aware semantic mapping, which involves reconstructing elements at the level of individual instances. Neural networks, the de facto solution for scene understanding, still face limitations such as overconfident incorrect predictions with out-of-distribution objects or generating inaccurate masks. Placing excessive reliance on these predictions makes the reconstruction susceptible to errors, reducing the robustness of the resulting maps and hampering robot operation. In this work, we propose Voxeland, a probabilistic framework for incrementally building instance-aware semantic maps. Inspired by the Theory of Evidence, Voxeland treats neural network predictions as subjective opinions regarding map instances at both geometric and semantic levels. These opinions are aggregated over time to form evidence, and are formalized through a probabilistic model. This enables us to quantify uncertainty in the reconstruction process, facilitating the identification of map areas requiring improvement (e.g. reobservation or reclassification). As a possible strategy to exploit this uncertainty quantification, we incorporate a Large Vision-Language Model (LVLM) to perform semantic level disambiguation for instances with high uncertainty. Results from the standard benchmarking on the publicly available SceneNN dataset demonstrate that Voxeland outperforms state-of-the-art methods, highlighting the benefits of incorporating and leveraging both instance- and semantic-level uncertainties to enhance reconstruction robustness. This is further validated through qualitative and quantitative experiments conducted on the real-world ScanNet dataset.