InterRSGaussian: Interactive Multi-view Instance Segmentation for 3D Remote Sensing Scenes
Organizations: School of Astronautics, Beihang University, Beijing, 100191, China · Key Laboratory of Spacecraft Design Optimization and Dynamic Simulation Technologies, Ministry of Education, Beihang University, Beijing, 100191, China
Abstract
Remote sensing image segmentation plays a critical role in various applications. Interactive instance segmentation, a specialized task within image segmentation, aims to generate precise segmentation results for target objects based on user input prompts. However, previous studies on interactive instance segmentation primarily focus on single-view images, limiting their ability to comprehensively understand complex 3D remote sensing scenes. In this study, we extend interactive instance segmentation from a single-view to a multi-view framework, which can also be easily converted to a 3D segmentation format, thus achieving more accurate and comprehensive 3D remote sensing scene perception across multiple views. First, we construct a multi-view interactive instance segmentation dataset for remote sensing, comprising five distinct scenes, each captured from about one hundred different views. Second, we propose InterRSGaussian, a novel instance segmentation method based on 3D Gaussians. InterRSGaussian enables users to provide point clicks on target objects from any arbitrary view and generates view-consistent instance segmentation results across all input views without requiring manual annotations. The resulting multi-view segmentation can further be projected onto 3D point clouds for 3D segmentation. To improve training efficiency and segmentation performance, we introduce a prior distillation network for initialization. Overall, this study extends traditional 2D remote sensing scene perception into a 3D multi-view framework, enabling a more comprehensive understanding of target objects in complex environments with user interactivity.
Figures & tables
| Scene | Type | View num. | Object num. |
| sys #1 | simulation data | 100 | 2 |
| sys #2 | simulation data | 80 | 2 |
| sys #3 | simulation data | 85 | 2 |
| real #1 | real-world data | 100 | 3 |
| real #2 | real-world data | 100 | 2 |
| Method | sys #1 | sys #2 | sys #3 | real #1 | real #2 |
| SAM | 94.77 | 83.86 | 50.62 | 89.88 | 96.79 |
| Gaussian Grouping-1k | 96.19 | 89.92 | 71.13 | 92.24 | 96.76 |
| Gaussian Grouping-10k | 96.36 | 89.94 | 59.43 | 92.49 | 90.58 |
| Ours-1k | 96.52 | 90.10 | 76.62 | 93.18 | 96.81 |
| Ours-10k | 96.40 | 90.14 | 56.67 | 91.43 | 90.83 |
| Method | sys #1 | sys #2 | sys #3 | real #1 | real #2 |
| 1-Layer | 96.21 | 89.96 | 62.95 | 92.24 | 96.17 |
| 2-Layer | 96.22 | 90.01 | 73.55 | 92.10 | 93.89 |
| 3-Layer | 96.23 | 90.13 | 75.00 | 92.39 | 92.40 |
| 4-Layer | 96.05 | 64.79 | 48.41 | 92.44 | 91.85 |
| Distillation-100 | 96.10 | 89.47 | 76.91 | 91.25 | 96.01 |
| Distillation-500 | 96.30 | 89.87 | 78.59 | 92.72 | 95.14 |
Instance Segmentation
Instance segmentation identifies each object and assigns it a separate pixel-level mask in images or video.
See all 178 papers