Conventional 3D instance segmentation methods rely on labor-intensive 3D annotations for supervised training, which limits their scalability and generalization to novel objects. Recent approaches leverage multi-view 2D masks from the Segment Anything Model (SAM) to guide the merging of 3D geometric primitives, thereby enabling zero-shot 3D instance segmentation. However, these methods typically process each frame independently and rely solely on 2D metrics, such as SAM prediction scores, to produce segmentation maps. This design overlooks multi-view correlations and inherent 3D priors, leading to inconsistent 2D masks across views and ultimately fragmented 3D segmentation. In this paper, we propose MV3DIS, a coarse-to-fine framework for zero-shot 3D instance segmentation that explicitly incorporates 3D priors. Specifically, we introduce a 3D-guided mask matching strategy that uses coarse 3D segments as a common reference to match 2D masks across views and consolidates multi-view mask consistency via 3D coverage distributions. Guided by these view-consistent 2D masks, the coarse 3D segments are further refined into precise 3D instances. Additionally, we introduce a depth consistency weighting scheme that quantifies projection reliability to suppress ambiguities from inter-object occlusions, thereby improving the robustness of 3D-to-2D correspondence. Extensive experiments on the ScanNetV2, ScanNet200, ScanNet++, Replica, and Matterport3D datasets demonstrate the effectiveness of MV3DIS, which achieves superior performance over previous methods
Figures & tables
Figure 1: View inconsistency in existing methods versus view consistency with MV3DIS. Top-right: 3D instance segmentations from existing approaches and MV3DIS. Bottom blue and yellow strips: 2D segmentation maps from different views produced by existing approaches and MV3DIS. Existing methods yield view-inconsistent masks and fragmented 3D instances, whereas MV3DIS enforces view consistency and produces coherent 3D instance segmentations.
Figure 2: Method overview. Our method consists of two stages: (a) Coarse 3D Instance Segmentation and (b) 3D Instance Refinement. We first over-segment the point cloud into superpoints and use SAM to generate per-frame 2D masks, which are organized into a global mask collection. These masks are consolidated into coarse 2D segmentation maps that guide the merging of superpoints into coarse 3D segments. We then project the coarse 3D segments onto images and perform 3D-guided mask matching to associate consistent 2D masks from the collection. Finally, we refine the coarse 3D segments into coherent 3D instances using the consistent 2D segmentation maps.
Figure 3: Depth Consistency and Visibility Weights. During 3D-to-2D projection, near-threshold occluded points (red boxes) may be misidentified as visible on foreground objects (a, c). To mitigate such projection errors, our method assigns a continuous depth consistency weight to each point (b, d), which decreases with larger depth discrepancies. Additionally, the visibility weight quantifies an object’s observability, assigning higher weights to frames with more visible points (e) than to frames affected by occlusion (f).
Method
VFM
mAP
AP 50
AP 25
Closed-vocabulary
Mask3D [ 46 ]
None
65.7
83.1
91.0
Open-vocabulary
Felzenszwalb [ 10 ]
None
5.0
12.7
38.9
SAM-graph [ 12 ]
SAM
24.1
40.3
65.9
SAM3D [ 61 ]
SAM
20.2
34.0
53.3
Table 1: Class-agnostic 3D instance segmentation on ScanNetV2. Best and second-best results are bold and underlined, respectively. VFM represents the 2D vision foundation model, and Sem-SAM represents Semantic-SAM [ 29 ] .
Method
VFM
mAP
AP 50
AP 25
Closed-vocabulary
Mask3D [ 46 ]
None
40.4
51.2
57.1
Open-vocabulary
Felzenszwalb [ 10 ]
None
4.8
9.8
27.5
SAM-graph [ 12 ]
SAM
22.1
41.7
62.8
SAM3D [ 61 ]
SAM
20.0
33.8
52.7
Table 2: Class-agnostic 3D instance segmentation on ScanNet200. Best and second-best results are bold and underlined, respectively. VFM represents the 2D vision foundation model, Sem-SAM represents Semantic-SAM, and CropFormer refers to [ 41 ] .
Method
VFM
mAP
AP 50
AP 25
Closed-vocabulary
Mask3D [ 46 ]
None
9.9
17.3
25.8
Open-vocabulary
Felzenszwalb [ 10 ]
None
4.1
9.2
25.3
SAM-graph [ 12 ]
SAM
12.9
25.3
43.6
SAM3D [ 61 ]
SAM
7.2
14.2
29.4
Table 3: Class-agnostic 3D instance segmentation on ScanNet++. Best and second-best results are bold and underlined, respectively. VFM represents the 2D vision foundation model, and SAM-HQ refers to [ 22 ] . Note that Mask3D is trained on the ScanNetV2 dataset.
Method
mAP
AP 50
AP 25
Head (AP)
Common (AP)
Tail (AP)
Sup. mask + Single-label
OpenMask3D [ 49 ]
15.4
19.9
23.1
17.1
14.1
14.9
OV. mask + Single-label
OVIR-3D [ 36 ]
9.3
18.7
25.0
9.8
9.4
8.5
SAM3D [ 61 ]
9.8
15.2
20.7
9.2
8.3
12.3
SAI3D [ 65 ]
12.7
18.8
24.1
12.1
10.4
16.2
Table 4: Semantic instance segmentation on ScanNet200. Head (AP), Common (AP), and Tail (AP) represent AP scores evaluated on frequent, common, and rare classes, respectively. “Sup.” and “OV.” represent “Supervised” and “Open-vocabulary”. Best results are in bold, and the second-best results are underlined.
Figure 4: Qualitative results of our method compared to SAM3D [ 61 ] and SAI3D [ 65 ] on the ScanNetV2 and ScanNet++ datasets.
RR
3DG-MM
DCW
mAP
AP 50
AP 25
33.3
51.3
70.1
✓
35.5
53.2
72.1
✓
✓
37.0
57.6
74.2
✓
✓
✓
38.5
60.2
76.2
Table 5: Ablation study of our main components. RR indicates region refinement, 3DG-MM indicates 3D-guided mask matching, and DCW indicates depth consistency weight.
Table 10
Figure 5: Correlation of performance with the number of images.