Conventional 3D instance segmentation methods rely on labor-intensive 3D annotations for supervised training, which limits their scalability and generalization to novel objects. Recent approaches leverage multi-view 2D masks from the Segment Anything Model (SAM) to guide the merging of 3D geometric primitives, thereby enabling zero-shot 3D instance segmentation. However, these methods typically process each frame independently and rely solely on 2D metrics, such as SAM prediction scores, to produce segmentation maps. This design overlooks multi-view correlations and inherent 3D priors, leading to inconsistent 2D masks across views and ultimately fragmented 3D segmentation. In this paper, we propose MV3DIS, a coarse-to-fine framework for zero-shot 3D instance segmentation that explicitly incorporates 3D priors. Specifically, we introduce a 3D-guided mask matching strategy that uses coarse 3D segments as a common reference to match 2D masks across views and consolidates multi-view mask consistency via 3D coverage distributions. Guided by these view-consistent 2D masks, the coarse 3D segments are further refined into precise 3D instances. Additionally, we introduce a depth consistency weighting scheme that quantifies projection reliability to suppress ambiguities from inter-object occlusions, thereby improving the robustness of 3D-to-2D correspondence. Extensive experiments on the ScanNetV2, ScanNet200, ScanNet++, Replica, and Matterport3D datasets demonstrate the effectiveness of MV3DIS, which achieves superior performance over previous methods
Figures & tables
Figure 1: View inconsistency in existing methods versus view consistency with MV3DIS. Top-right: 3D instance segmentations from existing approaches and MV3DIS. Bottom blue and yellow strips: 2D segmentation maps from different views produced by existing approaches and MV3DIS. Existing methods yield view-inconsistent masks and fragmented 3D instances, whereas MV3DIS enforces view consistency and produces coherent 3D instance segmentations.
Figure 2: Method overview. Our method consists of two stages: (a) Coarse 3D Instance Segmentation and (b) 3D Instance Refinement. We first over-segment the point cloud into superpoints and use SAM to generate per-frame 2D masks, which are organized into a global mask collection. These masks are consolidated into coarse 2D segmentation maps that guide the merging of superpoints into coarse 3D segments. We then project the coarse 3D segments onto images and perform 3D-guided mask matching to associate consistent 2D masks from the collection. Finally, we refine the coarse 3D segments into coherent 3D instances using the consistent 2D segmentation maps.
Figure 3: Depth Consistency and Visibility Weights. During 3D-to-2D projection, near-threshold occluded points (red boxes) may be misidentified as visible on foreground objects (a, c). To mitigate such projection errors, our method assigns a continuous depth consistency weight to each point (b, d), which decreases with larger depth discrepancies. Additionally, the visibility weight quantifies an object’s observability, assigning higher weights to frames with more visible points (e) than to frames affected by occlusion (f).
Method
VFM
mAP
AP 50
AP 25
Closed-vocabulary
Mask3D [ 46 ]
None
65.7
83.1
91.0
Open-vocabulary
Felzenszwalb [ 10 ]
None
5.0
12.7
38.9
SAM-graph [ 12 ]
SAM
24.1
40.3
65.9
SAM3D [ 61 ]
SAM
20.2
34.0
53.3
Table 1: Class-agnostic 3D instance segmentation on ScanNetV2. Best and second-best results are bold and underlined, respectively. VFM represents the 2D vision foundation model, and Sem-SAM represents Semantic-SAM [ 29 ] .
Method
VFM
mAP
AP 50
AP 25
Closed-vocabulary
Mask3D [ 46 ]
None
40.4
51.2
57.1
Open-vocabulary
Felzenszwalb [ 10 ]
None
4.8
9.8
27.5
SAM-graph [ 12 ]
SAM
22.1
41.7
62.8
SAM3D [ 61 ]
SAM
20.0
33.8
52.7
Table 2: Class-agnostic 3D instance segmentation on ScanNet200. Best and second-best results are bold and underlined, respectively. VFM represents the 2D vision foundation model, Sem-SAM represents Semantic-SAM, and CropFormer refers to [ 41 ] .
Method
VFM
mAP
AP 50
AP 25
Closed-vocabulary
Mask3D [ 46 ]
None
9.9
17.3
25.8
Open-vocabulary
Felzenszwalb [ 10 ]
None
4.1
9.2
25.3
SAM-graph [ 12 ]
SAM
12.9
25.3
43.6
SAM3D [ 61 ]
SAM
7.2
14.2
29.4
Table 3: Class-agnostic 3D instance segmentation on ScanNet++. Best and second-best results are bold and underlined, respectively. VFM represents the 2D vision foundation model, and SAM-HQ refers to [ 22 ] . Note that Mask3D is trained on the ScanNetV2 dataset.
Method
mAP
AP 50
AP 25
Head (AP)
Common (AP)
Tail (AP)
Sup. mask + Single-label
OpenMask3D [ 49 ]
15.4
19.9
23.1
17.1
14.1
14.9
OV. mask + Single-label
OVIR-3D [ 36 ]
9.3
18.7
25.0
9.8
9.4
8.5
SAM3D [ 61 ]
9.8
15.2
20.7
9.2
8.3
12.3
SAI3D [ 65 ]
12.7
18.8
24.1
12.1
10.4
16.2
Table 4: Semantic instance segmentation on ScanNet200. Head (AP), Common (AP), and Tail (AP) represent AP scores evaluated on frequent, common, and rare classes, respectively. “Sup.” and “OV.” represent “Supervised” and “Open-vocabulary”. Best results are in bold, and the second-best results are underlined.
Figure 4: Qualitative results of our method compared to SAM3D [ 61 ] and SAI3D [ 65 ] on the ScanNetV2 and ScanNet++ datasets.
RR
3DG-MM
DCW
mAP
AP 50
AP 25
33.3
51.3
70.1
✓
35.5
53.2
72.1
✓
✓
37.0
57.6
74.2
✓
✓
✓
38.5
60.2
76.2
Table 5: Ablation study of our main components. RR indicates region refinement, 3DG-MM indicates 3D-guided mask matching, and DCW indicates depth consistency weight.
Table 10
Figure 5: Correlation of performance with the number of images.
Consistent multi-view object segmentation is critical for 3D perception and robotics, yet remains challenging under severe viewpoint and occlusion changes. Existing methods typically perform 3D instance segmentation on point clouds or rely on offline 2D mask-matching pipelines. However, 3D instance segmentation is limited by scarce 3D annotations, while offline 2D matching suffers from object identity ambiguity across frames. To leverage strong 2D and 3D priors jointly, we propose SAM-V (Geometry-Aware Segment Anything for Multi-View Instance Segmentation). Instead of combining the two priors through post-hoc matching, SAM-V directly integrates features from a feed-forward geometry model (VGGT) into a 2D segmentation foundation model (SAM), trained end-to-end for cross-view instance prediction. SAM-V introduces a prompt-fusion mechanism that enriches sparse SAM prompt tokens with view-specific camera tokens and local VGGT features, making the prompt representation both view-aware and spatially grounded, together with a mask decoder that attends to dense 2D and 3D features. By conditioning the mask decoding directly on multi-view geometry, SAM-V produces consistent multi-view segmentation of a prompted object in a single forward pass without offline mask matching or explicit 3D reconstruction. On the IGGT 3D tracking benchmark, where consistent instance identity across frames directly determines performance, SAM-V improves overall IoU by 5 points and frame-level recall by 12 points on the ScanNet++ split over the state-of-the-art multi-view instance segmentation baseline and leads on all metrics in the zero-shot ScanNet split. Our code and pretrained models are available at https://github.com/gong208/SAM-V.git.
Class-agnostic 3D instance segmentation is critical for robotic systems operating in unknown environments, enabling perception of previously unseen objects for reliable manipulation and navigation. Existing approaches typically project per-frame 2D instance masks into 3D and merge them, which often breaks object identities across time and yields fragmented 3D instances. We introduce Cross-Dimensional Class-Agnostic 3D Instance Segmentation (CDIS), a zero-shot framework that explicitly tracks 2D instance masks across frames and associates them with 3D superpoints, creating a feedback loop between 2D and 3D. This cross-dimensional reasoning links temporally stable 2D tracks with spatially coherent 3D regions, producing globally consistent 3D instance labels without any 3D-specific training. Experiments on benchmark datasets demonstrate that CDIS achieves higher accuracy and consistency than state-of-the-art zero-shot methods, while remaining efficient and scalable to diverse real-world environments.
Juno Kim, Hye-Jung Yoon, Yesol Park +1
Interdisciplinary Program in AI, Seoul National University
3D instance segmentation methods typically rely on high-quality point clouds or posed RGB-D scans, requiring complex multi-stage processing pipelines, and are highly sensitive to reconstruction noise. While recent feed-forward transformers have revolutionized multi-view 3D reconstruction, they remain decoupled from high-level semantic understanding. In this work, we present SegVGGT, a unified end-to-end framework that simultaneously performs feed-forward 3D reconstruction and instance segmentation directly from multi-view RGB images. By introducing object queries that interact with multi-level geometric features, our method deeply integrates instance identification into the visual geometry grounded transformer. To address the severe attention dispersion problem caused by the massive number of global image tokens, we propose the Frame-level Attention Distribution Alignment (FADA) strategy. FADA explicitly guides object queries to attend to instance-relevant frames during training, providing structured supervision without extra inference overhead. Extensive experiments demonstrate that SegVGGT achieves the state-of-the-art performance on ScanNetv2 and ScanNet200, outperforming both recent joint models and RGB-D-based approaches, while exhibiting strong generalization capabilities on ScanNet++.
Jinyuan Qu, Hongyang Li, Lei Zhang
Tsinghua University · International Digital Economy Academy (IDEA) · South China University of Technology