3D instance segmentation methods typically rely on high-quality point clouds or posed RGB-D scans, requiring complex multi-stage processing pipelines, and are highly sensitive to reconstruction noise. While recent feed-forward transformers have revolutionized multi-view 3D reconstruction, they remain decoupled from high-level semantic understanding. In this work, we present SegVGGT, a unified end-to-end framework that simultaneously performs feed-forward 3D reconstruction and instance segmentation directly from multi-view RGB images. By introducing object queries that interact with multi-level geometric features, our method deeply integrates instance identification into the visual geometry grounded transformer. To address the severe attention dispersion problem caused by the massive number of global image tokens, we propose the Frame-level Attention Distribution Alignment (FADA) strategy. FADA explicitly guides object queries to attend to instance-relevant frames during training, providing structured supervision without extra inference overhead. Extensive experiments demonstrate that SegVGGT achieves the state-of-the-art performance on ScanNetv2 and ScanNet200, outperforming both recent joint models and RGB-D-based approaches, while exhibiting strong generalization capabilities on ScanNet++.
Figures & tables
Figure 1: Comparison of different 3D instance segmentation pipelines. (a) Segmentation methods that rely on high-quality point clouds require complex and time-consuming processing, limiting their applicability. (b) Feed-forward 3D reconstruction improves efficiency, but leads to severely degraded segmentation performance. (c) Our proposed SegVGGT performs 3D reconstruction and instance segmentation simultaneously in a single forward pass, achieving superior segmentation quality. (d) 3D instance segmentation performance on the ScanNet200 validation set.
Figure 2: Overall architecture of SegVGGT. SegVGGT takes multi-view unposed RGB images as input and outputs the camera parameters, depth maps, and instance masks in a single forward pass. In the visual geometry grounded transformer, image tokens and object queries are progressively refined to capture both multi-view geometry and instance semantics. The green arrows indicate the components related to the 3D instance segmentation. During training, the Frame-level Attention Distribution Alignment (FADA) module regularizes the cross-attention weights via Jensen-Shannon Divergence loss Ljs , effectively mitigating the attention dispersion problem.
ScanNetv2
ScanNet200
Method
Input Modality
mAP
mAP 50
mAP 25
mAP
mAP 50
mAP 25
Mask3D [ 46 ]
P
55.2
73.7
85.3
27.4
37.0
42.3
QueryFormer [ 32 ]
P
56.5
74.2
83.3
28.1
37.1
43.4
MAFT [ 25 ]
P
59.9
76.5
-
29.2
38.2
43.3
SPFormer [ 47 ]
P
56.3
73.9
82.9
-
-
-
SGIFormer-L [ 60 ]
P
61.0
81.2
88.9
29.2
39.4
44.2
Table 1: Comparison of SegVGGT with point cloud-based methods. Input modalities are denoted as follows: P (point cloud from the benchmark), I (RGB images), D (depth maps), and C (camera parameters). † indicates that the method first performs 3D reconstruction from the input images with VGGT [ 52 ] , and then performs 3D instance segmentation on the reconstructed point cloud.
ScanNetv2
ScanNet200
ScanNet++
Method
mAP
mAP 50
mAP 25
mAP
mAP 50
mAP 25
mAP
mAP 50
mAP 25
PanSt3R [ 66 ]
26.9
52.6
63.9
20.8
41.4
59.0
9.2
22.4
36.2
IGGT [ 28 ]
29.2
43.0
55.1
28.2
42.4
55.3
11.5
21.3
36.0
Ours
62.9
87.4
97.4
53.7
80.2
89.9
13.3
33.9
56.4
Table 2: Comparison of SegVGGT with RGB image-based methods.
Figure 3: Visualization of the FADA module’s effect. We visualize the similarity map between the query corresponding to the TV instance and the image tokens in the last cross-attention layer. Without the FADA module, the query attention is dispersed across the entire scene. After introducing FADA, the query focuses much more accurately on the TV, indicating that FADA improves the model’s ability to attend to target instances and thereby enhances instance segmentation performance.
Table 6
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Input Frames
Model Inference (s)
Postprocess (s)
Total Time (s)
Memory (GB)
Aggr.
Heads
Total
10
0.42
0.06
0.48
0.69
1.17
7.04
20
0.90
0.12
1.02
1.40
2.42
8.18
50
3.01
0.28
3.29
3.35
6.64
15.49
100
8.69
0.56
9.25
6.65
15.90
27.67
200
28.30
1.10
29.40
14.14
43.54
52.04
Appendix
Table 5: Scalability and efficiency analysis of SegVGGT. We report the runtime breakdown, including aggregator, decoding heads, post-processing, total runtime, and peak reserved memory under different numbers of input frames.
Method
Model Inference (s)
Postprocess (s)
Total Time (s)
Memory (GB)
PanSt3R [ 66 ]
41.80
340.16
381.96
14.32
PanSt3R † [ 66 ]
41.80
17.83
59.63
14.32
IGGT [ 28 ]
7.52
1113.64
1121.16
35.21
Ours
9.25
6.65
15.90
27.67
Appendix
Table 6: Efficiency comparison under 100 input frames. We benchmark the runtime and peak reserved memory against existing joint models. † denotes PanSt3R executed without its QUBO post-processing method.
Optimization Strategy
mAP
mAP 50
mAP 25
Freeze Frame & Global Attention
23.4
35.1
42.0
Joint Optimization (Ours)
31.9
45.7
53.7
Appendix
Table 7: Ablation on Joint Optimization. We evaluate the impact of jointly fine-tuning the pre-existing frame and global attention modules versus freezing them and only training the newly introduced instance reasoning parameters.
Method
mAP
mAP 50
mAP 25
VGGT (FT) + Mask Decoder with 6 Layers
22.4
33.7
40.1
VGGT (FT) + Mask Decoder with 12 Layers
23.4
35.1
41.4
VGGT (FT) + Mask Decoder with 18 Layers
16.7
25.7
31.3
SegVGGT (Our Unified Architecture)
31.9
45.7
53.7
Appendix
Table 8: Ablation on Unified Architecture. We compare our unified architecture with a decoupled two-stage alternative that performs 3D instance segmentation using a separate mask decoder based on Mask2Former [ 6 ] . (FT) denotes that the pre-trained VGGT backbone is jointly fine-tuned using LoRA [ 14 ] to ensure a fair comparison.
Figure 4: Visualization of the FADA module’s effect. We visualize the similarity maps between the query corresponding to the TV instance and the image tokens in the 15th and 23th cross-attention layers. Without the FADA module, the query attention is dispersed across the entire scene. After introducing FADA, the query focuses much more accurately on the TV, indicating that FADA improves the model’s ability to attend to target instances and thereby enhances instance segmentation performance.
Method
Abs Rel ↓
δ<1.25↑
VGGT (Pre-trained) [ 52 ]
0.0297
0.989
Ours (Geometry fine-tuning only)
0.0311
0.986
Ours (Joint fine-tuning with instance segmentation)
0.0307
0.987
Appendix
Table 9: Geometric Performance on ScanNet. We evaluate multi-view geometric consistency using dense depth estimation. A single global scale is estimated for each input sequence to align predictions with ground-truth depth.
Figure 5: Qualitative 2D results on ScanNet200. Given multi-view RGB inputs, SegVGGT simultaneously predicts dense depth maps, cross-view consistent instance-level feature maps, and 2D instance masks. Spatially adjacent objects of the same class are clearly separated both in the feature space and in the resulting instance masks.
Figure 6: Qualitative 3D results on ScanNet200. By directly unprojecting the 2D predictions, our method yields high-quality 3D scene reconstructions and precise instance segmentations. Note that instance colors are randomly assigned for visualization purposes and do not necessarily correspond between the prediction and ground truth.
Figure 7: Qualitative comparison with IGGT on ScanNet++. The blue and green bounding boxes highlight spatially adjacent instances of the same semantic class. While the clustering-based IGGT struggles to separate these closely located objects, our SegVGGT accurately disambiguates and segments them into distinct instances.
While recent feed-forward 3D reconstruction models provide a strong geometric foundation for scene understanding, extending them to 3D instance segmentation typically relies on a disjointed "lift-and-cluster" paradigm. Grouping dense pixel-wise embeddings via non-differentiable clustering scales poorly with the number of views and disconnects representation learning from the final segmentation objective. In this paper, we present a Feed-forward Anchored Scene Transformer for 3D Instance Segmentation (FAST3DIS), an end-to-end approach that effectively bypasses post-hoc clustering. We introduce a 3D-anchored, query-based Transformer architecture built upon a foundational depth backbone, adapted efficiently to learn instance-specific semantics while retaining its zero-shot geometric priors. We formulate a learned 3D anchor generator coupled with an anchor-sampling cross-attention mechanism for view-consistent 3D instance segmentation. By projecting 3D object queries directly into multi-view feature maps, our method samples context efficiently. Furthermore, we introduce a dual-level regularization strategy, that couples multi-view contrastive learning with a dynamically scheduled spatial overlap penalty to explicitly prevent query collisions and ensure precise instance boundaries. Experiments on complex indoor 3D datasets demonstrate that our approach achieves competitive segmentation accuracy with significantly improved memory scalability and inference speed over state-of-the-art clustering-based methods.
Consistent multi-view object segmentation is critical for 3D perception and robotics, yet remains challenging under severe viewpoint and occlusion changes. Existing methods typically perform 3D instance segmentation on point clouds or rely on offline 2D mask-matching pipelines. However, 3D instance segmentation is limited by scarce 3D annotations, while offline 2D matching suffers from object identity ambiguity across frames. To leverage strong 2D and 3D priors jointly, we propose SAM-V (Geometry-Aware Segment Anything for Multi-View Instance Segmentation). Instead of combining the two priors through post-hoc matching, SAM-V directly integrates features from a feed-forward geometry model (VGGT) into a 2D segmentation foundation model (SAM), trained end-to-end for cross-view instance prediction. SAM-V introduces a prompt-fusion mechanism that enriches sparse SAM prompt tokens with view-specific camera tokens and local VGGT features, making the prompt representation both view-aware and spatially grounded, together with a mask decoder that attends to dense 2D and 3D features. By conditioning the mask decoding directly on multi-view geometry, SAM-V produces consistent multi-view segmentation of a prompted object in a single forward pass without offline mask matching or explicit 3D reconstruction. On the IGGT 3D tracking benchmark, where consistent instance identity across frames directly determines performance, SAM-V improves overall IoU by 5 points and frame-level recall by 12 points on the ScanNet++ split over the state-of-the-art multi-view instance segmentation baseline and leads on all metrics in the zero-shot ScanNet split. Our code and pretrained models are available at https://github.com/gong208/SAM-V.git.
3D instance segmentation requires representations that are both geometrically structured and semantically discriminative. Pure 3D encoders model spatial structure well but often lack strong appearance semantics, while directly lifting 2D priors into 3D by concatenation injects rich semantics without ensuring that they align well with the underlying 3D structure. In this paper, we address this problem by formulating 2D-to-3D semantic transfer as a masked semantic reconstruction task. Specifically, we mask a subset of background lifted 2D priors and train the 3D encoder to recover the missing semantic priors, providing explicit supervision for distilling rich 2D semantics into the 3D backbone. We further propose geometry-modulated semantic injection, which uses raw 3D geometry to modulate lifted 2D priors before sparse encoding, so that geometry explicitly controls semantic injection and alleviates the distribution gap between 2D semantic features and 3D geometric features. Experiments on two baseline frameworks demonstrate that the proposed method achieves state-of-the-art results in both full-data and few-shot settings on ScanNetV2 and ScanNet200. Code is publicly available at https://github.com/yharoldsu0627/Geometry_Guided_Semantic_Reconstruction
Yuanhao Su, Shaofeng Zhang
University of Science and Technology of China · Fuzhou University