cs.CVMar 20, 2026
SaveSegVGGT: Joint 3D Reconstruction and Instance Segmentation from Multi-View Images
Organizations: Tsinghua University · International Digital Economy Academy (IDEA) · South China University of Technology
Abstract
3D instance segmentation methods typically rely on high-quality point clouds or posed RGB-D scans, requiring complex multi-stage processing pipelines, and are highly sensitive to reconstruction noise. While recent feed-forward transformers have revolutionized multi-view 3D reconstruction, they remain decoupled from high-level semantic understanding. In this work, we present SegVGGT, a unified end-to-end framework that simultaneously performs feed-forward 3D reconstruction and instance segmentation directly from multi-view RGB images. By introducing object queries that interact with multi-level geometric features, our method deeply integrates instance identification into the visual geometry grounded transformer. To address the severe attention dispersion problem caused by the massive number of global image tokens, we propose the Frame-level Attention Distribution Alignment (FADA) strategy. FADA explicitly guides object queries to attend to instance-relevant frames during training, providing structured supervision without extra inference overhead. Extensive experiments demonstrate that SegVGGT achieves the state-of-the-art performance on ScanNetv2 and ScanNet200, outperforming both recent joint models and RGB-D-based approaches, while exhibiting strong generalization capabilities on ScanNet++.
Figures & tables
Figure 1: Comparison of different 3D instance segmentation pipelines. (a) Segmentation methods that rely on high-quality point clouds require complex and time-consuming processing, limiting their applicability. (b) Feed-forward 3D reconstruction improves efficiency, but leads to severely degraded segmentation performance. (c) Our proposed SegVGGT performs 3D reconstruction and instance segmentation simultaneously in a single forward pass, achieving superior segmentation quality. (d) 3D instance segmentation performance on the ScanNet200 validation set.
Figure 2: Overall architecture of SegVGGT. SegVGGT takes multi-view unposed RGB images as input and outputs the camera parameters, depth maps, and instance masks in a single forward pass. In the visual geometry grounded transformer, image tokens and object queries are progressively refined to capture both multi-view geometry and instance semantics. The green arrows indicate the components related to the 3D instance segmentation. During training, the Frame-level Attention Distribution Alignment (FADA) module regularizes the cross-attention weights via Jensen-Shannon Divergence loss , effectively mitigating the attention dispersion problem.
| ScanNetv2 | ScanNet200 | ||||||
| Method | Input Modality | mAP | mAP 50 | mAP 25 | mAP | mAP 50 | mAP 25 |
| Mask3D [ 46 ] | P | 55.2 | 73.7 | 85.3 | 27.4 | 37.0 | 42.3 |
| QueryFormer [ 32 ] | P | 56.5 | 74.2 | 83.3 | 28.1 | 37.1 | 43.4 |
| MAFT [ 25 ] | P | 59.9 | 76.5 | - | 29.2 | 38.2 | 43.3 |
| SPFormer [ 47 ] | P | 56.3 | 73.9 | 82.9 | - | - | - |
| SGIFormer-L [ 60 ] | P | 61.0 | 81.2 | 88.9 | 29.2 | 39.4 | 44.2 |
Table 1: Comparison of SegVGGT with point cloud-based methods. Input modalities are denoted as follows: P (point cloud from the benchmark), I (RGB images), D (depth maps), and C (camera parameters). † indicates that the method first performs 3D reconstruction from the input images with VGGT [ 52 ] , and then performs 3D instance segmentation on the reconstructed point cloud.
| ScanNetv2 | ScanNet200 | ScanNet++ | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | mAP | mAP 50 | mAP 25 | mAP | mAP 50 | mAP 25 | mAP | mAP 50 | mAP 25 |
| PanSt3R [ 66 ] | 26.9 | 52.6 | 63.9 | 20.8 | 41.4 | 59.0 | 9.2 | 22.4 | 36.2 |
| IGGT [ 28 ] | 29.2 | 43.0 | 55.1 | 28.2 | 42.4 | 55.3 | 11.5 | 21.3 | 36.0 |
| Ours | 62.9 | 87.4 | 97.4 | 53.7 | 80.2 | 89.9 | 13.3 | 33.9 | 56.4 |
Table 2: Comparison of SegVGGT with RGB image-based methods.
Figure 3: Visualization of the FADA module’s effect. We visualize the similarity map between the query corresponding to the TV instance and the image tokens in the last cross-attention layer. Without the FADA module, the query attention is dispersed across the entire scene. After introducing FADA, the query focuses much more accurately on the TV, indicating that FADA improves the model’s ability to attend to target instances and thereby enhances instance segmentation performance.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Input Frames | Model Inference (s) | Postprocess (s) | Total Time (s) | Memory (GB) | ||
|---|---|---|---|---|---|---|
| Aggr. | Heads | Total | ||||
| 10 | 0.42 | 0.06 | 0.48 | 0.69 | 1.17 | 7.04 |
| 20 | 0.90 | 0.12 | 1.02 | 1.40 | 2.42 | 8.18 |
| 50 | 3.01 | 0.28 | 3.29 | 3.35 | 6.64 | 15.49 |
| 100 | 8.69 | 0.56 | 9.25 | 6.65 | 15.90 | 27.67 |
| 200 | 28.30 | 1.10 | 29.40 | 14.14 | 43.54 | 52.04 |
Table 5: Scalability and efficiency analysis of SegVGGT. We report the runtime breakdown, including aggregator, decoding heads, post-processing, total runtime, and peak reserved memory under different numbers of input frames.
| Method | Model Inference (s) | Postprocess (s) | Total Time (s) | Memory (GB) |
|---|---|---|---|---|
| PanSt3R [ 66 ] | 41.80 | 340.16 | 381.96 | 14.32 |
| PanSt3R † [ 66 ] | 41.80 | 17.83 | 59.63 | 14.32 |
| IGGT [ 28 ] | 7.52 | 1113.64 | 1121.16 | 35.21 |
| Ours | 9.25 | 6.65 | 15.90 | 27.67 |
Table 6: Efficiency comparison under 100 input frames. We benchmark the runtime and peak reserved memory against existing joint models. † denotes PanSt3R executed without its QUBO post-processing method.
| Optimization Strategy | mAP | mAP 50 | mAP 25 |
|---|---|---|---|
| Freeze Frame & Global Attention | 23.4 | 35.1 | 42.0 |
| Joint Optimization (Ours) | 31.9 | 45.7 | 53.7 |
Table 7: Ablation on Joint Optimization. We evaluate the impact of jointly fine-tuning the pre-existing frame and global attention modules versus freezing them and only training the newly introduced instance reasoning parameters.
| Method | mAP | mAP 50 | mAP 25 |
|---|---|---|---|
| VGGT (FT) + Mask Decoder with 6 Layers | 22.4 | 33.7 | 40.1 |
| VGGT (FT) + Mask Decoder with 12 Layers | 23.4 | 35.1 | 41.4 |
| VGGT (FT) + Mask Decoder with 18 Layers | 16.7 | 25.7 | 31.3 |
| SegVGGT (Our Unified Architecture) | 31.9 | 45.7 | 53.7 |
Table 8: Ablation on Unified Architecture. We compare our unified architecture with a decoupled two-stage alternative that performs 3D instance segmentation using a separate mask decoder based on Mask2Former [ 6 ] . (FT) denotes that the pre-trained VGGT backbone is jointly fine-tuned using LoRA [ 14 ] to ensure a fair comparison.
Figure 4: Visualization of the FADA module’s effect. We visualize the similarity maps between the query corresponding to the TV instance and the image tokens in the 15th and 23th cross-attention layers. Without the FADA module, the query attention is dispersed across the entire scene. After introducing FADA, the query focuses much more accurately on the TV, indicating that FADA improves the model’s ability to attend to target instances and thereby enhances instance segmentation performance.
| Method | Abs Rel | |
|---|---|---|
| VGGT (Pre-trained) [ 52 ] | 0.0297 | 0.989 |
| Ours (Geometry fine-tuning only) | 0.0311 | 0.986 |
| Ours (Joint fine-tuning with instance segmentation) | 0.0307 | 0.987 |
Table 9: Geometric Performance on ScanNet. We evaluate multi-view geometric consistency using dense depth estimation. A single global scale is estimated for each input sequence to align predictions with ground-truth depth.
Figure 5: Qualitative 2D results on ScanNet200. Given multi-view RGB inputs, SegVGGT simultaneously predicts dense depth maps, cross-view consistent instance-level feature maps, and 2D instance masks. Spatially adjacent objects of the same class are clearly separated both in the feature space and in the resulting instance masks.
Figure 6: Qualitative 3D results on ScanNet200. By directly unprojecting the 2D predictions, our method yields high-quality 3D scene reconstructions and precise instance segmentations. Note that instance colors are randomly assigned for visualization purposes and do not necessarily correspond between the prediction and ground truth.
Figure 7: Qualitative comparison with IGGT on ScanNet++. The blue and green bounding boxes highlight spatially adjacent instances of the same semantic class. While the clustering-based IGGT struggles to separate these closely located objects, our SegVGGT accurately disambiguates and segments them into distinct instances.