VGGT-Segmentor: Geometry-Enhanced Cross-View Segmentation

Yulu Gao; Bohao Zhang; Zongheng Tang; Jitong Liao; Wenjun Wu; Si Liu

cs.CVApr 15, 2026

VGGT-Segmentor: Geometry-Enhanced Cross-View Segmentation

Authors: Yulu Gao, Bohao Zhang, Zongheng Tang, Jitong Liao, Wenjun Wu, Si Liu

Organizations: Hangzhou International Innovation Institute of Beihang University · Beihang University

Abstract

Instance-level object segmentation across disparate egocentric and exocentric views is a fundamental challenge in visual understanding, critical for applications in embodied AI and remote collaboration. This task is exceptionally difficult due to severe changes in scale, perspective, and occlusion, which destabilize direct pixel-level matching. While recent geometry-aware models like VGGT provide a strong foundation for feature alignment, we find they often fail at dense prediction tasks due to significant pixel-level projection drift, even when their internal object-level attention remains consistent. To bridge this gap, we introduce VGGT-Segmentor (VGGT-S), a framework that unifies robust geometric modeling with pixel-accurate semantic segmentation. VGGT-S leverages VGGT's powerful cross-view feature representation and introduces a novel Union Segmentation Head. This head operates in three stages: mask prompt fusion, point-guided prediction, and iterative mask refinement, effectively translating high-level feature alignment into a precise segmentation mask. Furthermore, we propose a single-image self-supervised training strategy that eliminates the need for paired annotations and enables strong generalization. On the Ego-Exo4D benchmark, VGGT-S sets a new state-of-the-art, achieving 67.7% and 68.0% average IoU for Ego to Exo and Exo to Ego tasks, respectively, significantly outperforming prior methods. Notably, our correspondence-free pretrained model surpasses most fully-supervised baselines, demonstrating the effectiveness and scalability of our approach. Code is publicly available at: https://github.com/buaa-colalab/VGGT-S.

Figures & tables

Figure 1: Visualizing VGGT Cross-View Correspondence. Left: source image. Middle: target image with the projections of source-sampled points obtained by directly applying VGGT, which exhibit the systematic drift and misalignment. Right: star markers in the source image with the corresponding attention map on the target image, illustrating VGGT’s instance-consistent object alignment across views.

Figure 2: (A) Overall Architecture of VGGT-S , which integrates the original VGGT encoder with our Union Segmentation Head . (B) Mask Prompt Fusion stage, which injects the source mask $M_{s}$ into source feature map $F_{s}$ and target feature map $F_{t}$ via convolutional fusion and a Bottleneck Fusion module. (C) Point-Guided Prediction stage, which uses point sets $(P_{s},P_{t})$ to guide target mask prediction through bidirectional interactions between point embeddings and image features.

Ego $\to$ Exo				Exo $\to$ Ego
Method	ZSL	Type	IoU $\uparrow$	Method	ZSL	Type	IoU $\uparrow$
XSegTx [ 21 ]	$\checkmark$	S	0.3	XSegTx [ 21 ]	$\checkmark$	S	1.3
SEEM [ 55 ]	$\checkmark$	S	1.1	SEEM [ 55 ]	$\checkmark$	S	4.1
XSegTx [ 21 ]	$\times$	S	6.2	XSegTx [ 21 ]	$\times$	S	30.2
CMX [ 52 ]	$\times$	S	6.8	CMX [ 52 ]	$\times$	S	12.0
PSALM [ 54 ]	$\checkmark$	S	7.9	PSALM [ 54 ]	$\checkmark$	S	9.6

Table 1: Comparison with prior methods on Ego-Exo4D dataset. “ZSL” denotes the zero-shot learning results. “Type S” denotes spatial-only modeling, while “Type ST” denotes spatio-temporal modeling. Our VGGT-S provides both supervised and zero-shot learning results.

Method	AP
MvMHAT [ 20 ]	63.8
DOMR [ 31 ]	71.1
VGGT-S (Ours)	80.7

Table 2: Comparison with prior methods on MvMHAT dataset.

Method	IoU $\uparrow$		Time (ms)
Method	Ego $\to$ Exo	Exo $\to$ Ego	Time (ms)
Plain Head	35.5	37.1	105.8
+ BF	50.2	52.3	107.4
+ PGP	62.2	63.5	153.2
+ MR	67.7	68.0	161.4

Table 3: Component analysis. “BF” denotes the Bottleneck Fusion module in Mask Prompt Fusion stage. “PGP” denotes the Point-Guided Prediction. “MR” denotes Mask Refinement stage.

Fusion Size	IoU $\uparrow$		Time (ms)
Fusion Size	Ego $\to$ Exo	Exo $\to$ Ego	Time (ms)
37×37	67.7	68.0	161.4
74×74	68.4	68.5	180.9
518×518	OOM	OOM	OOM

Table 4: Effect of Bottleneck Fusion resolution.

#Points	IoU $\uparrow$		Time (ms)
#Points	Ego $\to$ Exo	Exo $\to$ Ego	Time (ms)
1	61.5	63.4	160.1
5	67.7	68.0	161.4
9	68.3	68.5	162.9

Table 5: Effect of the number of points used in Point-Guided Prediction.

#Refine Iters	IoU $\uparrow$		Time (ms)
#Refine Iters	Ego $\to$ Exo	Exo $\to$ Ego	Time (ms)
0	62.2	63.5	153.2
1	66.3	67.5	157.7
2	67.7	68.0	161.4
3	67.9	68.4	165.3

Table 6: Effect of iterations in Mask Refinement.

Image Size	IoU $\uparrow$		Time (ms)
Image Size	Ego $\to$ Exo	Exo $\to$ Ego	Time (ms)
420×420	66.1	66.3	136.8
518×518	67.7	68.0	161.4
700×700	68.5	68.9	225.4

Table 7: Effect of input image size.

#Blocks	IoU $\uparrow$		Time (ms)
#Blocks	Ego $\to$ Exo	Exo $\to$ Ego	Time (ms)
1	65.1	65.5	158.2
2	67.7	68.0	161.4
3	68.4	68.7	165.4
6	68.8	69.3	176.8

Table 8: Effect of the number of decoder blocks.

Figure 3: Visualization of VGGT-S vs. DOMR. The first row shows the Ego $\to$ Exo task. DOMR incorrectly takes the chopping board as the predicted result, while VGGT-S correctly identifies the pot. The second row illustrates the Exo $\to$ Ego task. Two similar bottles are nearby. Due to a lack of geometric information, DOMR mistakenly confuses them, whereas VGGT-S continues to make accurate predictions.

Figure 4: Visualization of the Effect of the Union Segmentation Head. Although VGGT projects points to incorrect locations, our Union Segmentation Head adjusts the predicted mask to geometrically consistent positions. Zooming in provides better results.

Image Segmentation

Image segmentation assigns pixels or voxels to objects and regions, supporting measurement, scene understanding, and decisions across image domains.

See all 399 papers

SAM-V: Geometry-Aware Segment Anything for Multi-View Instance Segmentation
Sep 21, 2026Jiangshan Gong, Yuqun Wu, Qiqian Fu +4Multi-View Consistency Instance Segmentation
University of Illinois at Urbana-Champaign · Meta
Learning Cross-View Object Correspondence via Cycle-Consistent Mask Prediction
Feb 22, 2026Shannan Yan, Leqi Zheng, Keyu Lv +7Egocentric Video Understanding Multi-View Consistency
Tsinghua University · WeChat Vision, Tencent Inc. · USTC
Beyond 2D Matching: A Unified Single-Stage Framework for Geometry-Aware Cross-View Object Geo-Localization
Jun 29, 2026Liyao Wang, Ruipu Wu, Haojun Xu +3Cross-View Geo-Localization Camera Pose Estimation
Beihang University, Beijing, China · Meituan, Beijing, China

Sep 21, 2026J/K move · Enter open · S save

SAM-V: Geometry-Aware Segment Anything for Multi-View Instance Segmentation

Jiangshan Gong, Yuqun Wu, Qiqian Fu, Yao Xiao+3

University of Illinois at Urbana-Champaign · Meta

Multi-View Consistency Instance Segmentation Segment Anything Models 3D Instance Segmentation

Open paper AskPDF arXiv

Consistent multi-view object segmentation is critical for 3D perception and robotics, yet remains challenging under severe viewpoint and occlusion changes. Existing methods typically perform 3D instance segmentation on point clouds or rely on offline 2D mask-matching pipelines. However, 3D instance segmentation is limited by scarce 3D annotations, while offline 2D matching suffers from object identity ambiguity across frames. To leverage strong 2D and 3D priors jointly, we propose SAM-V (Geometry-Aware Segment Anything for Multi-View Instance Segmentation). Instead of combining the two priors through post-hoc matching, SAM-V directly integrates features from a feed-forward geometry model (VGGT) into a 2D segmentation foundation model (SAM), trained end-to-end for cross-view instance prediction. SAM-V introduces a prompt-fusion mechanism that enriches sparse SAM prompt tokens with view-specific camera tokens and local VGGT features, making the prompt representation both view-aware and spatially grounded, together with a mask decoder that attends to dense 2D and 3D features. By conditioning the mask decoding directly on multi-view geometry, SAM-V produces consistent multi-view segmentation of a prompted object in a single forward pass without offline mask matching or explicit 3D reconstruction. On the IGGT 3D tracking benchmark, where consistent instance identity across frames directly determines performance, SAM-V improves overall IoU by 5 points and frame-level recall by 12 points on the ScanNet++ split over the state-of-the-art multi-view instance segmentation baseline and leads on all metrics in the zero-shot ScanNet split. Our code and pretrained models are available at https://github.com/gong208/SAM-V.git.

Abstract

Image Segmentation

Explore similar work

SAM-V: Geometry-Aware Segment Anything for Multi-View Instance Segmentation

Learning Cross-View Object Correspondence via Cycle-Consistent Mask Prediction

Beyond 2D Matching: A Unified Single-Stage Framework for Geometry-Aware Cross-View Object Geo-Localization