S4VY: Segment Anything in Feed-Forward 4D Visual Geometry

Jingdong Zhang; Xin Li; Jan Kautz; Wenping Wang; Chris Choy

cs.CVSep 29, 2026

S4VY: Segment Anything in Feed-Forward 4D Visual Geometry

Authors: Jingdong Zhang, Xin Li, Jan Kautz, Wenping Wang, Chris Choy

Organizations: Texas A&M University, College Station · NVIDIA

Abstract

Accurate instance segmentation in dynamic scenes is important for downstream applications such as robotics and autonomous driving. Existing Segment Anything models operate primarily on 2D image or video masks and preserve identity through sequential memory, while promptable 4D instance segmentation built upon feed-forward visual geometry remains underexplored. We introduce S4VY, a Segment Anything model built on feed-forward 4D visual geometry. From a set of RGB observations, S4VY transforms shared visual-geometric features into an exhaustive set of class-agnostic 4D instance masks through a space-time query decoder, with each persistent object query binding one entity across all observations. This representation supports prompt-independent segmentation as well as point- and box- conditioned selection, without requiring a seed mask or temporal ordering. We further develop an agentic harness for natural-language grounding in the large observation space of a 4D scene. Active tree search identifies relevant frames without scanning every fixed window; a dual-stream grounder combines fine-grained VLM visual priors with geometry-consistent instance features through complementary bounding-box prediction and object-query matching; and an independent critic selects the final 4D instance mask from their predictions. Extensive experiments demonstrate state-of-the-art 4D instance segmentation and strong language-guided grounding performance under a unified evaluation spanning static and dynamic scenes.

Figures & tables

Figure 1 : Overview of S4VY. Left: Given a set of RGB observations, S4VY predicts a consistent 4D mask for every instance in the scene. Right: Any instance can be grounded via either a natural-language prompt or an image-level point/box prompt.

Figure 2 : Architecture of S4VY. Top: The backbone encodes RGB observations into register and patch tokens. The Space-Time Query Decoder applies learnable object queries to the patch tokens to produce exhaustive instance masks across observations, while the Prompt Encoder maps image-level prompts to a promptable query. Bottom: For a language expression, Active Tree Search selects observation windows for the dual-stream Grounder, which predicts object-query matches and bounding boxes. The two predictions are decoded into mask candidates and compared by the Critic.

Figure 3 : (a) Prompt Encoder: Point or box prompts sample dense features derived from the contextualized patch tokens and are aggregated into a promptable object query; $T^{\prime}$ is the number of observations containing prompts. (b) Space-Time Query Decoder: The promptable and learnable object queries decode persistent instance masks over the complete observation set. The attention mask blocks learnable queries from reading the prompt query while retaining the reverse direction.

1:	$\mathcal{E}\leftarrow\textsc{Search}([1,T],0)$ .
2:	$(\hat{\mathbf{M}}^{q},\hat{\mathbf{M}}^{p})\leftarrow$ aggregate and decode both prediction streams in $\mathcal{E}$ .
3:	$\hat{\mathbf{M}}^{}\leftarrow f_{\mathrm{C}}(x,\hat{\mathbf{M}}^{q},\hat{\mathbf{M}}^{p})$ ; return $\hat{\mathbf{M}}^{}$ .
function Search ( $[a,b],d$ )
4:	Uniformly sample $N_{\mathrm{w}}$ indices $S$ from $[a,b]$ ; use every index if $b-a+1\leq N_{\mathrm{w}}$ .
5:	$\{(\hat{\mathbf{q}}_{i},\hat{p}_{i})\}_{i\in S}\leftarrow f_{\mathrm{G}}(x,\{I_{i}\}_{i\in S})$ ; add the predictions to $\mathcal{E}$ .
6:	if the indices in $S$ are adjacent, or $d=D$ , then return $\mathcal{E}$ .

Algorithm 1 Active Tree Search

Figure 4 : Agentic grounding harness. Active tree search locates relevant observations, the dual-stream grounder produces two 4D mask candidates, and the critic selects the final result. The illustrated example uses $N_{w}=4$ observations per grounder window.

Figure 5 : Training corpus. (a

\&

b) Annotation coverage and dominant scene types among the curated data. (c) Data composition and scale by supervision type.

Dataset	Supervision	Motion	Scene	Origin	Scenes/clips	Frames
Infinigen ( Raistrick et al., 2023 )	Exhaustive	Static	Indoor	Synthetic	1,466	146,034
ScanNet++ ( Yeshwanth et al., 2023 )	Exhaustive	Static	Indoor	Real	907	760,226
RealEstate10K ( Zhou et al., 2018 )	Exhaustive	Static	Mixed	Real	5,127	593,212
VIPSeg ( Miao et al., 2022 )	Exhaustive	Dynamic	Mixed	Real	2,806	66,767
Cityscapes-VPS ( Kim et al., 2020 )	Exhaustive	Dynamic	Outdoor	Real	394	1,970
KITTI-STEP ( Weber et al., 2021 )	Exhaustive	Dynamic	Outdoor	Real	12	5,027

Table 1 : Training-data composition. Counts report the sampled scenes or video clips and RGB observations used for training, rather than the full sizes of the source datasets.

Method	ScanNet			DAVIS			LVOS			VIPSeg		Latency
Method	T - mIoU	T - SR	$\mathcal{J}\&\mathcal{F}$	4D - IoU	4D - SR	$\mathcal{J}\&\mathcal{F}$	4D - IoU	4D - SR	$\mathcal{J}\&\mathcal{F}$	STQ	mAP	@100
SAM3D ( Yang et al., 2023b )	0.686	0.296	0.574	–	–	–	–	–	–	–	–	–
DEVA ( Cheng et al., 2023 )	0.180	0.029	0.219	0.531	0.584	0.531	0.373	0.312	0.412	0.583	0.047	32.3
GLEE ( Wu et al., 2024 )	0.127	0.038	0.178	0.710	0.870	0.754	0.329	0.225	0.367	0.207	0.214	36.8
OMG-Seg ( Li et al., 2024b )	0.189	0.067	0.218	0.632	0.789	0.691	0.167	0.102	0.191	0.554	0.205	13.7
SAM 2 ( Ravi et al., 2025 )	0.635	0.438	0.588	0.703	0.823	0.750	0.450	0.460	0.444	0.700	0.249	175

Table 2 : Class-agnostic instance segmentation. All methods are evaluated without target prompts or first-frame ground truth. Higher is better for segmentation metrics; latency is measured over 100 observations on an A100-80GB (s/clip).

Method	HOI4D			VISOR
Method	$\mathcal{J}\&\mathcal{F}$	T - mIoU	T - SR	$\mathcal{J}\&\mathcal{F}$	T - mIoU	T - SR
DEVA ( Cheng et al., 2023 )	0.594	0.557	0.250	0.213	0.111	0.018
GLEE ( Wu et al., 2024 )	0.427	0.404	0.261	–	–	–
OMG-Seg ( Li et al., 2024b )	0.272	0.248	0.192	0.020	0.016	0.000
SAM 2 ( Ravi et al., 2025 )	0.572	0.542	0.219	0.295	0.176	0.060
PanSt3R ( Zust et al., 2025 )	0.523	0.524	0.352	0.195	0.142	0.044

Table 3 : Egocentric and driving-scene segmentation. Left: results on HOI4D and VISOR. Right: results on the KITTI-STEP evaluation split. IGGT4D is omitted on HOI4D because its training mixture contains part of the evaluation split.

Table 10

$K$	Corruption	ScanNet					DAVIS
		Box		Point			Box		Point
		S4VY	SAM 2	S4VY	SAM 2	SAM-V	S4VY	SAM 2	S4VY	SAM 2	SAM-V
1	0%	0.806	0.753	0.793	0.649	0.567	0.790	0.809	0.793	0.771	0.134
2	0%	0.804	0.844	0.825	0.713	0.598	0.789	0.818	0.802	0.764	0.124
4	0%	0.800	0.877	0.825	0.760	0.589	0.779	0.827	0.805	0.760	0.145
8	0%	0.801	0.878	0.825	0.769	0.589	0.807	0.829	0.810	0.780	0.129

Table 6 : Point- and box-prompted segmentation on ScanNet and DAVIS.

K

is the maximum number of evenly spaced target-present observations used for prompting; corruption replaces the specified fraction with target-irrelevant prompts. All entries report

\mathcal{J}\&\mathcal{F}

for S4VY and SAM 2 ( Ravi et al., 2025 ) ; SAM-V ( Gong et al., 2026 ) supports point prompts only. For

K>1

, SAM-V independently decodes each prompted observation and retains the output with the highest predicted IoU.

K=8

covers all visible ScanNet views.

Method	VLM Size	ScanRefer			ScanNet++			Ref-DAVIS	MeViS
Method	VLM Size	3D mIoU	Box@.25	Mask@.25	3D mIoU	Box@.25	Mask@.25	J&F	J&F
VLM-Grounder ( Xu et al., 2024 )	30.5B	0.105	26.0	17.0	0.063	12.4	9.9	0.183	0.141
VLM $+$ SAM 2 ( Ravi et al., 2025 )	30.5B	0.308	34.9	42.6	0.170	15.1	26.2	0.709	0.483
Sa2VA-8B ( Yuan et al., 2026a )	8B	0.124	14.9	17.9	0.040	3.4	6.4	0.732	0.542
SAM 3 ( Carion et al., 2026 )	N/A	0.247	29.7	33.3	0.172	19.5	26.3	0.567	0.369
MVGGT ( Wu et al., 2026a )	N/A	0.325	35.3	50.1	0.090	7.8	14.8	0.005	0.003

Table 7 : Language-guided grounding. Static benchmarks report 3D mIoU and box/mask accuracy at IoU 0.25; dynamic benchmarks report J&F. Z3D is evaluated with either benchmark GT point clouds or reconstructed point clouds.

Method	Model Size	Configuration	3D mask mIoU	Mask@0.5	Box@0.5
Molmo 2 ( Clark et al., 2026 )	4B	Multi-frame point tracking	0.432	45.4	46.1
Molmo 2 ( Clark et al., 2026 )	4B	Multi-frame point localization	0.407	43.4	42.1
Qwen3-VL ( Bai et al., 2025 )	4B	Single-view box	0.316	32.2	32.9
Qwen3-VL ( Bai et al., 2025 )	4B	Multi-view box	0.202	21.1	20.4
GLM-4.1V ( Hong et al., 2025 )	9B	Per-view box	0.303	32.9	29.6
LocateAnything ( Wang et al., 2026b )	3B	Per-view box	0.287	31.6	31.6

Table 8 : Grounding-tool comparison on

160

queries randomly sampled from ScanRefer. Coordinates from each tool are decoded by the same promptable segmenter. Both configurations predict boxes. Model size refers to the grounding model only.

Method	Model Size	Precision	Recall	F1	IoU	Selected frames (%)
VideoAgent ( Wang et al., 2024b )	235B	0.305	0.108	0.144	0.082	9.5%
VideoTree ( Wang et al., 2025b )	235B	0.283	0.562	0.341	0.221	56.6%
Vidi 1.5 ( Team et al., 2025 )	9B	0.626	0.144	0.192	0.121	7.0%
LensWalk ( Li et al., 2026c )	235B	0.343	0.181	0.192	0.128	12.6%
Uniform top-8	N/A	0.291	0.144	0.173	0.099	13.8%
Uniform top-16	N/A	0.285	0.274	0.250	0.151	27.4%

Table 9 : Agentic frame localization on ScanRefer. Model size denotes the VLM used for frame selection; selected frames reports the returned fraction rather than a ranked metric.

(a) Grounder inputs
Input	3D mIoU	Mask@.5	Box@.5
Single w/o $\mathbf{z}^{\mathrm{reg}}$	0.206	22.9	20.7
Single	0.370	42.0	37.7
Multi	0.398	45.2	40.3
Multi $+$ CoT	0.407	46.0	41.1

Table 10 : Ablation studies. (a) Grounder inputs on ScanRefer. (b) Sliding-window and ATS traversal; Presence is the fraction of observations containing the target, and

\Delta Q

and

\Delta B

are the changes in query- and box-stream scores when using ATS. (c–d) Dual-stream and Critic decomposition on static and dynamic grounding benchmarks.

Panel of Figure 6 — Figure 6 : Qualitative comparison of class-agnostic segmentation. A consistent mask color denotes the same instance within each sequence. Ground-truth masks cover only annotated instances.

Panel of Figure 7 — Figure 7 : Qualitative comparison of language-guided grounding. ScanRefer ground-truth masks are projected from the annotated 3D point cloud.

Panel of Figure 8 — Figure 8 : Frame-space and reconstructed 4D-space visualizations. Selected observations are followed by aligned RGB and instance-colored point-cloud views. The results illustrate persistent instance identities across panoramic views and dynamic driving observations.

Panel of Figure 9 — Figure 9 : In-the-wild 4D segmentation. Uniformly sampled observations are shown with their masks and reconstructed 4D instances in the same temporal order.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 10 — Figure 10 : Representative training observations. Four RGB observations and corresponding masks from twelve training sources not used in our evaluation; DL3DV uses pseudo-masks.

Table 21

(a) Backbone adaptation
Strategy	mAP	AP 50	AP 75
Full fine-tuning	0.200	0.286	0.233
LoRA	0.210	0.287	0.260

Appendix

Table 13 : Segmenter ablations on IGGT-Benchmark ( Li et al., 2025a ) . (a) Backbone adaptation. (b) Frame positional encoding in the decoder.

Panel of Figure 11 — Figure 11 : Additional zero-shot panoramic segmentation. A Structured3D panorama ( Zheng et al., 2020 ) is decomposed into overlapping perspective patches; representative patches are shown above aligned RGB and instance-colored point clouds.

Panel of Figure 12 — Figure 12 : Additional 4D reconstruction and instance segmentation in a driving scene. Selected observations are followed by aligned RGB and instance-colored point-cloud views.

Panel of Figure 13 — Figure 13 : Additional qualitative comparisons of class-agnostic segmentation. A consistent mask color denotes the same instance within each sequence.

Panel of Figure 14 — Figure 14 : Feed-forward 3D reconstruction and instance segmentation. Eight RGB observations from held-out ARKitScenes are followed by aligned top-view RGB and instance-colored point clouds; all methods are evaluated zero-shot.

Panel of Figure 15 — Figure 15 : Feed-forward 4D reconstruction and instance segmentation in dynamic scenes. Selected RGB observations are followed by aligned RGB and instance-colored point-cloud views; earlier observations are rendered with lower opacity.

Panel of Figure 16 — Figure 16 : Additional qualitative comparisons of language-guided grounding (I). Ground-truth masks are projected from the annotated 3D point clouds.

Panel of Figure 17 — Figure 17 : Additional qualitative comparisons of language-guided grounding (II).

Panel of Figure 18 — Figure 18 : Additional in-the-wild 4D segmentation. Uniformly sampled observations are shown with their masks and reconstructed 4D instances in the same temporal order.

Explore similar work

IGGT4D: Streaming 4D Instance-Grounded Geometry Transformer
Jul 21, 2026Zhengyu Zou, Hao Li, Kuixuan Jiao +7Visual Geometry Grounded Transformer 4D Reconstruction
Horizon Robotics · S-Lab, Nanyang Technological University · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center
Dyna3: VLM-Guided Training-Free 4D Reconstruction via Depth Foundation Models
Oct 1, 2026Xinhao Xiang, Weiyang Li, Zhijie Zheng +24D Reconstruction Scene Reconstruction
IFM Lab, University of California, Davis, CA, USA · Genies Inc., CA, USA
SAM-V: Geometry-Aware Segment Anything for Multi-View Instance Segmentation
Sep 21, 2026Jiangshan Gong, Yuqun Wu, Qiqian Fu +4Amodal Segmentation 3D Perception
University of Illinois at Urbana-Champaign · Meta

Jul 21, 2026, cs.CVJ/K move · Enter open · S save

IGGT4D: Streaming 4D Instance-Grounded Geometry Transformer

Zhengyu Zou, Hao Li, Kuixuan Jiao, Liu Liu+6

Horizon Robotics · S-Lab, Nanyang Technological University · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center

Visual Geometry Grounded Transformer 4D Reconstruction Robust Geometric Model Estimation 4D Streaming Spatiotemporal

Open paper AskPDF arXiv

Real-world spatial intelligence requires agents to understand scenes from continuous video streams, where objects move, persist, disappear, and reappear over time. While recent spatial foundation models have enabled generalizable feed-forward 3D reconstruction, most streaming methods remain geometry-centric and lack temporally consistent object-level understanding. Meanwhile, existing semantic reconstruction and 3D-aware vision-language methods largely rely on externally extracted 2D semantic cues or loosely coupled geometry inputs, limiting unified geometry-instance learning in long dynamic scenes. In this paper, we propose IGGT4D, a streaming instance-grounded geometry Transformer for online 4D scene understanding. IGGT4D processes video frames sequentially, reuses historical context through causal spatial-temporal modeling, and incrementally updates a unified representation of camera motion, geometry, and object identity. This enables long-sequence feed-forward reconstruction with geometry-instance consistency in dynamic environments. To address the lack of high-quality 4D supervision, we further construct InsScene4D-147K, a large-scale dataset spanning real/synthetic and static/dynamic scenes, with RGB images, depth, poses, and temporally consistent instance masks generated by an automated geometry-guided annotation pipeline. Experiments on 3D reconstruction, pose estimation, instance spatial tracking, and open-vocabulary segmentation demonstrate that IGGT4D outperforms existing streaming baselines while maintaining scalable online inference for long dynamic sequences.

Abstract

Explore similar work

IGGT4D: Streaming 4D Instance-Grounded Geometry Transformer

Dyna3: VLM-Guided Training-Free 4D Reconstruction via Depth Foundation Models

SAM-V: Geometry-Aware Segment Anything for Multi-View Instance Segmentation