VGGT-World: Transforming VGGT into an Autoregressive Geometry World Model

Xiangyu Sun; Shijie Wang; Fengyi Zhang; Lin Liu; Caiyan Jia; Ziying Song; Zi Huang; Yadan Luo

cs.CVMar 13, 2026

VGGT-World: Transforming VGGT into an Autoregressive Geometry World Model

Authors: Xiangyu Sun, Shijie Wang, Fengyi Zhang, Lin Liu, Caiyan Jia, Ziying Song, Zi Huang, Yadan Luo

Organizations: UQMM Lab, The University of Queensland · Beijing Jiaotong University

Abstract

World models that forecast scene evolution by generating future video frames devote the bulk of their capacity to photometric details, yet the resulting predictions often remain geometrically inconsistent. We present VGGT-World, a geometry world model that side-steps video generation entirely and instead forecasts the temporal evolution of frozen geometry-foundation-model (GFM) features. Concretely, we repurpose the latent tokens of a frozen VGGT as the world state and train a lightweight temporal flow transformer to autoregressively predict their future trajectory. Two technical challenges arise in this high-dimensional (d=1024) feature space: (i) standard velocity-prediction flow matching collapses, and (ii) autoregressive rollout suffers from compounding exposure bias. We address the first with a clean-target (z-prediction) parameterization that yields a substantially higher signal-to-noise ratio, and the second with a two-stage latent flow-forcing curriculum that progressively conditions the model on its own partially denoised rollouts. Experiments on KITTI, Cityscapes, and TartanAir demonstrate that VGGT-World significantly outperforms the strongest baselines in depth forecasting while running 3.6-5 times faster with only 0.43B trainable parameters, establishing frozen GFM features as an effective and efficient predictive state for 3D world modeling.

Figures & tables

Figure 1: From video world models to geometry world models. Video world models predict future RGB in VAE latent space, coupling scene dynamics with appearance reconstruction. As a result, decoded predictions can remain geometrically invalid, with broken scene layout and mislocalized actors despite plausible video appearance. VGGT-World instead uses frozen geometry-foundation features as the latent state and models their temporal evolution directly, yielding a VAE-free, lightweight, and geometry-consistent alternative for future 3D forecasting.

Panel of Figure 2 — Figure 2: Comparison of $v$ -prediction and $z$ -prediction in frozen VGGT latent space. Left: $z$ -prediction consistently achieves substantially higher signal-to-noise ratio than $v$ -prediction in the layer-4 latent space. Middle: A target VGGT feature map and its corresponding RGB frame. Right: PCA visualizations of predicted latents during training. $v$ -prediction remains noisy even after prolonged training, whereas $z$ -prediction progressively recovers structured latent patterns aligned with the target VGGT feature.

Method	2011_09_26_drive_0002_sync				2011_10_03_drive_0047_sync				Mean (Dataset Avg.)
	AbsRel ( $\downarrow$ )		$\delta_{1}$ ( $\uparrow$ )		AbsRel ( $\downarrow$ )		$\delta_{1}$ ( $\uparrow$ )		AbsRel ( $\downarrow$ )		$\delta_{1}$ ( $\uparrow$ )
	Short	Mid	Short	Mid	Short	Mid	Short	Mid	Short	Mid	Short	Mid
Cosmos-4B [ 1 ]	0.138	0.148	84.0	81.4	0.155	0.156	77.5	75.2	0.155	0.165	77.1	74.1
Cosmos-12B [ 1 ]	0.131	0.139	84.5	79.7	0.149	0.176	77.6	75.9	0.154	0.185	77.5	71.5
Copy Last (VGGT [ 50 ] )	0.112	0.125	89.6	82.6	0.140	0.153	84.4	79.6	0.137	0.144	83.7	78.5
Aether [ 44 ]	0.074	0.102	94.2	88.0	0.113	0.111	87.4	85.2	0.096	0.124	87.9	84.7

Table 1: Depth forecasting on KITTI. We evaluate representative world-model baselines on future scene depth prediction, reporting AbsRel (

\downarrow

) and

\delta_{1}

(

\uparrow

) for short- and mid-term horizons.

Figure 3: The short- and mid-term depth forecasting results on KITTI. “Upper Bound” denotes the depth obtained by feeding the real future images into VGGT. “DINO-Foresight” predicts depth in the scale of Depth Anything [ 59 ] , leading to a different visualization appearance compared to others.

Method	AbsRel ( $\downarrow$ )		$\delta_{1}$ ( $\uparrow$ )
Method	Short	Mid	Short	Mid
Cosmos-4B [ 1 ]	0.189	0.247	76.3	70.5
Cosmos-12B [ 1 ]	0.181	0.241	78.2	72.4
Copy-Last (VGGT [ 50 ] )	0.154	0.212	84.1	77.8
VISTA ft [ 15 ]	0.124	0.153	86.4	82.8
DINO-Foresight [ 23 ]	0.114	0.136	88.6	85.4

Table 2: Depth forecasting on Citys- capes with AbsRel and depth accuracy

\delta_{1}

Method	Condition		1-view			2-views
Method	Text-free	Pose-free	Acc. $\downarrow$	Comp. $\downarrow$	CD $\downarrow$	Acc. $\downarrow$	Comp. $\downarrow$	CD $\downarrow$
Aether [ 44 ]	✓	✗	3.1547	4.5366	3.8457	2.7745	3.4420	3.1082
WVD [ 61 ]	✓	✗	4.3944	3.0660	3.7302	4.3794	2.5268	3.4531
Gen3R [ 19 ]	✗	✗	3.0250	2.5367	2.7809	2.2825	1.6643	1.9734
Gen3R [ 19 ]	✓	✗	3.4579	4.7384	4.0982	2.6029	3.1154	2.8591
Gen3R [ 19 ]	✓	✓	3.6427	7.8528	5.7478	2.7441	6.1423	4.4432

Table 3: Geometry generation on TartanAir with different conditioning (text, pose) under 1-view and 2-view settings.

Figure 4: Point cloud forecasting on TartanAir . Gen3R produces structurally disorganized geometry on walls and rooftops, whereas our method preserves coherent structure and yields more accurate predictions.

Figure 6: Qualitative study of camera parameter prediction on KITTI and TartanAir.

Figure 7: Efficiency comparison with VideoGen world models. (a) Inference latency per generated frame. (b) Trainable vs. frozen parameter footprint.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: The short- and mid-term depth forecasting results on Cityscapes. “Upper Bound” denotes the depth obtained by feeding the real future images into VGGT. “DINO-Foresight” predicts depth in the scale of Depth Anything [ 59 ] , leading to a different visual appearance from the other methods.

Figure 9: Point cloud forecasting on TartanAir. Gen3R exhibits noticeable artifacts in structurally homogeneous regions such as walls and windows.

Figure 10: Single-frame point cloud forecasting results on TartanAir. The gray regions in each point cloud indicate the ground truth.

Metric	KITTI	TartanAir
ATE $\downarrow$ (m)	0.4157	0.1464
RTE $\downarrow$ (m)	1.1700	0.1821
RRE $\downarrow$ ( ∘ )	0.5711	1.7350

Appendix

Table 5: Quantitative study of camera parameter prediction on KITTI and TartanAir.

3D Scene Prediction

See all 13 papers

GeoWM: Efficient Direct World Modeling in Explicit Geometry
Oct 5, 2026Mehrdad Noori, Guile Wu, Sam Hosseini +1World Model Learning World Models for Robotics
Huawei Noah’s Ark Lab, Canada
GeoState: Learning View-Decoupled Geometry States for 3D World Modeling
Oct 15, 2026Hongchang Liu, Jian Liu, Yujuan Ding +5World Model Learning 3D Scene Prediction
Tongji University · Hong Kong Polytechnic University · Tencent +1
GeoWorld: Providing Full-frame Geometry Features to Facilitate 3D Scene Generation
Nov 28, 2025Yuhao Wan, Lijuan Liu, Jingzhi Zhou +6Video Diffusion Models 3D Scene Generation
1VCIP & AAIS, Nankai University · 2ByteDance Inc. · 3Renmin University of China +2

Oct 5, 2026J/K move · Enter open · S save

GeoWM: Efficient Direct World Modeling in Explicit Geometry

Mehrdad Noori, Guile Wu, Sam Hosseini, Dongfeng Bai

Huawei Noah’s Ark Lab, Canada

World Model Learning World Models for Robotics World Models for Autonomous Driving

Open paper AskPDF arXiv

Modeling 3D scene geometry and its evolution over time is essential for autonomous driving and robotics. A common paradigm is to use world models to predict future images or latent representations of the environment and subsequently recover geometry from these predictions. However, this paradigm does not explicitly model geometric structure and typically relies on recursive rollouts to reach longer prediction horizons, leading to error accumulation and increasing computational cost. To address these limitations, we present GeoWM, a geometry world model that directly forecasts future scene geometry at specified future horizons without recursive rollout. The key idea is to leverage a geometry foundation model to transform observed RGB frames into a geometric history, which conditions a flow-matching transformer to predict the scene geometry at a specified future horizon. We further show that a lightweight camera-motion predictor can accurately estimate the future viewpoint, and that projecting the observed geometry into the predicted viewpoint provides an effective geometric prior for future geometry forecasting. Extensive experiments on four datasets spanning urban driving, aerial flight, and dynamic manipulation demonstrate that GeoWM outperforms the evaluated world models in forecasting depth, camera pose, and 3D scene geometry, while substantially reducing inference time at longer horizons.

Abstract

3D Scene Prediction

Explore similar work

GeoWM: Efficient Direct World Modeling in Explicit Geometry

GeoState: Learning View-Decoupled Geometry States for 3D World Modeling

GeoWorld: Providing Full-frame Geometry Features to Facilitate 3D Scene Generation