DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scale

Sicheng Zuo; Zixun Xie; Wenzhao Zheng; Shaoqing Xu; Fang Li; Hanbing Li; Long Chen; Zhi-Xin Yang

cs.CVApr 1, 2026

DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scale

Authors: Sicheng Zuo, Zixun Xie, Wenzhao Zheng, Shaoqing Xu, Fang Li, Hanbing Li, Long Chen, Zhi-Xin Yang, +1 more

Organizations: Tsinghua University · Xiaomi EV · Peking University · University of Macau

Abstract

End-to-end autonomous driving has evolved from the conventional paradigm based on sparse perception into vision-language-action (VLA) models, which focus on learning language descriptions as an auxiliary task to facilitate planning. In this paper, we propose an alternative Vision-Geometry-Action (VGA) paradigm that advocates dense 3D geometry as the critical cue for autonomous driving. As vehicles operate in a 3D world, we think dense 3D geometry provides the most comprehensive information for decision-making. However, most existing geometry reconstruction methods (e.g., DVGT) rely on computationally expensive batch processing of multi-frame inputs and cannot be applied to online planning. To address this, we introduce a streaming Driving Visual Geometry Transformer (DVGT-2), which processes inputs in an online manner and jointly outputs dense geometry and trajectory planning for the current frame. We employ temporal causal attention and cache historical features to support on-the-fly inference. To further enhance efficiency, we propose a sliding-window streaming strategy and use historical caches within a certain interval to avoid repetitive computations. Despite the faster speed, DVGT-2 achieves superior geometry reconstruction performance on various datasets. The same trained DVGT-2 can be directly applied to planning across diverse camera configurations without fine-tuning, including closed-loop NAVSIM and open-loop nuScenes benchmarks.

Figures & tables

Figure 1: DVGT-2 is a streaming visual geometry transformer specifically designed for autonomous driving . It inputs multi-view images and jointly predicts the 3D pointmaps, the ego poses, and future trajectory planning in an online manner.

Figure 2: Comparison of different paradigms for end-to-end autonomous driving . Conventional end-to-end models rely on sparse perception representations for scene understanding. VLA models predict language descriptions to interpret driving scenarios. Our VGA model reconstructs dense 3D geometry to facilitate safe planning.

Figure 3: Comparison of different paradigms for geometry reconstruction . Batch-processing models like DVGT [ 93 ] compute pair-wise relations across all frames, incurring an overall $\mathcal{O}(T^{2})$ complexity. Full-history streaming models like StreamVGGT [ 92 ] extract temporal cues from the entire history, leading to an $\mathcal{O}(T)$ per-frame complexity. In contrast, our sliding-window streaming strategy attends to a fixed-size cache of length $W$ , achieving a constant $\mathcal{O}(W)$ per-frame complexity.

Figure 4: Overall archetecture of DVGT-2 . Our model consists of an image encoder, a geometry transformer with temporal causal attention, and a set of prediction heads to jointly output geometry reconstruction and trajectory planning.

Figure 5: Efficient inference of DVGT-2 . Given the current frame multi-view input and the cache of past $W$ frames, our model performs efficient geometry reconstruction and trajectory planning in an online manner, avoiding recomputing historical frames.

Method	Paradigm	Acc $\downarrow$	Comp $\downarrow$	Abs Rel $\downarrow$	$\delta$ < $1.25\uparrow$	AUC $\uparrow$	Time
VGGT* [ 66 ]	Full-Seq.	1.705	1.711	0.280	0.669	74.4	$\sim$ 5.31s
MapAnything [ 29 ]	Full-Seq.	3.269	4.214	0.476	0.253	69.5	$\sim$ 2.28s
DVGT [ 93 ]	Full-Seq.	0.412	0.491	0.048	0.971	76.6	$\sim$ 1.88s
CUT3R* [ 67 ]	Streaming	1.858	2.245	0.275	0.596	36.9	$\sim$ 0.35 s
StreamVGGT* [ 92 ]	Streaming	2.209	2.060	0.303	0.620	74.1	$\sim$ 1.94s
Driv3R* [ 7 ]	Streaming	0.884	1.693	0.188	0.740	-	$\sim$ 0.56s

Table 1: Quantitative 3D geometry reconstruction results on OpenScene [ 5 ] . * denotes performing post-alignment with sparse LiDAR to recover the metric scale.

Method	Paradigm	Acc $\downarrow$	Comp $\downarrow$	Abs Rel $\downarrow$	$\delta$ < $1.25\uparrow$	AUC $\uparrow$
VGGT* [ 66 ]	Full-Seq.	1.944	2.071	0.348	0.573	83.9
MapAnything [ 29 ]	Full-Seq.	4.447	4.860	0.562	0.271	84.9
DVGT [ 93 ]	Full-Seq.	0.469	0.508	0.067	0.955	86.4
CUT3R* [ 67 ]	Streaming	2.055	2.611	0.330	0.547	44.5
StreamVGGT* [ 92 ]	Streaming	2.414	2.284	0.375	0.563	86.0
Driv3R* [ 7 ]	Streaming	0.742	1.345	0.189	0.721	–

Table 2: Quantitative 3D geometry reconstruction results on nuScenes [ 1 ] .

Method	Paradigm	Acc $\downarrow$	Comp $\downarrow$	Abs Rel $\downarrow$	$\delta$ < $1.25\uparrow$	AUC $\uparrow$
VGGT* [ 66 ]	Full-Seq.	3.202	3.390	0.333	0.567	84.3
MapAnything [ 29 ]	Full-Seq.	9.903	8.365	0.492	0.217	82.3
DVGT [ 93 ]	Full-Seq.	1.752	2.263	0.102	0.922	86.0
CUT3R* [ 67 ]	Streaming	3.369	4.192	0.290	0.563	50.9
StreamVGGT* [ 92 ]	Streaming	3.207	2.960	0.302	0.604	85.6
Driv3R* [ 7 ]	Streaming	0.800	1.311	0.168	0.770	–

Table 3: Quantitative 3D geometry reconstruction results on Waymo [ 62 ] .

Method	Paradigm	Acc $\downarrow$	Comp $\downarrow$	Abs Rel $\downarrow$	$\delta$ < $1.25\uparrow$	AUC $\uparrow$
VGGT* [ 66 ]	Full-Seq.	2.322	2.879	0.798	0.395	86.3
MapAnything [ 29 ]	Full-Seq.	7.442	7.928	1.836	0.207	87.4
DVGT [ 93 ]	Full-Seq.	0.751	1.017	0.145	0.848	95.3
CUT3R* [ 67 ]	Streaming	2.827	4.724	0.875	0.317	48.6
StreamVGGT* [ 92 ]	Streaming	2.655	2.731	0.810	0.419	91.9
Driv3R* [ 7 ]	Streaming	0.950	1.259	0.185	0.740	–

Table 4: Quantitative 3D geometry reconstruction results on DDAD [ 11 ] .

Method	Input	Aux. Sup.	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	Comf. $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
PARA-Drive [ 74 ]	C	Map & Mot. & Occ	97.9	92.4	93.0	99.8	79.3	84.0
VADv2 [ 3 ]	C	Map & Mot. & Traffic	97.2	89.1	91.6	100	76.0	80.9
UniAD [ 16 ]	C	Map & Box & Mot. & Occ	97.8	91.9	92.9	100	78.8	83.4
Transfuser [ 4 ]	C & L	Map & Box	97.7	92.8	92.8	100	79.2	84.0
Hydra-MDP [ 38 ]	C & L	Map & Box	98.3	96.0	94.6	100	78.7	86.5
GoalFlow [ 76 ]	C & L	Map & Box	98.3	93.8	94.3	100	79.8	85.7

Table 5: Closed-loop planning results on NAVSIM v1 navtest split .

\dagger

denotes using reinforcement learning to boost planning scores. Future states represent world-modeling-based methods. C and L denote camera and LiDAR, respectively.

Method	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	TL $\uparrow$	EP $\uparrow$	TTC $\uparrow$	LK $\uparrow$	HC $\uparrow$	EC $\uparrow$	EPDMS $\uparrow$
Ego Status MLP	93.1	77.9	92.7	99.6	86.0	91.5	89.4	98.3	85.4	64.0
Transfuser [ 4 ]	96.9	89.9	97.8	99.7	87.1	95.4	92.7	98.3	87.2	76.7
Hydra-MDP++ [ 33 ]	97.2	97.5	99.4	99.6	83.1	96.5	94.4	98.2	70.9	81.4
DriveSuprim [ 82 ]	97.5	96.5	99.4	99.6	88.4	96.6	95.5	98.3	77.0	83.1
ARTEMIS [ 8 ]	98.3	95.1	98.6	99.8	81.5	97.4	96.5	98.3	-	83.1
DiffusionDrive [ 41 ]	98.2	95.9	99.4	99.8	87.5	97.3	96.8	98.3	87.7	84.5

Table 6: Closed-loop planning results on NAVSIM v2 navtest split .

Figure 6: Qualitative visualizations . These results demonstrate that DVGT-2 can predict high-fidelity dense scene geometry and perform robust trajectory planning.

Method	Input	Aux. Sup.	L2 (m) $\downarrow$				Collision Rate (%) $\downarrow$
Method	Input	Aux. Sup.	1s	2s	3s	Avg.	1s	2s	3s	Avg.
IL [ 57 ]	LiDAR	None	0.44	1.15	2.47	1.35	0.08	0.27	1.95	0.77
NMP [ 83 ]	LiDAR	Box & Motion	0.53	1.25	2.67	1.48	0.04	0.12	0.87	0.34
FF [ 14 ]	LiDAR	Freespace	0.55	1.20	2.54	1.43	0.06	0.17	1.07	0.43
EO [ 30 ]	LiDAR	Freespace	0.67	1.36	2.78	1.60	0.04	0.09	0.88	0.33
ST-P3 [ 15 ]	Camera	Map & Box & Depth	1.33	2.11	2.90	2.11	0.23	0.62	1.27	0.71

Table 7: Open-looped planning results on nuScenes [ 1 ] .

\dagger

denotes the results computed with an average of previous frames as adopted in VAD [ 26 ] . Aux. Sup. represents auxiliary supervision. Avg. computes the average result of 1s, 2s, and 3s.

Panel of Figure 7 — Figure 7: Efficiency comparison of online inference . We report the per-frame latency and memory cost of different methods on multi-frame, 8-view sequences.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Table 15

Method	Paradigm	Acc $\downarrow$	Comp $\downarrow$	Abs Rel $\downarrow$	$\delta$ < $1.25\uparrow$	AUC@30 $\uparrow$
VGGT* [ 66 ]	Full-Seq.	1.154	1.294	0.158	0.801	96.91
MapAnything [ 29 ]	Full-Seq.	1.807	1.006	0.184	0.727	90.51
DVGT [ 93 ]	Full-Seq.	0.846	1.468	0.136	0.849	87.63
CUT3R* [ 67 ]	Streaming	0.973	2.054	0.217	0.660	51.82
StreamVGGT* [ 92 ]	Streaming	3.393	2.181	0.365	0.467	95.57
Driv3R* [ 7 ]	Streaming	0.864	1.083	0.164	0.784	-

Appendix

Table A.3: Quantitative 3D geometry reconstruction results on KITTI [ 10 ] . * denotes performing post-alignment with sparse LiDAR to recover the metric scale.

Dataset	Train Scenes	Test Scenes	Min Frames	Max Frames	Avg Frames	Avg Aspect Ratio	Num of Views
nuScenes	700	150	32	41	40	1.77	6
OpenScene	19376	2026	1	41	38	1.77	8
Waymo	798	202	34	40	40	1.77	5
KITTI	138	13	2	1033	62	3.31	2
DDAD	150	50	10	20	17	1.59	6

Appendix

Table A.4 : Detailed statistics of the datasets used in our experiments. All temporal statistics are reported at a 2Hz sampling rate.

Explore similar work

Oct 7, 2026

Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving

Vision-language-action~(VLA) models have emerged as a promising paradigm for autonomous driving. However, existing VLA models still suffer from a fundamental mismatch: driving actions require precise 3D geometric cues, while visual-language understanding and reasoning are largely conducted in a 2D semantic space. In this paper, we propose GeoCoTDrive, an explicit geometric chain-of-thought framework that grounds geometry in a planning-oriented manner. GeoCoTDrive follows a think with 2D first, drive with dedicated 3D priors paradigm. It first grounds 2D regions corresponding to decision-critical cues, and then retrieves localized 3D priors by sampling features from a geometric foundation model within the grounded regions. These localized geometric features are interleaved into the autoregressive context to support the trajectory generation. To supervise this process, we introduce planning-relevant grounding, a new region-level grounding task that focuses on local spatial cues directly affecting ego planning decisions, and construct the PlanningGrounding dataset to endow VLAs with planning-oriented grounding capability. Experiments across multiple end-to-end autonomous driving benchmarks show that GeoCoTDrive consistently improves safety-critical planning performance, demonstrating the effectiveness of the explicit geometric chain-of-thought process for VLA-based planning.

Xingtai Gui, Yucheng Zhou, Dongqian Guo +3

SKL-IOTSC, Department of AI, University of Macau · Afari Intelligent Drive

Autonomous Driving Planning Reasoning for VLA Models

Ask PDF

Jun 10, 2026

VLGA: Vision-Language-Geometry-Action Models for Autonomous Driving

Vision-language-action (VLA) models can describe scenes and reason about them in language, yet still struggle to ground their actions in the dense 3D world around them. Existing approaches either inject features from a frozen 3D foundation model without an objective that ensures the policy uses them, or constrain geometry with sparse box and map losses that provide no dense spatial signal. We introduce VLGA, the first vision-language-action model supervised to reconstruct the dense 3D world it drives through. VLGA introduces geometry as a fourth modality alongside vision, language, and action through a dedicated expert supervised by a per-pixel pointmap regression loss against LiDAR. Extensive experiments conducted on challenging nuScenes and Bench2Drive datasets for open-loop and closed-loop evaluations, respectively, show the superiority of VLGA over counterpart VLA methods. In particular, on open-loop nuScenes, VLGA sets a new state of the art among VLA methods without ego status, with the lowest L2 (0.50,m average) and 3-second collision rate (0.18%). On closed-loop Bench2Drive, VLGA attains the state-of-the-art driving score of 79.08, +0.71 over the strongest prior VLA, at comparable efficiency and comfort.

Jin Yao, Dhruva Dixith Kurra, Tom Lampo +3

Uber AV Labs · University of Virginia

VLMs for Autonomous Driving End-to-End Autonomous Driving Vision-Language-Action Models

Ask PDF

Aug 18, 2026

Geo-VLA: Geometry-Aware Vision-Language-Action Planning via Internalization of Map Semantics

Vision-language-action (VLA) models have advanced end-to-end autonomous driving by leveraging foundation models for semantic reasoning and long-tail generalization. However, their planning performance remains limited in complex driving environments because image-only representations inadequately capture planning-relevant road geometry and topology. In this paper, we propose Geo-VLA, a plug-and-play framework that enhances VLA models by learning geometry-aware visual representations. During training, Geo-VLA internalizes geometric map semantics to strengthen road-structure representations, while requiring no HD maps or additional lane information during inference. To support this approach, we introduce Geo-QA, a geometry-focused question-answering dataset that injects road geometry into vision-language representations through contrastive learning and instruction tuning. Experiments on NAVSIM v1 demonstrate that Geo-VLA consistently improves VLA planners with distinct action-generation architectures, achieving 92.1 PDMS and establishing a new state-of-the-art among single-camera VLA planners.

Ran Chen, Jiaxing Ren, Zhikun Zhang +3

University of Science and Technology Beijing Beijing, China

Vision-Language-Action Models Autonomous Driving Planning

Ask PDF