VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Driving

Jie Wang; Guang Li; Zhijian Huang; Chenxu Dang; Hangjun Ye; Yahong Han; Long Chen

cs.CVFeb 24, 2026

VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Driving

Authors: Jie Wang, Guang Li, Zhijian Huang, Chenxu Dang, Hangjun Ye, Yahong Han, Long Chen

Organizations: College of Intelligence and Computing, Tianjin University · Xiaomi EV

Abstract

The significance of cross-view 3D geometric modeling capabilities for autonomous driving is self-evident, yet existing Vision-Language Models (VLMs) inherently lack this capability, resulting in their mediocre performance. While some promising approaches attempt to mitigate this by constructing Q&A data for auxiliary training, they still fail to fundamentally equip VLMs with the ability to comprehensively handle diverse evaluation protocols. We thus chart a new course, advocating for the infusion of VLMs with the cross-view geometric grounding of mature 3D foundation models, closing this critical capability gap in autonomous driving. In this spirit, we propose a novel architecture, VGGDrive, which empowers Vision-language models with cross-view Geometric Grounding for autonomous Driving. Concretely, to bridge the cross-view 3D geometric features from the frozen visual 3D model with the VLM's 2D visual features, we introduce a plug-and-play Cross-View 3D Geometric Enabler (CVGE). The CVGE decouples the base VLM architecture and effectively empowers the VLM with 3D features through a hierarchical adaptive injection mechanism. Extensive experiments show that VGGDrive enhances base VLM performance across five autonomous driving benchmarks, including tasks like cross-view risk perception, motion prediction, and trajectory planning. It's our belief that mature 3D foundation models can empower autonomous driving tasks through effective integration, and we hope our initial exploration demonstrates the potential of this paradigm to the autonomous driving community.

Figures & tables

Figure 1: Existing relevant paradigms vs. our VGGDrive. (a) The VLA paradigm for trajectory planning. (b) Two existing paradigms for integrating 3D foundation models (VGGT [ 39 ] ) with VLMs: VGGT-Dist [ 11 ] and VGGT-Add [ 48 ] . (c) Our VGGDrive, which leverages the VGGT model to profoundly empower the basic VLM with cross-view geometric grounding capabilities, thereby handling diverse autonomous driving tasks.

Figure 2: Quantitative comparison of VGGDrive with specific sota methods across four autonomous driving benchmarks, covering evaluations of attributes such as cross-view risk perception , motion prediction and trajectory planning .

Figure 3: Overview of VGGDrive . Specifically, the frozen visual 3D foundation model (VGGT [ 39 ] ) extracts geometrically consistent 3D features $V^{3d}$ through cross-view analysis, while the base VLM is decomposed into multiple decoder layers. The proposed CVGE sequentially integrates the shared 3D features $V^{3d}$ with the 2D visual representations $V_{i}^{2d}$ , injecting them $V_{i}^{3d}$ through a hierarchical adaptive mechanism, thereby establishing geometric grounding and enabling deep enhancement of the VLM architecture.

NAVSIM	Base Model	Image	Lidar	NC↑	DAC↑	EP↑	TTC↑	Comf.↑	PDMS↑
TransFuser [ 4 ]	E2E	$\surd$	$\surd$	97.78	92.63	78.88	92.89	99.98	83.88
PARA-Drive [ 43 ]		$\surd$		97.90	92.40	79.30	93.00	99.8	84.00
DRAMA [ 46 ]		$\surd$	$\surd$	98.19	95.18	81.33	94.17	100	86.87
Hydra-MDP-V 8192 [ 29 ]		$\surd$	$\surd$	98.30	96.00	78.70	94.60	100	86.50
DiffusionDrive [ 31 ]		$\surd$	$\surd$	98.20	96.20	82.20	94.70	100	88.10
WoTE [ 27 ]		$\surd$	$\surd$	98.50	96.80	81.90	94.90	99.9	88.30

Table 1: The performance comparison on the NAVSIM navtest [ 5 ] benchmark, evaluated using closed-loop metrics, involves both SOTA E2E approaches and existing VLA models under supervised fine-tuning. This evaluation aims to reflect the performance gains achieved by VGGDrive in enhancing the closed-loop trajectory planning capability of the base VLM.

Dataset	Metrics	Zero-shot Models				Fine-tuned Models
Dataset	Metrics	GPT-4o [ 15 ]	LLAVA-OV [ 22 ]	RoboTron [ 12 ]	Qwen2.5-VL	Baseline	InMLLM [ 6 ]	VGGT-Dist	VGGT-Add	VGGDrive
NuInstruct	MAE ↓	9.93	87.04	19.36	24.10	4.35	9.08	3.73	3.63	3.08
	Accuracy ↑	10.64	3.75	2.57	0.63	47.71	32.48	56.21	56.35	56.37
	MAP↑	0	0	0	0	6.15	21.93	28.51	30.12	37.49
	BLEU↑	7.08	8.55	8.06	5.56	75.75	35.2	79.23	79.45	81.13
	Average*↑	1.95	0	0	0	31.32	20.13	40.06	40.57	42.98

Table 2: The performance comparison on the NuInstruct dataset [ 6 ] is conducted against existing SOTA methods. This experiment is crucial for evaluating the performance gains of VGGDrive in cross-view risk object perception ( MAP ), state predictio n, and ego-motion forecasting within autonomous driving scenarios. The symbol * indicates

\max\left(\frac{\text{Accuracy}+\text{MAP}+\text{BLEU}-\text{MAE}}{4},0\right).

Dataset	Metrics	Zero-shot Models			Fine-tuned Models
Dataset	Metrics	GPT-4o	LLAVA-OV	RoboTron	Baseline	DriveLM	TM-LMM [ 17 ]	OminiDrive	FSDrive [ 47 ]	I-VL4Drive [ 25 ]	VGGDrive
DriveLM	Accuracy ↑	38.55	25.03	52.94	64.35	52.30	59.60	70.00	71.77	73.39	77.50
	ChatGPT↑	67.27	65.70	51.11	65.30	55.35	58.44	65.00	63.42	65.25	64.76
	Language↑	8.97	14.44	7.12	44.90	45.13	46.38	-	52.77	48.56	56.69
	Match ↑	24.00	40.93	35.05	34.54	35.73	35.73	37.00	39.19	47.65	49.77
	Average↑	41.21	42.36	39.47	54.59	47.20	51.50	56.00	57.12	60.02	61.26

Table 3: The performance comparison on the DriveLM dataset [ 38 ] is conducted against existing SOTA methods. This experiment is crucial for evaluating the performance gains of VGGDrive in cross-view risk object perception ( Match ), action prediction and planning.

Dataset	Metrics	Zero-shot Models			Fine-tuned Models
Dataset	Metrics	GPT-4o	LLAVA-OV	RoboTron	OmniDrive	HERMES [ 49 ]	Baseline	VGGT-Dist	VGGT-Add	VGGDrive
OmniDrive	BLEU↑	10.91	16.14	20.30	38.00	-	37.29	37.28	37.23	37.58
	CIDEr↑	24.42	28.41	34.33	68.60	74.10	86.29	86.41	86.38	86.57
	ROUGE↑	22.34	22.14	23.67	32.60	32.70	34.33	34.32	34.26	34.40
	Average↑	19.22	22.23	26.10	46.40	-	52.64	52.67	52.62	52.85

Table 4: The performance comparison on the OmniDrive dataset [ 41 ] focuses on caption-related tasks in which base VLMs excel.

nuScenes		L2				Collision (%) ↓				Intersection (%) ↓
		1s	2s	3s	avg	1s	2s	3s	avg	1s	2s	3s	avg
UniAD [ 10 ]	E2E	0.20	0.42	0.75	0.46	0.02	0.25	0.84	0.37	0.20	1.33	3.24	1.59
VAD-Base [ 19 ]		0.17	0.34	0.60	0.37	0.04	0.27	0.67	0.33	0.21	2.13	5.06	2.47
BEV-Planner [ 30 ]		0.16	0.32	0.57	0.35	0.00	0.29	0.73	0.34	0.35	2.62	6.51	3.16
Baseline [ 30 ]	VLA	0.14	0.34	0.60	0.36	0.06	0.21	0.96	0.41	0.66	2.58	5.43	2.89
OmniDrive-Q [ 41 ]		0.14	0.29	0.55	0.33	0.00	0.13	0.78	0.30	0.56	2.48	5.96	3.00

Table 5: Performance comparison on nuScenes open-loop planning, with metrics from BEV-Planner’s reproduced results [ 30 ] .

Figure 4: Visualization of VGGDrive’s performance across various autonomous driving attribute evaluation tasks.

ID	Methods	NAVSIM		NuInstruct
ID	Methods	EP↑	PDMS↑	MAP↑	BLEU↑	Avg↑	Time (s)↓
1	Baseline	81.00	86.04	6.15	75.75	31.32	0.81
2	VGGT	79.56	84.45	3.34	74.69	30.35	0.84
3	VGGT-Dist	81.30	86.68	28.51	79.23	40.06	0.82
4	VGGT-Add	80.84	86.10	30.12	79.45	40.57	0.87
5	VGGT-MHCA	81.94	87.83	32.35	79.39	40.85	0.91

Table 6: Ablation study of various integration schemes between VGGT and VLM on the NAVSIM [ 5 ] and NuInstruct Datasets [ 6 ] . Inference speed is evaluated on the NuInstruct MAP task.

ID	Methods	NAVSIM		NuInstruct
ID	Methods	EP↑	PDMS↑	MAP↑	BLEU↑	Avg↑	Time (s)↓
1	Baseline	81.00	86.04	6.15	75.75	31.32	0.81
2	w/o MHCA	82.05	87.79	27.43	77.89	39.29	1.00
3	Share CVGE	82.48	88.05	30.85	79.55	40.95	0.96
4	w/o Residual	82.32	87.86	27.98	78.31	40.02	1.03
5	One-stage SFT	82.83	88.62	36.19	80.75	42.05	1.04

Table 7: Ablation Study of the Main Components of VGGDrive.

NAVSIM	NC↑	DAC↑	EP↑	TTC↑	Comf.↑	PDMS↑
Baseline	97.83	94.08	81.00	94.04	99.98	86.04
Fast3r [ 45 ]	98.34	95.83	82.62	95.58	99.98	88.36
VGGT [ 39 ]	98.55	96.30	82.92	95.59	99.98	88.76

Table 8: Ablation Study of the 3D Expert Model.

NAVSIM	NC↑	DAC↑	EP↑	TTC↑	Comf.↑	PDMS↑
Baseline	97.83	94.08	81.00	94.04	99.98	86.04
w/o $T_{i}^{img2lidar}$	98.33	95.93	82.59	95.42	99.99	88.32
s=8	98.17	96.15	82.66	95.26	99.99	88.36
s=2	98.50	96.16	82.80	95.78	99.99	88.70
OURS	98.55	96.30	82.92	95.59	99.98	88.76

Table 9: Further Ablation Analysis on the Navsim Benchmark.

Dataset	Train Samples	Test Samples	Cross-view perception	Action&States prediction	Trajectory planning	Scene understanding
NAVSIM [ 5 ]	103.3k	12.1k			✓
NuInstruct [ 6 ]	71.8k	16.1k	✓	✓
DriveLM [ 38 ]	376.2k	15.5k	✓	✓		✓
OmniDrive [ 41 ]	318.4k	54.1k				✓
nuScenes [ 30 ]	28.1k	6.0k			✓

Table S1 : Training and Testing Sample Statistics of Five Mainstream Autonomous Driving Datasets Used by VGGDrive, and the Model Capabilities Evaluated on Each Dataset.

Panel of Figure 15 — Figure S1 : Ablation analysis of closed-loop trajectory planning performance on the NAVSIM dataset when cross-view 3D geometric empowerment and adaptive injection are applied to individual decoding layers of the LLM.

Figure S2 : Qualitative results on the closed-loop trajectory planning task in the Navtest benchmark are presented, showcasing a typical left-turn example to demonstrate the performance of VGGDrive in complex driving scenarios.

Figure S3 : Qualitative results on the closed-loop trajectory planning task in the Navtest benchmark are presented, showcasing a typical right-turn example to demonstrate the performance of VGGDrive in complex driving scenarios.

Figure S4 : Qualitative results on the closed-loop trajectory planning task in the Navtest benchmark are presented, showcasing a typical straight-ahead example to demonstrate the performance of VGGDrive in complex driving scenarios.

Figure S5 : Qualitative results on the open-loop trajectory planning task in the nuScenes benchmark are presented.

Figure S6 : Qualitative results on Action $\&$ States prediction in autonomous driving scenarios are presented.

Multi-View Feature Fusion

See all 23 papers

Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving
Aug 31, 2026Xin Zhou, Zongchuang Zhao, Zhibo Yang +13VLMs for Autonomous Driving Autonomous Driving Perception
Qwen Team, Huazhong University of Science and Technology
Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving
Oct 7, 2026Xingtai Gui, Yucheng Zhou, Dongqian Guo +3Autonomous Driving Planning Reasoning for VLA Models
SKL-IOTSC, Department of AI, University of Macau · Afari Intelligent Drive
BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representations
Mar 6, 2026Thomas Monninger, Shaoyuan Xie, Qi Alfred Chen +1VLMs for Autonomous Driving End-to-End Autonomous Driving
Mercedes-Benz Research & Development North America, San Jose, USA · University of California, Irvine, USA

Aug 31, 2026J/K move · Enter open · S save

Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving

Xin Zhou, Zongchuang Zhao, Zhibo Yang, Mingsheng Li+12

Qwen Team, Huazhong University of Science and Technology

VLMs for Autonomous Driving Autonomous Driving Perception Autonomous Driving Planning 3D Scene Understanding

Open paper AskPDF arXiv

We present Qwen-Drive-1.0, an initial step towards a vision-language foundation model for autonomous driving. Qwen-Drive-1.0 retains the architecture of the pretrained vision-language model (VLM) and integrates 3D perception, visual question answering, and motion planning within a unified framework. An external bird's-eye-view (BEV) perception head jointly performs 3D object detection, semantic occupancy prediction, and BEV map segmentation. It serves as a probe of the 3D information accessible from the shared representations and provides an explicit, inspectable interface to 3D scene structure. A Planning Expert conditions on shared VLM representations to generate future ego trajectories. A staged training recipe combines driving supervision with general-purpose vision-language data to acquire driving-specific competence while helping preserve broad visual understanding and instruction-following capabilities. Experiments demonstrate strong 3D perception and driving scene understanding while largely preserving general vision-language capability. Comprehensive evaluations across open-loop, pseudo-closed-loop, and closed-loop settings further show highly competitive motion-planning performance.

Abstract

Multi-View Feature Fusion

Explore similar work

Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving

Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving

BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representations