The significance of cross-view 3D geometric modeling capabilities for autonomous driving is self-evident, yet existing Vision-Language Models (VLMs) inherently lack this capability, resulting in their mediocre performance. While some promising approaches attempt to mitigate this by constructing Q&A data for auxiliary training, they still fail to fundamentally equip VLMs with the ability to comprehensively handle diverse evaluation protocols. We thus chart a new course, advocating for the infusion of VLMs with the cross-view geometric grounding of mature 3D foundation models, closing this critical capability gap in autonomous driving. In this spirit, we propose a novel architecture, VGGDrive, which empowers Vision-language models with cross-view Geometric Grounding for autonomous Driving. Concretely, to bridge the cross-view 3D geometric features from the frozen visual 3D model with the VLM's 2D visual features, we introduce a plug-and-play Cross-View 3D Geometric Enabler (CVGE). The CVGE decouples the base VLM architecture and effectively empowers the VLM with 3D features through a hierarchical adaptive injection mechanism. Extensive experiments show that VGGDrive enhances base VLM performance across five autonomous driving benchmarks, including tasks like cross-view risk perception, motion prediction, and trajectory planning. It's our belief that mature 3D foundation models can empower autonomous driving tasks through effective integration, and we hope our initial exploration demonstrates the potential of this paradigm to the autonomous driving community.
Figures & tables
Figure 1: Existing relevant paradigms vs. our VGGDrive. (a) The VLA paradigm for trajectory planning. (b) Two existing paradigms for integrating 3D foundation models (VGGT [ 39 ] ) with VLMs: VGGT-Dist [ 11 ] and VGGT-Add [ 48 ] . (c) Our VGGDrive, which leverages the VGGT model to profoundly empower the basic VLM with cross-view geometric grounding capabilities, thereby handling diverse autonomous driving tasks.
Figure 2: Quantitative comparison of VGGDrive with specific sota methods across four autonomous driving benchmarks, covering evaluations of attributes such as cross-view risk perception , motion prediction and trajectory planning .
Figure 3: Overview of VGGDrive . Specifically, the frozen visual 3D foundation model (VGGT [ 39 ] ) extracts geometrically consistent 3D features V3d through cross-view analysis, while the base VLM is decomposed into multiple decoder layers. The proposed CVGE sequentially integrates the shared 3D features V3d with the 2D visual representations Vi2d , injecting them Vi3d through a hierarchical adaptive mechanism, thereby establishing geometric grounding and enabling deep enhancement of the VLM architecture.
NAVSIM
Base Model
Image
Lidar
NC↑
DAC↑
EP↑
TTC↑
Comf.↑
PDMS↑
TransFuser [ 4 ]
E2E
√
√
97.78
92.63
78.88
92.89
99.98
83.88
PARA-Drive [ 43 ]
√
97.90
92.40
79.30
93.00
99.8
84.00
DRAMA [ 46 ]
√
√
98.19
95.18
81.33
94.17
100
86.87
Hydra-MDP-V 8192 [ 29 ]
√
√
98.30
96.00
78.70
94.60
100
86.50
DiffusionDrive [ 31 ]
√
√
98.20
96.20
82.20
94.70
100
88.10
WoTE [ 27 ]
√
√
98.50
96.80
81.90
94.90
99.9
88.30
Table 1: The performance comparison on the NAVSIM navtest [ 5 ] benchmark, evaluated using closed-loop metrics, involves both SOTA E2E approaches and existing VLA models under supervised fine-tuning. This evaluation aims to reflect the performance gains achieved by VGGDrive in enhancing the closed-loop trajectory planning capability of the base VLM.
Dataset
Metrics
Zero-shot Models
Fine-tuned Models
GPT-4o [ 15 ]
LLAVA-OV [ 22 ]
RoboTron [ 12 ]
Qwen2.5-VL
Baseline
InMLLM [ 6 ]
VGGT-Dist
VGGT-Add
VGGDrive
NuInstruct
MAE ↓
9.93
87.04
19.36
24.10
4.35
9.08
3.73
3.63
3.08
Accuracy ↑
10.64
3.75
2.57
0.63
47.71
32.48
56.21
56.35
56.37
MAP↑
0
0
0
0
6.15
21.93
28.51
30.12
37.49
BLEU↑
7.08
8.55
8.06
5.56
75.75
35.2
79.23
79.45
81.13
Average*↑
1.95
0
0
0
31.32
20.13
40.06
40.57
42.98
Table 2: The performance comparison on the NuInstruct dataset [ 6 ] is conducted against existing SOTA methods. This experiment is crucial for evaluating the performance gains of VGGDrive in cross-view risk object perception ( MAP ), state predictio n, and ego-motion forecasting within autonomous driving scenarios. The symbol * indicates max(4Accuracy+MAP+BLEU−MAE,0).
Dataset
Metrics
Zero-shot Models
Fine-tuned Models
GPT-4o
LLAVA-OV
RoboTron
Baseline
DriveLM
TM-LMM [ 17 ]
OminiDrive
FSDrive [ 47 ]
I-VL4Drive [ 25 ]
VGGDrive
DriveLM
Accuracy ↑
38.55
25.03
52.94
64.35
52.30
59.60
70.00
71.77
73.39
77.50
ChatGPT↑
67.27
65.70
51.11
65.30
55.35
58.44
65.00
63.42
65.25
64.76
Language↑
8.97
14.44
7.12
44.90
45.13
46.38
-
52.77
48.56
56.69
Match ↑
24.00
40.93
35.05
34.54
35.73
35.73
37.00
39.19
47.65
49.77
Average↑
41.21
42.36
39.47
54.59
47.20
51.50
56.00
57.12
60.02
61.26
Table 3: The performance comparison on the DriveLM dataset [ 38 ] is conducted against existing SOTA methods. This experiment is crucial for evaluating the performance gains of VGGDrive in cross-view risk object perception ( Match ), action prediction and planning.
Dataset
Metrics
Zero-shot Models
Fine-tuned Models
GPT-4o
LLAVA-OV
RoboTron
OmniDrive
HERMES [ 49 ]
Baseline
VGGT-Dist
VGGT-Add
VGGDrive
OmniDrive
BLEU↑
10.91
16.14
20.30
38.00
-
37.29
37.28
37.23
37.58
CIDEr↑
24.42
28.41
34.33
68.60
74.10
86.29
86.41
86.38
86.57
ROUGE↑
22.34
22.14
23.67
32.60
32.70
34.33
34.32
34.26
34.40
Average↑
19.22
22.23
26.10
46.40
-
52.64
52.67
52.62
52.85
Table 4: The performance comparison on the OmniDrive dataset [ 41 ] focuses on caption-related tasks in which base VLMs excel.
nuScenes
L2
Collision (%) ↓
Intersection (%) ↓
1s
2s
3s
avg
1s
2s
3s
avg
1s
2s
3s
avg
UniAD [ 10 ]
E2E
0.20
0.42
0.75
0.46
0.02
0.25
0.84
0.37
0.20
1.33
3.24
1.59
VAD-Base [ 19 ]
0.17
0.34
0.60
0.37
0.04
0.27
0.67
0.33
0.21
2.13
5.06
2.47
BEV-Planner [ 30 ]
0.16
0.32
0.57
0.35
0.00
0.29
0.73
0.34
0.35
2.62
6.51
3.16
Baseline [ 30 ]
VLA
0.14
0.34
0.60
0.36
0.06
0.21
0.96
0.41
0.66
2.58
5.43
2.89
OmniDrive-Q [ 41 ]
0.14
0.29
0.55
0.33
0.00
0.13
0.78
0.30
0.56
2.48
5.96
3.00
Table 5: Performance comparison on nuScenes open-loop planning, with metrics from BEV-Planner’s reproduced results [ 30 ] .
Figure 4: Visualization of VGGDrive’s performance across various autonomous driving attribute evaluation tasks.
ID
Methods
NAVSIM
NuInstruct
EP↑
PDMS↑
MAP↑
BLEU↑
Avg↑
Time (s)↓
1
Baseline
81.00
86.04
6.15
75.75
31.32
0.81
2
VGGT
79.56
84.45
3.34
74.69
30.35
0.84
3
VGGT-Dist
81.30
86.68
28.51
79.23
40.06
0.82
4
VGGT-Add
80.84
86.10
30.12
79.45
40.57
0.87
5
VGGT-MHCA
81.94
87.83
32.35
79.39
40.85
0.91
Table 6: Ablation study of various integration schemes between VGGT and VLM on the NAVSIM [ 5 ] and NuInstruct Datasets [ 6 ] . Inference speed is evaluated on the NuInstruct MAP task.
ID
Methods
NAVSIM
NuInstruct
EP↑
PDMS↑
MAP↑
BLEU↑
Avg↑
Time (s)↓
1
Baseline
81.00
86.04
6.15
75.75
31.32
0.81
2
w/o MHCA
82.05
87.79
27.43
77.89
39.29
1.00
3
Share CVGE
82.48
88.05
30.85
79.55
40.95
0.96
4
w/o Residual
82.32
87.86
27.98
78.31
40.02
1.03
5
One-stage SFT
82.83
88.62
36.19
80.75
42.05
1.04
Table 7: Ablation Study of the Main Components of VGGDrive.
NAVSIM
NC↑
DAC↑
EP↑
TTC↑
Comf.↑
PDMS↑
Baseline
97.83
94.08
81.00
94.04
99.98
86.04
Fast3r [ 45 ]
98.34
95.83
82.62
95.58
99.98
88.36
VGGT [ 39 ]
98.55
96.30
82.92
95.59
99.98
88.76
Table 8: Ablation Study of the 3D Expert Model.
NAVSIM
NC↑
DAC↑
EP↑
TTC↑
Comf.↑
PDMS↑
Baseline
97.83
94.08
81.00
94.04
99.98
86.04
w/o Tiimg2lidar
98.33
95.93
82.59
95.42
99.99
88.32
s=8
98.17
96.15
82.66
95.26
99.99
88.36
s=2
98.50
96.16
82.80
95.78
99.99
88.70
OURS
98.55
96.30
82.92
95.59
99.98
88.76
Table 9: Further Ablation Analysis on the Navsim Benchmark.
Dataset
Train Samples
Test Samples
Cross-view perception
Action&States prediction
Trajectory planning
Scene understanding
NAVSIM [ 5 ]
103.3k
12.1k
✓
NuInstruct [ 6 ]
71.8k
16.1k
✓
✓
DriveLM [ 38 ]
376.2k
15.5k
✓
✓
✓
OmniDrive [ 41 ]
318.4k
54.1k
✓
nuScenes [ 30 ]
28.1k
6.0k
✓
Table S1 : Training and Testing Sample Statistics of Five Mainstream Autonomous Driving Datasets Used by VGGDrive, and the Model Capabilities Evaluated on Each Dataset.
Figure S1 : Ablation analysis of closed-loop trajectory planning performance on the NAVSIM dataset when cross-view 3D geometric empowerment and adaptive injection are applied to individual decoding layers of the LLM.
Figure S2 : Qualitative results on the closed-loop trajectory planning task in the Navtest benchmark are presented, showcasing a typical left-turn example to demonstrate the performance of VGGDrive in complex driving scenarios.
Figure S3 : Qualitative results on the closed-loop trajectory planning task in the Navtest benchmark are presented, showcasing a typical right-turn example to demonstrate the performance of VGGDrive in complex driving scenarios.
Figure S4 : Qualitative results on the closed-loop trajectory planning task in the Navtest benchmark are presented, showcasing a typical straight-ahead example to demonstrate the performance of VGGDrive in complex driving scenarios.
Figure S5 : Qualitative results on the open-loop trajectory planning task in the nuScenes benchmark are presented.
Figure S6 : Qualitative results on Action & States prediction in autonomous driving scenarios are presented.
We present Qwen-Drive-1.0, an initial step towards a vision-language foundation model for autonomous driving. Qwen-Drive-1.0 retains the architecture of the pretrained vision-language model (VLM) and integrates 3D perception, visual question answering, and motion planning within a unified framework. An external bird's-eye-view (BEV) perception head jointly performs 3D object detection, semantic occupancy prediction, and BEV map segmentation. It serves as a probe of the 3D information accessible from the shared representations and provides an explicit, inspectable interface to 3D scene structure. A Planning Expert conditions on shared VLM representations to generate future ego trajectories. A staged training recipe combines driving supervision with general-purpose vision-language data to acquire driving-specific competence while helping preserve broad visual understanding and instruction-following capabilities. Experiments demonstrate strong 3D perception and driving scene understanding while largely preserving general vision-language capability. Comprehensive evaluations across open-loop, pseudo-closed-loop, and closed-loop settings further show highly competitive motion-planning performance.
Xin Zhou, Zongchuang Zhao, Zhibo Yang +13
Qwen Team, Huazhong University of Science and Technology
Vision-language-action~(VLA) models have emerged as a promising paradigm for autonomous driving. However, existing VLA models still suffer from a fundamental mismatch: driving actions require precise 3D geometric cues, while visual-language understanding and reasoning are largely conducted in a 2D semantic space. In this paper, we propose GeoCoTDrive, an explicit geometric chain-of-thought framework that grounds geometry in a planning-oriented manner. GeoCoTDrive follows a think with 2D first, drive with dedicated 3D priors paradigm. It first grounds 2D regions corresponding to decision-critical cues, and then retrieves localized 3D priors by sampling features from a geometric foundation model within the grounded regions. These localized geometric features are interleaved into the autoregressive context to support the trajectory generation. To supervise this process, we introduce planning-relevant grounding, a new region-level grounding task that focuses on local spatial cues directly affecting ego planning decisions, and construct the PlanningGrounding dataset to endow VLAs with planning-oriented grounding capability. Experiments across multiple end-to-end autonomous driving benchmarks show that GeoCoTDrive consistently improves safety-critical planning performance, demonstrating the effectiveness of the explicit geometric chain-of-thought process for VLA-based planning.
Xingtai Gui, Yucheng Zhou, Dongqian Guo +3
SKL-IOTSC, Department of AI, University of Macau · Afari Intelligent Drive
The integration of Large Language Models (LLMs) into autonomous driving has attracted growing interest for their strong reasoning and semantic understanding abilities, which are essential for handling complex decision-making and long-tail scenarios. However, existing methods typically feed LLMs with tokens from multi-view and multi-frame images independently, leading to redundant computation and limited spatial consistency. This separation in visual processing hinders accurate 3D spatial reasoning and fails to maintain geometric coherence across views. On the other hand, Bird's-Eye View (BEV) representations learned from geometrically annotated tasks (e.g., object detection) provide spatial structure but lack the semantic richness of foundation vision encoders. To bridge this gap, we propose BEVLM, a framework that connects a spatially consistent and semantically distilled BEV representation with LLMs. Through extensive experiments, we show that BEVLM enables LLMs to reason more effectively in cross-view driving scenes, improving accuracy by 46.0%, by leveraging BEV features as unified inputs. Furthermore, by distilling semantic knowledge from LLMs into BEV representations, BEVLM significantly improves closed-loop end-to-end driving performance in safety-critical scenarios across UniAD and VAD, with gains of up to 28.2%.
Thomas Monninger, Shaoyuan Xie, Qi Alfred Chen +1
Mercedes-Benz Research & Development North America, San Jose, USA · University of California, Irvine, USA