Adapting general Vision-Language Models (VLMs) to Autonomous Driving (AD) is thought to require heavy 3D spatial priors or intrusive architectural overhauls to compensate for a lack of spatial grounding. In this work, we present a counterintuitive pilot study: off-the-shelf general VLMs already exhibit remarkable adaptability across diverse driving tasks, while precise metric depth remains the primary missing link, a bottleneck readily addressable via onboard sensors and depth foundation models. To this end, we introduce the Triplet Incremental Training Framework, a non-intrusive paradigm that seamlessly instills 3D geometric awareness into a general VLM backbone while fully preserving its open-world capabilities. Driven by automatically constructed (image, depth, QA pair) triplets, our framework operates via a unified tri-stream training pipeline: an unfrozen primary VLM stream to adapt to driving tasks, an auxiliary depth stream that guides the visual backbone to implicitly imagine 3D scene geometry via explicit geometric constraints, and a frozen twin VLM stream employing logit-level KL distillation to prevent semantic drift. Crucially, the auxiliary depth branch is discarded during deployment, ensuring the enhanced geometric awareness comes with zero inference overhead. We evaluate our framework across four major benchmarks targeting four diverse capabilities spanning 3D spatial perception, driving-oriented language reasoning, end-to-end planning, and general multimodal understanding. Our approach outperforms state-of-the-art driving VLMs, rivals domain-specific expert models in 3D perception, and successfully retains its original general reasoning capabilities. This demonstrates that Triplet Incremental Training offers a unified, deployment-ready paradigm for transforming general foundation models into robust driving intelligences.
Figures & tables
Figure 1: Comparison of driving VLM paradigms. (a) Task-head tuning paradigms suffer from fragile spatial boundaries without explicit depth supervision. (b) External expert fusion approaches introduce heavy computational and inference overhead. (c) Our Triplet Incremental Training aligns geometric features via a deployment-discarded transient depth branch, while regularizing semantics with a twin VLM. (d) Quantitative comparison demonstrating our state-of-the-art performance.
Figure 2: Overview of the Triplet Incremental Training pipeline. Specifically, the primary trainable VLM is optimized via dual constraints: a transient, deployment-discarded auxiliary depth branch for explicit geometric feature alignment, and a frozen twin VLM branch employing off-policy logit-level KL distillation to prevent semantic drift.
Figure 3: Visual demonstration of our model across multiple modalities and benchmarks.
Metrics
Metric3Dv2 [ 12 ]
UniDepth [ 38 ]
UniDepthV2 [ 37 ]
VGGT [ 44 ]
InternViT
InternViT-ours
δ1↑
84.1
84.6
87.6
60.0
77.2
80.6
Abs Rel ↓
23.6
12.7
14.3
28.9
19.1
16.7
Table 1: Monocular depth estimation probing on nuScenes.
Metric
PETR [ 29 ]
DETR3D [ 51 ]
FCOS3D [ 48 ]
PGD [ 47 ]
OmniDrive [ 46 ]
InternViT
InternViT-ours
mAP ↑
37.8
34.9
34.3
36.9
35.4
34.7
38.1
Table 2: 3D object detection probing on nuScenes.
Dataset
Metric
Zero-shot models
Finetuned models
GPT-4o [ 15 ]
InternVL3.5-4B [ 49 ]
VGGDrive [ 45 ]
Specialist Model
Baseline
Ours-4B
OmniDrive
BLEU ↑
10.91
25.64
37.58
38.00
40.94
42.55
ROUGE ↑
22.34
24.90
34.40
32.60
35.98
36.80
CIDEr ↑
24.42
24.73
86.57
68.60
86.51
92.09
Average ↑
19.22
25.09
52.85
46.40
54.47
57.15
NuInstruct
MAE ↓
9.93
20.56
3.08
9.08
1.67
1.44
Table 3: Driving language reasoning evaluation on OmniDrive and nuInstruct. ”Specialist Model” denotes domain-specific expert networks trained exclusively on target datasets [ 46 , 8 ] , ”Baseline” denotes InternVL3.5-4B optimized via standard SFT. * indicates max((Accuracy+mAP+BLEU−MAE)/4,0) .
Table 5: General multimodal capability preservation after driving adaptation.
Objectives
Perception
Driving Language
Planning
General VQA
Ldepth
LKD
Depth
3D Detection
OmniDrive
nuInstruct
NAVSIM
Avg. ↑
δ1↑
AbsRel ↓
mAP ↑
Avg. ↑
Avg. ↑
PDMS ↑
−
−
77.2
19.1
34.7
54.47
44.83
83.30
21.5
✓
−
80.8
16.3
38.2
56.79
44.90
86.42
21.8
−
✓
77.3
18.9
34.9
56.27
44.83
84.56
71.0
✓
✓
80.6
16.7
38.1
57.15
45.11
89.10
73.2
Table 6: Ablation of geometric supervision and semantic regularization on InternVL3.5-4B. All variants retain driving SFT; checkmarks indicate active auxiliary objectives. The last row is the full framework.
Method
MMMU
MMStar
POPE
OCRBench
ScienceQA
Avg.
InternVL3.5-4B
60.8
60.6
81.2
77.6
87.6
73.6
Ours-4B w/o KL
7.0
4.5
67.1
21.0
9.5
21.8
Ours-4B w/ KL
59.1
61.7
80.9
77.4
87.1
73.2
Δ w/ KL
+52.1
+57.2
+13.8
+56.4
+77.6
+51.4
Table 7: Ablation study on the off-policy logit-level KL distillation constraint. Δ reports gains over w/o KL.
We present Qwen-Drive-1.0, an initial step towards a vision-language foundation model for autonomous driving. Qwen-Drive-1.0 retains the architecture of the pretrained vision-language model (VLM) and integrates 3D perception, visual question answering, and motion planning within a unified framework. An external bird's-eye-view (BEV) perception head jointly performs 3D object detection, semantic occupancy prediction, and BEV map segmentation. It serves as a probe of the 3D information accessible from the shared representations and provides an explicit, inspectable interface to 3D scene structure. A Planning Expert conditions on shared VLM representations to generate future ego trajectories. A staged training recipe combines driving supervision with general-purpose vision-language data to acquire driving-specific competence while helping preserve broad visual understanding and instruction-following capabilities. Experiments demonstrate strong 3D perception and driving scene understanding while largely preserving general vision-language capability. Comprehensive evaluations across open-loop, pseudo-closed-loop, and closed-loop settings further show highly competitive motion-planning performance.
Xin Zhou, Zongchuang Zhao, Zhibo Yang +13
Qwen Team, Huazhong University of Science and Technology
The significance of cross-view 3D geometric modeling capabilities for autonomous driving is self-evident, yet existing Vision-Language Models (VLMs) inherently lack this capability, resulting in their mediocre performance. While some promising approaches attempt to mitigate this by constructing Q&A data for auxiliary training, they still fail to fundamentally equip VLMs with the ability to comprehensively handle diverse evaluation protocols. We thus chart a new course, advocating for the infusion of VLMs with the cross-view geometric grounding of mature 3D foundation models, closing this critical capability gap in autonomous driving. In this spirit, we propose a novel architecture, VGGDrive, which empowers Vision-language models with cross-view Geometric Grounding for autonomous Driving. Concretely, to bridge the cross-view 3D geometric features from the frozen visual 3D model with the VLM's 2D visual features, we introduce a plug-and-play Cross-View 3D Geometric Enabler (CVGE). The CVGE decouples the base VLM architecture and effectively empowers the VLM with 3D features through a hierarchical adaptive injection mechanism. Extensive experiments show that VGGDrive enhances base VLM performance across five autonomous driving benchmarks, including tasks like cross-view risk perception, motion prediction, and trajectory planning. It's our belief that mature 3D foundation models can empower autonomous driving tasks through effective integration, and we hope our initial exploration demonstrates the potential of this paradigm to the autonomous driving community.
Jie Wang, Guang Li, Zhijian Huang +4
College of Intelligence and Computing, Tianjin University · Xiaomi EV
End-to-end autonomous driving methods built on vision language models (VLMs) have undergone rapid development driven by their universal visual understanding and strong reasoning capabilities obtained from the large-scale pretraining. However, we find that current VLMs struggle to understand fine-grained 3D spatial relationships which is a fundamental requirement for systems interacting with the physical world. To address this issue, we propose SpaceDrive, a spatial-aware VLM-based driving framework that treats spatial information as explicit positional encodings (PEs) instead of textual digit tokens, enabling joint reasoning over semantic and spatial representations. SpaceDrive employs a universal positional encoder to all 3D coordinates derived from multi-view depth estimation, historical ego-states, and text prompts. These 3D PEs are first superimposed to augment the corresponding 2D visual tokens. Meanwhile, they serve as a task-agnostic coordinate representation, replacing the digit-wise numerical tokens as both inputs and outputs for the VLM. This mechanism enables the model to better index specific visual semantics in spatial reasoning and directly regress trajectory coordinates rather than generating digit-by-digit, thereby enhancing planning accuracy. Extensive experiments validate that SpaceDrive achieves state-of-the-art open-loop performance on the nuScenes dataset and the second-best Driving Score of 78.02 on the Bench2Drive closed-loop benchmark over existing VLM-based methods. Code is available at: https://github.com/zhenghao2519/SpaceDrive.
Peizheng Li, Zhenghao Zhang, David Holtz +6
Mercedes-Benz AG · TU Munich · Karlsruhe Institute of Technology +4