Adapting general Vision-Language Models (VLMs) to Autonomous Driving (AD) is thought to require heavy 3D spatial priors or intrusive architectural overhauls to compensate for a lack of spatial grounding. In this work, we present a counterintuitive pilot study: off-the-shelf general VLMs already exhibit remarkable adaptability across diverse driving tasks, while precise metric depth remains the primary missing link, a bottleneck readily addressable via onboard sensors and depth foundation models. To this end, we introduce the Triplet Incremental Training Framework, a non-intrusive paradigm that seamlessly instills 3D geometric awareness into a general VLM backbone while fully preserving its open-world capabilities. Driven by automatically constructed (image, depth, QA pair) triplets, our framework operates via a unified tri-stream training pipeline: an unfrozen primary VLM stream to adapt to driving tasks, an auxiliary depth stream that guides the visual backbone to implicitly imagine 3D scene geometry via explicit geometric constraints, and a frozen twin VLM stream employing logit-level KL distillation to prevent semantic drift. Crucially, the auxiliary depth branch is discarded during deployment, ensuring the enhanced geometric awareness comes with zero inference overhead. We evaluate our framework across four major benchmarks targeting four diverse capabilities spanning 3D spatial perception, driving-oriented language reasoning, end-to-end planning, and general multimodal understanding. Our approach outperforms state-of-the-art driving VLMs, rivals domain-specific expert models in 3D perception, and successfully retains its original general reasoning capabilities. This demonstrates that Triplet Incremental Training offers a unified, deployment-ready paradigm for transforming general foundation models into robust driving intelligences.
Figures & tables
Figure 1: Comparison of driving VLM paradigms. (a) Task-head tuning paradigms suffer from fragile spatial boundaries without explicit depth supervision. (b) External expert fusion approaches introduce heavy computational and inference overhead. (c) Our Triplet Incremental Training aligns geometric features via a deployment-discarded transient depth branch, while regularizing semantics with a twin VLM. (d) Quantitative comparison demonstrating our state-of-the-art performance.
Figure 2: Overview of the Triplet Incremental Training pipeline. Specifically, the primary trainable VLM is optimized via dual constraints: a transient, deployment-discarded auxiliary depth branch for explicit geometric feature alignment, and a frozen twin VLM branch employing off-policy logit-level KL distillation to prevent semantic drift.
Figure 3: Visual demonstration of our model across multiple modalities and benchmarks.
Metrics
Metric3Dv2 [ 12 ]
UniDepth [ 38 ]
UniDepthV2 [ 37 ]
VGGT [ 44 ]
InternViT
InternViT-ours
δ1↑
84.1
84.6
87.6
60.0
77.2
80.6
Abs Rel ↓
23.6
12.7
14.3
28.9
19.1
16.7
Table 1: Monocular depth estimation probing on nuScenes.
Metric
PETR [ 29 ]
DETR3D [ 51 ]
FCOS3D [ 48 ]
PGD [ 47 ]
OmniDrive [ 46 ]
InternViT
InternViT-ours
mAP ↑
37.8
34.9
34.3
36.9
35.4
34.7
38.1
Table 2: 3D object detection probing on nuScenes.
Dataset
Metric
Zero-shot models
Finetuned models
GPT-4o [ 15 ]
InternVL3.5-4B [ 49 ]
VGGDrive [ 45 ]
Specialist Model
Baseline
Ours-4B
OmniDrive
BLEU ↑
10.91
25.64
37.58
38.00
40.94
42.55
ROUGE ↑
22.34
24.90
34.40
32.60
35.98
36.80
CIDEr ↑
24.42
24.73
86.57
68.60
86.51
92.09
Average ↑
19.22
25.09
52.85
46.40
54.47
57.15
NuInstruct
MAE ↓
9.93
20.56
3.08
9.08
1.67
1.44
Table 3: Driving language reasoning evaluation on OmniDrive and nuInstruct. ”Specialist Model” denotes domain-specific expert networks trained exclusively on target datasets [ 46 , 8 ] , ”Baseline” denotes InternVL3.5-4B optimized via standard SFT. * indicates max((Accuracy+mAP+BLEU−MAE)/4,0) .
Table 5: General multimodal capability preservation after driving adaptation.
Objectives
Perception
Driving Language
Planning
General VQA
Ldepth
LKD
Depth
3D Detection
OmniDrive
nuInstruct
NAVSIM
Avg. ↑
δ1↑
AbsRel ↓
mAP ↑
Avg. ↑
Avg. ↑
PDMS ↑
−
−
77.2
19.1
34.7
54.47
44.83
83.30
21.5
✓
−
80.8
16.3
38.2
56.79
44.90
86.42
21.8
−
✓
77.3
18.9
34.9
56.27
44.83
84.56
71.0
✓
✓
80.6
16.7
38.1
57.15
45.11
89.10
73.2
Table 6: Ablation of geometric supervision and semantic regularization on InternVL3.5-4B. All variants retain driving SFT; checkmarks indicate active auxiliary objectives. The last row is the full framework.
Method
MMMU
MMStar
POPE
OCRBench
ScienceQA
Avg.
InternVL3.5-4B
60.8
60.6
81.2
77.6
87.6
73.6
Ours-4B w/o KL
7.0
4.5
67.1
21.0
9.5
21.8
Ours-4B w/ KL
59.1
61.7
80.9
77.4
87.1
73.2
Δ w/ KL
+52.1
+57.2
+13.8
+56.4
+77.6
+51.4
Table 7: Ablation study on the off-policy logit-level KL distillation constraint. Δ reports gains over w/o KL.