cs.CVNov 12, 2026

From Foundation VLM to Versatile Driving VLM: A Triplet Incremental Training Approach

Authors: Shengtao Wu, Zichao Dong, Weixuan Qiao, Xiaolu Liu, Xin Zhan, Junbo Chen, Jianke Zhu

Organizations: Zhejiang University

Abstract

Adapting general Vision-Language Models (VLMs) to Autonomous Driving (AD) is thought to require heavy 3D spatial priors or intrusive architectural overhauls to compensate for a lack of spatial grounding. In this work, we present a counterintuitive pilot study: off-the-shelf general VLMs already exhibit remarkable adaptability across diverse driving tasks, while precise metric depth remains the primary missing link, a bottleneck readily addressable via onboard sensors and depth foundation models. To this end, we introduce the Triplet Incremental Training Framework, a non-intrusive paradigm that seamlessly instills 3D geometric awareness into a general VLM backbone while fully preserving its open-world capabilities. Driven by automatically constructed (image, depth, QA pair) triplets, our framework operates via a unified tri-stream training pipeline: an unfrozen primary VLM stream to adapt to driving tasks, an auxiliary depth stream that guides the visual backbone to implicitly imagine 3D scene geometry via explicit geometric constraints, and a frozen twin VLM stream employing logit-level KL distillation to prevent semantic drift. Crucially, the auxiliary depth branch is discarded during deployment, ensuring the enhanced geometric awareness comes with zero inference overhead. We evaluate our framework across four major benchmarks targeting four diverse capabilities spanning 3D spatial perception, driving-oriented language reasoning, end-to-end planning, and general multimodal understanding. Our approach outperforms state-of-the-art driving VLMs, rivals domain-specific expert models in 3D perception, and successfully retains its original general reasoning capabilities. This demonstrates that Triplet Incremental Training offers a unified, deployment-ready paradigm for transforming general foundation models into robust driving intelligences.

Figures & tables

Explore similar work

CardsList
  1. Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving

    Aug 31, 2026Xin Zhou, Zongchuang Zhao, Zhibo Yang +13VLMs for Autonomous DrivingAutonomous Driving Perception

  2. VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Driving

    Feb 24, 2026Jie Wang, Guang Li, Zhijian Huang +4Multi-View Feature FusionVLMs for Autonomous Driving

  3. SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving

    Dec 11, 2025Peizheng Li, Zhenghao Zhang, David Holtz +6Vision-Language ModelsVLMs for Autonomous Driving