The integration of vision-language models (VLMs) into end-to-end (E2E) autonomous driving has demonstrated strong potential by providing scene description and reasoning capabilities. However, VLMs are computationally expensive, and switching language reasoning on and off at runtime without hurting closed-loop performance has not been demonstrated, which hinders practical adoption. To address these issues, we propose DRIFT, an E2E autonomous driving framework that integrates a lightweight language model (LM) for visual question answering and action commentary. By efficiently projecting intermediate driving representations into the language embedding space, we demonstrate that a compact LM (<150M parameters) can achieve strong scene description and action commentary. Furthermore, we introduce a runtime-flexible planning module that is trained on surrogate reasoning features distilled from the LM and can be conditioned on either the LM or the surrogate reasoning features at inference. This design enables bypassing the LM entirely when required by deployment constraints, without sacrificing driving performance. With only 255M parameters, DRIFT reaches up to a Driving Score (DS) of 97.03 and a Success Rate of 89.39% (mean over three runs) on Bench2Drive by interleaving its LM and surrogate reasoning at a chosen reasoning rate. This is on par with the strongest non-language models while being far more compact than existing VLM-based models. DRIFT further achieves competitive results on the real-world NAVSIM and AlpaSim benchmarks at significantly lower computational cost than existing VLM-based approaches. Project page: https://drift-e2e.github.io/
Figures & tables
Figure 1: Overview of our proposed method architecture (left) and qualitative driving results (right).
Method
Params
Frames
DS ↑
SR (%) ↑
Language-based models
DriveMoE [ 39 ]
3B
234K
74.22
48.64
ETA [ 10 ]
7B
234K
74.33±0.96
48.33±3.56
VLR-Driver [ 17 ]
8.1B
234K
75.01
50.00
ORION [ 8 ]
7B
234K
77.74
54.62
SpaceDrive+ [ 19 ]
7.4B
-
78.02
55.11
Table 1: Closed-loop results on Bench2Drive. We compare DRIFT with language-based and non-language models in Driving Score (DS) and Success Rate (SR). Frames: size of the training set (“-”: not reported). DRIFT results are mean ±std over 3 closed-loop runs (sample standard deviation); DRIFT w/ LM uses a reasoning rate of 2.
Dataset
Model
B-4 ↑
S ↑
C ↑
R-L ↑
M ↑
DriveLM-CARLA
DriveLM-CARLA (LLaMA-7B)
-
72.71
1.15
71.20
-
SimLingo (Qwen2-0.5B)
72.07
87.24
7.08
84.77
50.57
Ours (SmolLM2-135M)
72.63
85.56
6.96
84.59
50.08
Commentary
Ours (SmolLM2-135M)
83.90
82.20
6.93
89.10
57.01
Table 2: Language evaluation results. B-4 = BLEU-4, S = SPICE, C = CIDEr, R-L = ROUGE-L, M = METEOR.
Figure 2: Grad-CAM maps for the LM and surrogate paths in a junction left turn scenario. While the LM maintains broader attention across all critical scene objects, the surrogate path focuses more narrowly on driving-relevant segments.
Figure 3: Effect of the LM reasoning rate (RR) on Bench2Drive. Bars show the mean and error bars the sample standard deviation over 3 closed-loop runs; dots are individual runs. The best setting (RR=2, selected on this benchmark) is highlighted.
DRIFT
Decoupled
Constant answer
Bench2Drive (closed loop)
Longitudinal
93.2
87.2
65.6
Lateral
85.2
85.0
73.9
NAVSIM (open loop)
Longitudinal
79.4
76.0
33.3
Lateral
89.4
88.9
33.3
Table 3: Reasoning–action alignment. Agreement (%) between DRIFT’s generated reasoning and its plan on Bench2Drive (closed loop) and NAVSIM (open loop). Decoupled: same language model trained on the same commentary, but the planner never reads its reasoning. Constant answer: best score from always giving the same answer. Protocol details in the supplementary.
Method
Params (M)
DS ↑
Latency (ms) ↓
FPS ↑
GPU mem. (GiB) ↓
FLOPs (G) ↓
DRIFT w/o LM
120
96.07
35.07
28.52
0.48 †
150.6
DRIFT w/ LM (RR=1)
255
96.03
172.03
5.81
1.69
308.5
SimLingo [ 31 ]
1000
86.81
417.81
2.39
2.89
2338.6
Table 4: Embedded inference efficiency on Jetson AGX Orin (batch size one, MAXN, mixed precision). Latency averages closed-loop inference calls, excluding simulation and communication; memory and FLOPs use fixed-input profiling. DS is from one complete 220-route Bench2Drive evaluation. † LM unloaded. Runtime switching uses 1.69 GiB.
Method
Params
PDMS ↑
EPDMS ↑
AlpaSim Score ↑
FSDrive [ 40 ]
2B
85.1
-
-
AdaThinkDrive [ 25 ]
8B
86.2
-
-
FLARE [ 37 ]
4B
86.9
-
-
SGDrive [ 18 ]
2B
87.4
27.1
-
Qwen-Drive-1.0, SFT [ 43 ]
4B
87.9
30.5
91.7
LTFv6 (reported) [ 27 ]
69M
85.4
28.3
-
Table 5: Real-world benchmarks. PDMS (NAVSIM navtest), EPDMS (NAVSIM navhard) and AlpaSim score (nuPlan track). ‡ : with a test-time heading post-processing (heading set to the tangent of the predicted waypoints). Best in bold, second best underlined, and only methods without direct optimization on NAVSIM score are listed.
Variant
B-4 ↑
S ↑
M ↑
Features
LiDAR only
89.47
94.88
63.47
RGB only
90.51
95.37
64.38
RGB + LiDAR
90.62
95.41
64.48
+ speed
90.80
95.54
64.65
+ speed, tp †
90.87
95.57
64.66
Projector
Cross-attention
88.38
94.31
62.45
Table 6: Ablation on feature selection (top) and projector design (bottom). Evaluated after Stage 2 on the in-distribution LEAD validation split, hence not comparable to Tab. 2 . LiDAR refers to FbevL and RGB to FimgL . † marks the adopted default. B-4 = BLEU-4, S = SPICE, M = METEOR.
Epochs
DS
Variant
S2
S3a
S3b
S
w/o LM
w/ LM
DRIFT-Base
0
31
0
−
95.45±0.26
−
+ VQA pre-training
2
31
0
76.90
95.94±0.29
−
+ Commentary training ( ours )
2
15
16
82.20
95.97±0.65
97.03±0.09
Table 7: Effect of training stages. Epochs of Stage 2 (VQA), 3a (E2E driving) and 3b (driving with commentary training). S: SPICE of the commentary on the LEAD evaluation samples. DS on Bench2Drive; w/ LM uses RR=2.
Figure 4: Perception and reasoning in front of a crossing pedestrian. Top: front camera and GT BEV. Bottom: predicted BEV semantics of each training variant, with the LM’s answer to “What should the ego do next?” below.
Integrating vision-language models (VLMs) into end-to-end (E2E) autonomous driving (AD) systems has shown promise in improving scene understanding. However, existing integration strategies suffer from several limitations: they either struggle to resolve distribution misalignment between reasoning and action spaces, underexploit the general reasoning capabilities of pretrained VLMs, or incur substantial inference latency during action policy generation, which degrades driving performance. To address these challenges, we propose AutoMoT in this work, an end-to-end AD framework that unifies reasoning and action generation within a single vision-language-action (VLA) model. Our approach leverages a mixture-of-transformer (MoT) architecture with joint attention sharing, which preserves the general reasoning capabilities of pre-trained VLMs while enabling efficient fast-slow inference through asynchronous execution at different task frequencies. Extensive experiments on multiple benchmarks, under both open- and closed-loop settings, demonstrate that AutoMoT achieves competitive performance compared to state-of-the-art methods. We further investigate the functional boundary of pre-trained VLMs in AD, examining when AD-tailored fine-tuning is necessary. Our results show that pre-trained VLMs can achieve competitive multi-task scene understanding performance through semantic prompting alone, while fine-tuning remains essential for action-level tasks such as decision-making and trajectory planning. We refer to https://automot-website.github.io/ for the demonstration videos and qualitative results.
Wenhui Huang, Songyan Zhang, Qihang Huang +6
Nanyang Technological University, Singapore · Harvard University, US · Xiaomi EV, China
Vision-language models (VLMs) have emerged as a promising direction for end-to-end autonomous driving (AD) by jointly modeling visual observations, driving context, and language-based reasoning. However, existing VLM-based systems face a trade-off between high-level reasoning and motion planning: large models offer strong semantic understanding but are costly to adapt for precise control, whereas small VLM models can be fine-tuned efficiently but often exhibit weaker reasoning. We propose NaviDriveVLM, a decoupled framework that separates reasoning from action generation using a large-scale Navigator and a lightweight trainable Driver. This design preserves reasoning ability, reduces training cost, and provides an explicit interpretable intermediate representation for downstream planning. Experiments on the nuScenes benchmark show that NaviDriveVLM outperforms large VLM baselines in end-to-end motion planning.
Ximeng Tao, Pardis Taghavi, Dimitar Filev +2
J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · The Department of Engineering Technology and Industrial Distribution Texas A&M University, College Station, TX 77843, USA
End-to-end autonomous driving requires models to understand traffic scenes, infer driving intent, and generate executable motion plans. Recent vision-language-action (VLA) models inherit semantic priors from large-scale vision-language pretraining, yet still face a coupling trade-off: fully shared backbones preserve multimodal interaction but may entangle language reasoning and trajectory prediction, whereas decou pled reasoning-action pipelines reduce task conflict but weaken semantic-motion coupling. We propose VECTOR-DRIVE, a tightly coupled VLA framework built on Qwen2.5-VL-3B. VECTOR-DRIVE keeps all tokens coupled through shared self attention and routes feed-forward computation according to token semantics. Vision and language tokens are processed by a Vision-Language Expert to preserve semantic priors, while target-point, ego-state, and noisy action tokens are routed to a Trajectory Expert for motion-specific computation. On the action-token pathway, a flow-matching planner refines noisy action tokens into future waypoints and speed profiles. This design couples semantic reasoning and motion planning within a single multimodal Transformer while separating task-specific FFN computation. On Bench2Drive, VECTOR-DRIVE achieves 88.91 Driving Score and outperforms representative end-to end and VLA-based baselines. Qualitative results and ablations further validate the benefits of shared attention, semantic-aware expert routing, progressive training, and flow-based action de coding.
Rui Zhao, Jianlin Yu, Zhenhai Gao +2
College of Automotive Engineering, Jilin University · 2The National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University · 3ReeFocus AI Technology