The integration of vision-language models (VLMs) into end-to-end (E2E) autonomous driving has demonstrated strong potential by providing scene description and reasoning capabilities. However, VLMs are computationally expensive, and switching language reasoning on and off at runtime without hurting closed-loop performance has not been demonstrated, which hinders practical adoption. To address these issues, we propose DRIFT, an E2E autonomous driving framework that integrates a lightweight language model (LM) for visual question answering and action commentary. By efficiently projecting intermediate driving representations into the language embedding space, we demonstrate that a compact LM (<150M parameters) can achieve strong scene description and action commentary. Furthermore, we introduce a runtime-flexible planning module that is trained on surrogate reasoning features distilled from the LM and can be conditioned on either the LM or the surrogate reasoning features at inference. This design enables bypassing the LM entirely when required by deployment constraints, without sacrificing driving performance. With only 255M parameters, DRIFT reaches up to a Driving Score (DS) of 97.03 and a Success Rate of 89.39% (mean over three runs) on Bench2Drive by interleaving its LM and surrogate reasoning at a chosen reasoning rate. This is on par with the strongest non-language models while being far more compact than existing VLM-based models. DRIFT further achieves competitive results on the real-world NAVSIM and AlpaSim benchmarks at significantly lower computational cost than existing VLM-based approaches. Project page: https://drift-e2e.github.io/
Figures & tables
Figure 1: Overview of our proposed method architecture (left) and qualitative driving results (right).
Method
Params
Frames
DS ↑
SR (%) ↑
Language-based models
DriveMoE [ 39 ]
3B
234K
74.22
48.64
ETA [ 10 ]
7B
234K
74.33±0.96
48.33±3.56
VLR-Driver [ 17 ]
8.1B
234K
75.01
50.00
ORION [ 8 ]
7B
234K
77.74
54.62
SpaceDrive+ [ 19 ]
7.4B
-
78.02
55.11
Table 1: Closed-loop results on Bench2Drive. We compare DRIFT with language-based and non-language models in Driving Score (DS) and Success Rate (SR). Frames: size of the training set (“-”: not reported). DRIFT results are mean ±std over 3 closed-loop runs (sample standard deviation); DRIFT w/ LM uses a reasoning rate of 2.
Dataset
Model
B-4 ↑
S ↑
C ↑
R-L ↑
M ↑
DriveLM-CARLA
DriveLM-CARLA (LLaMA-7B)
-
72.71
1.15
71.20
-
SimLingo (Qwen2-0.5B)
72.07
87.24
7.08
84.77
50.57
Ours (SmolLM2-135M)
72.63
85.56
6.96
84.59
50.08
Commentary
Ours (SmolLM2-135M)
83.90
82.20
6.93
89.10
57.01
Table 2: Language evaluation results. B-4 = BLEU-4, S = SPICE, C = CIDEr, R-L = ROUGE-L, M = METEOR.
Figure 2: Grad-CAM maps for the LM and surrogate paths in a junction left turn scenario. While the LM maintains broader attention across all critical scene objects, the surrogate path focuses more narrowly on driving-relevant segments.
Figure 3: Effect of the LM reasoning rate (RR) on Bench2Drive. Bars show the mean and error bars the sample standard deviation over 3 closed-loop runs; dots are individual runs. The best setting (RR=2, selected on this benchmark) is highlighted.
DRIFT
Decoupled
Constant answer
Bench2Drive (closed loop)
Longitudinal
93.2
87.2
65.6
Lateral
85.2
85.0
73.9
NAVSIM (open loop)
Longitudinal
79.4
76.0
33.3
Lateral
89.4
88.9
33.3
Table 3: Reasoning–action alignment. Agreement (%) between DRIFT’s generated reasoning and its plan on Bench2Drive (closed loop) and NAVSIM (open loop). Decoupled: same language model trained on the same commentary, but the planner never reads its reasoning. Constant answer: best score from always giving the same answer. Protocol details in the supplementary.
Method
Params (M)
DS ↑
Latency (ms) ↓
FPS ↑
GPU mem. (GiB) ↓
FLOPs (G) ↓
DRIFT w/o LM
120
96.07
35.07
28.52
0.48 †
150.6
DRIFT w/ LM (RR=1)
255
96.03
172.03
5.81
1.69
308.5
SimLingo [ 31 ]
1000
86.81
417.81
2.39
2.89
2338.6
Table 4: Embedded inference efficiency on Jetson AGX Orin (batch size one, MAXN, mixed precision). Latency averages closed-loop inference calls, excluding simulation and communication; memory and FLOPs use fixed-input profiling. DS is from one complete 220-route Bench2Drive evaluation. † LM unloaded. Runtime switching uses 1.69 GiB.
Method
Params
PDMS ↑
EPDMS ↑
AlpaSim Score ↑
FSDrive [ 40 ]
2B
85.1
-
-
AdaThinkDrive [ 25 ]
8B
86.2
-
-
FLARE [ 37 ]
4B
86.9
-
-
SGDrive [ 18 ]
2B
87.4
27.1
-
Qwen-Drive-1.0, SFT [ 43 ]
4B
87.9
30.5
91.7
LTFv6 (reported) [ 27 ]
69M
85.4
28.3
-
Table 5: Real-world benchmarks. PDMS (NAVSIM navtest), EPDMS (NAVSIM navhard) and AlpaSim score (nuPlan track). ‡ : with a test-time heading post-processing (heading set to the tangent of the predicted waypoints). Best in bold, second best underlined, and only methods without direct optimization on NAVSIM score are listed.
Variant
B-4 ↑
S ↑
M ↑
Features
LiDAR only
89.47
94.88
63.47
RGB only
90.51
95.37
64.38
RGB + LiDAR
90.62
95.41
64.48
+ speed
90.80
95.54
64.65
+ speed, tp †
90.87
95.57
64.66
Projector
Cross-attention
88.38
94.31
62.45
Table 6: Ablation on feature selection (top) and projector design (bottom). Evaluated after Stage 2 on the in-distribution LEAD validation split, hence not comparable to Tab. 2 . LiDAR refers to FbevL and RGB to FimgL . † marks the adopted default. B-4 = BLEU-4, S = SPICE, M = METEOR.
Epochs
DS
Variant
S2
S3a
S3b
S
w/o LM
w/ LM
DRIFT-Base
0
31
0
−
95.45±0.26
−
+ VQA pre-training
2
31
0
76.90
95.94±0.29
−
+ Commentary training ( ours )
2
15
16
82.20
95.97±0.65
97.03±0.09
Table 7: Effect of training stages. Epochs of Stage 2 (VQA), 3a (E2E driving) and 3b (driving with commentary training). S: SPICE of the commentary on the LEAD evaluation samples. DS on Bench2Drive; w/ LM uses RR=2.
Figure 4: Perception and reasoning in front of a crossing pedestrian. Top: front camera and GT BEV. Bottom: predicted BEV semantics of each training variant, with the LM’s answer to “What should the ego do next?” below.
J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · The Department of Engineering Technology and Industrial Distribution Texas A&M University, College Station, TX 77843, USA
College of Automotive Engineering, Jilin University · 2The National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University · 3ReeFocus AI Technology