Organizations: J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · The Department of Engineering Technology and Industrial Distribution Texas A&M University, College Station, TX 77843, USA
Vision-language models (VLMs) have emerged as a promising direction for end-to-end autonomous driving (AD) by jointly modeling visual observations, driving context, and language-based reasoning. However, existing VLM-based systems face a trade-off between high-level reasoning and motion planning: large models offer strong semantic understanding but are costly to adapt for precise control, whereas small VLM models can be fine-tuned efficiently but often exhibit weaker reasoning. We propose NaviDriveVLM, a decoupled framework that separates reasoning from action generation using a large-scale Navigator and a lightweight trainable Driver. This design preserves reasoning ability, reduces training cost, and provides an explicit interpretable intermediate representation for downstream planning. Experiments on the nuScenes benchmark show that NaviDriveVLM outperforms large VLM baselines in end-to-end motion planning.
Figures & tables
Fig. 1: (a) Large-scale VLMs show strong reasoning ability but fail to generate accurate driving actions without fine-tuning. (b) Lightweight VLMs can be fine-tuned for future waypoints prediction, but their reasoning ability degrades. (c) We decouple reasoning and motion planning into two modules, using a large-scale VLM as the Navigator for reasoning and a lightweight VLM as the Driver for waypoint prediction, preserving reasoning while optimizing driving performance.
Fig. 2: Overview of NaviDriveVLM. The framework is decoupled into a large VLM serving as the Navigator and a lightweight VLM serving as the Driver. (a) Multi-view surround images are encoded into visual tokens (blue). The Navigator prompt and ego state are tokenized as text tokens (pink and orange). (b) The Navigator VLM generates reasoning tokens (green), which are concatenated with the front-view image tokens, Driver prompt, and ego state tokens, and then fed into the Driver VLM. (c) The Driver VLM is fine-tuned to predict future waypoints or driving actions. The reasoning tokens can be decoded into text for interpretability.
Fig. 3: Prompt design for the Navi-VLM and Driver-VLM.
Fig. 4: Qualitative results across three different driving scenarios. Predicted waypoints are visualized as blue masks and blue dots, while ground-truth waypoints are shown in green. The minimum average L2 error in meters over 6 seconds is shown in brackets. The first row represents a non-fine-tuned large VLM, which is capable of generating reasonable high-level reasoning outputs. However, the predicted future waypoints deviate significantly from the ground truth. The second row corresponds to a smaller fine-tuned VLM, which can generate accurate future waypoints suitable for control. However, it lacks strong scene understanding and reasoning capabilities. The third row presents our proposed NaviDriveVLM framework, which combines reliable high-level reasoning with accurate future waypoint prediction.
Fig. 5: Additional qualitative results from NaviDriveVLM. Predicted waypoints are visualized as blue masks and blue dots, while ground-truth waypoints are shown in green. The minimum average L2 error in meters over 6 seconds is shown in brackets. Scenes D, E, and F illustrate the reasoning results and the corresponding predicted future waypoints predicted by NaviDriveVLM across three scenarios: waiting at a red traffic light, following another vehicle, and braking.
Model
VLM
L2 (1s)
L2 (2s)
L2 (3s)
Avg. L2 (m) ↓
OpenEMMA [ 25 ]
LLaVa-7B [ 26 ]
1.45
3.21
3.76
2.81
ST-P3 [ 27 ]
-
1.33
2.11
2.90
2.11
GenAD [ 28 ]
-
0.36
0.83
1.55
0.91
Ego-MLP [ 29 ]
-
0.46
0.76
1.12
0.78
VAD-Base [ 30 ]
-
0.41
0.70
1.05
0.72
UniAD [ 2 ]
-
0.44
0.67
0.96
0.69
TABLE I: End-to-end motion planning experiments on nuScenes
Model
Output
L2 (1s)
L2 (2s)
L2 (3s)
L2 (6s)
Avg. L2 (m) ↓
NaviDriveVLM
Waypoint (x,y)
0.200
0.495
0.934
3.245
1.285
Action (α,κ)
0.259
0.571
1.007
2.911
1.201
TABLE II: Comparison of Waypoint and Action Prediction Performance
Model
Reason
Command
Images
L2 (1s)
L2 (2s)
L2 (3s)
L2 (6s)
Avg. L2 (m) ↓
NaviDriveVLM
✓
0.204
0.518
1.029
3.977
1.515
✓
✓
0.200
0.516
1.012
3.861
1.476
✓
✓
0.200
0.496
0.934
3.254
1.288
✓
✓
✓
0.200
0.495
0.934
3.245
1.285
TABLE III: Ablation Study on Input Components of NaviDriveVLM