Bird's-Eye-View (BEV)-conditioned large language models (LLMs) enable language-interactive driving, but onboard deployment requires efficient perception and continuous planning alongside autoregressive language generation. We present GridDriver, an efficient BEV-conditioned LLM framework combining trajectory prediction, structured driving indicators, natural-language navigation, driving explanations, and scene question answering. Lightweight spatio-temporal BEV perception and spatially structured tokens provide scene context to a shared LLM, while frame-wise scheduling interleaves trajectory prediction with asynchronous language generation. GridDriver achieves competitive performance on public benchmarks, reaching 87.9% PDMS on NAVSIM and 57.6% accuracy on nuScenes-QA with a shared planning and QA checkpoint. Spatial representation analyses demonstrate improved geometric information retention and trajectory prediction compared with Q-Former compression. The system achieves 32.7 ms standalone planning latency and 9.4 plans/s in an onboard ROS 2 pipeline. We further evaluate planning on our own data recordings and demonstrate open-loop onboard operation on a research vehicle. Across public benchmarks and real-world evaluations, we identify strengths and limitations of the BEV-based approach.
Figures & tables
Fig. 1 : GridDriver combines spatial BEV tokens and natural-language navigation in an LLM for trajectory planning and driving explanations, with an onboard implementation evaluated on a research vehicle.
Fig. 2 : Framework architecture. BEVFormer-tiny [ 8 ] encodes multi-camera images and CAN state into temporally aligned BEV features, augmented with traffic-light predictions. BEVFormer’s DETR-based 3D detection head and our drivable-area head support perception training. Grid tokens and navigation instructions are passed to Qwen3-1.7B, optionally with explicit ego-state. The LLM generates explanations, while the planning-token is decoded into future waypoints and indicators.
Method
Planner Ego
avg. L2 ↓
avg. Col. ↓
FSDrive ∗ [ 14 ]
0.53
0.17
DrivePI ∗ [ 6 ]
0.49
0.38
FSDrive ∗ [ 14 ]
✓
0.28
0.10
AutoVLA (CoT) ∗ [ 2 ]
✓
0.40
0.20
RDADriver [ 9 ]
✓
0.40
0.10
DrivePI ∗ [ 6 ]
✓
0.40
0.11
TABLE I : Open-loop nuScenes planning (mean 1–3s L2 and collision rate). Planner Ego denotes explicit ego-state input to the planner; GridDriver retains BEVFormer CAN conditioning. External scores may differ in evaluation protocol; ∗ marks unverified ST-P3 variants. GridDriver: cmd: standard navigation command, NL: natural-language instruction.
Method
NC
DAC
TTC
EP
C
PDMS
Constant Velocity
68.0
57.8
50.0
19.4
100.0
20.7
DiffusionDrive [ 26 ]
98.2
96.2
94.7
82.2
100.0
88.1
FSDrive [ 14 ]
98.2
93.8
93.3
80.1
99.9
85.1
AutoVLA [ 2 ]
96.9
92.4
88.1
75.8
99.9
80.5
ReCogDrive-IL [ 16 ]
98.1
94.7
94.2
80.9
100.0
86.5
SGDrive (SFT) [ 11 ]
98.6
95.1
95.4
81.2
100.0
87.4
TABLE II : NAVSIM v1.1 navtest PDMS (% ↑ ). NC: no at-fault collisions; DAC: drivable area compliance; TTC: time to collision; EP: ego progress; C: comfort. GridDriver: cmd: standard navigation command, NL: natural-language instruction. PDMS cmd std.: 0.15 pp. PDMS NL std.: 0.06 pp
Method
NC
DAC
DDC
TLC
EP
TTC
LK
HC
EC
EPDMS
Constant velocity
88.9/83.2
42.9/59.1
70.7/76.5
99.3/98.1
77.5/71.4
87.3/81.1
78.7/48.0
97.1/97.2
60.4/62.0
11.5
DiffusionDrive [ 26 ]
96.8/80.1
86.0/72.8
98.8/84.4
99.3/98.4
84.0/85.9
95.8/76.6
96.7/46.4
97.6/96.3
79.6/72.8
27.5
SimWAM [ 27 ]
98.0/81.8
92.0/78.6
99.7/87.3
99.6/98.4
83.8/86.3
96.2/78.6
97.3/49.5
97.8/96.3
71.6/69.9
37.6
DriveVLA-W0 [ 12 ]
96.8/76.8
83.3/64.3
99.0/79.9
99.6/98.3
84.6/89.2
95.3/75.0
96.4/46.8
97.6/95.8
78.2/53.1
24.4
SGDrive-IL [ 11 ]
95.8/79.4
87.6/65.4
97.8/79.1
99.8/98.9
84.4/88.9
94.7/75.3
92.9/42.7
97.8/96.4
28.9/29.6
25.5
ReCogDrive [ 16 ] -IL
96.4/80.2
78.9/65.0
98.7/82.4
99.8/98.7
82.6/85.2
95.6/76.9
94.4/43.8
97.6/96.6
74.2/71.8
25.7
TABLE III : NAVSIM v2 navhard two-stage EPDMS (% ↑ ). Each component is Stage 1/2; NC: no at-fault collisions, DAC: drivable area compliance, DDC: driving direction compliance, TLC: traffic light compliance, EP: ego progress, TTC: time to collision, LK: lane keeping, HC: history comfort, EC: extended comfort. EPDMS std.: 1.1 pp.
Method
P
Ext.
Obj.
Sts.
Cnt.
Cmp.
Acc.
OpenDriveVLA-0.5B [ 4 ]
✗
83.9
50.2
57.0
22.0
68.4
58.4
OpenDriveVLA-3B [ 4 ]
✗
84.0
50.3
56.9
22.3
68.5
58.5
DriveIK-0.5B [ 15 ]
✓
83.7
47.1
52.8
19.5
67.3
56.4
DrivePI [ 6 ]
✓
85.3
57.5
59.1
22.4
68.3
60.7
GridDriver (ours) (NL)
✓
83.3
51.4
54.6
20.6
66.9
57.6
TABLE IV : nuScenes-QA val. accuracy (%). Ext.: existence, Obj.: object, Sts.: status, Cnt.: counting, Cmp.: comparison, P: same checkpoint for QA and planning. Accuracy std.: 0.6.
Fig. 3 : Qualitative examples. Interface showing model predictions across three driving scenarios (anonymized).
Fig. 4 : Model predictions on (a) different navigation instructions and (b) injected counterfactual reasoning on nuScenes.
With ego
No ego
Language intervention
N
Δd
Δy
Δd
Δy
Explanation edits
Remove sentence
5,119
-0.10
+0.01
-0.24
0.00
Generic text while moving
2,691
+0.07
0.00
+0.14
-0.01
while stopped
879
+0.96
-0.01
+1.10
+0.01
Hazard: Insert pedestrian warning
2,691
-0.48
+0.05
-0.75
+0.06
TABLE V : Language intervention effects on nuScenes 3s plans (m). N : frames; Δd : path-length change; Δy : lateral shift (turn-aligned for instruction edits). Bold indicates statistical significance at the 95% confidence level. With/no ego: with or without explicit ego-state input.
Vehicle occupancy
Drivable area
Stage
Q-Former
Grid
Q-Former
Grid
BEV features
0.35 ± 0.00
0.60 ± 0.00
LLM input (tokens)
0.10 ± 0.01
0.37 ± 0.00
0.32 ± 0.00
0.62 ± 0.00
LLM layer 14 (tokens)
0.04 ± 0.00
0.31 ± 0.02
0.19 ± 0.01
0.59 ± 0.01
LLM layer 28 (tokens)
0.02 ± 0.01
0.18 ± 0.01
0.14 ± 0.02
0.52 ± 0.01
Plan token (layer 28)
0.03 ± 0.01
0.08 ± 0.00
0.15 ± 0.01
0.30 ± 0.01
TABLE VI : Scene-disjoint linear-probe R2 for vehicle occupancy and drivable area (mean ± std., 3 seeds). Comparison of Q-Former and spatial grid representations across LLM layers.
Fig. 5 : Illustration of frame-wise scheduling of trajectory prediction and asynchronous language generation across two GPUs on our research vehicle (not to scale).
Config.
Enc.
LLM
Plan
1st tok.
Sent.
LoRA merged
38.4
17.6
56.0
56.1
168
+ FP16
22.5
16.4
38.9
39.0
156
+ CUDA graphs
17.2
15.5
32.7
32.8
153
TABLE VII : Inference latency (ms) on RTX 4090. Median values except full-sentence mean.
Fig. 6 : Live urban driving showing onboard visualization and planned trajectories. Generated reasoning below each image.
Vision-Language Models(VLMs) excel at autoregressive text generation, yet end-to-end autonomous driving requires multi-task learning with structured outputs and heterogeneous decoding behaviors, such as autoregressive language generation, parallel object detection and trajectory regression. To accommodate these differences, existing systems typically introduce separate or cascaded decoders, resulting in architectural fragmentation and limited backbone reuse. In this work, we present a unified autonomous driving framework built upon a pretrained VLM, where heterogeneous decoding behaviors are reconciled within a single transformer decoder. We demonstrate that pretrained VLM attention exhibits strong transferability beyond pure language modeling. By organizing visual and structured query tokens within a single causal decoder, structured queries can naturally condition on visual context through the original attention mechanism. Textual and structured outputs share a common attention backbone, enabling stable joint optimization across heterogeneous tasks. Trajectory planning is realized within the same causal LLM decoder by introducing structured trajectory queries. This unified formulation enables planning to share the pretrained attention backbone with images and perception tokens. Extensive experiments on end-to-end autonomous driving benchmarks demonstrate state-of-the-art performance, including 0.28 L2 and 0.18 collision rate on nuScenes open-loop evaluation and competitive results (86.8 PDMS) on NAVSIM closed-loop evaluation. The full model preserves multi-modal generation capability, while an efficient inference mode achieves approximately 40% lower latency. Code and models are available at https://github.com/Z1zyw/OneDrive
Yiwei Zhang, Xuesong Chen, Jin Gao +5
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, University of Chinese Academy of Sciences · AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University +2
We present Qwen-Drive-1.0, an initial step towards a vision-language foundation model for autonomous driving. Qwen-Drive-1.0 retains the architecture of the pretrained vision-language model (VLM) and integrates 3D perception, visual question answering, and motion planning within a unified framework. An external bird's-eye-view (BEV) perception head jointly performs 3D object detection, semantic occupancy prediction, and BEV map segmentation. It serves as a probe of the 3D information accessible from the shared representations and provides an explicit, inspectable interface to 3D scene structure. A Planning Expert conditions on shared VLM representations to generate future ego trajectories. A staged training recipe combines driving supervision with general-purpose vision-language data to acquire driving-specific competence while helping preserve broad visual understanding and instruction-following capabilities. Experiments demonstrate strong 3D perception and driving scene understanding while largely preserving general vision-language capability. Comprehensive evaluations across open-loop, pseudo-closed-loop, and closed-loop settings further show highly competitive motion-planning performance.
Xin Zhou, Zongchuang Zhao, Zhibo Yang +13
Qwen Team, Huazhong University of Science and Technology
Vision-language models (VLMs) have emerged as a promising direction for end-to-end autonomous driving (AD) by jointly modeling visual observations, driving context, and language-based reasoning. However, existing VLM-based systems face a trade-off between high-level reasoning and motion planning: large models offer strong semantic understanding but are costly to adapt for precise control, whereas small VLM models can be fine-tuned efficiently but often exhibit weaker reasoning. We propose NaviDriveVLM, a decoupled framework that separates reasoning from action generation using a large-scale Navigator and a lightweight trainable Driver. This design preserves reasoning ability, reduces training cost, and provides an explicit interpretable intermediate representation for downstream planning. Experiments on the nuScenes benchmark show that NaviDriveVLM outperforms large VLM baselines in end-to-end motion planning.
Ximeng Tao, Pardis Taghavi, Dimitar Filev +2
J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · The Department of Engineering Technology and Industrial Distribution Texas A&M University, College Station, TX 77843, USA