LAD-Drive: Bridging Language and Trajectory with Action-Aware Diffusion Transformers
Authors: Fabian Schmidt, Karol Fedurko, Markus Enzweiler, Abhinav Valada
Organizations: Institute for Intelligent Systems, Esslingen University of Applied Sciences, Germany. · Department of Computer Science, University of Freiburg, Germany.
While multimodal large language models (MLLMs) provide advanced reasoning for autonomous driving, translating their discrete semantic knowledge into continuous trajectories remains a fundamental challenge. Existing methods often rely on unimodal planning heads that inherently limit their ability to represent multimodal driving behavior. Furthermore, most generative approaches frequently condition on one-hot encoded actions, discarding the nuanced navigational uncertainty critical for complex scenarios. To resolve these limitations, we introduce LAD-Drive, a generative framework that structurally disentangles high-level intention from low-level spatial planning. LAD-Drive employs an action decoder to infer a probabilistic meta-action distribution, establishing an explicit belief state that preserves the nuanced intent typically lost by one-hot encodings. This distribution, fused with the vehicle's kinematic state, conditions an action-aware diffusion decoder that utilizes a truncated denoising process to refine learned motion anchors into safe, kinematically feasible trajectories. Extensive evaluations on the LangAuto benchmark demonstrate that LAD-Drive achieves state-of-the-art results, outperforming competitive baselines by up to 59% in Driving Score while significantly reducing route deviations and collisions. We will publicly release the code and models on https://github.com/iis-esslingen/lad-drive.
Figures & tables
Fig. 1: LAD-Drive disentangles high-level semantic reasoning from low-level spatial planning across three modules. First, the Language module uses a multimodal LLM (AD-MLLM) to synthesize sensor data and navigation instructions into contextualized hidden states. Second, the Action decoder infers a discrete meta-action distribution directly from these states. Finally, the Diffusion decoder utilizes this probabilistic conditioning to iteratively refine noisy priors into kinematically consistent, multimodal trajectories.
Fig. 2: Detailed architecture of LAD-Drive. The framework utilizes an AD-MLLM backbone to process multi-view images, LiDAR point clouds, and navigation instructions into contextualized hidden states. A dedicated action decoder (green) infers a meta-action distribution from these states, which is fused with the vehicle’s ego-status to form a joint state-intent representation. The diffusion decoder (orange) employs a shared transformer-based block to iteratively refine noisy trajectory anchors over two denoising steps, with each step’s output updating the input for the next. This refinement process utilizes sequential multi-head cross-attention (MHCA) layers to ground trajectory generation in both the global visual-language context and the predicted probabilistic intent, outputting refined trajectories and their corresponding confidence scores.
Fig. 3: Anchor trajectories generated via k-means clustering on training trajectories to provide motion priors for the initialization of the truncated diffusion process.
Method
Venue
Tiny
Short
Long
Mean
DS ↑
RC ↑
IS ↑
DS ↑
RC ↑
IS ↑
DS ↑
RC ↑
IS ↑
DS ↑
RC ↑
IS ↑
LMDrive [ 6 ] (Reported)
CVPR’24
66.5
77.9
0.85
50.6
60.0
0.84
36.2
46.6
0.81
51.1
61.5
0.83
LMDrive [ 6 ] (Checkpoint)
CVPR’24
60.7
71.0
0.82
41.3
57.0
0.79
26.8
36.5
0.77
42.9
54.8
0.79
AD-H [ 15 ]
arXiv’24
77.5
85.1
0.91
56.1
68.0
0.78
44.0
53.2
0.83
59.2
68.8
0.84
BEVDriver [ 7 ]
IROS’25
70.2
81.3
0.87
66.7
77.8
0.87
48.9
59.7
0.82
61.9
72.9
0.85
SToRM [ 27 ]
arXiv’26
78.8
86.9
0.92
64.5
74.7
0.88
44.2
56.8
0.82
62.5
72.8
0.87
TABLE I: LangAuto Performance. Best results highlighted in bold.
Method
DS ↑
RC ↑
IS ↑
CP ↓
CV ↓
CL ↓
RL ↓
SS ↓
Off ↓
RD ↓
TO ↓
AB ↓
DP ↓
Latency ↓
LMDrive
42.9
54.8
0.80
0.08
2.83
4.28
2.31
0.00
3.79
11.95
2.93
1.63
33.61
40.15
LAD-Drive (Ours)
68.2
74.5
0.90
0.02
0.67
1.20
1.60
0.11
1.10
2.31
2.68
2.48
32.13
47.04
TABLE II: Detailed Infraction and Efficiency Analysis, including mean total model latency [ms] and decoder parameter count [M].
Fig. 4: Qualitative comparison between LMDrive (top row) and LAD-Drive (bottom row). For LAD-Drive, the generated multimodal trajectories are visualized using a heat map color scheme, with paths ranging from white (low confidence) to red (high confidence). The predicted meta-action distribution from the action decoder is displayed in the top right of each LAD-Drive panel (L: Left, CLL: Change Lane Left, S: Straight, LF: Lane Follow, CLR: Change Lane Right, R: Right). Column 1 (Route Deviation): Despite the instruction to make a left turn, LMDrive turns right. LAD-Drive executes the correct intention while safely avoiding a cyclist. Column 2 (Off-Road & Collision Layout): LMDrive fails to interpret the road layout and instruction, driving off-road into a fence, whereas LAD-Drive successfully continues straight. Columns 3 & 4 (Collision Vehicle): In the third scenario, LMDrive collides head-on with a truck, while LAD-Drive safely navigates the lane. In the fourth scenario, LMDrive initiates an unsafe left lane change into traffic, ignoring the right-turn navigation instruction. In contrast, LAD-Drive effectively handles dynamic agents and remains safely on the road, demonstrating robustness even when the action decoder exhibits slight uncertainty.
Recent advancements in Language Models (LMs) have demonstrated strong semantic reasoning capabilities, enabling their application in high-level decision-making for autonomous driving (AD). However, LMs operate over discrete token spaces and lack the ability to generate continuous, physically feasible trajectories required for motion planning. Meanwhile, diffusion models have proven effective at generating reliable and dynamically consistent trajectories, but often lack semantic interpretability and alignment with scene-level understanding. To address these limitations, we propose KnowDiffuser, a knowledge-guided motion planning framework that tightly integrates the semantic understanding of language models with the generative power of diffusion models. The framework employs a language model to infer context-aware meta-actions from structured scene representations, which are then mapped to prior trajectories that anchor the subsequent denoising process. A two-stage truncated denoising mechanism refines these trajectories efficiently, preserving both semantic alignment and physical feasibility. Experiments on the nuPlan benchmark demonstrate that KnowDiffuser significantly outperforms existing planners in both open-loop and closed-loop evaluations, establishing a robust and interpretable framework that effectively bridges the semantic-to-physical gap in AD systems.
Fan Ding, Xuewen Luo, Fengze Yang +4
Monash University Malaysia. · University of Utah, USA.
Vision-Language Models(VLMs) excel at autoregressive text generation, yet end-to-end autonomous driving requires multi-task learning with structured outputs and heterogeneous decoding behaviors, such as autoregressive language generation, parallel object detection and trajectory regression. To accommodate these differences, existing systems typically introduce separate or cascaded decoders, resulting in architectural fragmentation and limited backbone reuse. In this work, we present a unified autonomous driving framework built upon a pretrained VLM, where heterogeneous decoding behaviors are reconciled within a single transformer decoder. We demonstrate that pretrained VLM attention exhibits strong transferability beyond pure language modeling. By organizing visual and structured query tokens within a single causal decoder, structured queries can naturally condition on visual context through the original attention mechanism. Textual and structured outputs share a common attention backbone, enabling stable joint optimization across heterogeneous tasks. Trajectory planning is realized within the same causal LLM decoder by introducing structured trajectory queries. This unified formulation enables planning to share the pretrained attention backbone with images and perception tokens. Extensive experiments on end-to-end autonomous driving benchmarks demonstrate state-of-the-art performance, including 0.28 L2 and 0.18 collision rate on nuScenes open-loop evaluation and competitive results (86.8 PDMS) on NAVSIM closed-loop evaluation. The full model preserves multi-modal generation capability, while an efficient inference mode achieves approximately 40% lower latency. Code and models are available at https://github.com/Z1zyw/OneDrive
Yiwei Zhang, Xuesong Chen, Jin Gao +5
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, University of Chinese Academy of Sciences · AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University +2
End-to-end autonomous driving must balance multimodal maneuver generation against real-time inference constraints. Diffusion planners capture diverse behaviors, but their iterative denoising incurs considerable inference cost in safety-critical deployment, while one-step alternatives still imitate only the single expert demonstration, so neither family certifies both the diversity and the quality of its outputs. To address this, we propose Vault, a framework that couples one-step latent generation with sample-based reinforcement learning guidance. Built on a V-JEPA 2.1 visual representation pretrained on the full NAVSIM training imagery, Vault generates trajectory candidates in a single forward pass by drifting in a VAE latent space. During training, a reward-gated positive pool collects the maximum-scoring trajectories that the model itself discovers under the official evaluator and anchors the drift targets to these certified samples. This closes a reinforcement-learning loop purely through sample-based target construction, without policy gradients, learned reward models, or extra interaction: the generator imitates the frontier of its own certified successes, so quality and multimodality improve together under a single fixed sampling configuration. A learned scorer that predicts the official score and its sub-metrics then selects the executed trajectory, and all training-only modules are discarded at inference. On the NAVSIM benchmarks, Vault achieves state-of-the-art performance at real-time inference cost, reaching 94.6 PDMS on v1 and 91.2 EPDMS on v2.