LAD-Drive: Bridging Language and Trajectory with Action-Aware Diffusion Transformers

Fabian Schmidt; Karol Fedurko; Markus Enzweiler; Abhinav Valada

cs.ROMar 2, 2026

LAD-Drive: Bridging Language and Trajectory with Action-Aware Diffusion Transformers

Authors: Fabian Schmidt, Karol Fedurko, Markus Enzweiler, Abhinav Valada

Organizations: Institute for Intelligent Systems, Esslingen University of Applied Sciences, Germany. · Department of Computer Science, University of Freiburg, Germany.

Abstract

While multimodal large language models (MLLMs) provide advanced reasoning for autonomous driving, translating their discrete semantic knowledge into continuous trajectories remains a fundamental challenge. Existing methods often rely on unimodal planning heads that inherently limit their ability to represent multimodal driving behavior. Furthermore, most generative approaches frequently condition on one-hot encoded actions, discarding the nuanced navigational uncertainty critical for complex scenarios. To resolve these limitations, we introduce LAD-Drive, a generative framework that structurally disentangles high-level intention from low-level spatial planning. LAD-Drive employs an action decoder to infer a probabilistic meta-action distribution, establishing an explicit belief state that preserves the nuanced intent typically lost by one-hot encodings. This distribution, fused with the vehicle's kinematic state, conditions an action-aware diffusion decoder that utilizes a truncated denoising process to refine learned motion anchors into safe, kinematically feasible trajectories. Extensive evaluations on the LangAuto benchmark demonstrate that LAD-Drive achieves state-of-the-art results, outperforming competitive baselines by up to 59% in Driving Score while significantly reducing route deviations and collisions. We will publicly release the code and models on https://github.com/iis-esslingen/lad-drive.

Figures & tables

Fig. 1: LAD-Drive disentangles high-level semantic reasoning from low-level spatial planning across three modules. First, the Language module uses a multimodal LLM (AD-MLLM) to synthesize sensor data and navigation instructions into contextualized hidden states. Second, the Action decoder infers a discrete meta-action distribution directly from these states. Finally, the Diffusion decoder utilizes this probabilistic conditioning to iteratively refine noisy priors into kinematically consistent, multimodal trajectories.

Fig. 2: Detailed architecture of LAD-Drive. The framework utilizes an AD-MLLM backbone to process multi-view images, LiDAR point clouds, and navigation instructions into contextualized hidden states. A dedicated action decoder (green) infers a meta-action distribution from these states, which is fused with the vehicle’s ego-status to form a joint state-intent representation. The diffusion decoder (orange) employs a shared transformer-based block to iteratively refine noisy trajectory anchors over two denoising steps, with each step’s output updating the input for the next. This refinement process utilizes sequential multi-head cross-attention (MHCA) layers to ground trajectory generation in both the global visual-language context and the predicted probabilistic intent, outputting refined trajectories and their corresponding confidence scores.

Fig. 3: Anchor trajectories generated via k-means clustering on training trajectories to provide motion priors for the initialization of the truncated diffusion process.

Method	Venue	Tiny			Short			Long			Mean
		DS $\uparrow$	RC $\uparrow$	IS $\uparrow$	DS $\uparrow$	RC $\uparrow$	IS $\uparrow$	DS $\uparrow$	RC $\uparrow$	IS $\uparrow$	DS $\uparrow$	RC $\uparrow$	IS $\uparrow$
LMDrive [ 6 ] (Reported)	CVPR’24	66.5	77.9	0.85	50.6	60.0	0.84	36.2	46.6	0.81	51.1	61.5	0.83
LMDrive [ 6 ] (Checkpoint)	CVPR’24	60.7	71.0	0.82	41.3	57.0	0.79	26.8	36.5	0.77	42.9	54.8	0.79
AD-H [ 15 ]	arXiv’24	77.5	85.1	0.91	56.1	68.0	0.78	44.0	53.2	0.83	59.2	68.8	0.84
BEVDriver [ 7 ]	IROS’25	70.2	81.3	0.87	66.7	77.8	0.87	48.9	59.7	0.82	61.9	72.9	0.85
SToRM [ 27 ]	arXiv’26	78.8	86.9	0.92	64.5	74.7	0.88	44.2	56.8	0.82	62.5	72.8	0.87

TABLE I: LangAuto Performance. Best results highlighted in bold.

Method	DS $\uparrow$	RC $\uparrow$	IS $\uparrow$	CP $\downarrow$	CV $\downarrow$	CL $\downarrow$	RL $\downarrow$	SS $\downarrow$	Off $\downarrow$	RD $\downarrow$	TO $\downarrow$	AB $\downarrow$	DP $\downarrow$	Latency $\downarrow$
LMDrive	42.9	54.8	0.80	0.08	2.83	4.28	2.31	0.00	3.79	11.95	2.93	1.63	33.61	40.15
LAD-Drive (Ours)	68.2	74.5	0.90	0.02	0.67	1.20	1.60	0.11	1.10	2.31	2.68	2.48	32.13	47.04

TABLE II: Detailed Infraction and Efficiency Analysis, including mean total model latency [ms] and decoder parameter count [M].

Fig. 4: Qualitative comparison between LMDrive (top row) and LAD-Drive (bottom row). For LAD-Drive, the generated multimodal trajectories are visualized using a heat map color scheme, with paths ranging from white (low confidence) to red (high confidence). The predicted meta-action distribution from the action decoder is displayed in the top right of each LAD-Drive panel (L: Left, CLL: Change Lane Left, S: Straight, LF: Lane Follow, CLR: Change Lane Right, R: Right). Column 1 (Route Deviation): Despite the instruction to make a left turn, LMDrive turns right. LAD-Drive executes the correct intention while safely avoiding a cyclist. Column 2 (Off-Road & Collision Layout): LMDrive fails to interpret the road layout and instruction, driving off-road into a fence, whereas LAD-Drive successfully continues straight. Columns 3 & 4 (Collision Vehicle): In the third scenario, LMDrive collides head-on with a truck, while LAD-Drive safely navigates the lane. In the fourth scenario, LMDrive initiates an unsafe left lane change into traffic, ignoring the right-turn navigation instruction. In contrast, LAD-Drive effectively handles dynamic agents and remains safely on the road, demonstrating robustness even when the action decoder exhibits slight uncertainty.

Table 7

Language-Conditioned Robot Planning

See all 16 papers

KnowDiffuser: A Knowledge-Guided Diffusion Planner with LLM Reasoning
Mar 11, 2026Fan Ding, Xuewen Luo, Fengze Yang +4Autonomous Driving Planning Diffusion-Based Planning
Monash University Malaysia. · University of Utah, USA.
OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models
Apr 20, 2026Yiwei Zhang, Xuesong Chen, Jin Gao +5VLMs for Autonomous Driving End-to-End Autonomous Driving
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, University of Chinese Academy of Sciences · AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University +2
Vault: One-Step Latent Generation with Positive-Anchored Rewards for Autonomous Driving
Jun 4, 2026Yining Xing, Zehong Ke, Zhiyuan Liu +1End-to-End Autonomous Driving One-Step Generative Modeling

Mar 11, 2026J/K move · Enter open · S save

KnowDiffuser: A Knowledge-Guided Diffusion Planner with LLM Reasoning

Fan Ding, Xuewen Luo, Fengze Yang, Bo Yu+3

Monash University Malaysia. · University of Utah, USA.

Autonomous Driving Planning Diffusion-Based Planning Diffusion Models for Motion Generation

Open paper AskPDF arXiv

Recent advancements in Language Models (LMs) have demonstrated strong semantic reasoning capabilities, enabling their application in high-level decision-making for autonomous driving (AD). However, LMs operate over discrete token spaces and lack the ability to generate continuous, physically feasible trajectories required for motion planning. Meanwhile, diffusion models have proven effective at generating reliable and dynamically consistent trajectories, but often lack semantic interpretability and alignment with scene-level understanding. To address these limitations, we propose KnowDiffuser, a knowledge-guided motion planning framework that tightly integrates the semantic understanding of language models with the generative power of diffusion models. The framework employs a language model to infer context-aware meta-actions from structured scene representations, which are then mapped to prior trajectories that anchor the subsequent denoising process. A two-stage truncated denoising mechanism refines these trajectories efficiently, preserving both semantic alignment and physical feasibility. Experiments on the nuPlan benchmark demonstrate that KnowDiffuser significantly outperforms existing planners in both open-loop and closed-loop evaluations, establishing a robust and interpretable framework that effectively bridges the semantic-to-physical gap in AD systems.

Abstract

Language-Conditioned Robot Planning

Explore similar work

KnowDiffuser: A Knowledge-Guided Diffusion Planner with LLM Reasoning

OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models

Vault: One-Step Latent Generation with Positive-Anchored Rewards for Autonomous Driving