UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving

Yongkang Li; Lijun Zhou; Sixu Yan; Bencheng Liao; Tianyi Yan; Kaixin Xiong; Long Chen; Hongwei Xie

cs.CVApr 2, 2026

UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving

Authors: Yongkang Li, Lijun Zhou, Sixu Yan, Bencheng Liao, Tianyi Yan, Kaixin Xiong, Long Chen, Hongwei Xie, +6 more

Organizations: Huazhong University of Science and Technology · Xiaomi EV · SKL-IOTSC, University of Macau

Abstract

Vision-Language-Action (VLA) models have recently emerged in autonomous driving, with the promise of leveraging rich world knowledge to improve the cognitive capabilities of driving systems. However, adapting such models for driving tasks currently faces a critical dilemma between spatial perception and semantic reasoning. Consequently, existing VLA systems are forced into suboptimal compromises: directly adopting 2D Vision-Language Models yields limited spatial perception, whereas enhancing them with 3D spatial representations often impairs the native reasoning capacity of VLMs. We argue that this dilemma largely stems from the coupled optimization of spatial perception and semantic reasoning within shared model parameters. To overcome this, we propose UniDriveVLA, a Unified Driving Vision-Language-Action model based on Mixture-of-Transformers that addresses the perception-reasoning conflict via expert decoupling. Specifically, it comprises three experts for driving understanding, scene perception, and action planning, which are coordinated through masked joint attention. In addition, we combine a sparse perception paradigm with a three-stage progressive training strategy to improve spatial perception while maintaining semantic reasoning capability. Extensive experiments show that UniDriveVLA achieves state-of-the-art performance in open-loop evaluation on nuScenes and closed-loop evaluation on Bench2Drive. Moreover, it demonstrates strong performance across a broad range of perception, prediction, and understanding tasks, including 3D detection, online mapping, motion forecasting, and driving-oriented VQA, highlighting its broad applicability as a unified model for autonomous driving. Code and model have been released at https://github.com/xiaomi-research/unidrivevla

Figures & tables

Figure 1: Comparison of VLA paradigms for autonomous driving. (a) Vanilla 2D VLA provides strong semantic reasoning but limited spatial perception. (b) 3D-enhanced VLA improves spatial perception but may degrade semantic reasoning. (c) UniDriveVLA decouples understanding, perception, and action with the Mixture-of-Transformers architecture, achieving both.

Figure 2: Analysis of representation interference and model performance. (a) Cosine similarity between LLM tokens and perception tokens across layers. In the shared-weight decoder, the similarity progressively increases toward 1, indicating feature collapse into nearly identical representations, whereas MoT maintains low similarity and preserves task decoupling. (b) Performance comparison. By mitigating optimization conflicts, UniDriveVLA consistently outperforms the shared-weight baseline across perception, reasoning, and planning metrics.

Figure 3: Architecture overview of UniDriveVLA. UniDriveVLA adopts a Mixture-of-Transformers architecture with three specialized experts for driving understanding, scene perception, and action planning. By decoupling heterogeneous tokens into dedicated experts and coordinating them through masked joint attention, the model mitigates optimization conflicts and unifies understanding, perception, and planning within a single framework.

Figure 4: Illustration of Masked Joint Attention.

Method	Open-loop Metric	Closed-loop Metric
Method	Avg. L2 $\downarrow$	Driving Score $\uparrow$	Success Rate (%) $\uparrow$	Efficiency $\uparrow$	Comfortness $\uparrow$
AutoVLA $\dagger$ [NeurIPS25] [ 120 ]	-	78.84	57.73	146.93	39.33
SimLingo $\dagger$ [CVPR25] [ 77 ]	-	85.94	66.82	244.18	25.49
R2SE $\dagger$ [T-PAMI26] [ 56 ]	-	86.28	69.54	243.89	23.26
AD-MLP [arXiv23] [ 112 ]	3.64	18.05	0.00	48.45	22.63
VAD [ICCV23] [ 42 ]	0.91	42.35	15.00	157.94	46.01

Table 1: Open-loop and closed-loop planning results on Bench2Drive. Avg. L2 is computed over predictions within a 2-second horizon at 2 Hz.

\dagger

Models trained on PDM-Lite, which provides higher-quality expert demonstrations than Think2Drive. Bold numbers indicate the best results among methods trained without PDM-Lite.

Method	Ability (%) $\uparrow$
Method	Merging	Overtaking	Emergency Brake	Give Way	Traffic Sign	Mean
AD-MLP [arXiv23] [ 112 ]	0.00	0.00	0.00	0.00	4.35	0.87
UniAD [CVPR23] [ 29 ]	14.10	17.78	21.67	10.00	14.21	15.55
VAD [ICCV23] [ 42 ]	8.11	24.44	18.64	20.00	19.15	18.07
ThinkTwice [CVPR22] [ 37 ]	13.72	22.93	52.99	50.00	47.78	37.48
DriveAdapter [ICCV23] [ 36 ]	14.55	22.61	54.04	50.00	50.45	38.33

Table 2: Multi-ability results of end-to-end autonomous driving methods on Bench2Drive.

Method	ST-P3 metrics								UniAD metrics								LLM
	L2 (m) $\downarrow$				Collision (%) $\downarrow$				L2 (m) $\downarrow$				Collision (%) $\downarrow$
	1s	2s	3s	Avg.	1s	2s	3s	Avg.	1s	2s	3s	Avg.	1s	2s	3s	Avg.
Methods with Ego Status
ST-P3* [ 28 ]	1.33	2.11	2.90	2.11	0.23	0.62	1.27	0.71	-	-	-	-	-	-	-	-		-
VAD* [ 42 ]	0.17	0.34	0.60	0.37	0.04	0.27	0.67	0.33	-	-	-	-	-	-	-	-	-
UniAD* [ 29 ]	-	-	-	-	-	-	-	-	0.20	0.42	0.75	0.46	0.02	0.25	0.84	0.37	-

Table 3: End-to-end trajectory planning performance on nuScenes. We evaluate planning quality using L2 displacement error and collision rate across two mainstream protocols: ST-P3 [ 28 ] and UniAD [ 29 ] . * denotes methods using ego-state information. Results for SparseDrive ‡ are re-evaluated following the protocol established by GPT-Driver [ 69 ] .

Method	Detection		Map				Motion
Method	mAP $\uparrow$	NDS $\uparrow$	AP ${}_{ped}\uparrow$	AP ${}_{divider}\uparrow$	AP ${}_{boundary}\uparrow$	mAP $\uparrow$	minADE $(m)\downarrow$	minFDE $(m)\downarrow$
UniAD [CVPR23] [ 29 ]	0.380	0.359	–	–	–	–	0.710	1.020
VAD [ICCV23] [ 42 ]	0.276	0.397	0.406	0.515	0.506	0.476	–	–
SparseDrive [ICRA25] [ 86 ]	0.418	0.525	0.499	0.570	0.584	0.551	0.600	0.960
EgoFSD [ICRA26] [ 85 ]	0.410	0.528	0.549	0.557	0.573	0.560	–	–
HiP-AD [ICCV25] [ 88 ]	0.424	0.535	–	–	–	0.571	0.610	–

Table 4: Comparison of perception, mapping, and motion prediction performance on the nuScenes validation dataset.

Table 9

Architecture	Understanding		Perception		Planning
Architecture	General VQA (%) $\uparrow$	DriveBench (%) $\uparrow$	Det (NDS) $\uparrow$	Map (mAP) $\uparrow$	L2 (m) $\downarrow$	CR (%) $\downarrow$
Shared-Weight Decoder	31.1	50.8	0.437	0.516	0.641	0.175
Mixture-of-Transformers	45.5	54.9	0.439	0.516	0.533	0.140

Table 7: Performance comparison between a shared-weight decoder and Mixture-of-Transformers (MoT) across understanding, perception, and planning tasks. General VQA is averaged over general multimodal benchmarks, DriveBench is used for driving-oriented understanding, and perception and planning metrics are reported on nuScenes.

Model	Params	MMStar $\uparrow$	MMMU $\uparrow$	RealWorldQA $\uparrow$	AI2D $\uparrow$	MME $\uparrow$	VLMsAreBlind $\uparrow$	ChartQA $\uparrow$
Qwen2.5-VL [ 4 ]	7B	63.7	51.1	69.0	82.5	2317	40.3	82.5
Qwen3-VL [ 3 ]	8B	63.0	52.8	69.0	83.2	2364	61.9	83.8
InternVL3 [ 122 ]	8B	69.1	54.8	67.8	83.9	2393	42.0	86.6
InternVL3.5 [ 97 ]	8B	66.3	57.3	63.0	82.3	2357	45.4	87.1
GPT-5 Nano [ 83 ]	—	41.3	57.6	60.7	65.7	-	40.2	48.6
UniDriveVLA	8B	43.3	47.3	49.9	76.3	1876	26.6	76.3

Table 8: Comparison of UniDriveVLA with general-purpose and driving-specific models on general multimodal understanding benchmarks.

Explore similar work

Teaching Vision-Language-Action Models What to See and Where to Look
Jul 2, 2026Yuguang Yang, Canyu Chen, Zhewen Tan +10End-to-End Autonomous Driving VLM Distillation
School of Electronic Information Engineering, Beihang University · Institute for AI Industry Research (AIR), Tsinghua University · National College for Excellent Engineers, Beihang University +5
MindVLA-U1: VLA Beats VA with Unified Streaming Architecture for Autonomous Driving
May 12, 2026Yuzhou Huang, Benjin Zhu, Hengtong Lu +6VLMs for Autonomous Driving End-to-End Autonomous Driving
CUHK MMLab · Li Auto · Tsinghua University
LVDrive: Latent Visual Representation Enhanced Vision-Language-Action Autonomous Driving Model
May 21, 2026Xiaodong Mei, Diankun Zhang, Hongwei Xie +3VLMs for Autonomous Driving World Models
The Hong Kong University of Science and Technology · Xiaomi EV

Jul 2, 2026J/K move · Enter open · S save

Teaching Vision-Language-Action Models What to See and Where to Look

Yuguang Yang, Canyu Chen, Zhewen Tan, Yizhi Wang+9

School of Electronic Information Engineering, Beihang University · Institute for AI Industry Research (AIR), Tsinghua University · National College for Excellent Engineers, Beihang University+5

End-to-End Autonomous Driving VLM Distillation Vision-Language-Action Models Autonomous Driving Planning Vehicle Trajectory Prediction

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing VLAs' training relies heavily on text-centric visual question answering and chain-of-thought reasoning data, which emphasizes linguistic reasoning rather than action-grounded planning. As a result, the learned representations capture semantic knowledge but lack spatial dependencies crucial for reliable trajectory prediction. We propose DriveTeach-VLA, a framework that explicitly teaches VLAs what to see and where to look. Driving-aware Vision Distillation (DVD) injects driving-specific perceptual priors into the vision encoder, while 2D Trajectory-Guided Prompts (2D-TGP) provide spatial conditioning aligned with feasible driving trajectories. Together, they form a vision-guided learning pipeline: what to see (DVD pretraining) - where to look (TGP-guided SFT) - how to act (TGP-guided GRPO). DriveTeach-VLA achieves the state-of-the-art performance on NAVSIM and nuScenes. Our code is available at: https://github.com/ShivaTeam/DriveTeach-VLA.

Abstract

Explore similar work

Teaching Vision-Language-Action Models What to See and Where to Look

MindVLA-U1: VLA Beats VA with Unified Streaming Architecture for Autonomous Driving

LVDrive: Latent Visual Representation Enhanced Vision-Language-Action Autonomous Driving Model