Autonomous driving fundamentally demands dense understanding of visual, semantic, and geometric information, yet existing Vision-Language-Action (VLA) models are typically trained with sparse supervision such as language instructions or trajectory signals. This direct mapping from dense visual observations to sparse representations forces the model to compress or even discard critical scene information essential for safe driving. We argue that this limitation does not stem from insufficient model capacity, but from the lack of explicit mechanisms that encourage the formation of dense world representations in the latent space. In this spirit, we propose a novel paradigm for driving intelligence, Think Densely, Act Sparsely, and instantiate it with an efficient and interpretable framework, LECDrive (Latent Expert Cognitive Chains for Vision-Language-Action autonomous Driving). The core idea is to mimic the hierarchical perception process of humans by embedding a progressive chain of latent expert cognition within the VLA model. Concretely, we inject a compact set of task-specific latent tokens as carriers, enabling the model to internalize dense knowledge from foundational vision models, including visual representations (DINOv3), semantic structures (SAM3), and spatial geometry (DepthAnything3), while maintaining efficient inference. This interdependent cognitive chain follows a curriculum learning principle, where supervision is progressively structured from representation to semantics and geometry. Extensive experiments demonstrate that LECDrive consistently improves the performance of base VLMs across multiple autonomous driving benchmarks, covering key tasks such as object perception, state prediction, and trajectory planning. These findings suggest that constructing dense world models through structured latent expert chains is a promising direction for VLA-based autonomous driving.
Figures & tables
Figure 1: Mainstream paradigms vs. LECDrive. (a) Decoupled modeling [ 15 , 16 , 22 ] . (b) Geometry-enhanced modeling [ 24 , 25 , 26 ] . (c) Our Think Densely, Act Sparsely paradigm with latent expert chains, enabling progressive dense world modeling via a compact set of task-specific latent tokens.
Figure 2: Quantitative comparison of LECDrive with state-of-the-art methods across three autonomous driving benchmarks, covering closed-loop planning, perception and prediction tasks.
Figure 3: Overview of LECDrive . Three frozen dense expert models first generate task-specific pseudo-labels. A compact set of task-specific latent tokens is then injected into the visual tokens of each view. Task-specific latent embeddings from four VLM layers are aggregated and fed into dedicated perception heads for dense world prediction. For trajectory planning, LECDrive combines discrete motion primitives with continuous offset refinement to improve robustness and precision.
NAVSIM-v1
Input
NC↑
DAC↑
EP↑
TTC↑
Comf.↑
PDMS↑
PARA-Drive [ 35 ]
Camera
97.9
92.4
79.3
93.0
99.8
84.0
DRAMA [ 36 ]
Camera & Lidar
98.19
95.18
81.33
94.17
100.0
86.87
Hydra-MDP [ 37 ]
Camera & Lidar
98.3
96.0
78.7
94.6
100.0
86.5
DiffusionDrive [ 38 ]
Camera & Lidar
98.2
96.2
82.2
94.7
100.0
88.1
WoTE [ 7 ]
Camera & Lidar
98.5
96.8
81.9
94.9
99.9
88.3
Hydra-NeXt [ 37 ]
Camera & Lidar
98.1
97.7
81.8
94.6
100.0
88.6
Table 1: Performance comparison on the NAVSIM-v1 benchmark [ 34 ] under closed-loop trajectory evaluation, covering both E2E methods (gray) and supervised fine-tuned VLA models (green).
NAVSIM-v2
NC↑
DAC↑
DCC↑
TLC↑
EP↑
TTC↑
LK.↑
HC↑
EC↑
EPDMS↑
Ego-MLP [ 43 ]
93.1
77.9
92.7
99.6
86.0
91.5
89.4
98.3
85.4
64.0
TransFuser
96.9
89.9
97.8
99.7
87.1
95.4
92.7
98.3
87.2
76.7
Hydra-MDP++
97.2
97.5
99.4
99.6
83.1
96.5
94.4
98.2
70.9
81.4
DriveSuprim [ 44 ]
97.5
96.5
99.4
99.6
88.4
96.6
95.5
98.3
77.0
83.1
ReCogDrive
98.3
95.2
99.5
99.8
87.1
97.5
96.6
98.3
86.5
83.6
DiffusionDrive
98.2
96.3
99.4
99.8
87.4
97.4
97.0
98.3
87.7
88.3
Table 2: The performance comparison on the NAVSIM-v2 navtest [ 34 ] benchmark considers both SOTA end-to-end approaches and existing VLA models under supervised fine-tuning.
Dataset
Metrics
Zero-shot Models
Fine-tuned Models
GPT-4o [ 45 ]
RoboTron [ 46 ]
Qwen2.5-VL
Baseline
InMLLM [ 31 ]
VGGDrive
OURS
NuInstruct
MAP↑
0
0
0
6.15
21.93
37.49
38.25
Accuracy↑
10.64
2.57
0.63
47.71
32.48
56.37
57.02
MAE↓
9.93
19.36
24.10
4.35
9.08
3.08
3.57
BLEU↑
7.08
8.06
5.56
75.75
35.20
81.13
82.38
Average*↑
1.95
0
0
31.32
20.13
42.98
43.52
Table 3: Comparison with sota methods on the NuInstruct dataset [ 31 ] for risk object perception and state prediction. * indicates max((Accuracy+MAP+BLEU−MAE)/4,0) .
Dataset
Metrics
Fine-tuned Models
Baseline
DriveLM
TMLMM [ 47 ]
OminiD [ 23 ]
FSDrive [ 48 ]
IVL4Drive [ 49 ]
VGGDrive
OURS
DriveLM
Accuracy↑
64.35
52.30
59.60
70.00
71.77
73.39
77.50
79.81
ChatGPT↑
65.30
55.35
58.44
65.00
63.42
65.25
64.76
65.69
Language↑
44.90
45.13
46.38
-
52.77
48.56
56.69
56.53
Match↑
34.54
35.73
35.73
37.00
39.19
47.65
49.77
47.73
Average↑
54.59
47.20
51.50
56.00
57.12
60.02
61.26
61.41
Table 4: The performance comparison on the DriveLM dataset [ 32 ] is conducted against existing state-of-the-art (SOTA) methods to evaluate risk object perception, action prediction, and planning.
Figure 4: Visualization of LECDrive across various driving tasks. Besides trajectory planning, we showcase the model’s internal dense world modeling results, including semantic segmentation and depth estimation generated by dedicated heads, along with risk object perception and state prediction.
ID
Baseline
MT
Offset
DINO v3
SAM 3
DA 3
NC↑
DAC↑
EP↑
TTC↑
Comf.↑
PDMS↑
1
✓
97.83
94.08
81.00
94.04
99.98
86.04
2
✓
✓
98.44
94.95
81.45
95.72
99.99
87.44
3
✓
✓
✓
98.62
96.06
82.38
95.78
100.00
88.49
4
✓
✓
✓
✓
98.53
96.57
82.99
95.97
99.99
89.07
5
✓
✓
✓
✓
✓
98.68
96.76
83.27
95.97
99.98
89.31
6
✓
✓
✓
✓
✓
98.65
96.85
83.54
96.08
99.99
89.53
Table 5: Ablation study on the effectiveness of the proposed key components for closed-loop planning.
ID
Methods
NC↑
DAC↑
EP↑
TTC↑
Comf.↑
PDMS↑
1
Baseline
97.83
94.08
81.00
94.04
99.98
86.04
2
Visual tokens + Task heads
98.58
96.28
82.73
95.82
99.99
88.73
3
Task tokens ( l28 ) + Task heads
98.59
96.78
83.14
95.94
99.99
89.26
4
Task tokens = 2
98.57
96.54
82.93
96.16
99.99
89.13
5
Qwen2.5-VL-3B
98.66
96.66
83.19
95.99
100.00
89.21
6
OURS
98.72
96.97
83.68
96.39
99.99
89.81
Table 6: Ablation Study on Architectural Variants and Key Hyperparameters in LECDrive.
Configuration
NC ↑
DAC ↑
EP ↑
TTC ↑
PDMS ↑
(a) Expert supervision without MT+Offset
Baseline
97.83
94.08
81.00
94.04
86.04
+ DINOv3
98.38
95.49
82.71
95.01
87.98
+ SAM 3
98.21
95.80
83.03
97.85
88.17
+ DepthAnything 3
98.60
96.01
82.39
95.81
88.47
+ All Experts
98.59
96.51
82.97
96.05
89.15
Table 7: Further ablations on expert knowledge internalization and causal ordering on NAVSIM-v1.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
nuScenes
L2 ↓
Collision (%) ↓
Intersection (%) ↓
1s
2s
3s
avg
1s
2s
3s
avg
1s
2s
3s
avg
UniAD [ 50 ]
E2E
0.20
0.42
0.75
0.46
0.02
0.25
0.84
0.37
0.20
1.33
3.24
1.59
VAD-Base [ 51 ]
0.17
0.34
0.60
0.37
0.04
0.27
0.67
0.33
0.21
2.13
5.06
2.47
BEV-Planner
0.16
0.32
0.57
0.35
0.00
0.29
0.73
0.34
0.35
2.62
6.51
3.16
Baseline
VLA
0.14
0.34
0.60
0.36
0.06
0.21
0.96
0.41
0.66
2.58
5.43
2.89
OmniDrive-Q
0.14
0.29
0.55
0.33
0.00
0.13
0.78
0.30
0.56
2.48
5.96
3.00
Appendix
Table 8: Performance comparison on nuScenes open-loop planning, with metrics from BEV-Planner’s reproduced results [ 43 ] .
Dataset
Metrics
Zero-shot Models
Fine-tuned Models
GPT-4o
RoboTron
Ominidrive
Hermes [ 5 ]
Baseline
VGGDrive
LECDrive
Omnidrive
BLEU↑
10.91
20.30
38.00
-
37.29
37.58
37.73
CIDEr↑
24.42
34.33
68.60
74.10
86.29
86.57
86.34
ROUGE↑
22.34
23.67
32.60
32.70
34.33
34.40
34.42
Average↑
19.22
26.10
46.40
-
52.64
52.85
52.83
Appendix
Table 9: The performance comparison on the OmniDrive dataset [ 23 ] focuses on caption-related tasks in autonomous driving scenarios, where base VLMs excel.
Figure 5: Visualization of clustered motion primitives.
Figure 6: Visualization of the proposed hybrid trajectory formulation. Green denotes the human-driving ground-truth trajectory, yellow denotes the trajectory decoded solely from discrete motion tokens, and red denotes the final prediction after continuous offset refinement.
Figure 7: Visualization of the proposed hybrid trajectory formulation in representative autonomous driving scenarios, including left turns, right turns, and straight driving.
Vision-Language-Action (VLA) models have emerged as a promising framework for end-to-end autonomous driving. However, existing VLAs typically rely on sparse action supervision, which underutilizes their powerful scene understanding and reasoning capabilities. Recent attempts to incorporate dense visual supervision via world modeling often overemphasize pixel-level image reconstruction, neglecting semantically meaningful scene representation learning. In this work, we propose LVDrive, a Latent Visual representation enhanced VLA framework for autonomous driving. LVDrive introduces a future scene prediction task into the VLA paradigm, where future representations are learned entirely in a high-level latent space under auxiliary supervision from a pretrained vision backbone. Departing from inefficient autoregressive generation, we jointly model future scene and motion prediction within a unified embedding space, processed in a single forward pass to conduct the future-aware reasoning. We further design a two-stage trajectory decoding strategy that explicitly leverages the learned latent future representations to refine trajectory generation. Extensive experiments on the challenging Bench2Drive benchmark demonstrate that LVDrive achieves significant improvements in closed-loop driving performance, outperforming both action supervised methods and image-reconstruction-based world model approaches.
Xiaodong Mei, Diankun Zhang, Hongwei Xie +3
The Hong Kong University of Science and Technology · Xiaomi EV
Vision-Language-Action (VLA) models have recently emerged in autonomous driving, with the promise of leveraging rich world knowledge to improve the cognitive capabilities of driving systems. However, adapting such models for driving tasks currently faces a critical dilemma between spatial perception and semantic reasoning. Consequently, existing VLA systems are forced into suboptimal compromises: directly adopting 2D Vision-Language Models yields limited spatial perception, whereas enhancing them with 3D spatial representations often impairs the native reasoning capacity of VLMs. We argue that this dilemma largely stems from the coupled optimization of spatial perception and semantic reasoning within shared model parameters. To overcome this, we propose UniDriveVLA, a Unified Driving Vision-Language-Action model based on Mixture-of-Transformers that addresses the perception-reasoning conflict via expert decoupling. Specifically, it comprises three experts for driving understanding, scene perception, and action planning, which are coordinated through masked joint attention. In addition, we combine a sparse perception paradigm with a three-stage progressive training strategy to improve spatial perception while maintaining semantic reasoning capability. Extensive experiments show that UniDriveVLA achieves state-of-the-art performance in open-loop evaluation on nuScenes and closed-loop evaluation on Bench2Drive. Moreover, it demonstrates strong performance across a broad range of perception, prediction, and understanding tasks, including 3D detection, online mapping, motion forecasting, and driving-oriented VQA, highlighting its broad applicability as a unified model for autonomous driving. Code and model have been released at https://github.com/xiaomi-research/unidrivevla
Yongkang Li, Lijun Zhou, Sixu Yan +11
Huazhong University of Science and Technology · Xiaomi EV · SKL-IOTSC, University of Macau
While Vision-Language-Action (VLA) models have revolutionized autonomous driving by unifying perception and planning, their reliance on explicit textual Chain-of-Thought (CoT) leads to semantic-perceptual decoupling and perceptual-symbolic conflicts. Recent shifts toward latent reasoning attempt to bypass these bottlenecks by thinking in continuous hidden space. However, without explicit intermediate constraints, standard latent CoT often operates as a physics-agnostic representation. To address this, we propose the Latent Spatio-Temporal VLA (LaST-VLA), a framework shifting the reasoning paradigm from discrete symbolic processing into a physically grounded Latent Spatio-Temporal CoT. By implementing a dual-feature alignment mechanism, we distill geometric constraints from 3D foundation models and dynamic foresight from world models directly into the latent space. Coupled with a progressive SFT training strategy that transitions from feature alignment to trajectory generation, and refined via Reinforcement Learning with Group Relative Policy Optimization (GRPO) to ensure safety and rule compliance. \method~setting a new record on NAVSIM v1 (91.3 PDMS) and NAVSIM v2 (87.1 EPDMS), while excelling in spatial-temporal reasoning on SURDS and NuDynamics benchmarks.
Yuechen Luo, Fang Li, Shaoqing Xu +10
1Tsinghua University · 2Xiaomi EV · University of Macau