Think Densely, Act Sparsely: Latent Expert Cognitive Chains for Vision-Language-Action Autonomous Driving

Jie Wang; Guang Li; Zhijian Huang; Jinlong Li; Chenxu Dang; Hangjun Ye; Yahong Han; Long Chen

cs.CVOct 19, 2026

Think Densely, Act Sparsely: Latent Expert Cognitive Chains for Vision-Language-Action Autonomous Driving

Authors: Jie Wang, Guang Li, Zhijian Huang, Jinlong Li, Chenxu Dang, Hangjun Ye, Yahong Han, Long Chen

Organizations: School of Artificial Intelligence, Tianjin University · School of Artificial Intelligence, Shanxi University · Xiaomi EV

Abstract

Autonomous driving fundamentally demands dense understanding of visual, semantic, and geometric information, yet existing Vision-Language-Action (VLA) models are typically trained with sparse supervision such as language instructions or trajectory signals. This direct mapping from dense visual observations to sparse representations forces the model to compress or even discard critical scene information essential for safe driving. We argue that this limitation does not stem from insufficient model capacity, but from the lack of explicit mechanisms that encourage the formation of dense world representations in the latent space. In this spirit, we propose a novel paradigm for driving intelligence, Think Densely, Act Sparsely, and instantiate it with an efficient and interpretable framework, LECDrive (Latent Expert Cognitive Chains for Vision-Language-Action autonomous Driving). The core idea is to mimic the hierarchical perception process of humans by embedding a progressive chain of latent expert cognition within the VLA model. Concretely, we inject a compact set of task-specific latent tokens as carriers, enabling the model to internalize dense knowledge from foundational vision models, including visual representations (DINOv3), semantic structures (SAM3), and spatial geometry (DepthAnything3), while maintaining efficient inference. This interdependent cognitive chain follows a curriculum learning principle, where supervision is progressively structured from representation to semantics and geometry. Extensive experiments demonstrate that LECDrive consistently improves the performance of base VLMs across multiple autonomous driving benchmarks, covering key tasks such as object perception, state prediction, and trajectory planning. These findings suggest that constructing dense world models through structured latent expert chains is a promising direction for VLA-based autonomous driving.

Figures & tables

Figure 1: Mainstream paradigms vs. LECDrive. (a) Decoupled modeling [ 15 , 16 , 22 ] . (b) Geometry-enhanced modeling [ 24 , 25 , 26 ] . (c) Our Think Densely, Act Sparsely paradigm with latent expert chains, enabling progressive dense world modeling via a compact set of task-specific latent tokens.

Figure 2: Quantitative comparison of LECDrive with state-of-the-art methods across three autonomous driving benchmarks, covering closed-loop planning, perception and prediction tasks.

Figure 3: Overview of LECDrive . Three frozen dense expert models first generate task-specific pseudo-labels. A compact set of task-specific latent tokens is then injected into the visual tokens of each view. Task-specific latent embeddings from four VLM layers are aggregated and fed into dedicated perception heads for dense world prediction. For trajectory planning, LECDrive combines discrete motion primitives with continuous offset refinement to improve robustness and precision.

NAVSIM-v1	Input	NC↑	DAC↑	EP↑	TTC↑	Comf.↑	PDMS↑
PARA-Drive [ 35 ]	Camera	97.9	92.4	79.3	93.0	99.8	84.0
DRAMA [ 36 ]	Camera & Lidar	98.19	95.18	81.33	94.17	100.0	86.87
Hydra-MDP [ 37 ]	Camera & Lidar	98.3	96.0	78.7	94.6	100.0	86.5
DiffusionDrive [ 38 ]	Camera & Lidar	98.2	96.2	82.2	94.7	100.0	88.1
WoTE [ 7 ]	Camera & Lidar	98.5	96.8	81.9	94.9	99.9	88.3
Hydra-NeXt [ 37 ]	Camera & Lidar	98.1	97.7	81.8	94.6	100.0	88.6

Table 1: Performance comparison on the NAVSIM-v1 benchmark [ 34 ] under closed-loop trajectory evaluation, covering both E2E methods (gray) and supervised fine-tuned VLA models (green).

NAVSIM-v2	NC↑	DAC↑	DCC↑	TLC↑	EP↑	TTC↑	LK.↑	HC↑	EC↑	EPDMS↑
Ego-MLP [ 43 ]	93.1	77.9	92.7	99.6	86.0	91.5	89.4	98.3	85.4	64.0
TransFuser	96.9	89.9	97.8	99.7	87.1	95.4	92.7	98.3	87.2	76.7
Hydra-MDP++	97.2	97.5	99.4	99.6	83.1	96.5	94.4	98.2	70.9	81.4
DriveSuprim [ 44 ]	97.5	96.5	99.4	99.6	88.4	96.6	95.5	98.3	77.0	83.1
ReCogDrive	98.3	95.2	99.5	99.8	87.1	97.5	96.6	98.3	86.5	83.6
DiffusionDrive	98.2	96.3	99.4	99.8	87.4	97.4	97.0	98.3	87.7	88.3

Table 2: The performance comparison on the NAVSIM-v2 navtest [ 34 ] benchmark considers both SOTA end-to-end approaches and existing VLA models under supervised fine-tuning.

Dataset	Metrics	Zero-shot Models			Fine-tuned Models
Dataset	Metrics	GPT-4o [ 45 ]	RoboTron [ 46 ]	Qwen2.5-VL	Baseline	InMLLM [ 31 ]	VGGDrive	OURS
NuInstruct	MAP↑	0	0	0	6.15	21.93	37.49	38.25
	Accuracy↑	10.64	2.57	0.63	47.71	32.48	56.37	57.02
	MAE↓	9.93	19.36	24.10	4.35	9.08	3.08	3.57
	BLEU↑	7.08	8.06	5.56	75.75	35.20	81.13	82.38
	Average*↑	1.95	0	0	31.32	20.13	42.98	43.52

Table 3: Comparison with sota methods on the NuInstruct dataset [ 31 ] for risk object perception and state prediction. * indicates

\max((\text{Accuracy}+\text{MAP}+\text{BLEU}-\text{MAE})/4,0)

Dataset	Metrics	Fine-tuned Models
Dataset	Metrics	Baseline	DriveLM	TMLMM [ 47 ]	OminiD [ 23 ]	FSDrive [ 48 ]	IVL4Drive [ 49 ]	VGGDrive	OURS
DriveLM	Accuracy↑	64.35	52.30	59.60	70.00	71.77	73.39	77.50	79.81
	ChatGPT↑	65.30	55.35	58.44	65.00	63.42	65.25	64.76	65.69
	Language↑	44.90	45.13	46.38	-	52.77	48.56	56.69	56.53
	Match↑	34.54	35.73	35.73	37.00	39.19	47.65	49.77	47.73
	Average↑	54.59	47.20	51.50	56.00	57.12	60.02	61.26	61.41

Table 4: The performance comparison on the DriveLM dataset [ 32 ] is conducted against existing state-of-the-art (SOTA) methods to evaluate risk object perception, action prediction, and planning.

Figure 4: Visualization of LECDrive across various driving tasks. Besides trajectory planning, we showcase the model’s internal dense world modeling results, including semantic segmentation and depth estimation generated by dedicated heads, along with risk object perception and state prediction.

ID	Baseline	MT	Offset	DINO v3	SAM 3	DA 3	NC↑	DAC↑	EP↑	TTC↑	Comf.↑	PDMS↑
1	$\checkmark$						97.83	94.08	81.00	94.04	99.98	86.04
2	$\checkmark$	$\checkmark$					98.44	94.95	81.45	95.72	99.99	87.44
3	$\checkmark$	$\checkmark$	$\checkmark$				98.62	96.06	82.38	95.78	100.00	88.49
4	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$			98.53	96.57	82.99	95.97	99.99	89.07
5	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$		98.68	96.76	83.27	95.97	99.98	89.31
6	$\checkmark$	$\checkmark$	$\checkmark$		$\checkmark$	$\checkmark$	98.65	96.85	83.54	96.08	99.99	89.53

Table 5: Ablation study on the effectiveness of the proposed key components for closed-loop planning.

ID	Methods	NC↑	DAC↑	EP↑	TTC↑	Comf.↑	PDMS↑
1	Baseline	97.83	94.08	81.00	94.04	99.98	86.04
2	Visual tokens + Task heads	98.58	96.28	82.73	95.82	99.99	88.73
3	Task tokens ( $l_{28}$ ) + Task heads	98.59	96.78	83.14	95.94	99.99	89.26
4	Task tokens = 2	98.57	96.54	82.93	96.16	99.99	89.13
5	Qwen2.5-VL-3B	98.66	96.66	83.19	95.99	100.00	89.21
6	OURS	98.72	96.97	83.68	96.39	99.99	89.81

Table 6: Ablation Study on Architectural Variants and Key Hyperparameters in LECDrive.

(a) Expert supervision without MT+Offset
Configuration	NC $\uparrow$	DAC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	PDMS $\uparrow$
Baseline	97.83	94.08	81.00	94.04	86.04
+ DINOv3	98.38	95.49	82.71	95.01	87.98
+ SAM 3	98.21	95.80	83.03	97.85	88.17
+ DepthAnything 3	98.60	96.01	82.39	95.81	88.47
+ All Experts	98.59	96.51	82.97	96.05	89.15

Table 7: Further ablations on expert knowledge internalization and causal ordering on NAVSIM-v1.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

nuScenes		L2 ↓				Collision (%) ↓				Intersection (%) ↓
nuScenes		1s	2s	3s	avg	1s	2s	3s	avg	1s	2s	3s	avg
UniAD [ 50 ]	E2E	0.20	0.42	0.75	0.46	0.02	0.25	0.84	0.37	0.20	1.33	3.24	1.59
VAD-Base [ 51 ]		0.17	0.34	0.60	0.37	0.04	0.27	0.67	0.33	0.21	2.13	5.06	2.47
BEV-Planner		0.16	0.32	0.57	0.35	0.00	0.29	0.73	0.34	0.35	2.62	6.51	3.16
Baseline	VLA	0.14	0.34	0.60	0.36	0.06	0.21	0.96	0.41	0.66	2.58	5.43	2.89
OmniDrive-Q		0.14	0.29	0.55	0.33	0.00	0.13	0.78	0.30	0.56	2.48	5.96	3.00

Appendix

Table 8: Performance comparison on nuScenes open-loop planning, with metrics from BEV-Planner’s reproduced results [ 43 ] .

Dataset	Metrics	Zero-shot Models		Fine-tuned Models
Dataset	Metrics	GPT-4o	RoboTron	Ominidrive	Hermes [ 5 ]	Baseline	VGGDrive	LECDrive
Omnidrive	BLEU↑	10.91	20.30	38.00	-	37.29	37.58	37.73
	CIDEr↑	24.42	34.33	68.60	74.10	86.29	86.57	86.34
	ROUGE↑	22.34	23.67	32.60	32.70	34.33	34.40	34.42
	Average↑	19.22	26.10	46.40	-	52.64	52.85	52.83

Appendix

Table 9: The performance comparison on the OmniDrive dataset [ 23 ] focuses on caption-related tasks in autonomous driving scenarios, where base VLMs excel.

Figure 5: Visualization of clustered motion primitives.

Figure 6: Visualization of the proposed hybrid trajectory formulation. Green denotes the human-driving ground-truth trajectory, yellow denotes the trajectory decoded solely from discrete motion tokens, and red denotes the final prediction after continuous offset refinement.

Figure 7: Visualization of the proposed hybrid trajectory formulation in representative autonomous driving scenarios, including left turns, right turns, and straight driving.

VLMs for Autonomous Driving

Vision-language models for autonomous driving connect camera and sensor data with language to interpret scenes, predict hazards, and plan actions.

See all 192 papers

LVDrive: Latent Visual Representation Enhanced Vision-Language-Action Autonomous Driving Model
May 21, 2026Xiaodong Mei, Diankun Zhang, Hongwei Xie +3VLMs for Autonomous Driving World Models
The Hong Kong University of Science and Technology · Xiaomi EV
UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving
Apr 2, 2026Yongkang Li, Lijun Zhou, Sixu Yan +11Mixture-of-Transformers Vision-Language-Action Models
Huazhong University of Science and Technology · Xiaomi EV · SKL-IOTSC, University of Macau
LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving
Mar 2, 2026Yuechen Luo, Fang Li, Shaoqing Xu +10VLMs for Autonomous Driving Vision-Language-Action Models
1Tsinghua University · 2Xiaomi EV · University of Macau

May 21, 2026J/K move · Enter open · S save

LVDrive: Latent Visual Representation Enhanced Vision-Language-Action Autonomous Driving Model

Xiaodong Mei, Diankun Zhang, Hongwei Xie, Guang Chen+2

The Hong Kong University of Science and Technology · Xiaomi EV

VLMs for Autonomous Driving World Models Latent Visual Reasoning Vision-Language-Action Models World Models for Autonomous Driving Autonomous Driving Planning

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have emerged as a promising framework for end-to-end autonomous driving. However, existing VLAs typically rely on sparse action supervision, which underutilizes their powerful scene understanding and reasoning capabilities. Recent attempts to incorporate dense visual supervision via world modeling often overemphasize pixel-level image reconstruction, neglecting semantically meaningful scene representation learning. In this work, we propose LVDrive, a Latent Visual representation enhanced VLA framework for autonomous driving. LVDrive introduces a future scene prediction task into the VLA paradigm, where future representations are learned entirely in a high-level latent space under auxiliary supervision from a pretrained vision backbone. Departing from inefficient autoregressive generation, we jointly model future scene and motion prediction within a unified embedding space, processed in a single forward pass to conduct the future-aware reasoning. We further design a two-stage trajectory decoding strategy that explicitly leverages the learned latent future representations to refine trajectory generation. Extensive experiments on the challenging Bench2Drive benchmark demonstrate that LVDrive achieves significant improvements in closed-loop driving performance, outperforming both action supervised methods and image-reconstruction-based world model approaches.

Abstract

VLMs for Autonomous Driving

Explore similar work

LVDrive: Latent Visual Representation Enhanced Vision-Language-Action Autonomous Driving Model

UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving

LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving