LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving

Yuechen Luo; Fang Li; Shaoqing Xu; Yang Ji; Zehan Zhang; Bing Wang; Yuannan Shen; Jianwei Cui

cs.CVMar 2, 2026

LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving

Authors: Yuechen Luo, Fang Li, Shaoqing Xu, Yang Ji, Zehan Zhang, Bing Wang, Yuannan Shen, Jianwei Cui, +5 more

Organizations: 1Tsinghua University · 2Xiaomi EV · University of Macau

Abstract

While Vision-Language-Action (VLA) models have revolutionized autonomous driving by unifying perception and planning, their reliance on explicit textual Chain-of-Thought (CoT) leads to semantic-perceptual decoupling and perceptual-symbolic conflicts. Recent shifts toward latent reasoning attempt to bypass these bottlenecks by thinking in continuous hidden space. However, without explicit intermediate constraints, standard latent CoT often operates as a physics-agnostic representation. To address this, we propose the Latent Spatio-Temporal VLA (LaST-VLA), a framework shifting the reasoning paradigm from discrete symbolic processing into a physically grounded Latent Spatio-Temporal CoT. By implementing a dual-feature alignment mechanism, we distill geometric constraints from 3D foundation models and dynamic foresight from world models directly into the latent space. Coupled with a progressive SFT training strategy that transitions from feature alignment to trajectory generation, and refined via Reinforcement Learning with Group Relative Policy Optimization (GRPO) to ensure safety and rule compliance. \method~setting a new record on NAVSIM v1 (91.3 PDMS) and NAVSIM v2 (87.1 EPDMS), while excelling in spatial-temporal reasoning on SURDS and NuDynamics benchmarks.

Figures & tables

Figure 1: Comparison of VLA Paradigms. (a) Direct VLA is efficient but lacks reasoning. (b) Explicit Textual CoT is interpretable but suffers high latency and hallucinations. (c) Naive Latent CoT (w/o supervision) is efficient but unstable (model collapse). (d) Our Spatio-Temporal Latent CoT (supervision) aligns latent features with physical priors, achieving efficiency, stability, and grounding.

Figure 2: Overview of the LaST-VLA framework. (a) Model Architecture: The model constructs a Latent CoT by aligning hidden states with dynamic and geometric priors distilled from foundation models (Cosmos and VGGT) via specialized adapters. (b) Progressive Training Strategy: The pipeline features a two-stage SFT phase that utilizes structured causal masking to enforce physically grounded reasoning, followed by RL fine-tuning to directly optimize the policy for driving safety and compliance.

Figure 3: Architecture of the Dynamics (a) and Geometry (b) Adapters. Random mask is used only during training.

Method	Image	Lidar	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	CF $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
Constant Velocity			68.0	57.8	50.0	100	19.4	20.6
Ego Status MLP			93.0	77.3	83.6	100	62.8	65.6
UniAD ( Hu et al., 2023 )	✓		97.8	91.9	92.9	100	78.8	83.4
DiffusionDrive ( Liao et al., 2025 )	✓	✓	98.2	96.2	94.7	100	82.2	88.1
WoTE ( Li et al., 2025d )	✓	✓	98.5	96.8	94.9	99.9	81.9	88.3
DriveVLA-W0-7B ( Li et al., 2025c )	✓		98.7	99.1	94.9	99.3	83.3	90.2

Table 1: Comparison with state-of-the-art methods on NAVSIMv1 with PDMS. * indicates models utilizing Textual CoT.

Method	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	TLC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	LK $\uparrow$	HC $\uparrow$	EC $\uparrow$	EPDMS $\uparrow$
Ego Status	93.1	77.9	92.7	99.6	86.0	91.5	89.4	98.3	85.4	64.0
TransFuser ( Chitta et al., 2022 )	96.9	89.9	97.8	99.7	87.1	95.4	92.7	98.3	87.2	76.7
DiffusionDrive ( Liao et al., 2025 )	98.2	95.9	99.4	99.8	87.5	97.3	96.8	98.3	87.7	84.5
HydraMDP++ ( Li et al., 2024 )	98.5	98.5	99.5	99.7	87.4	97.9	95.8	98.2	75.7	85.6
DriveVLA-W0-7B ( Li et al., 2025c )	98.5	99.1	98.0	99.7	86.4	98.1	93.2	97.9	58.9	86.1
LaST-VLA-2B (RL)	98.6	97.7	99.1	99.7	90.2	98.2	96.6	98.3	86.8	86.8

Table 2: Comparison with state-of-the-art methods on NAVSIMv2 with EPDMS.

Method	Single-object (%) $\uparrow$			Multi-object(%) $\uparrow$			Score(%) $\uparrow$	Motion (%) $\uparrow$
Method	Yaw	Pixel	Depth	Dis	L/R	F/B	Score(%) $\uparrow$	Motion (%) $\uparrow$
Qwen2.5-VL-72B-Instruct	11.57	6.13	44.00	58.05	66.16	14.92	33.47	54.79
Qwen2.5-VL-7B-Instruct	7.57	3.46	25.95	11.46	17.95	9.30	12.61	48.77
Qwen2.5-VL-3B-Instruct	6.27	3.81	27.68	17.84	14.81	10.49	13.48	35.34
SURDS-3B* ( Guo et al., 2024 )	20.97	44.81	69.84	49.30	51.35	8.54	40.80	-
InternVL3-2B $\dagger$	37.30	48.90	99.90	80.54	82.81	77.95	71.23	66.54

Table 3: Comparison with state-of-the-art methods on the SURDS and NuDynamics benchmarks. Score represents the average performance on SURDS. Motion denotes the Motion State Estimation accuracy on NuDynamics. * indicates models trained with SFT+RL, and

\dagger

indicates models trained with Only SFT. All methods except LaST-VLA utilize Textual CoT.

Mode	WM	3D	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	CF $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
SFT			98.3	92.3	94.7	100	77.2	83.9
	✓		98.6	94.4	95.5	100	78.4	85.8
		✓	98.6	94.8	95.7	100	79.1	86.4
	✓	✓	98.7	95.4	95.7	100	80.5	87.3
RL			98.3	94.0	94.7	100	83.0	87.2
	✓		98.5	97.7	95.2	100	86.7	90.9

Table 4: Ablation study analyzing the impact of geometric (3D) and dynamic (WM) latent CoT across SFT and RL training phases.

Method	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	CF $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
No Text	98.5	94.3	95.5	100	79.6	86.0
Text	98.3	94.0	94.7	100	83.0	87.2
Latent (w/o sup.)	98.6	96.8	95.8	100	84.6	89.8
Latent (with sup.)	98.7	97.9	95.6	100	86.7	91.3

Table 5: Ablation study on the effectiveness of Textual CoT (Text), Latent CoT (Latent), and Latent Token Supervision(sup.) after RL training phase. “w/o” denotes “without”.

Figure 4: PDMS performance varying with training steps during the RL phase.

Figure 5: Qualitative visualization comparing the Textual CoT baseline (Red) and LaST-VLA (Green) . (a) Drivable Area Compliance (DAC): Our method maintains precise lane adherence, whereas the baseline violates spatial boundaries. (b) Time-to-Collision (TTC): Our method accurately anticipates dynamics to avoid rear-end collisions, while the baseline fails to brake effectively.

Mode	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	CF $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
Standard Mask	98.1	97.3	94.6	100	86.6	90.4
Structured Causal Mask	98.7	97.9	95.6	100	86.7	91.3

Table 6: Ablation study on the effectiveness of the Structured Causal Mask after RL training phase.

ID	WM/N	3D/N	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	CF $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
1	3*12	6	98.6	97.5	95.6	100	86.5	90.9
2	3*12	24	98.4	97.7	95.2	100	86.5	90.8
3	3*6	12	98.6	97.5	95.4	100	86.5	90.8
4	3*24	12	98.9	97.6	96.0	100	85.9	90.9
5	3*12	12	98.7	97.9	95.6	100	86.7	91.3

Table 7: Ablation study on the number of geometric(3D) and dynamic(WM) latent tokens after RL training phase.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Method	VLM Model	L2 (m) $\downarrow$				Collision (%) $\downarrow$
Method	VLM Model	1s	2s	3s	Avg.	1s	2s	3s	Avg.
Conventional End-to-end Methods
UniAD* ( Hu et al., 2023 )	-	0.20	0.42	0.75	0.46	0.02	0.25	0.84	0.37
VLMs-based Methods
Doe-1 ( Zheng et al., 2024 )	Lumina-mGPT-7B	0.50	1.18	2.11	1.26	0.04	0.37	1.19	0.53
AutoVLA* ( Zhou et al., 2025b )	Qwen2.5-VL-3B	0.28	0.66	1.16	0.70	0.14	0.25	0.53	0.31

Appendix

Table 8: Comparison with state-of-the-art methods on the nuScenes. * indicates that the ego status is use.

ID	Supervised Features	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	CF $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
1	Feature (4)	98.3	97.3	94.8	100	86.5	90.4
2	Feature (11)	98.2	97.3	94.6	100	86.5	90.3
3	Feature (17)	98.4	97.5	95.3	100	86.4	90.7
4	Feature (4, 11, 17, 23)	98.4	97.7	94.8	100	86.8	90.8
5	Feature (23) (Ours)	98.6	97.7	95.6	100	86.4	91.0

Appendix

Table 9: Ablation study on the effectiveness of supervising features from different layers of the VGGT aggregator. The Feature (23) is the last layer. Note: These methods only have geometric (3D) latent CoT, without dynamic (WM) latent CoT.

Figure 6: Qualitative visualization comparing the Textual CoT baseline (Red) and LaST-VLA (Green) on NAVSIM benchmark. (a) Drivable Area Compliance (DAC): Our method maintains precise lane adherence, whereas the baseline violates spatial boundaries. (b) Time-to-Collision (TTC): Our method accurately anticipates dynamics to avoid rear-end collisions, while the baseline fails to brake effectively.

Figure 7: Qualitative visualization of failure cases on NAVSIM benchmark. The red line represents the trajectory predicted by LaST-VLA, while the green line denotes the Ground Truth. We observe that failures predominantly occur when the planned path extends beyond the front-view camera’s field of view (FOV). Lacking sufficient visual context in these peripheral regions, the model struggles with precise spatial grounding, leading to potential collisions or deviations from the drivable area.

Figure 8: Examples of the Yaw Angle Determination task on the SURDS benchmark.

Figure 9: Examples of the Pixel Location Estimation task on the SURDS benchmark.

Figure 10: Examples of the Depth Range Determination on the SURDS benchmark.

Figure 11: Examples of the Distance Estimation task on the SURDS benchmark.

Figure 12: Examples of the Left/Right Determination task on the SURDS benchmark.

Figure 13: Examples of the Front/Behind Determination task on the SURDS benchmark.

Figure 14: Examples of the Motion State Estimation task on the NuDynamics benchmark.

Explore similar work

S-squared-VLA: Decoupling Semantic and Spatial Streams in Vision-Language-Action Models for Autonomous Driving
Jul 15, 2026Jianguo Yu, Rukang Wang, Duanfeng Chu +3VLMs for Autonomous Driving Vision-Language-Action Models
School of Mechanical and Electronic Engineering, Wuhan University of Technology, Wuhan, Hubei, China · Intelligent Transportation Systems Research Center, Wuhan University of Technology, Wuhan, Hubei, China · School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan, Hubei, China
Continuous Actions from Discrete Minds: Latent-Aligned Planning for End-to-End Autonomous Driving
Sep 3, 2026Ruoyu Yao, Yusen Xie, Qingzhao Liu +5Latent Action Learning End-to-End Autonomous Driving
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · Central Media Technology Institute, Huawei
Xiaomi OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation
Apr 20, 2026Jinghui Lu, Jiayi Guan, Zhijian Huang +47Trajectory Generation VLMs for Autonomous Driving

Jul 15, 2026J/K move · Enter open · S save

S-squared-VLA: Decoupling Semantic and Spatial Streams in Vision-Language-Action Models for Autonomous Driving

Jianguo Yu, Rukang Wang, Duanfeng Chu, Chen Wang+2

School of Mechanical and Electronic Engineering, Wuhan University of Technology, Wuhan, Hubei, China · Intelligent Transportation Systems Research Center, Wuhan University of Technology, Wuhan, Hubei, China · School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan, Hubei, China

VLMs for Autonomous Driving Vision-Language-Action Models Autonomous Driving Planning

Open paper AskPDF arXiv

Vision-Language Models (VLMs) have demonstrated remarkable potential for high-level reasoning in autonomous driving, yet they fundamentally struggle to generate precise, low-level control actions. This limitation is rooted in a semantic-physical gap caused by the inherent mismatch between discrete language tokens and continuous trajectory planning. While Vision-Language-Action (VLA) architectures attempt to bridge this gap by unifying perception and control into a single policy, this entanglement creates a new bottleneck. Standard VLAs experience a severe spatial representation collapse, which irreversibly degrades the fine-grained spatial and geometric priors essential for safe, boundary-aware navigation. To address this limitation, we propose the S-squared-VLA, which explicitly decouples the semantic and spatial streams in Vision-Language-Action models. The semantic stream leverages hierarchical bridging to extract multi-scale VLM features for robust intent reasoning. In parallel, an independent spatial stream bypasses the autoregressive language bottleneck, directly preserving uncompressed spatial features from the visual encoder. By integrating auxiliary perception supervision, this stream explicitly equips the model with rich spatial and geometric priors. Finally, a dual-stream planning adapter fuses high-level semantic intent with precise spatial constraints via cascaded attention mechanisms. Evaluations on the NAVSIM closed-loop benchmark show that S-squared-VLA achieves a Predictive Driver Model Score (PDMS) of 87.1, establishing a new state-of-the-art for VLA models under a purely supervised fine-tuning (SFT) setting. By mitigating the spatial representation collapse of traditional VLMs, our framework significantly outperforms baselines, achieving the highest No Collision (NC) rate of 98.4 among all evaluated methods.

Abstract

Explore similar work

S-squared-VLA: Decoupling Semantic and Spatial Streams in Vision-Language-Action Models for Autonomous Driving

Continuous Actions from Discrete Minds: Latent-Aligned Planning for End-to-End Autonomous Driving

Xiaomi OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation