Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Driving

Qiqi Liu; Huan Xu; Jingyu Li; Bin Sun; Zhihui Hao; Dangen She; Xiatian Zhu; Li Zhang

cs.ROMar 28, 2026

Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Driving

Authors: Qiqi Liu, Huan Xu, Jingyu Li, Bin Sun, Zhihui Hao, Dangen She, Xiatian Zhu, Li Zhang

Organizations: Fudan University · Shanghai Innovation Institute · Li Auto Inc. · University of Surrey

Abstract

Autonomous driving requires reasoning about how the environment evolves and planning actions accordingly. Existing world-model-based approaches typically predict future scenes first and plan afterwards, resulting in open-loop imagination that may drift from the actual decision process. In this paper, we present Uni-World VLA, a unified vision-language-action (VLA) model that tightly interleaves future frame prediction and trajectory planning. Instead of generating a full world rollout before planning, our model alternates between predicting future frames and ego actions step by step, allowing planning decisions to be continuously conditioned on the imagined future observations. This interleaved generation forms a closed-loop interaction between world modeling and control, enabling more adaptive decision-making in dynamic traffic scenarios. In addition, we incorporate monocular depth information into frames to provide stronger geometric cues for world modeling, improving long-horizon scene prediction. Experiments on the NAVSIM benchmark show that our approach achieves competitive closed-loop planning performance while producing high-fidelity future frame predictions. These results demonstrate that tightly coupling world prediction and planning is a promising direction for scalable VLA driving systems.

Figures & tables

Figure 1: Different generative paradigms of unified world models for autonomous driving. (a) Unified world models perform video generation and planning as separate tasks; (b) World-conditioned trajectory prediction, where future trajectories are predicted conditioned on the generated world states; (c) Interleaved world modeling and planning (ours). Visual tokens and action queries are generated alternately, forming a closed-loop interaction that respects the temporal causality of driving.

Figure 2: Overview of the paradigm of alternative generation in Uni-World VLA. (a) The construction of multi-model historical information; (b) The interleaved frame-action generative paradigm.

Figure 3: Schematic illustration of training and inference process. (a) Interleaved sequence for joint video generation and trajectory supervision. (b) Causal attention mask. (c) Autoregressive interleaved inference with KV-cache reuse.

Traditional End-to-End Methods
Method	Input	NC $\uparrow$	DAC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	Comf. $\uparrow$	PDMS $\uparrow$
VADv2- $\nu_{8192}$ [ 6 ]	C	97.2	89.1	76.0	91.6	100.0	80.9
UniAD [ 17 ]	C	97.8	91.9	78.8	92.9	100.0	83.4
TransFuser [ 9 ]	C&L	97.7	92.8	79.2	92.8	100.0	84.0
ReCogDrive-IL [ 30 ]	SC	98.1	94.7	80.9	94.2	100.0	86.5
DiffusionDrive [ 33 ]	C&L	98.2	96.2	82.2	94.7	100.0	88.1

Table 1: Comparison with state-of-the-art methods on the NAVSIM test split. PDMS and its sub-metrics (NC, DAC, EP, TTC, and Comfort; see Sec. 4.1 ) evaluate closed-loop driving performance. Input modalities include multi-view cameras (C), single-view camera (SC), and multi-view cameras with LiDAR (C&L). Best results are shown in bold .

Metric & Settings	WoVoGen [ 35 ]	DriveDreamer [ 48 ]	SVD [ 2 , 7 ]	DrivingGPT [ 7 ]	GenAD [ 55 ]	Ours
FVD $\downarrow$	417.7	340.8	227.5	142.6	184.0	141.8
Max Duration/Fps	2.5 s/2 Hz	4 s/2 Hz	4 s/2 Hz	4 s/2 Hz	4 s/2 Hz	4 s/2 Hz
Dataset	nuScenes	nuScenes	NAVSIM	NAVSIM	OpenDV	NAVSIM
View	Multi	Multi	Front	Front	Front	Front

Table 2: Comparison of video generation quality across driving world models. FVD 4.1 measures the realism of generated future video sequences. We additionally report the maximum prediction horizon, frame rate, dataset, and camera view used by each method.

Figure 4: Visualization of predicted frames and BEV trajectories

Pretrain	Future Frames	Depth	NC $\uparrow$	DAC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	Comf. $\uparrow$	PDMS $\uparrow$	FVD $\downarrow$
$\times$	$\times$	$\times$	97.1	91.4	77.4	91.5	100.0	82.1	-
$\checkmark$	$\times$	$\times$	98.8	95.8	82.0	95.8	100.0	88.2	-
$\checkmark$	$\checkmark$	$\times$	98.8	96.5	82.9	96.4	100.0	89.2	164.2
$\checkmark$	$\checkmark$	$\checkmark$	98.7	96.7	83.2	96.1	100.0	89.4	141.8

Table 3: Ablation study on the effects of pretraining, future-frame modeling, and depth conditioning. NC, DAC, EP, TTC, Comfort, and PDMS measure planning quality, while FVD evaluates the quality of generated future frames.

Figure 5: Comparison of predicted future frames with and without depth fusion

Scheme	NC $\uparrow$	DAC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	Comf. $\uparrow$	PDMS $\uparrow$
A-cross-frequency alternation	98.8	96.4	81.2	96.0	100.0	88.3
B-high-frequency actions-frames	98.7	95.1	77.3	96.4	100.0	86.1
C-hybrid dense-then-coarse	98.7	95.8	80.6	96.0	100.0	87.8
D-sliding 1s action windows	98.6	94.5	78.4	95.2	99.7	85.7
E-2Hz aligned interleaving(Ours)	98.8	96.5	82.9	96.4	100.0	89.2

Table 4: Ablation over five alternative generation schemes (without depth fusion).

Historical Visual Info.	NC $\uparrow$	DAC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	Comf. $\uparrow$	PDMS $\uparrow$	FVD $\downarrow$
2.0 s Context+Dynamic (Ours)	98.8	96.5	82.9	96.4	100.0	89.2	164.2
1.0 s Context+Dynamic	99.0	96.4	81.4	96.7	100.0	88.8	170.7
Context Only	98.6	96.8	82.3	96.2	100.0	89.1	165.5
Dynamic Only	97.4	90.8	76.2	92.3	100.0	81.7	203.6

Table 5: Effect of historical visual information on planning (without depth fusion).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Details of contextual, dynamic and action tokens

Figure 7: Visualization of predicted future frames (a) and the corresponding ground truth (b) in representative and challenging driving scenarios.

Figure 8: BEV visualization of the corresponding six scenarios (from left to right and top to bottom). Green polyline: ground-truth trajectory; red polyline: planned future trajectory.

Explore similar work

Learning Vision-Language-Action World Models for Autonomous Driving
Apr 10, 2026Guoqing Wang, Pin Tang, Xiangxuan Ren +3World Models for Autonomous Driving Vision-Language-Action Models
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University · Central Research Institute, Huawei
UniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Driving
Jan 7, 2026Zhexiao Xiong, Xin Ye, Burhan Yaman +5Vision-Language Models World Model Learning
Bosch Research North America · Bosch Center for Artificial Intelligence (BCAI) · Arizona State University +2
DriveWLA: Unifying Future Dynamics and Scene Semantics through World-Language-Action Modeling for Autonomous Driving
Oct 13, 2026Shuzheng Wang, Haoxuan Li, Shengzhen Li +5Autonomous Driving Planning VLMs for Autonomous Driving
Huazhong University of Science and Technology · Yinwang Intelligent Technology Co. Ltd.

Apr 10, 2026J/K move · Enter open · S save

Learning Vision-Language-Action World Models for Autonomous Driving

Guoqing Wang, Pin Tang, Xiangxuan Ren, Guodongfang Zhao+2

MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University · Central Research Institute, Huawei

World Models for Autonomous Driving Vision-Language-Action Models Autonomous Driving Planning

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have recently achieved notable progress in end-to-end autonomous driving by integrating perception, reasoning, and control within a unified multimodal framework. However, they often lack explicit modeling of temporal dynamics and global world consistency, which limits their foresight and safety. In contrast, world models can simulate plausible future scenes but generally struggle to reason about or evaluate the imagined future they generate. In this work, we present VLA-World, a simple yet effective VLA world model that unifies predictive imagination with reflective reasoning to improve driving foresight. VLA-World first uses an action-derived feasible trajectory to guide the generation of the next-frame image, capturing rich spatial and temporal cues that describe how the surrounding environment evolves. The model then reasons over this self-generated future imagined frame to refine the predicted trajectory, achieving higher performance and better interpretability. To support this pipeline, we curate nuScenes-GR-20K, a generative reasoning dataset derived from nuScenes, and employ a three-stage training strategy that includes pretraining, supervised fine-tuning, and reinforcement learning. Extensive experiments demonstrate that VLA-World consistently surpasses state-of-the-art VLA and world-model baselines on both planning and future-generation benchmarks. Project page: https://vlaworld.github.io

Abstract

Explore similar work

Learning Vision-Language-Action World Models for Autonomous Driving

UniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Driving

DriveWLA: Unifying Future Dynamics and Scene Semantics through World-Language-Action Modeling for Autonomous Driving