cs.ROMar 10, 2026

Latent World Models for Automated Driving: A Unified Taxonomy, Evaluation Framework, and Open Challenges

Authors: Rongxiang Zeng, Yongqi Dong

Organizations: RWTH Aachen University, Templergraben 55, 52062 Aachen, Germany · Delft University of Technology, 2628 CN Delft, The Netherlands

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Emerging generative world models and vision-language-action (VLA) systems are rapidly reshaping automated driving by enabling scalable simulation, long-horizon forecasting, and capability-rich decision making. Across these directions, latent representations serve as the central computational substrate: they compress high-dimensional multi-sensor observations, enable temporally coherent rollouts, and provide interfaces for planning, reasoning, and controllable generation. This paper proposes a unifying latent-space framework that synthesizes recent progress in world models for automated driving. The framework organizes the design space by the target and form of latent representations (latent worlds, latent actions, latent generators; continuous states, discrete tokens, and hybrids) and by structural priors for geometry, topology, and semantics. Building on this taxonomy, the paper articulates five cross-cutting internal mechanics (i.e, structural isomorphism, long-horizon temporal stability, semantic and reasoning alignment, value-aligned objectives and post-training, as well as adaptive computation and deliberation) and connects these design choices to robustness, generalization, and deployability. The work also proposes concrete evaluation prescriptions, including a closed-loop metric suite and a resource-aware deliberation cost, designed to reduce the open-loop / closed-loop mismatch. Finally, the paper identifies actionable research directions toward advancing latent world model for decision-ready, verifiable, and resource-efficient automated driving.

Figures & tables

Fig. 1: A visual roadmap visualization of the paper.
Fig. 2: Taxonomy of world models for automated driving : A conceptual overview of Neural Simulation , Latent Planning , Data Synthesis , and Cognitive Reasoning within a unified latent-centric framework.
Method Name AbbreviationLatent StructureLatent Dynamics ModelKey MechanismInput Source →\to Output
Part 1: Neural Simulation & World Modeling
BEVWorld [ 29 ]Multimodal BEV GridTransformerMultimodal Tokenization via Ray-casting Rendering and DiffusionCamera, LiDAR →\to Point Cloud, Video
CoGen [ 36 ]2D + 3D ConditionsDiffusion3D Consistent Video Generation via Adaptive Geometric ConditioningCamera →\to Video
Drive-OccWorld [ 37 ]3D OccupancyTransformer4D Occupancy Forecasting with Semantic and Motion NormalizationCamera →\to Occupancy, Flow
DriveDreamer [ 1 ]2D LatentDiffusionTwo-Stage Training for Structured Traffic Constraints ComprehensionCamera, 3D Box →\to Video
DriveScape [ 38 ]2D + 3D Road PriorTransformerBi-Directional Modulated Feature Fusion for Multi-View SynthesisCamera →\to Video
TABLE I: Comparison of Representative World Model Methods: paradigms and attributes
Fig. 3: Internal mechanics of latent world models for automated driving.
Fig. 4: Evaluation paradigms for latent world models in automated driving : open-loop evaluation (left), and closed-loop evaluation (right).
Benchmark Datasets or PlatformsTypePrimary Evaluation FocusPros (+) & Cons (-)Representative Methods
Category 1: Static Real-World Datasets (Open-Loop)
nuScenesReal-world LogPerception, Generation (FID), Prediction (L2)+ Multi-modal standard; Rich annotations - Causal gap (No interaction)GenAD [ 46 ] , DriveWorld [ 26 ] , Drive-OccWorld [ 37 ]
Waymo Open Dataset (WOD)Real-world LogLarge-scale Scaling, Long-horizon Prediction+ High volume for foundation models - High computational costGAIA-2 [ 39 ] , LatentDriver [ 48 ]
Category 2: Interactive Simulation Platforms (Closed-Loop)
CARLA (LB 2.0)Physics EnginePlanning Policy, Safety, Rule Compliance+ Fully interactive; Reproducible dynamics - Visual Sim-to-Real gapThink2Drive [ 4 ] , Raw2Drive [ 28 ] , MomAD [ 50 ] , Alpamayo-R1 [ 56 ]
LGSVL (SVL Simulator)Physics EngineFull AV Stack Integration+ High-fidelity sensor models; ROS compatibility - Heavy system dependency–
TABLE II: Evaluation ecosystem for latent world models : Representative datasets and simulation platforms.
Fig. 5: Open challenges for latent world models in autonomous driving.
Fig. 6: Illustration of future horizons and research directions for latent world models in automated driving.
Figure 9
Figure 10
More in this topic

World Models for Autonomous Driving

World models for autonomous driving predict how scenes and traffic evolve under candidate actions to support planning, control, and simulation.

See all 142 papers

Explore similar work

CardsList
Jul 23, 2026

HyWorldVLA: A Vision-Language-Action Model with Hybrid World Modeling for Autonomous Driving

Vision-Language-Action (VLA) models augmented with world modeling represent a promising paradigm for end-to-end autonomous driving. While pixel-level future prediction enables fine-grained spatiotemporal reasoning, it compromises robustness in noisy driving scenarios. Conversely, latent-based world models alleviate this sensitivity but often incur limited interpretability and representational degradation due to absent pixel-level grounding. To reconcile this trade-off, we propose HyWorldVLA, a hybrid world-VLA framework that unifies pixel-level supervision and latent representation learning. In the pre-training stage, HyWorldVLA predicts video latents encoded by a pre-trained video VAE, while simultaneously reconstructing video frames to provide precise pixel-level grounding. During the subsequent co-fine-tuning phase, the model exclusively predicts latent features, which are fed into an action expert to generate trajectories. Extensive experiments on NAVSIM v1 and v2 benchmarks demonstrate that HyWorldVLA significantly outperforms both pixel-based and latent-based world model baselines. Notably, we present the first comprehensive qualitative and quantitative analysis of world model noise robustness in autonomous driving, establishing a new benchmark for evaluating future architectures.

Quanfu Yu, Xian Wu, Hao Xu +1

Automotive New Technology Research Institute, BYD Company Limited

World ModelsAutonomous DrivingVision-Language-Action ModelsWorld Models for Autonomous Driving
AskPDF
Mar 25, 2026

Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Driving

We introduce Latent-WAM, an efficient end-to-end autonomous driving framework that achieves strong trajectory planning through spatially-aware and dynamics-informed latent world representations. Existing world-model-based planners suffer from inadequately compressed representations, limited spatial understanding, and underutilized temporal dynamics, resulting in sub-optimal planning under constrained data and compute budgets. Latent-WAM addresses these limitations with two core modules: a Spatial-Aware Compressive World Encoder (SCWE) that distills geometric knowledge from a foundation model and compresses multi-view images into compact scene tokens via learnable queries, and a Dynamic Latent World Model (DLWM) that employs a causal Transformer to autoregressively predict future world status conditioned on historical visual and ego status representations. Extensive experiments on NAVSIM v2 and HUGSIM demonstrate new state-of-the-art results: 89.3 EPDMS on NAVSIM v2 and 28.9 HD-Score on HUGSIM, surpassing the best prior perception-free method by 3.2 EPDMS with significantly less training data and a compact 104M-parameter model. The code and model weights have been open-sourced at https://github.com/yukki555/Latent-WAM.

Linbo Wang, Yupeng Zheng, Qiang Chen +13

Institute of Automation, Chinese Academy of Sciences · Chongqing Chang’an Technology Co., Ltd · Zhongguancun Academy +2

End-to-End Autonomous DrivingWorld Models for Autonomous DrivingAutonomous Driving PlanningLatent World Models
AskPDF
Sep 3, 2026

Drive-HWM: Hierarchical World Models for Dynamic-Latent Guided Autonomous Driving

World models offer a promising paradigm for autonomous driving by predicting how traffic scenes may evolve and using such predictions to support action generation. However, existing approaches either separate future prediction from action generation or jointly predict them at the same temporal scale, making it difficult to simultaneously achieve long-horizon anticipation and responsive, observation-grounded decision making. We present Drive-HWM, a hierarchical slow--fast world modeling framework that organizes future representation prediction and action generation at complementary temporal scales. The slow world model predicts multi-step future representations to capture extended scene evolution. To explicitly model the abundant motion dynamics in driving environments, we introduce Dynamic-Aware Latents learned through optical-flow prediction. Guided by these future representations, the fast model uses a lightweight multimodal backbone and an autoregressive expert to jointly predict the next frame and the immediate action from the latest observation. Next-frame prediction encourages the fast model to capture imminent scene evolution, while one-step action generation allows decisions to be continuously updated as new observations arrive. Extensive experiments on NAVSIM v1 and v2 demonstrate the strong driving performance of Drive-HWM. Comprehensive ablation studies further validate the effectiveness of the hierarchical slow--fast design, dynamics-aware future representations, and joint next-frame and action prediction.

Zhaoxin Fan, Tianbao Zhang, Wenjun Wu +5

School of Artificial Intelligence, Beihang University, Beijing, China · Shanghai Jiao Tong University, Shanghai, China · Dim12 AI +3

World ModelsHierarchical World ModelsWorld Models for Autonomous DrivingLatent World ModelsAutonomous Driving Planning
AskPDF