Latent World Models for Automated Driving: A Unified Taxonomy, Evaluation Framework, and Open Challenges
Organizations: RWTH Aachen University, Templergraben 55, 52062 Aachen, Germany · Delft University of Technology, 2628 CN Delft, The Netherlands
Abstract
Emerging generative world models and vision-language-action (VLA) systems are rapidly reshaping automated driving by enabling scalable simulation, long-horizon forecasting, and capability-rich decision making. Across these directions, latent representations serve as the central computational substrate: they compress high-dimensional multi-sensor observations, enable temporally coherent rollouts, and provide interfaces for planning, reasoning, and controllable generation. This paper proposes a unifying latent-space framework that synthesizes recent progress in world models for automated driving. The framework organizes the design space by the target and form of latent representations (latent worlds, latent actions, latent generators; continuous states, discrete tokens, and hybrids) and by structural priors for geometry, topology, and semantics. Building on this taxonomy, the paper articulates five cross-cutting internal mechanics (i.e, structural isomorphism, long-horizon temporal stability, semantic and reasoning alignment, value-aligned objectives and post-training, as well as adaptive computation and deliberation) and connects these design choices to robustness, generalization, and deployability. The work also proposes concrete evaluation prescriptions, including a closed-loop metric suite and a resource-aware deliberation cost, designed to reduce the open-loop / closed-loop mismatch. Finally, the paper identifies actionable research directions toward advancing latent world model for decision-ready, verifiable, and resource-efficient automated driving.
Figures & tables
| Method Name Abbreviation | Latent Structure | Latent Dynamics Model | Key Mechanism | Input Source Output |
| Part 1: Neural Simulation & World Modeling | ||||
| BEVWorld [ 29 ] | Multimodal BEV Grid | Transformer | Multimodal Tokenization via Ray-casting Rendering and Diffusion | Camera, LiDAR Point Cloud, Video |
| CoGen [ 36 ] | 2D + 3D Conditions | Diffusion | 3D Consistent Video Generation via Adaptive Geometric Conditioning | Camera Video |
| Drive-OccWorld [ 37 ] | 3D Occupancy | Transformer | 4D Occupancy Forecasting with Semantic and Motion Normalization | Camera Occupancy, Flow |
| DriveDreamer [ 1 ] | 2D Latent | Diffusion | Two-Stage Training for Structured Traffic Constraints Comprehension | Camera, 3D Box Video |
| DriveScape [ 38 ] | 2D + 3D Road Prior | Transformer | Bi-Directional Modulated Feature Fusion for Multi-View Synthesis | Camera Video |
| Benchmark Datasets or Platforms | Type | Primary Evaluation Focus | Pros (+) & Cons (-) | Representative Methods |
| Category 1: Static Real-World Datasets (Open-Loop) | ||||
| nuScenes | Real-world Log | Perception, Generation (FID), Prediction (L2) | + Multi-modal standard; Rich annotations - Causal gap (No interaction) | GenAD [ 46 ] , DriveWorld [ 26 ] , Drive-OccWorld [ 37 ] |
| Waymo Open Dataset (WOD) | Real-world Log | Large-scale Scaling, Long-horizon Prediction | + High volume for foundation models - High computational cost | GAIA-2 [ 39 ] , LatentDriver [ 48 ] |
| Category 2: Interactive Simulation Platforms (Closed-Loop) | ||||
| CARLA (LB 2.0) | Physics Engine | Planning Policy, Safety, Rule Compliance | + Fully interactive; Reproducible dynamics - Visual Sim-to-Real gap | Think2Drive [ 4 ] , Raw2Drive [ 28 ] , MomAD [ 50 ] , Alpamayo-R1 [ 56 ] |
| LGSVL (SVL Simulator) | Physics Engine | Full AV Stack Integration | + High-fidelity sensor models; ROS compatibility - Heavy system dependency | – |
World Models for Autonomous Driving
World models for autonomous driving predict how scenes and traffic evolve under candidate actions to support planning, control, and simulation.
See all 142 papers