cs.AIAug 24, 2026
SaveFrom Generation to Simulation: How Far Are World Models from Being True Simulators?
Organizations: Institute of Systems Engineering, Academy of Military Sciences, Beijing 100101, China
Abstract
With the rapid progress of diffusion models and large-scale video generation, generative world models are increasingly expected to replace traditional simulators, including physics engines, game engines, and reinforcement-learning environments. Yet the remaining distance from generation to simulation lacks a systematic assessment. We present a capability-based study using an external yardstick: eight capabilities of a traditional simulator, namely asset construction, physics engine, interaction, controllability, stability, state feedback, diversity, and evaluation metrics. We trace three main technical routes--latent dynamics, video generation, and joint-embedding prediction--and map exactly 200 representative works published from 2018 to June 2026 onto these capabilities. Our analysis shows that world models have achieved functional substitution in interaction and controllability for specific scenarios, but remain short of traditional simulators in formal guarantees of physical laws, structured state feedback, and reproducible long-horizon evolution. State feedback is the most neglected cross-route shortcoming: only 6 of 163 implementation papers expose a runtime interface for querying entity states or physical parameters. We identify six research directions: formalized physics, a unified action interface, first-class state feedback, long-horizon stability, downstream-utility evaluation, and cross-route hybridization. Project page: https://github.com/AtongWang/world-model-simulators
Figures & tables
Fig. 1: Overview of the survey. Centred on “world model as simulator”, the eight capabilities of a traditional simulator are arranged around it: asset construction, physics engine, interaction, controllability, stability, state feedback, diversity, and evaluation metrics. Each capability panel reports its coverage as a principal contribution across the 200-paper corpus, a status tag ( strength / structural gap / neutral ), and representative works. Controllability (62.5%), interaction (40%), and stability (40%) are tagged as strengths; asset construction (19%), the physics engine (17%), and state feedback (22.5%) as cross-route structural gaps; and diversity (23%) and evaluation metrics (25.5%) remain neutral.
Fig. 2: Research attention across the eight simulator capabilities and the corpus composition, 2018–2026. (a) Cumulative papers contributing each capability, drawn as nested year slices (to 2020/2022/2024/2026); the radial value is the number of papers for which that capability is a principal contribution, and the number at each vertex is its 2026 total. Controllability, interaction, and stability dominate, while asset construction, the physics engine, and state feedback stay low. (b) The 200-paper corpus by publication year, split into published (72) and preprint / other (128); the corpus is increasingly preprint-dominated in 2025–2026.
Fig. 3: Three principal routes to world modelling, distinguished by prediction space. A latent-dynamics model encodes observations into a compact state and rolls that state forward for decoding and control; a video-generative model predicts future observations directly in pixel or token space through diffusion or autoregressive generation; and a joint-embedding predictive model predicts a future representation against a target embedding without reconstructing pixels. The routes therefore expose different simulator interfaces: compact state for imagination and control, rendered observations for direct inspection, and abstract embeddings for efficient downstream prediction and planning.
| Method | Venue | Route | C1 | C2 | C3 | C4 | C5 | C6 | C7 | C8 | Category | Highlight |
| C1 – Asset Construction | ||||||||||||
| Lyra2 [ 132 ] | arXiv’26 | 3D/4D | ✓ | ✓ | ✓ | Asset generation | Memory routing removes drift | |||||
| DecMem [ 180 ] | arXiv’26 | Diff. | ✓ | ✓ | Asset generation | Decoupled memory, minute-scale | ||||||
| CP4D [ 211 ] | arXiv’26 | 3D/4D | ✓ | ✓ | ✓ | Asset generation | Physics-aware compositional 4D | |||||
| Prisma-World [ 134 ] | arXiv’26 | Diff. | ✓ | ✓ | ✓ | Asset as condition | Multi-agent cross-view consistency | |||||
| MoVerse [ 208 ] | arXiv’26 | 3D/4D | ✓ | ✓ | ✓ | Asset generation | Panorama-Gaussian navigable world | |||||
TABLE I: Representative simulator-capable world models, grouped by their primary simulator capability. Each work is marked on every capability it substantively contributes (C1–C8), with a Category following the taxonomy of Section IV and a one-line Highlight . For C8 we list representative evaluation benchmarks. Routes: Latent (latent dynamics), AR (autoregressive), Diff. (diffusion), 3D/4D, Occ. (occupancy), Bench. (benchmark); arXiv venues are preprints as of the search cutoff.
| Route | C1 | C2 | C3 | C4 | C5 | C6 | C7 | C8 |
| Latent dynamics/action | 2/29 | 5/29 | 11/29 | 18/29 | 11/29 | 10/29 | 9/29 | 3/29 |
| Autoregressive | 2/37 | 3/37 | 21/37 | 25/37 | 23/37 | 7/37 | 8/37 | 7/37 |
| Diffusion | 12/63 | 11/63 | 26/63 | 52/63 | 27/63 | 13/63 | 14/63 | 9/63 |
| JEPA predictive | 0/5 | 0/5 | 1/5 | 5/5 | 4/5 | 0/5 | 1/5 | 0/5 |
| Explicit 3D/4D | 20/25 | 5/25 | 6/25 | 11/25 | 8/25 | 6/25 | 7/25 | 1/25 |
| Occupancy | 1/4 | 0/4 | 1/4 | 3/4 | 1/4 | 3/4 | 1/4 | 1/4 |
TABLE II: Model-family principal-topic counts for the 163 implementation papers. Columns denote C1 asset construction, C2 physics engine, C3 interaction, C4 controllability, C5 stability, C6 state feedback, C7 diversity, and C8 evaluation metrics. Each cell reports , where is the complete family denominator and is the number carrying that principal-topic label. Counts measure research attention, not capability attainment.
Fig. 4: Structured assets as conditional input to a world model. BEV road layouts, HD maps, and 3D bounding boxes specify the geometry and traffic configuration supplied to a video-diffusion world model, which produces a temporally evolving, multi-view street-scene rollout consistent with those conditions.
Fig. 5: Generation of a navigable 3D/4D world from sparse input. A single image or text description conditions a world model that constructs an environment with spatial extent and renders consistent observations along a user-controlled camera path.
Fig. 6: A unified platform for persistent and editable world assets. Terrain, buildings, vegetation, roads, and dynamic agents are maintained as separable layers; instance-level edits update the persistent scene before the world model generates a recomposed rollout.
Fig. 7: A world model with an explicit differentiable physics core. Object-centric position, velocity, and force states are advanced under actions with gradients propagated through the dynamics; a state-guided neural renderer then converts the physical trajectory into observed frames.
Fig. 8: Soft physical constraints in a generative world model. A pretrained geometry or physics model supplies a dense scene-flow or 4D-correspondence prior as a training constraint, while the conditioned video-diffusion model remains responsible for generating the evolving observations.
Fig. 9: Physics evaluation through paired causal intervention. A baseline and an intervened rollout share the scene and action sequence but differ in one physical variable, such as object mass; evaluation asks whether the resulting difference follows the expected physical consequence.
Fig. 10: Camera-level interaction in a real-time world model. Controller inputs such as movement, turning, and zoom update the model state and produce the next observation, forming an action-in–frame-out loop for interactive viewpoint navigation.
Fig. 11: Object-level interaction with persistent state. An object action updates the world state, and the modified state must remain valid across subsequent viewpoint changes; in the illustrated sequence, the door remains open after the camera turns away and returns.
Fig. 12: A world model as a programmable environment. A policy interacts through an API carrying actions, observations, rewards, reset, random seed, and state queries, thereby closing the rollout and policy-update loop required for reinforcement-learning training.
Fig. 13: Controllability of world models across action-representation levels. Explicit signals such as keyboard input, camera pose, and 3D trajectories provide precise control but transfer poorly; language and multimodal conditions express human-readable intent while leaving physical execution ambiguous; and latent actions can be learned from passive video and transferred across environments, but are not directly interpretable. Moving from explicit to latent control therefore trades physical precision for abstraction and generality, making cross-level action mapping an open problem.
Fig. 14: Self-forcing training against autoregressive drift. Training only on clean history creates a train–test mismatch under which errors compound during rollout; conditioning on the model’s own past outputs teaches recovery from imperfect context and improves long-horizon geometric consistency.
Fig. 15: Implicit versus explicit memory for consistent world rollouts. Implicit memory retrieves view-relevant internal states but may fade with horizon length; explicit 3D memory stores observations in a persistent geometric map and retrieves them when the camera revisits a location.
Fig. 16: Causal consistency under viewpoint change. After an object is moved and the camera turns away, a model without persistent state may place the object back at its former location; a consistent world model retains the intervened state when the object is observed again.
| Variable | Operational meaning | Present | Absent | Unresolved |
| B1 | Ego information: camera/ego pose, vehicle state, robot joint or end-effector state | 65 | 74 | 24 |
| B2 | Sensor output: depth, semantics, LiDAR, normals, occupancy, or comparable sensor modality | 45 | 113 | 5 |
| B3 | Task signal: reward, termination, or an equivalent task-evaluation signal | 20 | 137 | 6 |
| B4 | Runtime entity/physics annotations: named non-ego entity states or physical parameters | 6 | 157 | 0 |
| B5 | Closed-loop interaction: repeated action-conditioned generation or model-mediated control loop | 87 | 56 | 20 |
TABLE III: State-interface audit for all 163 implementation papers. “Present”, “absent”, and “unresolved” are reported separately; each row sums to 163. B4 requires a runtime, queryable entity/physics annotation interface and excludes sensor-space estimates, latent vectors, and training-only labels. B5 is a cross-cutting closed-loop attribute.
Fig. 17: Structured feedback through multi-task auxiliary outputs. Parallel prediction heads decode a shared temporal representation into aligned RGB, depth, semantic segmentation, occupancy, and LiDAR outputs, providing downstream systems with complementary views of the same simulated state.
Fig. 18: Structured state derived from physical simulation. An explicit object-centric trajectory exposes position, velocity, force, and mass over time, from which a controller can directly consume quantities such as contact force and per-pixel 3D pointmaps.
Fig. 19: Latent representation as machine-facing state feedback. The world model predicts a compact latent trajectory rather than reconstructing pixels, and a policy or planner selects actions by optimizing directly in that representation space; efficiency is gained at the cost of interpretability.
Fig. 20: Scale-driven diversity in a world foundation model. Heterogeneous data across environments, conditions, and embodiments are compressed into a pretrained model that produces diverse evolving rollouts, including zero-shot coverage beyond the scene types emphasized during training.
Fig. 21: Compositional diversity through language. A language parser decomposes instructions into reusable primitives whose new combinations condition a world model to generate novel or rare rollouts, including long-tail interaction and accident scenarios.
Fig. 22: Parametrically controllable diversity. Independent controls over appearance, camera pose, and scene layout allow a world model to sweep one factor while keeping the others fixed, separating purposeful variation from unconstrained stochastic generation.
Fig. 23: Three complementary lenses for evaluating world models as simulators. Generation-quality evaluation measures visual fidelity, temporal consistency, and action following, but remains predominantly open loop. Closed-loop embodied evaluation tests multi-step interaction, task success, and policy transfer. Physical-plausibility evaluation uses causal interventions to assess whether rollouts preserve physical response and state over time. Together, these lenses shift the target from surface similarity toward downstream simulator utility.