With the rapid progress of diffusion models and large-scale video generation, generative world models are increasingly expected to replace traditional simulators, including physics engines, game engines, and reinforcement-learning environments. Yet the remaining distance from generation to simulation lacks a systematic assessment. We present a capability-based study using an external yardstick: eight capabilities of a traditional simulator, namely asset construction, physics engine, interaction, controllability, stability, state feedback, diversity, and evaluation metrics. We trace three main technical routes--latent dynamics, video generation, and joint-embedding prediction--and map exactly 200 representative works published from 2018 to June 2026 onto these capabilities. Our analysis shows that world models have achieved functional substitution in interaction and controllability for specific scenarios, but remain short of traditional simulators in formal guarantees of physical laws, structured state feedback, and reproducible long-horizon evolution. State feedback is the most neglected cross-route shortcoming: only 6 of 163 implementation papers expose a runtime interface for querying entity states or physical parameters. We identify six research directions: formalized physics, a unified action interface, first-class state feedback, long-horizon stability, downstream-utility evaluation, and cross-route hybridization. Project page: https://github.com/AtongWang/world-model-simulators
Figures & tables
Fig. 1: Overview of the survey. Centred on “world model as simulator”, the eight capabilities of a traditional simulator are arranged around it: asset construction, physics engine, interaction, controllability, stability, state feedback, diversity, and evaluation metrics. Each capability panel reports its coverage as a principal contribution across the 200-paper corpus, a status tag ( strength / structural gap / neutral ), and representative works. Controllability (62.5%), interaction (40%), and stability (40%) are tagged as strengths; asset construction (19%), the physics engine (17%), and state feedback (22.5%) as cross-route structural gaps; and diversity (23%) and evaluation metrics (25.5%) remain neutral.
Fig. 2: Research attention across the eight simulator capabilities and the corpus composition, 2018–2026. (a) Cumulative papers contributing each capability, drawn as nested year slices (to 2020/2022/2024/2026); the radial value is the number of papers for which that capability is a principal contribution, and the number at each vertex is its 2026 total. Controllability, interaction, and stability dominate, while asset construction, the physics engine, and state feedback stay low. (b) The 200-paper corpus by publication year, split into published (72) and preprint / other (128); the corpus is increasingly preprint-dominated in 2025–2026.
Fig. 3: Three principal routes to world modelling, distinguished by prediction space. A latent-dynamics model encodes observations into a compact state and rolls that state forward for decoding and control; a video-generative model predicts future observations directly in pixel or token space through diffusion or autoregressive generation; and a joint-embedding predictive model predicts a future representation against a target embedding without reconstructing pixels. The routes therefore expose different simulator interfaces: compact state for imagination and control, rendered observations for direct inspection, and abstract embeddings for efficient downstream prediction and planning.
Method
Venue
Route
C1
C2
C3
C4
C5
C6
C7
C8
Category
Highlight
C1 – Asset Construction
Lyra2 [ 132 ]
arXiv’26
3D/4D
✓
✓
✓
Asset generation
Memory routing removes drift
DecMem [ 180 ]
arXiv’26
Diff.
✓
✓
Asset generation
Decoupled memory, minute-scale
CP4D [ 211 ]
arXiv’26
3D/4D
✓
✓
✓
Asset generation
Physics-aware compositional 4D
Prisma-World [ 134 ]
arXiv’26
Diff.
✓
✓
✓
Asset as condition
Multi-agent cross-view consistency
MoVerse [ 208 ]
arXiv’26
3D/4D
✓
✓
✓
Asset generation
Panorama-Gaussian navigable world
TABLE I: Representative simulator-capable world models, grouped by their primary simulator capability. Each work is marked on every capability it substantively contributes (C1–C8), with a Category following the taxonomy of Section IV and a one-line Highlight . For C8 we list representative evaluation benchmarks. Routes: Latent (latent dynamics), AR (autoregressive), Diff. (diffusion), 3D/4D, Occ. (occupancy), Bench. (benchmark); arXiv venues are preprints as of the search cutoff.
Route
C1
C2
C3
C4
C5
C6
C7
C8
Latent dynamics/action
2/29
5/29
11/29
18/29
11/29
10/29
9/29
3/29
Autoregressive
2/37
3/37
21/37
25/37
23/37
7/37
8/37
7/37
Diffusion
12/63
11/63
26/63
52/63
27/63
13/63
14/63
9/63
JEPA predictive
0/5
0/5
1/5
5/5
4/5
0/5
1/5
0/5
Explicit 3D/4D
20/25
5/25
6/25
11/25
8/25
6/25
7/25
1/25
Occupancy
1/4
0/4
1/4
3/4
1/4
3/4
1/4
1/4
TABLE II: Model-family × principal-topic counts for the 163 implementation papers. Columns denote C1 asset construction, C2 physics engine, C3 interaction, C4 controllability, C5 stability, C6 state feedback, C7 diversity, and C8 evaluation metrics. Each cell reports n/N , where N is the complete family denominator and n is the number carrying that principal-topic label. Counts measure research attention, not capability attainment.
Fig. 4: Structured assets as conditional input to a world model. BEV road layouts, HD maps, and 3D bounding boxes specify the geometry and traffic configuration supplied to a video-diffusion world model, which produces a temporally evolving, multi-view street-scene rollout consistent with those conditions.
Fig. 5: Generation of a navigable 3D/4D world from sparse input. A single image or text description conditions a world model that constructs an environment with spatial extent and renders consistent observations along a user-controlled camera path.
Fig. 6: A unified platform for persistent and editable world assets. Terrain, buildings, vegetation, roads, and dynamic agents are maintained as separable layers; instance-level edits update the persistent scene before the world model generates a recomposed rollout.
Fig. 7: A world model with an explicit differentiable physics core. Object-centric position, velocity, and force states are advanced under actions with gradients propagated through the dynamics; a state-guided neural renderer then converts the physical trajectory into observed frames.
Fig. 8: Soft physical constraints in a generative world model. A pretrained geometry or physics model supplies a dense scene-flow or 4D-correspondence prior as a training constraint, while the conditioned video-diffusion model remains responsible for generating the evolving observations.
Fig. 9: Physics evaluation through paired causal intervention. A baseline and an intervened rollout share the scene and action sequence but differ in one physical variable, such as object mass; evaluation asks whether the resulting difference follows the expected physical consequence.
Fig. 10: Camera-level interaction in a real-time world model. Controller inputs such as movement, turning, and zoom update the model state and produce the next observation, forming an action-in–frame-out loop for interactive viewpoint navigation.
Fig. 11: Object-level interaction with persistent state. An object action updates the world state, and the modified state must remain valid across subsequent viewpoint changes; in the illustrated sequence, the door remains open after the camera turns away and returns.
Fig. 12: A world model as a programmable environment. A policy interacts through an API carrying actions, observations, rewards, reset, random seed, and state queries, thereby closing the rollout and policy-update loop required for reinforcement-learning training.
Fig. 13: Controllability of world models across action-representation levels. Explicit signals such as keyboard input, camera pose, and 3D trajectories provide precise control but transfer poorly; language and multimodal conditions express human-readable intent while leaving physical execution ambiguous; and latent actions can be learned from passive video and transferred across environments, but are not directly interpretable. Moving from explicit to latent control therefore trades physical precision for abstraction and generality, making cross-level action mapping an open problem.
Fig. 14: Self-forcing training against autoregressive drift. Training only on clean history creates a train–test mismatch under which errors compound during rollout; conditioning on the model’s own past outputs teaches recovery from imperfect context and improves long-horizon geometric consistency.
Fig. 15: Implicit versus explicit memory for consistent world rollouts. Implicit memory retrieves view-relevant internal states but may fade with horizon length; explicit 3D memory stores observations in a persistent geometric map and retrieves them when the camera revisits a location.
Fig. 16: Causal consistency under viewpoint change. After an object is moved and the camera turns away, a model without persistent state may place the object back at its former location; a consistent world model retains the intervened state when the object is observed again.
Variable
Operational meaning
Present
Absent
Unresolved
B1
Ego information: camera/ego pose, vehicle state, robot joint or end-effector state
Task signal: reward, termination, or an equivalent task-evaluation signal
20
137
6
B4
Runtime entity/physics annotations: named non-ego entity states or physical parameters
6
157
0
B5
Closed-loop interaction: repeated action-conditioned generation or model-mediated control loop
87
56
20
TABLE III: State-interface audit for all 163 implementation papers. “Present”, “absent”, and “unresolved” are reported separately; each row sums to 163. B4 requires a runtime, queryable entity/physics annotation interface and excludes sensor-space estimates, latent vectors, and training-only labels. B5 is a cross-cutting closed-loop attribute.
Fig. 17: Structured feedback through multi-task auxiliary outputs. Parallel prediction heads decode a shared temporal representation into aligned RGB, depth, semantic segmentation, occupancy, and LiDAR outputs, providing downstream systems with complementary views of the same simulated state.
Fig. 18: Structured state derived from physical simulation. An explicit object-centric trajectory exposes position, velocity, force, and mass over time, from which a controller can directly consume quantities such as contact force and per-pixel 3D pointmaps.
Fig. 19: Latent representation as machine-facing state feedback. The world model predicts a compact latent trajectory rather than reconstructing pixels, and a policy or planner selects actions by optimizing directly in that representation space; efficiency is gained at the cost of interpretability.
Fig. 20: Scale-driven diversity in a world foundation model. Heterogeneous data across environments, conditions, and embodiments are compressed into a pretrained model that produces diverse evolving rollouts, including zero-shot coverage beyond the scene types emphasized during training.
Fig. 21: Compositional diversity through language. A language parser decomposes instructions into reusable primitives whose new combinations condition a world model to generate novel or rare rollouts, including long-tail interaction and accident scenarios.
Fig. 22: Parametrically controllable diversity. Independent controls over appearance, camera pose, and scene layout allow a world model to sweep one factor while keeping the others fixed, separating purposeful variation from unconstrained stochastic generation.
Fig. 23: Three complementary lenses for evaluating world models as simulators. Generation-quality evaluation measures visual fidelity, temporal consistency, and action following, but remains predominantly open loop. Closed-loop embodied evaluation tests multi-step interaction, task success, and policy transfer. Physical-plausibility evaluation uses causal interventions to assess whether rollouts preserve physical response and state over time. Together, these lenses shift the target from surface similarity toward downstream simulator utility.
A simulator summarizes an environment in a state that users can read and change. Generative models learn representations of the world without explicit specification, but offer no built-in way to edit that world's state. We investigate the conditions under which a generative model can act as a simulator through editability: whether changing a model's latent representation produces the next-step outputs we get by editing a simulator's state. Plainly stated, our goal is to find a mapping between a generative model's latent state and the simulator state, where the model is trained on observations alone and the mapping is learned afterward. We test two kinds of mappings, both learned from observations paired with simulator states. In one, we train probes to decode the simulator state from the model, then steer along them. In the other, we learn the mapping directly with an inverse map, which can succeed even where steering fails. We study both in two contrasting environments: the discrete board game Othello, and Rayworld, our own continuous world of moving discs seen through partial renderings, closer to physical scenes and video. We find that which mapping succeeds depends strongly on the environment, even though the state is decodable in every variant. In Othello, editability falls smoothly as the game is simplified, and even small rule changes have large consequences. In Rayworld, probes fail to steer along the simulator's continuous state, yet an inverse map sets the model from that state anyway. A model can therefore be driven by a simulator it may have never learned.
World simulators can provide safe and scalable environments for training Physical AI systems before real-world deployment. Large video generation models are emerging as a promising basis for such simulators because they can generate diverse and realistic visual futures. However, using them as world simulators requires physically faithful video continuations, namely, generated videos that preserve the physical state implied by the conditioning input, and evolve in ways consistent with basic physical principles. We propose PhyWorld, a video generation world model designed to produce temporally coherent and physically faithful scene continuations through two-stage post-training. In the first stage, we improve video-to-video continuation with flow matching fine-tuning, encouraging stable visual attributes and coherent motion dynamics across frames. In the second stage, we align generated dynamics with physical principles using Direct Preference Optimization (DPO) over physics preference pairs, guiding the model toward outputs with higher physical plausibility. To evaluate PhyWorld, we use both standard video-quality benchmarks and a dedicated physical-faithfulness benchmark with per-law scoring. Experiments show that PhyWorld improves video consistency, achieving an average score of 0.769 on VBench compared with 0.756 or below for state-of-the-art baselines. PhyWorld also improves physical plausibility, reaching an average score of 3.09 on our physical-faithfulness benchmark compared with 2.99 for the strongest baseline. These results suggest that post-training large video generation models with continuation and physics-preference signals can make them more effective world simulators for Physical AI.
Pu Zhao, Juyi Lin, Timothy Rupprecht +10
Northeastern University · University of Georgia · Tulane University +1
World models have emerged as a powerful paradigm for building interactive simulation environments, with recent video-based approaches demonstrating impressive progress in generating visually plausible dynamics. However, because these models typically infer dynamics from video and represent them in latent states, they do not explicitly enforce physical constraints. As a result, the generated video rollouts are not physically plausible, exhibiting unstable contacts, distorted shapes, or inconsistent motion. In this paper, we present an agentic framework constructing physics-based world models through executable simulation code. The framework coordinates planning, code generation, visual review, and physics analysis agents. The planning agent converts the natural language prompt into a structured scene plan, the code agent implements it as executable simulation code, and the visual review agent provide visual feedback while the physics analysis agent checks physical consistency. The code is iteratively revised based on the feedback until the simulation matches the prompt reqirements and physical constraints. Experimental results show that our framework outperforms advanced video-based models in physical accuracy, instruction fidelity and visual quality, which could be applied to various scenarios including driving simulation and embodied robot tasks.
Hongyu Wang, Jingquan Wang, Bocheng Zou +2
Department of Mechanical & Aerospace Engineering, University of Wisconsin-Madison · School of Computer, Data, and Information Sciences, University of Wisconsin-Madison