We present a Generative World Simulation framework that couples persistent shared world state with continuous first-person audiovisual generation. The framework integrates DAO, a computable world engine, and JING, an Egocentric Interactive Experience Model, to support sustained interaction for humans and agents. DAO maintains the states of characters, objects, and spaces and their relationships, advancing the world according to interaction rules and world time while preserving the consequences of past actions. From this evolving state, DAO derives observer-specific conditions for JING, which combines these conditions with character, object, and scene references, observation history, and current actions to generate each observer's audiovisual experience. Users can explore environments, manipulate objects, and communicate with characters through viewpoint control and text instructions, with their actions shaping both the shared state and subsequent experiences. By connecting world-state evolution with experience generation in a continuous interaction loop, the framework offers an approach to building interactive worlds that retain their history, evolve through action, and persist beyond any individual observer.
Figures & tables
Figure 2: Data pipeline overview. Internally collected and generated egocentric videos and open-source datasets undergo automated media checks, model-assisted semantic filtering, temporal segmentation, and captioning. Optional camera trajectories and extracted or generated reference images are aligned with the processed clips and validated before accepted samples are organized by capability and conditioning configuration and packaged into latent representations and LMDB shards for training.
Figure 3: Diverse aligned training samples. Illustrative samples covering real-world and virtual-world navigation, scene revisits, object interaction, dialogue, and rendered environments. Within each panel, video frames, captions, audio, and available camera poses follow a common temporal axis, with optional reference images for character, object, or scene conditioning.
Figure 4: JING architecture. (a) Reference images provide visual conditions through the video VAE encoder and (b) the Qwen3-VL conditioner, which also encodes text instructions. Camera trajectories are processed by a separate control encoder. (c) Current tokens attend to cached keys and values (K/V) from reference and historical context. (d) Camera features modulate video tokens within each Transformer block. (e) Retrieved audiovisual memories supplement the persistent first chunk and recent context. The diffusion sampler produces clean audiovisual latents, which the video and audio VAE decoders convert into synchronized output.
Figure 5: DAO world engine. (a) User and agent actions pass through action admission to produce actor intents and state edits. (b) A world definition initializes a shared state comprising an entity graph, entity attributes, actor states, a clock, and active events. An ordered update cycle transforms Wt into Wt+1 by applying edits, advancing motion, resolving spatial constraints, and advancing activities, while carrying unchanged state forward. (c) Local readout combines the world state with an observer’s pose to provide observations for an experience interface. (d) World snapshots and event history preserve state across interactions.
Rank
Model
Avg.
Video quality
Setting adherence
Interaction adherence
Consistency
Physics
Full (289 cases)
1
JING (Bidirectional) (ours)
81.0
80.9
88.5
76.7
85.7
73.3
2
JING (4-step AR) (ours)
79.9
79.8
88.9
79.5
81.4
69.9
3
Kling 3.0 [ 45 ]
79.4
80.0
91.0
73.1
83.9
69.2
4
Wan 2.7 [ 46 ]
78.4
81.0
91.4
72.1
75.8
71.6
5
Seedance 1.5 [ 47 ]
76.2
81.9
82.9
68.3
79.9
68.2
Table 1: Quantitative comparison on WBench [ 44 ] . 1 1 1 Data as of September 17, 2026. https://meituan-longcat.github.io/WBench/ Ranks refer to the comparison pools of 15 Full and 39 Navigation entries. Bold and underlined values indicate the best and second-best results among the models shown within each split; shaded rows denote our models.
Figure 6: Reference-conditioned generation. Input references and sampled outputs for three configurations: scene only (a), scene and character (b), and scene, character, and object (c).
Figure 7: Egocentric exploration. Sampled frames from navigation through desert-city alleys (a), a forest shrine (b), and a garden at beetle scale (c). Overlaid controls indicate user inputs during generation.
Figure 8: Object manipulation. Sampled frames showing a hammer striking heated metal (a), a young dragon igniting a woodpile (b), and hands releasing a sky lantern (c).
Figure 9: Character communication. Sampled frames from conversations in Chinese on a rose terrace (a), English in a fantasy setting (b), and Japanese on a train (c). The corresponding videos include generated speech.
Figure 10: Action-conditioned branching. A shared fountain opening leads to three continuations: looking into the water (a), tossing a coin (b), and meeting a fairy (c).
Figure 11: Reference-conditioned branching. A shared bakery opening is paired with character references for Clara (a), Tom (b), and Eli (c), producing different encounters.
Figure 12: Embodied interaction. Generated first-person observations of mug placement (a), ball retrieval for a dog (b), and plant watering (c), viewed from a household robot.
Figure 13: Agents act autonomously and change the world. Six successive events driven by agent intentions in DAO: (a) path blocked, (b) intent misread, (c) partial witness, (d) Clara approaches, (e) evidence checked, and (f) social repair. Local observations and accumulated evidence guide autonomous actions from conflict to reconciliation.
Figure 14: Different viewpoints of agents in a shared world. A shared interaction, shown from the viewpoints of Clara, Eli, and Nori.
Figure 15: JING generation results with and without DAO.
Generative video world models can now synthesize open-ended environments that agents can navigate and interact with in simple ways. Yet open-ended generation does not imply full interaction: as a generated world expands, newly created content through navigation should expand what the agent can act upon, and as the agent changes the world, those changes should become persistent parts of the environment rather than transient visual effects. We characterize these two requirements as Open-World Interactivity, where newly generated or encountered entities are incorporated into the actionable world, and Persistent State, where interaction outcomes are committed to the world state and continue to influence subsequent observations and interactions. We present Oneira, an interactive video world model that closes the loop between generation and interaction through an explicit, extensible world state managed by a coding agent. Given the current observation and an action or high-level goal, the agent reads the world state, grounds the relevant entities, plans the interaction, and writes its outcome back into a world state table. When exploration reveals new objects, the agent incorporates them from generated observations, allowing the interaction space to expand with the generated world. Meanwhile, previously induced state changes are carried across video segments, making the consequences of interaction persistent parts of subsequent world evolution. The updated world state is rendered along the camera action trajectory into a coarse conditioning video, from which a video generator fills in the appearance, motion, and interaction details not represented in the state. Experiments show that Oneira enables direct and consistent interaction with newly generated objects, while preserving the effects of prior interactions over long horizons. Project page: https://madaoer.github.io/projects/oneira
Xindi Yang, Baolu Li, Liam Lee +8
Monash University · Dalian University of Technology · The Chinese University of Hong Kong, Shenzhen +2
Recent video world models generate increasingly realistic and interactive visual experiences, yet lack reliable mechanisms for maintaining persistent world state and enforcing programmable rules over extended interactions. We introduce Programmable World Model, a framework that decouples world-state evolution from visual observation generation. An agent translates natural-language instructions into executable programs that specify entity states and state-transition rules, enabling direct control over individual entities and their interactions. A lightweight engine executes these programs to update and maintain an explicit, persistent global world state, including off-screen entities and non-visual attributes. To connect world state with visual generation, we introduce state-augmented 3D oriented bounding boxes (OBBs) as an intermediate representation. This representation, together with the target camera trajectory, is deterministically compiled into pixel-aligned spatiotemporal conditioning signals for a pretrained video model serving as the generative renderer. This design allows users to create playable games with predefined mechanics, direct control over individual entities, and persistent world state throughout gameplay. We further introduce CombatStateBench, a benchmark for evaluating programmable world models. On CombatStateBench, our method achieves 94% Count Accuracy and 98% State Accuracy, substantially outperforming existing interactive video world models while supporting coherent long-horizon generation. These results demonstrate the effectiveness of separating explicit state evolution from generative rendering for building persistent, programmable worlds.
Game worlds have traditionally been built through labor-intensive production pipelines, making them costly to develop, difficult to customization, and expensive to modify after deployment. Recent advances in video world models offer a fundamentally different paradigm. Rather than explicitly authoring every component of a virtual environment, these models autoregressively synthesize future observations conditioned on the current world state and user interactions, enabling playable worlds to be generated online. Trained on both gameplay recordings and real-world videos, they can capture diverse visual appearances and physical dynamics, opening new opportunities for interactive applications beyond gaming, including embodied intelligence. In this paper, we present AlayaWorld, a full-stack open-source framework for building interactive generative worlds. AlayaWorld enables open-ended real-time interaction, allowing users to freely navigate and perform diverse actions such as combat, spell casting, and monster summoning. The framework unifies the complete development-from data preparation model architecture, model training, inference acceleration, and deployment-within a modular and extensible architecture. Alongside the framework, we release reproducible pipelines, reference implementations, evaluation tools, and comprehensive documentation, establishing a practical foundation for future research and real-time applications of generative world models.