Generative World Simulation with DAO and JING

XGEN Team; Guohao Cai; Haoxuan Che; Frank Fan; Chenran Huang; Junjia Huang; Jiayi Jiang; Yanheng Li

cs.CVOct 14, 2026

Generative World Simulation with DAO and JING

Authors: XGEN Team, Guohao Cai, Haoxuan Che, Frank Fan, Chenran Huang, Junjia Huang, Jiayi Jiang, Yanheng Li,

Abstract

We present a Generative World Simulation framework that couples persistent shared world state with continuous first-person audiovisual generation. The framework integrates DAO, a computable world engine, and JING, an Egocentric Interactive Experience Model, to support sustained interaction for humans and agents. DAO maintains the states of characters, objects, and spaces and their relationships, advancing the world according to interaction rules and world time while preserving the consequences of past actions. From this evolving state, DAO derives observer-specific conditions for JING, which combines these conditions with character, object, and scene references, observation history, and current actions to generate each observer's audiovisual experience. Users can explore environments, manipulate objects, and communicate with characters through viewpoint control and text instructions, with their actions shaping both the shared state and subsequent experiences. By connecting world-state evolution with experience generation in a continuous interaction loop, the framework offers an approach to building interactive worlds that retain their history, evolve through action, and persist beyond any individual observer.

Figures & tables

Figure 2: Data pipeline overview. Internally collected and generated egocentric videos and open-source datasets undergo automated media checks, model-assisted semantic filtering, temporal segmentation, and captioning. Optional camera trajectories and extracted or generated reference images are aligned with the processed clips and validated before accepted samples are organized by capability and conditioning configuration and packaged into latent representations and LMDB shards for training.

Figure 3: Diverse aligned training samples. Illustrative samples covering real-world and virtual-world navigation, scene revisits, object interaction, dialogue, and rendered environments. Within each panel, video frames, captions, audio, and available camera poses follow a common temporal axis, with optional reference images for character, object, or scene conditioning.

Figure 4: JING architecture. (a) Reference images provide visual conditions through the video VAE encoder and (b) the Qwen3-VL conditioner, which also encodes text instructions. Camera trajectories are processed by a separate control encoder. (c) Current tokens attend to cached keys and values (K/V) from reference and historical context. (d) Camera features modulate video tokens within each Transformer block. (e) Retrieved audiovisual memories supplement the persistent first chunk and recent context. The diffusion sampler produces clean audiovisual latents, which the video and audio VAE decoders convert into synchronized output.

Figure 5: DAO world engine. (a) User and agent actions pass through action admission to produce actor intents and state edits. (b) A world definition initializes a shared state comprising an entity graph, entity attributes, actor states, a clock, and active events. An ordered update cycle transforms $W_{t}$ into $W_{t+1}$ by applying edits, advancing motion, resolving spatial constraints, and advancing activities, while carrying unchanged state forward. (c) Local readout combines the world state with an observer’s pose to provide observations for an experience interface. (d) World snapshots and event history preserve state across interactions.

Full (289 cases)
Rank	Model	Avg.	Video quality	Setting adherence	Interaction adherence	Consistency	Physics
1	JING (Bidirectional) (ours)	81.0	80.9	88.5	76.7	85.7	73.3
2	JING (4-step AR) (ours)	79.9	79.8	88.9	79.5	81.4	69.9
3	Kling 3.0 [ 45 ]	79.4	80.0	91.0	73.1	83.9	69.2
4	Wan 2.7 [ 46 ]	78.4	81.0	91.4	72.1	75.8	71.6
5	Seedance 1.5 [ 47 ]	76.2	81.9	82.9	68.3	79.9	68.2

Table 1: Quantitative comparison on WBench [ 44 ] . 1 1 1 Data as of September 17, 2026. https://meituan-longcat.github.io/WBench/ Ranks refer to the comparison pools of 15 Full and 39 Navigation entries. Bold and underlined values indicate the best and second-best results among the models shown within each split; shaded rows denote our models.

Figure 6: Reference-conditioned generation. Input references and sampled outputs for three configurations: scene only (a), scene and character (b), and scene, character, and object (c).

Figure 7: Egocentric exploration. Sampled frames from navigation through desert-city alleys (a), a forest shrine (b), and a garden at beetle scale (c). Overlaid controls indicate user inputs during generation.

Figure 8: Object manipulation. Sampled frames showing a hammer striking heated metal (a), a young dragon igniting a woodpile (b), and hands releasing a sky lantern (c).

Figure 9: Character communication. Sampled frames from conversations in Chinese on a rose terrace (a), English in a fantasy setting (b), and Japanese on a train (c). The corresponding videos include generated speech.

Figure 10: Action-conditioned branching. A shared fountain opening leads to three continuations: looking into the water (a), tossing a coin (b), and meeting a fairy (c).

Figure 11: Reference-conditioned branching. A shared bakery opening is paired with character references for Clara (a), Tom (b), and Eli (c), producing different encounters.

Figure 12: Embodied interaction. Generated first-person observations of mug placement (a), ball retrieval for a dog (b), and plant watering (c), viewed from a household robot.

Figure 13: Agents act autonomously and change the world. Six successive events driven by agent intentions in DAO: (a) path blocked, (b) intent misread, (c) partial witness, (d) Clara approaches, (e) evidence checked, and (f) social repair. Local observations and accumulated evidence guide autonomous actions from conflict to reconciliation.

Figure 14: Different viewpoints of agents in a shared world. A shared interaction, shown from the viewpoints of Clara, Eli, and Nori.

Figure 15: JING generation results with and without DAO.

Interactive World Models

Interactive world models predict and generate how environments change in response to actions, supporting simulation, planning, and control.

See all 120 papers

Oneira: From Open-Ended Generation to Open-World Interaction in Video World Models
Oct 1, 2026Xindi Yang, Baolu Li, Liam Lee +8Long-Horizon Video Generation Action-Conditioned World Models
Monash University · Dalian University of Technology · The Chinese University of Hong Kong, Shenzhen +2
Programmable World Model
Sep 9, 2026Zheng-Hui Huang, Guixu Lin, Jiacheng Lin +8World Models Controllable Video Generation
Alaya Lab
AlayaWorld: Long-Horizon and Playable Video World Generation
Jul 7, 2026AlayaWorld Team, Kaipeng Zhang, Chuanhao Li +14Real-Time Interactive Video Generation Long-Horizon Video Generation
Alaya Lab

Oct 1, 2026J/K move · Enter open · S save

Oneira: From Open-Ended Generation to Open-World Interaction in Video World Models

Xindi Yang, Baolu Li, Liam Lee, Zhenfei Yin+7

Monash University · Dalian University of Technology · The Chinese University of Hong Kong, Shenzhen+2

Long-Horizon Video Generation Action-Conditioned World Models Interactive World Models Video World Models

Open paper AskPDF arXiv

Generative video world models can now synthesize open-ended environments that agents can navigate and interact with in simple ways. Yet open-ended generation does not imply full interaction: as a generated world expands, newly created content through navigation should expand what the agent can act upon, and as the agent changes the world, those changes should become persistent parts of the environment rather than transient visual effects. We characterize these two requirements as Open-World Interactivity, where newly generated or encountered entities are incorporated into the actionable world, and Persistent State, where interaction outcomes are committed to the world state and continue to influence subsequent observations and interactions. We present Oneira, an interactive video world model that closes the loop between generation and interaction through an explicit, extensible world state managed by a coding agent. Given the current observation and an action or high-level goal, the agent reads the world state, grounds the relevant entities, plans the interaction, and writes its outcome back into a world state table. When exploration reveals new objects, the agent incorporates them from generated observations, allowing the interaction space to expand with the generated world. Meanwhile, previously induced state changes are carried across video segments, making the consequences of interaction persistent parts of subsequent world evolution. The updated world state is rendered along the camera action trajectory into a coarse conditioning video, from which a video generator fills in the appearance, motion, and interaction details not represented in the state. Experiments show that Oneira enables direct and consistent interaction with newly generated objects, while preserving the effects of prior interactions over long horizons. Project page: https://madaoer.github.io/projects/oneira

Abstract

Interactive World Models

Explore similar work

Oneira: From Open-Ended Generation to Open-World Interaction in Video World Models

Programmable World Model

AlayaWorld: Long-Horizon and Playable Video World Generation