Oneira: From Open-Ended Generation to Open-World Interaction in Video World Models

Xindi Yang; Baolu Li; Liam Lee; Zhenfei Yin; Songxin Zhang; Zhuoyang Song; Xu Jia; Jianfei Cai

cs.CVOct 1, 2026

Oneira: From Open-Ended Generation to Open-World Interaction in Video World Models

Authors: Xindi Yang, Baolu Li, Liam Lee, Zhenfei Yin, Songxin Zhang, Zhuoyang Song, Xu Jia, Jianfei Cai, +3 more

Organizations: Monash University · Dalian University of Technology · The Chinese University of Hong Kong, Shenzhen · Oxford University · University of Bristol

Abstract

Generative video world models can now synthesize open-ended environments that agents can navigate and interact with in simple ways. Yet open-ended generation does not imply full interaction: as a generated world expands, newly created content through navigation should expand what the agent can act upon, and as the agent changes the world, those changes should become persistent parts of the environment rather than transient visual effects. We characterize these two requirements as Open-World Interactivity, where newly generated or encountered entities are incorporated into the actionable world, and Persistent State, where interaction outcomes are committed to the world state and continue to influence subsequent observations and interactions. We present Oneira, an interactive video world model that closes the loop between generation and interaction through an explicit, extensible world state managed by a coding agent. Given the current observation and an action or high-level goal, the agent reads the world state, grounds the relevant entities, plans the interaction, and writes its outcome back into a world state table. When exploration reveals new objects, the agent incorporates them from generated observations, allowing the interaction space to expand with the generated world. Meanwhile, previously induced state changes are carried across video segments, making the consequences of interaction persistent parts of subsequent world evolution. The updated world state is rendered along the camera action trajectory into a coarse conditioning video, from which a video generator fills in the appearance, motion, and interaction details not represented in the state. Experiments show that Oneira enables direct and consistent interaction with newly generated objects, while preserving the effects of prior interactions over long horizons. Project page: https://madaoer.github.io/projects/oneira

Figures & tables

Figure 1: Oneira conditions video generation on an explicit world state table that a coding agent updates. Given an input image and a goal, the agent plans each interaction and writes its outcome into the table, and the video model renders the updated state into realistic video. This design enables navigation and interaction beyond the input image (a), instance-level control (b), generalization to novel objects (c), and effects beyond the target object (d). The small image in the corner of each frame shows the coarse conditioning video rendered from the table. Purple marks the default state, yellow a changed state, and a dashed outline a box removed from the table.

Figure 2: Pipeline of Oneira for one segment. Given the input image and a goal, the coding agent $A$ reads the world state table and plans each action $P_{k}$ . The engine $E$ writes each planned action into the table and renders the result $T_{k}$ along the camera path $C_{k}$ as the coarse conditioning video $V_{k}$ (Section 3.2 ). An interaction appears as a change of the object box, such as its removal or recoloring. Objects that the interaction moves lose their boxes and are registered again from the generated frames once they come to rest, as are objects that it uncovers. The video model $G$ generates the segment conditioned on the first frame, optional memory frames, $V_{k}$ , and the chunk captions $c_{k}$ (Section 3.3 ). The last frame and the updated table start the next segment. The sphere is a schematic. The conditioning and generated frames come from one segment produced by Oneira .

Figure 3: Qualitative comparison on a 30-second chain of three segments. The label above each column gives the action at that moment, orange cells mark the chunks whose captions carry an interaction, and the insets show the conditioning video of Oneira . Red tags name the failure in a baseline frame. Oneira controls both interactions at the specified chunks and maintains the changed states when the camera turns back.

Figure 4: Data pipeline. We recover the camera path, the walkable ground, and the object boxes from the GT video, and a VLM judge annotates the state change. Gates keep the clean clips, fixed rules assemble the world state table, and the engine renders the conditioning video from it.

	Interaction part $\uparrow$			Memory part $\uparrow$			VBench $\uparrow$
Method	Interaction	Timing	Target	PSNR	SSIM	State	SC	BC	MS	TF	DD	AQ	IQ	OC
MiniMax-H3 Ref2VA	0.30	0.57	0.53	10.96	0.34	0.68	0.83	0.90	0.99	0.97	1.00	0.53	0.73	0.19
LingBot-World-V2	0.49	0.40	0.39	9.08	0.27	0.44	0.79	0.87	0.97	0.98	0.97	0.54	0.72	0.20
YUME 1.5	0.05	0.43	0.22	10.09	0.35	0.41	0.83	0.89	0.99	0.97	0.95	0.52	0.74	0.20
AlayaWorld	0.08	0.16	0.26	9.19	0.30	0.39	0.79	0.88	0.99	0.95	0.65	0.49	0.69	0.18
Oneira	0.78	0.76	0.82	21.11	0.64	0.80	0.84	0.90	0.99	0.97	1.00	0.54	0.73	0.22

Table 1: Results on InteractionBench. Interaction, Timing, and Target are pass rates on the 100 interaction cases. PSNR in dB and SSIM compare the visit and revisit frames, and State is the pass rate on the 50 memory chains. VBench reports eight dimensions. Bold marks the best entry based on unrounded scores.

Variant	Interaction $\uparrow$	Timing $\uparrow$	Target $\uparrow$	RotErr $\downarrow$	TransErr $\downarrow$
Ours w/o sphere	0.63	0.69	0.62	3.32	0.20
Ours w/o boxes	0.61	0.74	0.61	2.78	0.15
Ours w/o chunk captions	0.73	0.66	0.80	2.71	0.15
Ours (full)	0.78	0.76	0.82	2.60	0.13

Table 2: Ablation of the conditions on the interaction part of InteractionBench. RotErr in degrees and TransErr are Umeyama-aligned Pi3X camera metrics.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Composition of the training set. Left, the share of navigation clips per category of camera motion and their source. Right, the share of interaction clips per action category.

Figure 6: The prompt of the VLM judge for one chunk. Placeholders in braces are filled per case, and the questions on hand-held tools and two-step spells are omitted.

Figure 7: Interaction with unseen objects. The target is absent from the first frame and comes into view while the camera explores in the first segment. The coding agent adds it to the world state table before the second segment, so its box appears in the conditioning video from then on, and a hand interacts with it. The row under each chain is its conditioning video.

Figure 8: Interaction with identical objects. Each scene holds two kinds of objects, three identical objects of each kind, and two chunk captions name one object each by its place in the row. Insets show the conditioning video of Oneira , and orange labels mark the interaction columns.

Explore similar work

Sep 9, 2026cs.CV

Programmable World Model

Recent video world models generate increasingly realistic and interactive visual experiences, yet lack reliable mechanisms for maintaining persistent world state and enforcing programmable rules over extended interactions. We introduce Programmable World Model, a framework that decouples world-state evolution from visual observation generation. An agent translates natural-language instructions into executable programs that specify entity states and state-transition rules, enabling direct control over individual entities and their interactions. A lightweight engine executes these programs to update and maintain an explicit, persistent global world state, including off-screen entities and non-visual attributes. To connect world state with visual generation, we introduce state-augmented 3D oriented bounding boxes (OBBs) as an intermediate representation. This representation, together with the target camera trajectory, is deterministically compiled into pixel-aligned spatiotemporal conditioning signals for a pretrained video model serving as the generative renderer. This design allows users to create playable games with predefined mechanics, direct control over individual entities, and persistent world state throughout gameplay. We further introduce CombatStateBench, a benchmark for evaluating programmable world models. On CombatStateBench, our method achieves 94% Count Accuracy and 98% State Accuracy, substantially outperforming existing interactive video world models while supporting coherent long-horizon generation. These results demonstrate the effectiveness of separating explicit state evolution from generative rendering for building persistent, programmable worlds.

Zheng-Hui Huang, Guixu Lin, Jiacheng Lin +8

Alaya Lab

Video World Models World Models World Visual Generation 3D Generation

Ask PDF

Jul 7, 2026cs.CV

AlayaWorld: Long-Horizon and Playable Video World Generation

Game worlds have traditionally been built through labor-intensive production pipelines, making them costly to develop, difficult to customization, and expensive to modify after deployment. Recent advances in video world models offer a fundamentally different paradigm. Rather than explicitly authoring every component of a virtual environment, these models autoregressively synthesize future observations conditioned on the current world state and user interactions, enabling playable worlds to be generated online. Trained on both gameplay recordings and real-world videos, they can capture diverse visual appearances and physical dynamics, opening new opportunities for interactive applications beyond gaming, including embodied intelligence. In this paper, we present \textbf{AlayaWorld}, a full-stack open-source framework for building interactive generative worlds. AlayaWorld enables open-ended real-time interaction, allowing users to freely navigate and perform diverse actions such as combat, spell casting, and monster summoning. The framework unifies the complete development-from data preparation model architecture, model training, inference acceleration, and deployment-within a modular and extensible architecture. Alongside the framework, we release reproducible pipelines, reference implementations, evaluation tools, and comprehensive documentation, establishing a practical foundation for future research and real-time applications of generative world models.

AlayaWorld Team, Kaipeng Zhang, Chuanhao Li +14

Alaya Lab

Playable Video World Generation Video World Models Generative Video Models Embodied Artificial Intelligence Long-Horizon Task Planning

Ask PDF

Jul 20, 2026cs.AI

AlayaWorld: Interactive Long-Horizon World Modeling -- Full Technical Report

Unlike conventional video game development, which relies on labor-intensive pipelines for asset production, animation, physics, and programming, video world models generate interactive environments from user inputs instantly. It enable us to create customized, explorable, and continuously evolving virtual world from text, an image, or video. Realizing this vision requires four tightly coupled capabilities: interaction, persistent spatiotemporal consistency, stable long-horizon generation, and efficient response. We present AlayaWorld, an interactive long-horizon video world model that generates 24-fps video at 540p and 720p. Built on a 15B video diffusion transformer, AlayaWorld generates short latent chunks autoregressively under camera trajectories and switchable text prompts. Its bounded visual context combines a persistent sink frame, compressed temporal history, geometry-aligned spatial memory, and recent-frame conditioning. To reduce long-term drift, the model is trained with corrupted histories and prediction residuals collected from its own roll-outs. We further introduce a discrete autoregressive distillation formulation that combines distribution-matching distillation, self-forcing++, and consistency distillation, reducing inference from approximately 30 sampling steps to four steps per chunk. On iWorld-Bench, AlayaWorld achieves the best performance over long-horizon generation. Conceived as a full-stack, open-source, and long-term project, AlayaWorld is intended to provide an extensible foundation for future research on interactive video world models.

AlayaWorld Team, Kaipeng Zhang, Chuanhao Li +15

Alaya Lab

Video World Models World Technical Report

Ask PDF