BoxWorld: Let World Engines Control and Video Generators Render

Zike Wu; Ahmed Hatem; Chintan Trivedi; Wallace Michel Pinto Lira; Xuebin Qin; Lele Wang; Renjie Liao

cs.CVOct 26, 2026

BoxWorld: Let World Engines Control and Video Generators Render

Authors: Zike Wu, Ahmed Hatem, Chintan Trivedi, Wallace Michel Pinto Lira, Xuebin Qin, Lele Wang, Renjie Liao

Organizations: University of British Columbia · Vector Institute for AI · Huawei Canada · Canada CIFAR AI Chair

Abstract

We tackle the challenge of building real-time interactive video worlds with user-defined behaviors. Such worlds must track object states, apply interaction rules, and render the resulting consequences as users act. Yet existing end-to-end video world models typically learn interaction dynamics within the generator, making it difficult to specify object-level rules and reason about their consequences. In this work, we introduce BoxWorld, which separates interaction reasoning from generative rendering. Given user-specified rules, a coding agent creates a Box World Engine that determines interaction outcomes and exposes the resulting object states through a lightweight interface of layout boxes, persistent object attributes, and dynamic status variables, leaving appearance and physical effects to generative rendering. A Box-State Generative Renderer then renders this evolving state stream into video autoregressively, enabling online control over object motion, state switches, and conditional events. Rendering this state stream is non-trivial: the renderer must respond to object-level state updates in real time while remaining coherent with the generated history. To this end, we introduce a three-stage training strategy that learns object grounding from open-domain videos and state transitions from gameplay videos, both annotated automatically, and finally performs rollout distillation on self-generated histories. We further propose PairedStateBench, a paired benchmark for fine-grained interaction control. BoxWorld substantially outperforms end-to-end world models and remains competitive across multiple public benchmarks. Project page: https://boxworld-web.github.io.

Figures & tables

Figure 1: BoxWorld separates interaction reasoning from generative rendering. A coding agent creates a Box World Engine from user-defined rules. The engine updates object states from user actions, and the Box-State Generative Renderer renders them into video.

Figure 2: Interfaces for interactive video worlds. (a) Implicit-state approaches benefit from broad visual generalization, but provide only coarse control. (b) Dense-state approaches expose finer control through simulators, but such interfaces are often domain-specific and costly to construct and simulate. (c) Our BoxWorld exposes an explicit-but-minimal Object-Centric State Interface for object-level control, while the video generator completes the remaining physical effects.

Figure 3: Box-State Generative Renderer. (a) The State Grounding Module encodes each object’s box, attribute, and dynamic status into grounding tokens, and its masked cross-attention lets each visual token attend only to the objects whose boxes cover it. (b) The module is inserted between self-attention and cross-attention in each block of the renderer.

Overall
Method	FPS ↑	Trigger ↑	Suppress ↑	Paired-CF ↑	State-Fid ↑	OnTime ↑	Task ↑
LingBot-v1	0.05	53.5	11.1	16.2	43.1	44.5	25.3
LingBot-v1 (Oracle)	0.05	45.5	25.4	16.2	42.3	27.7	18.4
LingBot-v2 †	13.3	21.3	63.5	10.8	46.1	14.3	11.5
LingBot-v2 † (Oracle)	13.3	19.0	69.8	16.2	57.1	17.5	11.5
LongLive	20.7	10.7	50.8	13.5	36.2	5.7	8.0

Table 1: Results on PairedStateBench. † Camera-control version, the only publicly released LingBot-v2 variant. Oracle variants additionally receive the interaction consequences as text.

Figure 4: Qualitative comparison on PairedStateBench. Each case compares two branches from the same scene that differ in whether the precondition is met: nurturing the flower before it blooms (top) and collecting the key before opening the door (bottom). BoxWorld renders the event only once the precondition is met, whereas LingBot-v1 renders it without the precondition, and its oracle variant still opens the door without the key and blooms the flower before nurturing. More cases are in Appendix C.2 . Please refer to the project page for videos.

Figure 5: Qualitative results of BoxWorld. Each row shows a Box World Engine (left) and two videos rendered from the same state stream with different object attributes and scene descriptions. Please refer to the project page for videos.

VBench
Variant	Total ↑	Long Total ↑
Stage 1	83.30	81.72
Stage 2	83.57	81.82
Stage 3	83.76	83.00
YouTube-VIS
Variant	mIOU ↑	FVD ↓

Table 2: Ablations of training stages and the state interface. Full tables are in Appendix C .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Video-to-Structure Annotation Pipeline.

Panel of Figure 7 — Figure 7: An example of our data annotation results from VideoGen-1M ( Tan et al., 2024 ) .

Panel of Figure 8 — Figure 8: An example of our data annotation results from gameplay videos.

Figure 9: Additional qualitative comparisons on PairedStateBench with LingBot-v1, LingBot-v2, and their oracle variants: jumping and moving, occupying a pressure plate to open a door, stopping to lie down, and controlling altitude. Each case compares two branches with different user actions. Please refer to the project page for videos.

Figure 10: Additional qualitative VBench results comparing LongLive with the three stages of BoxWorld.

		VBench			VBench-Long (30 s)
Model	FPS ↑	Total ↑	Quality ↑	Semantic ↑	Total ↑	Quality ↑	Semantic ↑
LTX-Video ( HaCohen et al., 2024 )	8.98	80.00	82.30	70.79	–	–	–
Wan2.1 ( Team Wan et al., 2025 )	0.78	84.26	85.30	80.09	–	–	–
SkyReels-V2 ( Chen et al., 2025a )	0.49	82.67	84.70	74.53	75.29	80.77	53.37
MAGI-1 ( Sand.ai et al., 2025 )	0.19	79.18	82.04	67.74	–	–	–
NOVA ( Deng et al., 2025 )	0.88	80.12	80.39	79.05	–	–	–

Appendix

Table 3: Video quality on VBench and 30-second VBench-Long. FPS measures generation throughput; baseline scores follow their reported evaluation settings.

Method	Real-time	FVD ↓	mIOU ↑	rCLIP ${}_{t}^{\uparrow}$	rCLIP ${}_{i}^{\uparrow}$	rCFC ↑
TrackDiffusion ( Li et al., 2025a )	No	464	0.492	0.273	0.804	0.940
LVD ( Lian et al., 2024 )	No	558	0.281	0.261	0.803	0.861
VideoTetris ( Tian et al., 2024 )	No	590	0.166	0.267	0.799	0.423
BlobGEN-Vid ( Feng et al., 2025b )	No	317	0.598	0.289	0.836	0.958
BoxWorld (Stage 3)	Yes	62	0.572	0.291	0.837	0.965

Appendix

Table 4: Layout-to-video generation on YouTube-VIS 2021. Baseline scores are quoted from prior work, and all metrics follow the protocol of BlobGEN-Vid ( Feng et al., 2025b ) .

Variant	Quality ↑	Semantic ↑	Total ↑
Stage 3	84.82	79.54	83.76
w/o spatial mask	81.78	74.51	80.33
Stage 2	84.22	80.97	83.57
w/o dynamics LoRA (Stage 1)	84.46	78.62	83.30
High-noise-only LoRA	83.84	77.44	82.56
Open-domain video LoRA	82.68	79.16	81.98

Appendix

Table 5: Quantitative VBench ablations. The mask ablation is applied to Stage 3, and the adapter variants are compared with Stage 2; all variants use the four-step streaming sampler.

Variant	FVD ↓	mIOU ↑	rCLIP ${}_{t}^{\uparrow}$	rCLIP ${}_{i}^{\uparrow}$	rCFC ↑
Stage 3	62	0.572	0.291	0.837	0.965
w/o spatial mask	79	0.291	0.264	0.783	0.955
w/o object attributes	68	0.323	0.273	0.785	0.963

Appendix

Table 6: Grounding ablations on YouTube-VIS. Both ablations are applied to Stage 3; the attribute ablation replaces every per-box caption with “object,” retaining the global prompt, boxes, and original evaluation targets. All rows use four-step causal autoregressive generation.

Variant	Trigger ↑	Suppress ↑	Paired-CF ↑	State-Fid ↑	OnTime ↑	Task ↑
Stage 1	87.3	60.3	64.9	83.4	83.8	49.4
Stage 2	82.0	85.7	67.6	83.6	81.5	64.4
Stage 3	92.2	90.5	97.3	91.6	91.2	83.9

Appendix

Table 7: Training stages on PairedStateBench. All variants use the same engine-exported state sequences.

Real-Time Interactive Video Generation

Real-Time Interactive Video Generation creates low-latency video streams that respond continuously to user actions, camera motion, text, or audio.

See all 79 papers

Programmable World Model
Sep 9, 2026Zheng-Hui Huang, Guixu Lin, Jiacheng Lin +8World Models Controllable Video Generation
Alaya Lab
Oneira: From Open-Ended Generation to Open-World Interaction in Video World Models
Oct 1, 2026Xindi Yang, Baolu Li, Liam Lee +8Long-Horizon Video Generation Action-Conditioned World Models
Monash University · Dalian University of Technology · The Chinese University of Hong Kong, Shenzhen +2
Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends
May 31, 2026Jiuming Liu, Chaojun Ni, Mengmeng Liu +7Action-Conditioned Video Generation Long-Horizon Video Generation
Department of Engineering, University of Cambridge, U.K. · Peking University, China · University of Twente, Netherlands +4

Sep 9, 2026J/K move · Enter open · S save

Programmable World Model

Zheng-Hui Huang, Guixu Lin, Jiacheng Lin, Yi-Chuan Huang+7

Alaya Lab

World Models Controllable Video Generation World Model Evaluation Interactive World Models Video World Models

Open paper AskPDF arXiv

Recent video world models generate increasingly realistic and interactive visual experiences, yet lack reliable mechanisms for maintaining persistent world state and enforcing programmable rules over extended interactions. We introduce Programmable World Model, a framework that decouples world-state evolution from visual observation generation. An agent translates natural-language instructions into executable programs that specify entity states and state-transition rules, enabling direct control over individual entities and their interactions. A lightweight engine executes these programs to update and maintain an explicit, persistent global world state, including off-screen entities and non-visual attributes. To connect world state with visual generation, we introduce state-augmented 3D oriented bounding boxes (OBBs) as an intermediate representation. This representation, together with the target camera trajectory, is deterministically compiled into pixel-aligned spatiotemporal conditioning signals for a pretrained video model serving as the generative renderer. This design allows users to create playable games with predefined mechanics, direct control over individual entities, and persistent world state throughout gameplay. We further introduce CombatStateBench, a benchmark for evaluating programmable world models. On CombatStateBench, our method achieves 94% Count Accuracy and 98% State Accuracy, substantially outperforming existing interactive video world models while supporting coherent long-horizon generation. These results demonstrate the effectiveness of separating explicit state evolution from generative rendering for building persistent, programmable worlds.

Abstract

Real-Time Interactive Video Generation

Explore similar work

Programmable World Model

Oneira: From Open-Ended Generation to Open-World Interaction in Video World Models

Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends