Precise Editing and Flexible Referencing for Interactable Worlds

Xinyao Liao; Xianfang Zeng; Zhu Liang; Zhoujie Fu; Qianxun Xu; Jiachi Liu; Gang Yu; Guosheng Lin

cs.CVSep 28, 2026

Precise Editing and Flexible Referencing for Interactable Worlds

Authors: Xinyao Liao, Xianfang Zeng, Zhu Liang, Zhoujie Fu, Qianxun Xu, Jiachi Liu, Gang Yu, Guosheng Lin

Organizations: Nanyang Technological University · StepFun

Abstract

We present EditWorld, a video world model for precise editing and flexible referencing in interactable worlds. Existing video world models primarily focus on navigation, letting users explore generated worlds but offering limited control over how existing world content is modified. EditWorld extends world modeling from exploration to precise modification by streaming editing instructions and reference images during autoregressive generation. To support these capabilities, EditWorld introduces Gated Causal Attention for temporally varying editing conditions and reference images, together with a Sparse Context mechanism that maintains a bounded historical context for long-horizon inference. We further adopt joint autoregressive and bidirectional training with annealed self-resampling, and construct a dedicated data synthesis and annotation pipeline that provides supervision for world editing. We also present WBench-Editing to systematically evaluate streaming world editing capabilities. EditWorld achieves the best overall performance on WBench-Editing with an overall score of 73.8 and an editing score of 80.0, substantially outperforming existing methods on editing-related metrics. https://github.com/leoisufa/EditWorld

Figures & tables

Figure 1: Data pipeline overview. (1) Our raw data is collected from two main sources: open-source navigation datasets and video editing datasets. (2) Sections 2.1 and 2.2 illustrate the synthesis pipelines for constructing global and local editable world data from the source videos. (3) Each synthesized training sample is annotated with four components: scene description, editing instruction, editing state, and camera poses. (4) Reference images are further extracted from the synthesized videos, together with rewritten editing instructions to prevent semantic leakage.

Figure 2: EditWorld. The left panel illustrates how our model uses an initial image as the world prior and autoregressively generates and edits the world in response to user-provided actions, textual prompts, and reference images. The right panel visualizes the attention patterns of our Gated Causal Attention and Sparse Context mechanisms. For simplicity, we illustrate the case with one sink chunk, one recent chunk, one preceding prompt, and one reference image.

Model	Overall	Editing	Navigation	Quality	Setting	Consistency	Physical
MatrixGame3 ( Wang et al., 2026b )	54.0	18.1	65.7	74.0	42.8	82.5	49.7
JoyAI-Echo-1.5 ( Zhang et al., 2026b )	53.8	21.4	48.0	71.9	51.4	80.0	62.0
minWM(Wan 2.1) ( Zhao et al., 2026a )	53.2	36.2	48.1	62.8	48.6	66.7	61.5
Astra ( Zhu et al., 2026c )	56.1	27.1	65.5	68.5	44.3	87.4	53.4
Alaya-EVOKE ( Yin et al., 2026 )	57.7	47.2	63.6	71.2	40.2	59.8	56.6
HY-World 1.5 ( Sun et al., 2025 )	56.7	21.8	62.0	64.8	57.2	89.0	61.7

Table 1: Comparison results on WBench-Editing. All evaluated methods use their autoregressive model variants, and all test cases are evaluated using text-only multi-turn interactive editing instructions. The top four results in each column are highlighted with progressively darker colors.

Figure 3: Comparison results with video world models on WBench-Editing. The upper panel presents multi-turn world editing with text-only instructions, while the lower panel shows multi-turn editing with reference-image conditioning. Our model follows the editing instructions more accurately while better preserving the consistency of the overall world. Zoom in for the best view.

Model	Overall	Editing	Navigation	Quality	Setting	Consistency	Physical
ABot-World ( Jiang et al., 2026 )	57.3	25.0	66.9	72.0	45.4	80.5	60.9
SolarWM(H3) ( Huang et al., 2026a )	64.6	19.8	85.9	74.8	67.6	87.5	64.9
XGEN-JING(BI) ( XGEN-JING, 2026 )	68.7	63.6	70.4	72.4	70.0	79.8	62.3
EditWorld (Ours)	74.0	74.6	80.5	73.3	65.4	83.1	67.3

Table 2: Comparison results on WBench-Editing (Reference only).

Model	Average	Quality	Setting	Interaction	Consistency	Physical
Astra ( Zhu et al., 2026c )	63.7	67.1	59.6	66.9	73.3	51.4
HY-GameCraft ( Li et al., 2025 )	68.2	73.0	66.6	66.3	72.6	62.4
MatrixGame2 ( He et al., 2025b )	68.7	73.8	67.1	80.3	65.1	57.2
Kairos 3.0 ( Kairos et al., 2026 )	70.3	74.0	70.3	64.1	82.6	60.4
MatrixGame3 ( Wang et al., 2026b )	71.3	75.5	63.6	83.6	74.5	59.3
Infinite-World ( Wu et al., 2026b )	72.8	77.0	69.3	75.4	80.0	62.1

Table 3: Comparison results on WBench. All baseline results are collected from the official WBench Leaderboard, accessed on September 21, 2026. The top four results in each column are highlighted with progressively darker colors.

Figure 4: (a) Unidirectional reference-video attention better preserves fine-grained reference details than bidirectional patterns. (b) Short-term textual context improves temporal continuity across prompt transitions; without preceding prompts, switching editing instructions causes abrupt scene changes and weaker consistency with the preceding video content.

Model	Editing Overall	Editing Presence	Semantic Alignment	Editing Completion	Detail Accuracy	Edit Cleanliness
AR Only	69.7	82.6	79.0	73.5	34.9	78.3
AR + BI	80.0	93.0	89.2	86.2	53.2	78.1

Table 4: Ablation of the training objective on WBench-Editing. Detailed editing-related metrics are reported for models trained with joint objectives and with the autoregressive objective only.

Figure 5: Comparison results with existing video world models on WBench-Edit (text only).

Figure 6: Comparison results with existing video world models on WBench-Edit (text only).

Figure 7: Comparison results with video world models on WBench-Edit (with reference image).

Figure 8: Comparison results with video world models on WBench-Edit (with reference image).

Figure 9: Visualization results of our model. The first image on the left is the reference input.

Figure 10: Visualization results of our model. The first image on the left is the reference input.

Figure 11: Visualization results of multi-turn interactive editing of our model.

Figure 12: Visualization results of multi-turn interactive editing of our model.

Explore similar work

World in World: Explore the World with World Models
Sep 11, 2026Chenxi Song, Yanming Yang, Chi ZhangVideo World Models Autoregressive Video Generation
Westlake AGI Lab · AGI Lab, Westlake University
AlayaWorld: Interactive Long-Horizon World Modeling -- Full Technical Report
Jul 20, 2026AlayaWorld Team, Kaipeng Zhang, Chuanhao Li +15Video World Models World
Alaya Lab
DreamX-World 1.0: A General-Purpose Interactive World Model
Jun 15, 2026DreamX Team, Yancheng Bai, Rui Chen +20Video World Models World Models
DreamX Team

Sep 11, 2026, cs.CVJ/K move · Enter open · S save

World in World: Explore the World with World Models

Chenxi Song, Yanming Yang, Chi Zhang

Westlake AGI Lab · AGI Lab, Westlake University

Video World Models Autoregressive Video Generation World Models World Long-Horizon Task Planning

Open paper AskPDF arXiv

Autoregressive video world models enable interactive, long-horizon exploration, but flexible control remains challenging. Exploring a source video from new viewpoints requires the generated rollout to remain synchronised with the recorded event, place observed content in the requested view, plausibly complete newly exposed regions, and recover previously generated appearance on revisits. Existing methods typically address these requirements through task-specific modules or additional training. We present World in World, a training-free inference-time interface that converts heterogeneous control evidence into camera- and time-labelled clean visual states, which are read through the native self attention of a frozen causal video model. The evidence comprises source-video observations, target-view scene projections, geometry renderings that guide completion of newly exposed subject regions, and retrieved generated states beyond the rolling cache. Each evidence source carries token-level support and its own availability schedule. A correspondence router combines persistent point identities with geometry to establish token correspondences, guiding supported queries towards matching source-video tokens. Evidence-wise attention CFG (EWA) then independently regulates each auxiliary channel's additional contribution using attention responses from the same denoising forward pass. The shared interface supports camera-controlled rerendering, long-horizon revisiting, and human-motion transfer with the same frozen backbone. We evaluate World in World on camera-controlled video rerendering under diverse viewpoint changes, assessing perceptual quality, temporal consistency, and camera-following accuracy.

Abstract

Explore similar work

World in World: Explore the World with World Models

AlayaWorld: Interactive Long-Horizon World Modeling -- Full Technical Report

DreamX-World 1.0: A General-Purpose Interactive World Model