ReWorld: An Interactive World Model with Long-Horizon Memory

Zhifei Chen; Luozhou Wang; Guibao Shen; Dongyu Yan; Shuai Yang; Tianshuo Xu; Yihua Du; Wei Wang

cs.AIAug 24, 2026

ReWorld: An Interactive World Model with Long-Horizon Memory

Authors: Zhifei Chen, Luozhou Wang, Guibao Shen, Dongyu Yan, Shuai Yang, Tianshuo Xu, Yihua Du, Wei Wang,

Organizations: HKUST(GZ) · ATH, Alibaba

Abstract

An interactive world model must follow the user's actions, remember the places it has shown, and stream in real time. The tension is structural: control wants a short horizon, memory wants an unbounded one. ReWorld separates the two during training and bounds them at inference. Mixed per-head attention windows confine most heads to the recent past while a small set of global heads attends over the entire history, and random head routing keeps either capability from binding to particular heads; random chunk dropping makes sparse histories in-distribution. At inference the whole past lives under a fixed budget: a bounded KV cache backed by a pose-indexed landmark bank, from which the model retrieves the landmarks nearest the current pose. A metric-scale-aligned data engine places eight sources -- Unreal-rendered fly-throughs, game roaming, and real-world footage -- on one physical action scale, so the same key press moves the camera the same distance in every source, and palindrome trajectories supply the revisit evidence that memory training needs. Distribution-matching distillation confined to a LoRA adapter then compresses sampling to four steps: one backbone serves both a high-fidelity multi-step mode and a real-time interactive one, streaming 704x1280 video across photorealistic, game-style, and stylized worlds. Under a three-axis protocol covering action following, long-horizon recall, and video quality, against six recent interactive world models it attains the best control fidelity ( $11.95^\circ$ rotation error and the best camera-motion consistency) and the best generation quality; and on minute-long out-and-back rollouts ( $64$ ,s, $384$ latents), its fixed 12-chunk cache still regenerates the starting view -- at rollout lengths where a sliding window has long evicted the evidence and full-KV attention runs out of memory.

Figures & tables

Figure 2: Overview of ReWorld. Left: a metric-aligned data pipeline places UE-rendered, real-world, and game footage on a single physical action scale, and palindrome routes—the camera retracing its own path—supply revisit supervision (Sec. 3 ). Middle: teacher-forcing training turns the bidirectional backbone into a streaming world model, with a DMD LoRA trained alongside for few-step real-time inference (Secs. 2.2 and 2.4 ); control (action injection, local short-window heads) and memory (MRoPE pose-indexed attention, global long-window heads) are decoupled, and the partition into global and local heads is switched every step through a fixed pool of random partitions so that neither capability binds to particular heads (Sec. 2.2 ); chunk-drop training masks random parts of the history, matching the sparse cache the model will read at inference (Sec. 2.3 ). Right: at inference, a fixed cache budget holds a sink chunk, pose-retrieved landmarks, and a recent window next to the chunk being generated; chunks aging out of the recent window are consolidated into a bounded landmark bank and retrieved by pose proximity upon revisits, so spatial memory persists over unbounded rollouts at constant cost (Sec. 2.3 ).

Design	Attention index	Where pose enters	Attn. passes	Extra cost
Temporal RoPE	time & space	—	1	—
HY-World 1.5 (two-pass) [ 30 ]	time $+$ projective pose	separate camera-aware pass, fused	2	${\approx}2\times$ attention FLOPs
E-PRoPE [ 7 ]	time $+$ projective pose	separate reduced-token camera pass, added back	2	one reduced pass ( ${\sim}4.5\times$ fewer tokens)
MRoPE (ours)	time, space & pose	Q/K phase $+$ zero-init V/O residuals	1	1 small MLP $+$ 2 linears

Table 1: Pose conditioning in attention: design comparison. Two-pass designs such as HY-World 1.5 add a camera-aware attention pass next to the temporal-RoPE pass and fuse the outputs; E-PRoPE trims the second pass to spatially reduced tokens; MRoPE folds pose into the same pass, leaving the attention kernel, mask layout, and per-head windows unchanged.

Figure 3: Mixed per-head attention windows, routed from a pattern pool. Left: local heads attend to the last $w{=}12$ frames (reactive control); global heads attend to the full causal history (long-range memory). Right: each optimizer step switches to the next of $|\mathcal{P}|{=}12$ random head partitions, so neither ability binds to particular heads.

Figure 4: Memory consolidation under a fixed KV budget. Left: at inference the cache holds one sink, six pose-retrieved landmarks, and five recent chunks ( $B{=}12$ ); a chunk aging out of the recent window is consolidated into the bounded landmark bank—kept sparse and diverse—and re-enters the cache when the camera returns nearby (a full bank evicts its most redundant member). Right: chunk-drop training randomly masks past chunks at every step, so such sparse caches are in-distribution.

Figure 5: Real-time distillation. A caption-domain DMD LoRA plugs into the frozen backbone; the control path carries no LoRA.

Figure 6: The four-stage data pipeline. Eight UE, real-world, and game sources; an automated UE service renders $337$ environments with NavMesh-based trajectory planning (the control pattern anchors each path at two points of interest and fills the segment between them with direction-balanced random motion); a curation funnel filters each pool and evens out on-screen speed by dropping too-fast and too-slow clips; unification then rescales the pose numbers to one shared metric scale — the same action moves the camera the same distance — before clip-level sampling.

Source	#Clips	Duration	Pose	Caption
UE control (ours)	$95{,}895$	$15$ s	engine GT (c2w, cm)	pipeline-generated
UE expansion (ours)	$42{,}536$	$30$ – $60$ s	engine GT (c2w, cm)	scene summary + camera
DL3DV	$29{,}864$	$15$ s	VIPE	Qwen-VL
RealEstate10K	$12{,}065$	$\geq 8$ s	VIPE (re-estimated)	pre-computed
Sekai real-walking-hq	$14{,}730$	$60$ s	MegaSaM (c2w)	official CSV
Game roaming ( $79$ games)	$18{,}387$	$15$ s	VIPE (metric)	Qwen-VL

Table 2: The eight-source joint training corpus. #Clips are counts after per-source filtering (Sec. 3.4 ); Duration is the per-clip length; Pose gives the camera-trajectory provenance; Caption gives the text-annotation provenance.

	Overall			RotErr ∘ $\downarrow$ per trajectory
Method	RotErr ∘ $\downarrow$	TransErr $\downarrow$	CamMC $\downarrow$	dolly	strafe	arc_yaw	arc_pitch	s_curve	palindrome
SANA-WM [ 48 ]	13.02	0.123	0.388	4.95	1.51	32.01	21.33	17.54	0.80
DreamX [ 7 ]	13.10	0.114	0.381	2.50	1.21	33.00	20.71	18.06	3.13
HY-WorldPlay [ 30 ]	14.66	0.114	0.427	0.26	0.24	43.53	22.00	21.73	0.20
Matrix-Game 3.0 [ 35 ]	15.45	0.075	0.394	8.19	1.00	39.93	20.59	17.75	5.24
LingBot-World [ 29 ]	12.59	0.107	0.354	3.10	6.01	26.21	26.83	12.23	1.16

Table 3: Camera controllability on the 40-image

\times

6-trajectory benchmark (240 clips per method). Overall RotErr, TransErr, and CamMC (

\downarrow

) averaged over all trajectories, followed by per-trajectory RotErr ( ∘ ,

\downarrow

) after

\mathrm{Sim}(3)

and global orientation alignment. Best per column in bold.

Figure 7: Constructed palindrome trajectories. The camera moves out (blue) and retraces its path (orange), so the needle view at $t{=}0$ must be regenerated at $t{=}k{-}1$ ; scored mirror pairs $(t,\,k{-}1{-}t)$ span at least half the rollout.

	$k{=}48$ ( $\approx$ 8 s)					$k{=}96$ ( $\approx$ 16 s)
Method	Path Length	SSIM $\uparrow$	LPIPS $\downarrow$	DINO $\uparrow$	ORB $\uparrow$	Path Length	SSIM $\uparrow$	LPIPS $\downarrow$	DINO $\uparrow$	ORB $\uparrow$
SANA-WM [ 48 ]	296	0.318	0.433	0.810	0.174	499	0.313	0.528	0.700	0.161
DreamX [ 7 ]	279	0.214	0.553	0.740	0.180	454	0.194	0.627	0.603	0.159
HY-WorldPlay [ 30 ]	113 †	0.448	0.163	0.969	0.464	210 †	0.427	0.247	0.942	0.460
Matrix-Game 3.0 [ 35 ]	312	0.318	0.369	0.891	0.292	724	0.275	0.478	0.850	0.247
LingBot-World [ 29 ]	352	0.274	0.557	0.663	0.177	834	0.251	0.635	0.509	0.162

Table 4: Long-horizon memory against the six interactive world-model baselines on the palindromic revisit bench: absolute revisit-similarity scores (SSIM, LPIPS, DINO, ORB) at rollout lengths

k{=}48

and

k{=}96

latents, alongside the executed path length (accumulated median optical flow of the outbound half, px). Every method receives the same action commands, but how far it actually travels differs, and a shorter path generally yields higher revisit scores.

k{=}48

is a sanity length that several baselines can solve from context alone;

k{=}96

forces recall from beyond every bounded window. Best per column in bold , second best underlined ; path length is descriptive, not ranked.

Figure 8: Qualitative memory comparison on a strafe-and-return rollout. All methods start from the same input frame and execute the same lateral palindrome—the camera strafes left, then retraces the same line back to the start, without ever turning—so the last column should reproduce the first: the same landmarks, in the same places. Columns are latents $0/8/28/48/72/94$ of a $k{=}96$ rollout; each baseline is driven through its native control interface. ReWorld (top row; bounded cache $B{=}12$ with the landmark bank, four-step distilled) returns to the input view with the layout intact, whereas the baselines either drift off the commanded path or regenerate the revisited region with altered geometry.

Method	Mean $\uparrow$	Imaging	Aesthetic	Subject Cons.	Background Cons.	Temporal Flick.	Dynamic Degree	Motion Smooth.
SANA-WM [ 48 ]	0.835	0.668	0.537	0.908	0.919	0.942	0.900	0.969
DreamX [ 7 ]	0.828	0.670	0.542	0.880	0.904	0.909	0.946	0.946
HY-WorldPlay [ 30 ]	0.842	0.670	0.601	0.976	0.957	0.973	0.733	0.987
Matrix-Game 3.0 [ 35 ]	0.836	0.713	0.461	0.894	0.919	0.921	0.983	0.962
LingBot-World [ 29 ]	0.841	0.692	0.577	0.905	0.915	0.918	0.933	0.947
Yume-1.5 [ 26 ]	0.844	0.669	0.565	0.878	0.911	0.934	0.988	0.964

Table 5: VBench quality on the seven video-intrinsic dimensions (higher is better), computed on the shared benchmark clips normalized to 32 frames /

1280\times 704

/ 16 fps. Best per column in bold.

Figure 9: Inference cache policies, qualitatively. On out-and-back trajectories, sliding window and KV compression drift or reinvent the revisited scene; the landmark cache (ours) returns to the start view.

Training recipe (consolidation arm)
	Revisit SSIM $\uparrow$
Method	$k{=}96$	$k{=}192$	$k{=}288$	$k{=}384$
base	0.4019	0.3747	0.3698	0.3387
+ CD	0.4043	0.3941	0.3846	0.3463
+ RH	0.4319	0.3787	0.3836	0.3565
+ CD + RH (ours)	0.4358	0.3929	0.3943	0.3752

Table 6: Revisit SSIM

\uparrow

on the long-rollout memory test (Sec. 4.3 ) at four rollout lengths

k

. Upper block: the training recipe is varied with the inference arm fixed to our consolidation. Lower block: the inference cache policy is varied on the final recipe; full-KV is the unbounded upper bound (OOM beyond

k{=}192

Interactive World Models

Interactive world models predict and generate how environments change in response to actions, supporting simulation, planning, and control.

See all 117 papers

Sep 28, 2026

WorldPlay2: Extending Real-Time Interactive World Models in Control and Horizon

Interactive world models require responding in real time to versatile controls and maintaining long-horizon consistency. However, modeling heterogeneous controls remains difficult, while explosive contexts and unstable distillation impede achieving both long-horizon consistency and real-time responsiveness. In this paper, we present WorldPlay2, an interactive world model that couples a factorized hybrid control interface with a co-design of compressed memory and stable distillation. 1) Our factorized hybrid control interface integrates frame-aligned action control with structured semantic control that explicitly disentangles scene appearance, character identity, and dynamic semantic events, thereby facilitating effective control learning. 2) To achieve efficient long-horizon modeling, we compress historical contexts into compact memory tokens shared by the autoregressive student and the bidirectional teacher. This design enables clip-wise, memory-conditioned score evaluation instead of jointly processing an entire long rollout, substantially reducing distillation overhead. 3) We further propose Stable Forcing, which initializes the autoregressive student via a few-step strategy and leverages full-rollout replay to preserve the quality of long-horizon rollouts, ensuring robust and stable distillation. Extensive experiments demonstrate the strong generalizability of our model and its superior performance compared to existing methods.

Haiyu Zhang, Wenqiang Sun, Tengfei Wang +5

Long-Horizon Video Generation World Models Action-Conditioned World Models Interactive World Models Video World Models

Ask PDF

Jun 16, 2026

ActWorld: From Explorable to Interactive World Model via Action-Aware Memory

Interactive world models aim to simulate environment dynamics under real-time user actions. However, their action vocabulary is largely confined to navigation: most actions correspond to motion (e.g., walk, turn, look around), while interaction with objects in the scene (e.g., pick up plates, open doors, or trigger physical responses) is either absent, restricted to game domains, or relegated to prompt-to-full-video scenarios. The resulting worlds are visually explorable but not truly actionable. In this work, we present ActWorld, an interactive world model that extends prior navigation-centric generators to support mid-rollout object interaction within a chunk-autoregressive framework. We argue that the navigation-interaction gap stems from two bottlenecks. First, a data bottleneck: the lack of human-object interaction data with accurate, dense labels. Second, a memory bottleneck: recency-biased history compression in existing world models discards the event-transition frames that causally determine subsequent object states, leading to an action-forgetting pathology. On the data side, we construct a 100K interaction video dataset, each annotated with per-chunk captions via chain-of-thought reasoning. On the model side, we introduce a hierarchical action-aware memory design that routes history compression by interaction importance, complemented by a persistent memory bank that maintains event-update and object-identity tokens across long rollouts. Experiments show that ActWorld supports both flexible navigation and rich object interaction within a single model, substantially improving interaction fidelity over navigation-only baselines without sacrificing viewpoint control. Project page is available at https://interactwm.github.io/ActWorld.

Zhexiao Xiong, Yizhi Song, Hao Kang +11

1Washington University in St. Louis · 2Intelligent Creation, ByteDance · *Work done during internship at ByteDance

HOI Generation World Model Learning Action-Conditioned World Models Interactive World Models Memory-Augmented Agents Video World Models

Ask PDF

Sep 10, 2026

World in World: Explore the World with World Models

Autoregressive video world models enable interactive, long-horizon exploration, but flexible control remains challenging. Exploring a source video from new viewpoints requires the generated rollout to remain synchronised with the recorded event, place observed content in the requested view, plausibly complete newly exposed regions, and recover previously generated appearance on revisits. Existing methods typically address these requirements through task-specific modules or additional training. We present World in World, a training-free inference-time interface that converts heterogeneous control evidence into camera- and time-labelled clean visual states, which are read through the native self attention of a frozen causal video model. The evidence comprises source-video observations, target-view scene projections, geometry renderings that guide completion of newly exposed subject regions, and retrieved generated states beyond the rolling cache. Each evidence source carries token-level support and its own availability schedule. A correspondence router combines persistent point identities with geometry to establish token correspondences, guiding supported queries towards matching source-video tokens. Evidence-wise attention CFG (EWA) then independently regulates each auxiliary channel's additional contribution using attention responses from the same denoising forward pass. The shared interface supports camera-controlled rerendering, long-horizon revisiting, and human-motion transfer with the same frozen backbone. We evaluate World in World on camera-controlled video rerendering under diverse viewpoint changes, assessing perceptual quality, temporal consistency, and camera-following accuracy.

Chenxi Song, Yanming Yang, Chi Zhang

Westlake AGI Lab · AGI Lab, Westlake University

Camera-Controlled Video Generation Long-Horizon Video Generation Video Generation Temporal Consistency in Video Generation Video World Models

Ask PDF

Abstract

Interactive World Models

Explore similar work

WorldPlay2: Extending Real-Time Interactive World Models in Control and Horizon

ActWorld: From Explorable to Interactive World Model via Action-Aware Memory

World in World: Explore the World with World Models