ReWorld: An Interactive World Model with Long-Horizon Memory

Zhifei Chen; Luozhou Wang; Guibao Shen; Dongyu Yan; Shuai Yang; Tianshuo Xu; Yihua Du; Wei Wang

cs.AIAug 24, 2026

ReWorld: An Interactive World Model with Long-Horizon Memory

Authors: Zhifei Chen, Luozhou Wang, Guibao Shen, Dongyu Yan, Shuai Yang, Tianshuo Xu, Yihua Du, Wei Wang,

Organizations: HKUST(GZ) · ATH, Alibaba

Abstract

An interactive world model must follow the user's actions, remember the places it has shown, and stream in real time. The tension is structural: control wants a short horizon, memory wants an unbounded one. ReWorld separates the two during training and bounds them at inference. Mixed per-head attention windows confine most heads to the recent past while a small set of global heads attends over the entire history, and random head routing keeps either capability from binding to particular heads; random chunk dropping makes sparse histories in-distribution. At inference the whole past lives under a fixed budget: a bounded KV cache backed by a pose-indexed landmark bank, from which the model retrieves the landmarks nearest the current pose. A metric-scale-aligned data engine places eight sources -- Unreal-rendered fly-throughs, game roaming, and real-world footage -- on one physical action scale, so the same key press moves the camera the same distance in every source, and palindrome trajectories supply the revisit evidence that memory training needs. Distribution-matching distillation confined to a LoRA adapter then compresses sampling to four steps: one backbone serves both a high-fidelity multi-step mode and a real-time interactive one, streaming 704x1280 video across photorealistic, game-style, and stylized worlds. Under a three-axis protocol covering action following, long-horizon recall, and video quality, against six recent interactive world models it attains the best control fidelity ( $11.95^\circ$ rotation error and the best camera-motion consistency) and the best generation quality; and on minute-long out-and-back rollouts ( $64$ ,s, $384$ latents), its fixed 12-chunk cache still regenerates the starting view -- at rollout lengths where a sliding window has long evicted the evidence and full-KV attention runs out of memory.

Figures & tables

Figure 2: Overview of ReWorld. Left: a metric-aligned data pipeline places UE-rendered, real-world, and game footage on a single physical action scale, and palindrome routes—the camera retracing its own path—supply revisit supervision (Sec. 3 ). Middle: teacher-forcing training turns the bidirectional backbone into a streaming world model, with a DMD LoRA trained alongside for few-step real-time inference (Secs. 2.2 and 2.4 ); control (action injection, local short-window heads) and memory (MRoPE pose-indexed attention, global long-window heads) are decoupled, and the partition into global and local heads is switched every step through a fixed pool of random partitions so that neither capability binds to particular heads (Sec. 2.2 ); chunk-drop training masks random parts of the history, matching the sparse cache the model will read at inference (Sec. 2.3 ). Right: at inference, a fixed cache budget holds a sink chunk, pose-retrieved landmarks, and a recent window next to the chunk being generated; chunks aging out of the recent window are consolidated into a bounded landmark bank and retrieved by pose proximity upon revisits, so spatial memory persists over unbounded rollouts at constant cost (Sec. 2.3 ).

Design	Attention index	Where pose enters	Attn. passes	Extra cost
Temporal RoPE	time & space	—	1	—
HY-World 1.5 (two-pass) [ 30 ]	time $+$ projective pose	separate camera-aware pass, fused	2	${\approx}2\times$ attention FLOPs
E-PRoPE [ 7 ]	time $+$ projective pose	separate reduced-token camera pass, added back	2	one reduced pass ( ${\sim}4.5\times$ fewer tokens)
MRoPE (ours)	time, space & pose	Q/K phase $+$ zero-init V/O residuals	1	1 small MLP $+$ 2 linears

Table 1: Pose conditioning in attention: design comparison. Two-pass designs such as HY-World 1.5 add a camera-aware attention pass next to the temporal-RoPE pass and fuse the outputs; E-PRoPE trims the second pass to spatially reduced tokens; MRoPE folds pose into the same pass, leaving the attention kernel, mask layout, and per-head windows unchanged.

Figure 3: Mixed per-head attention windows, routed from a pattern pool. Left: local heads attend to the last $w{=}12$ frames (reactive control); global heads attend to the full causal history (long-range memory). Right: each optimizer step switches to the next of $|\mathcal{P}|{=}12$ random head partitions, so neither ability binds to particular heads.

Figure 4: Memory consolidation under a fixed KV budget. Left: at inference the cache holds one sink, six pose-retrieved landmarks, and five recent chunks ( $B{=}12$ ); a chunk aging out of the recent window is consolidated into the bounded landmark bank—kept sparse and diverse—and re-enters the cache when the camera returns nearby (a full bank evicts its most redundant member). Right: chunk-drop training randomly masks past chunks at every step, so such sparse caches are in-distribution.

Figure 5: Real-time distillation. A caption-domain DMD LoRA plugs into the frozen backbone; the control path carries no LoRA.

Figure 6: The four-stage data pipeline. Eight UE, real-world, and game sources; an automated UE service renders $337$ environments with NavMesh-based trajectory planning (the control pattern anchors each path at two points of interest and fills the segment between them with direction-balanced random motion); a curation funnel filters each pool and evens out on-screen speed by dropping too-fast and too-slow clips; unification then rescales the pose numbers to one shared metric scale — the same action moves the camera the same distance — before clip-level sampling.

Source	#Clips	Duration	Pose	Caption
UE control (ours)	$95{,}895$	$15$ s	engine GT (c2w, cm)	pipeline-generated
UE expansion (ours)	$42{,}536$	$30$ – $60$ s	engine GT (c2w, cm)	scene summary + camera
DL3DV	$29{,}864$	$15$ s	VIPE	Qwen-VL
RealEstate10K	$12{,}065$	$\geq 8$ s	VIPE (re-estimated)	pre-computed
Sekai real-walking-hq	$14{,}730$	$60$ s	MegaSaM (c2w)	official CSV
Game roaming ( $79$ games)	$18{,}387$	$15$ s	VIPE (metric)	Qwen-VL

Table 2: The eight-source joint training corpus. #Clips are counts after per-source filtering (Sec. 3.4 ); Duration is the per-clip length; Pose gives the camera-trajectory provenance; Caption gives the text-annotation provenance.

	Overall			RotErr ∘ $\downarrow$ per trajectory
Method	RotErr ∘ $\downarrow$	TransErr $\downarrow$	CamMC $\downarrow$	dolly	strafe	arc_yaw	arc_pitch	s_curve	palindrome
SANA-WM [ 48 ]	13.02	0.123	0.388	4.95	1.51	32.01	21.33	17.54	0.80
DreamX [ 7 ]	13.10	0.114	0.381	2.50	1.21	33.00	20.71	18.06	3.13
HY-WorldPlay [ 30 ]	14.66	0.114	0.427	0.26	0.24	43.53	22.00	21.73	0.20
Matrix-Game 3.0 [ 35 ]	15.45	0.075	0.394	8.19	1.00	39.93	20.59	17.75	5.24
LingBot-World [ 29 ]	12.59	0.107	0.354	3.10	6.01	26.21	26.83	12.23	1.16

Table 3: Camera controllability on the 40-image

\times

6-trajectory benchmark (240 clips per method). Overall RotErr, TransErr, and CamMC (

\downarrow

) averaged over all trajectories, followed by per-trajectory RotErr ( ∘ ,

\downarrow

) after

\mathrm{Sim}(3)

and global orientation alignment. Best per column in bold.

Figure 7: Constructed palindrome trajectories. The camera moves out (blue) and retraces its path (orange), so the needle view at $t{=}0$ must be regenerated at $t{=}k{-}1$ ; scored mirror pairs $(t,\,k{-}1{-}t)$ span at least half the rollout.

	$k{=}48$ ( $\approx$ 8 s)					$k{=}96$ ( $\approx$ 16 s)
Method	Path Length	SSIM $\uparrow$	LPIPS $\downarrow$	DINO $\uparrow$	ORB $\uparrow$	Path Length	SSIM $\uparrow$	LPIPS $\downarrow$	DINO $\uparrow$	ORB $\uparrow$
SANA-WM [ 48 ]	296	0.318	0.433	0.810	0.174	499	0.313	0.528	0.700	0.161
DreamX [ 7 ]	279	0.214	0.553	0.740	0.180	454	0.194	0.627	0.603	0.159
HY-WorldPlay [ 30 ]	113 †	0.448	0.163	0.969	0.464	210 †	0.427	0.247	0.942	0.460
Matrix-Game 3.0 [ 35 ]	312	0.318	0.369	0.891	0.292	724	0.275	0.478	0.850	0.247
LingBot-World [ 29 ]	352	0.274	0.557	0.663	0.177	834	0.251	0.635	0.509	0.162

Table 4: Long-horizon memory against the six interactive world-model baselines on the palindromic revisit bench: absolute revisit-similarity scores (SSIM, LPIPS, DINO, ORB) at rollout lengths

k{=}48

and

k{=}96

latents, alongside the executed path length (accumulated median optical flow of the outbound half, px). Every method receives the same action commands, but how far it actually travels differs, and a shorter path generally yields higher revisit scores.

k{=}48

is a sanity length that several baselines can solve from context alone;

k{=}96

forces recall from beyond every bounded window. Best per column in bold , second best underlined ; path length is descriptive, not ranked.

Figure 8: Qualitative memory comparison on a strafe-and-return rollout. All methods start from the same input frame and execute the same lateral palindrome—the camera strafes left, then retraces the same line back to the start, without ever turning—so the last column should reproduce the first: the same landmarks, in the same places. Columns are latents $0/8/28/48/72/94$ of a $k{=}96$ rollout; each baseline is driven through its native control interface. ReWorld (top row; bounded cache $B{=}12$ with the landmark bank, four-step distilled) returns to the input view with the layout intact, whereas the baselines either drift off the commanded path or regenerate the revisited region with altered geometry.

Method	Mean $\uparrow$	Imaging	Aesthetic	Subject Cons.	Background Cons.	Temporal Flick.	Dynamic Degree	Motion Smooth.
SANA-WM [ 48 ]	0.835	0.668	0.537	0.908	0.919	0.942	0.900	0.969
DreamX [ 7 ]	0.828	0.670	0.542	0.880	0.904	0.909	0.946	0.946
HY-WorldPlay [ 30 ]	0.842	0.670	0.601	0.976	0.957	0.973	0.733	0.987
Matrix-Game 3.0 [ 35 ]	0.836	0.713	0.461	0.894	0.919	0.921	0.983	0.962
LingBot-World [ 29 ]	0.841	0.692	0.577	0.905	0.915	0.918	0.933	0.947
Yume-1.5 [ 26 ]	0.844	0.669	0.565	0.878	0.911	0.934	0.988	0.964

Table 5: VBench quality on the seven video-intrinsic dimensions (higher is better), computed on the shared benchmark clips normalized to 32 frames /

1280\times 704

/ 16 fps. Best per column in bold.

Figure 9: Inference cache policies, qualitatively. On out-and-back trajectories, sliding window and KV compression drift or reinvent the revisited scene; the landmark cache (ours) returns to the start view.

Training recipe (consolidation arm)
	Revisit SSIM $\uparrow$
Method	$k{=}96$	$k{=}192$	$k{=}288$	$k{=}384$
base	0.4019	0.3747	0.3698	0.3387
+ CD	0.4043	0.3941	0.3846	0.3463
+ RH	0.4319	0.3787	0.3836	0.3565
+ CD + RH (ours)	0.4358	0.3929	0.3943	0.3752

Table 6: Revisit SSIM

\uparrow

on the long-rollout memory test (Sec. 4.3 ) at four rollout lengths

k

. Upper block: the training recipe is varied with the inference arm fixed to our consolidation. Lower block: the inference cache policy is varied on the final recipe; full-KV is the unbounded upper bound (OOM beyond

k{=}192

Interactive World Models

Interactive world models predict and generate how environments change in response to actions, supporting simulation, planning, and control.

See all 117 papers

WorldPlay2: Extending Real-Time Interactive World Models in Control and Horizon
Sep 28, 2026Haiyu Zhang, Wenqiang Sun, Tengfei Wang +5Long-Horizon Video Generation World Models
ActWorld: From Explorable to Interactive World Model via Action-Aware Memory
Jun 16, 2026Zhexiao Xiong, Yizhi Song, Hao Kang +11HOI Generation World Model Learning
1Washington University in St. Louis · 2Intelligent Creation, ByteDance · *Work done during internship at ByteDance
World in World: Explore the World with World Models
Sep 10, 2026Chenxi Song, Yanming Yang, Chi ZhangCamera-Controlled Video Generation Long-Horizon Video Generation
Westlake AGI Lab · AGI Lab, Westlake University

Sep 28, 2026J/K move · Enter open · S save

WorldPlay2: Extending Real-Time Interactive World Models in Control and Horizon

Haiyu Zhang, Wenqiang Sun, Tengfei Wang, Junta Wu+4

Long-Horizon Video Generation World Models Action-Conditioned World Models Interactive World Models Video World Models

Open paper AskPDF arXiv

Interactive world models require responding in real time to versatile controls and maintaining long-horizon consistency. However, modeling heterogeneous controls remains difficult, while explosive contexts and unstable distillation impede achieving both long-horizon consistency and real-time responsiveness. In this paper, we present WorldPlay2, an interactive world model that couples a factorized hybrid control interface with a co-design of compressed memory and stable distillation. 1) Our factorized hybrid control interface integrates frame-aligned action control with structured semantic control that explicitly disentangles scene appearance, character identity, and dynamic semantic events, thereby facilitating effective control learning. 2) To achieve efficient long-horizon modeling, we compress historical contexts into compact memory tokens shared by the autoregressive student and the bidirectional teacher. This design enables clip-wise, memory-conditioned score evaluation instead of jointly processing an entire long rollout, substantially reducing distillation overhead. 3) We further propose Stable Forcing, which initializes the autoregressive student via a few-step strategy and leverages full-rollout replay to preserve the quality of long-horizon rollouts, ensuring robust and stable distillation. Extensive experiments demonstrate the strong generalizability of our model and its superior performance compared to existing methods.

Abstract

Interactive World Models

Explore similar work

WorldPlay2: Extending Real-Time Interactive World Models in Control and Horizon

ActWorld: From Explorable to Interactive World Model via Action-Aware Memory

World in World: Explore the World with World Models