Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Models

Kaijin Chen; Dingkang Liang; Xin Zhou; Yikang Ding; Xiaoqiang Liu; Pengfei Wan; Xiang Bai

cs.CVMar 26, 2026

Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Models

Authors: Kaijin Chen, Dingkang Liang, Xin Zhou, Yikang Ding, Xiaoqiang Liu, Pengfei Wan, Xiang Bai

Organizations: Huazhong University of Science and Technology · Kling Team, Kuaishou Technology

Abstract

Video world models have shown immense potential in simulating the physical world, yet existing memory mechanisms primarily treat environments as static canvases. When dynamic subjects hide out of sight and later re-emerge, current methods often struggle, leading to frozen, distorted, or vanishing subjects. To address this, we introduce Hybrid Memory, a novel paradigm requiring models to simultaneously act as precise archivists for static backgrounds and vigilant trackers for dynamic subjects, ensuring motion continuity during out-of-view intervals. To facilitate research in this direction, we construct HM-World, the first large-scale video dataset dedicated to hybrid memory. It features 59K high-fidelity clips with decoupled camera and subject trajectories, encompassing 17 diverse scenes, 49 distinct subjects, and meticulously designed exit-entry events to rigorously evaluate hybrid coherence. Furthermore, we propose HyDRA, a specialized memory architecture that compresses memory into tokens and utilizes a spatiotemporal relevance-driven retrieval mechanism. By selectively attending to relevant motion cues, HyDRA effectively preserves the identity and motion of hidden subjects. Extensive experiments on HM-World demonstrate that our method significantly outperforms state-of-the-art approaches in both dynamic subject consistency and overall generation quality. Code is publicly available at https://github.com/H-EmbodVis/HyDRA.

Figures & tables

Figure 2: Instances of exit-entry camera motion.

Figure 3: Construction Procedure of HM-World. We combine (a) 3D scenes, (b) subjects, (c) subject trajectories, and (d) camera trajectories to render data containing dynamics in Unreal Engine 5.

Dataset	Reference	Dynamic Subject	Subject Exit-Enter	Subject Pose	Camera Movable	Total Num.
WorldScore [ 7 ]	ICCV 25	✓	✗	✗	✓	3K
Context-As-Memory [ 37 ]	SIGGRAPH Asia 25	✗	✗	✗	✓	10K
Multi-Cam Video [ 1 ]	ICCV 25	✓	✗	✗	✓	136K
360°-Motion [ 32 ]	ICLR 25	✓	✗	✓	✗	5.4K
HM-World (ours)	-	✓	✓	✓	✓	59K

Table 1: The comparison between existing datasets and HM-World dataset. "Dynamic Subject" means including moving subjects, "Exit-Enter" refers to containing exit-entry events in clips, and "Subject Pose" denotes including annotated 3D poses of subjects.

Figure 5: Overview of HyDRA . (a) Memory Tokenization Module. (b) Dynamic retrieval attention computes relevance between the target query and memory tokens to retrieve the top-k relevant tokens, enabling the model to recall associated hybrid memory.

Method	Reference	PSNR	SSIM	LPIPS	$\text{DSC}_{ctx}$	$\text{DSC}_{GT}$	Subj. Cons.	Bg. Cons.
Baseline	-	18.696	0.517	0.356	0.812	0.837	0.903	0.925
DFoT [ 26 ]	ICML 25	17.693	0.482	0.410	0.803	0.826	0.893	0.913
Context-as-Memory [ 37 ]	SIGGRAPH Asia 25	18.921	0.530	0.342	0.816	0.839	0.911	0.922
HyDRA (ours)	-	20.357	0.606	0.289	0.827	0.849	0.926	0.932

Table 2: Quantitative comparison with other methods.

Method	PSNR	SSIM	LPIPS	$\text{DSC}_{ctx}$	$\text{DSC}_{GT}$	Subject Consistency	Background Consistency
WorldPlay [ 27 ]	14.855	0.355	0.500	0.822	0.832	0.910	0.925
HyDRA (ours)	20.357	0.606	0.289	0.827	0.849	0.926	0.932

Table 3: Quantitative comparison against the state-of-the-art commercial model.

Figure 6: Qualitative comparison with other methods. The green boxes in the figure represent consistently generated subjects, while the red boxes stand for failure cases.

$T$	$H\times W$	PSNR	SSIM	LPIPS	$\text{DSC}_{ctx}$	$\text{DSC}_{GT}$	Subject Consistency	Background Consistency
$2$	$2\times 2$	20.113	0.599	0.299	0.820	0.843	0.919	0.929
$2$	$4\times 4$	20.357	0.606	0.289	0.827	0.849	0.926	0.932
$2$	$8\times 8$	20.230	0.610	0.292	0.822	0.843	0.923	0.927
$1$	$4\times 4$	19.076	0.554	0.337	0.819	0.841	0.912	0.925

Table 4: Kernel Size of Memory Tokenizer.

Setting	PSNR	SSIM	LPIPS	$\text{DSC}_{ctx}$	$\text{DSC}_{GT}$	Subject Consistency	Background Consistency
5	19.309	0.566	0.339	0.817	0.836	0.913	0.927
10	20.357	0.606	0.289	0.827	0.849	0.926	0.932
15	20.333	0.612	0.291	0.828	0.842	0.925	0.935

Table 5: Number of retrieved tokens.

Method	PSNR	SSIM	LPIPS	$\text{DSC}_{ctx}$	$\text{DSC}_{GT}$	Subject Consistency	Background Consistency
FOV Overlap	19.776	0.586	0.300	0.820	0.844	0.908	0.930
Dynamic Affinity	20.357	0.606	0.289	0.827	0.849	0.926	0.932

Table 6: Approaches to retrieve tokens.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Qualitative comparison between retrieval methods. The upper displays frames selected by different methods, while the lower shows the generation results. Selected frames are the source frames of the selected tokens.

Figure 10: Distribution comparison of different retrieval methods. The x-axis and y-axis represent the token index and DiT layers, respectively. The bubble size and color reflect the selection frequency of each token during the entire denoising process. (a) The FOV overlap method yields a fixed token selection. (b) Our dynamic affinity method exhibits a diverse retrieval distribution, enabling the perception of richer memory contexts.

Figure 11: Qualitative comparison between different kernel sizes of the Memory Tokenizer. The red bounding boxes annotate the inconsistent region.

Panel of Figure 12 — Figure 12: Qualitative comparison between the number of retrieved tokens. The red bounding boxes annotate the inconsistent region.

Retrieval-Augmented Video Generation

See all 11 papers

LiveWorld: Simulating Out-of-Sight Dynamics in Generative Video World Models
Mar 7, 2026Zicheng Duan, Jiatong Xia, Zeyu Zhang +7Video World Models World Models
Adelaide University · The Australian National University · Monash University +2
Compression and Retrieval: Implicit Memory Retrieval for Video World Models
Jun 22, 2026Zhan Peng, Jie Ma, Huiqiang Sun +6Long-Horizon Video Generation Memory-Augmented Video Generation
Huazhong University of Science and Technology, China · HUJING Digital Media & Entertainment Group, China · Sun Yat-sen University, China
WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models
Oct 13, 2026Nan Zhou, Jianxiong Gao, Qiuyu Wang +10World Model Evaluation Video World Models
Zhejiang University · Ant Group · Fudan University +1

Mar 7, 2026J/K move · Enter open · S save

LiveWorld: Simulating Out-of-Sight Dynamics in Generative Video World Models

Zicheng Duan, Jiatong Xia, Zeyu Zhang, Wenbo Zhang+6

Adelaide University · The Australian National University · Monash University+2

Video World Models World Models Interactive World Models Latent Dynamics Modeling Long-Horizon Video Generation

Open paper AskPDF arXiv

Recent generative video world models aim to simulate visual environment evolution, allowing an observer to interactively explore the scene via camera control. However, they implicitly assume that the world only evolves within the observer's field of view. Once an object leaves the observer's view, its state is "frozen" in memory, and revisiting the same region later often fails to reflect events that should have occurred in the meantime. In this work, we identify and formalize this overlooked limitation as the "out-of-sight dynamics" problem, which impedes video world models from representing a continuously evolving world. To address this issue, we propose LiveWorld, a novel framework that extends video world models to support persistent world evolution. Instead of treating the world as static observational memory, LiveWorld models a persistent global state composed of a static 3D background and dynamic entities that continue evolving even when unobserved. To maintain these unseen dynamics, LiveWorld introduces a monitor-based mechanism that autonomously simulates the temporal progression of active entities and synchronizes their evolved states upon revisiting, ensuring spatially coherent rendering. For evaluation, we further introduce LiveBench, a dedicated benchmark for the task of maintaining out-of-sight dynamics. Extensive experiments show that LiveWorld enables persistent event evolution and long-term scene consistency, bridging the gap between existing 2D observation-based memory and true 4D dynamic world simulation. The baseline and benchmark will be publicly available at https://zichengduan.github.io/LiveWorld/index.html.

Abstract

Retrieval-Augmented Video Generation

Explore similar work

LiveWorld: Simulating Out-of-Sight Dynamics in Generative Video World Models

Compression and Retrieval: Implicit Memory Retrieval for Video World Models

WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models