Spatial Memory Intelligence: Endowing World Models with Understanding-Driven Long-Term Memory

Ying Yang; Guiyu Zhang; Lianghua Huang; Chang Nie; Chenyang Si; Haofan Wang; Shaoshuai Shi; Li Jiang

cs.CVOct 1, 2026

Spatial Memory Intelligence: Endowing World Models with Understanding-Driven Long-Term Memory

Authors: Ying Yang, Guiyu Zhang, Lianghua Huang, Chang Nie, Chenyang Si, Haofan Wang, Shaoshuai Shi, Li Jiang

Organizations: The Chinese University of Hong Kong, Shenzhen · Alibaba Group · Nanjing University · Lovart AI ·

Abstract

Long-video generation and world models have shown strong potential for interactive entertainment and embodied simulation by predicting future observations conditioned on user actions and historical memory. However, as memory sequences grow longer and their structures become increasingly complex, managing long-range spatial context becomes increasingly challenging, calling for a more intelligent and systematic memory-management strategy. Building on the advancing spatial reasoning capabilities of multimodal large language models (MLLMs) and the broader vision of unified models, we propose Spatial Memory Intelligence (SMI), the first framework to systematically employ an understanding model for spatial-memory management in long-video world models. SMI introduces four coordinated atomic operations: spatial clustering, within-cluster sparsification, action-aware retrieval, and reliability-aware filtering. Extensive experiments across multiple baselines, benchmarks, and world-model backbones demonstrate the effectiveness and generalizability of SMI, achieving comprehensive improvements in memory sparsity, generation stability, and spatial consistency.

Figures & tables

Figure 1: Motivation and benefits of SMI. Understanding-driven spatial-memory management organizes disorganized and redundant histories and retrieves relevant memories, enabling 83.68% memory sparsification with improved spatial consistency and generation stability.

Figure 2: Overview of SMI. SMI uses an MLLM to manage spatial memory $\mathcal{M}_{k}$ through four atomic operations: spatial clustering, within-cluster sparsification, action-aware retrieval, and reliability-aware filtering.

Figure 3: Failure modes of geometry- and semantic-based memory retrieval.

Figure 4: Motivation for spatial clustering: redundancy and spatial disorder in long-term memory.

Method	Background Consistency $\uparrow$	Motion Smoothness $\uparrow$	Aesthetic Quality $\uparrow$	Camera-Constrained Consistency $\uparrow$	Overall Visual Quality $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	Sparsity $\uparrow$	Latency $\downarrow$
	VBench			GPT-5.6-sol evaluation		Reconstruction		Efficiency
Base (FoV) ( Sun et al., 2025 )	0.937983	0.991386	0.445389	51.6053	53.1538	12.478528	0.603425	0.0000%	$\times 1.00$
FramePack ( Zhang et al., 2025a )	0.939390	0.989362	0.432000	43.2800	46.6474	11.718369	0.621979	94.3820%	$\times 1.00$
Deep Forcing ( Yi et al., 2025 )	0.946473	0.982810	0.440462	37.6760	33.6297	11.260350	0.664867	79.2135%	$\mathbf{\times 0.81}$
MoC ( Cai et al., 2025 )	0.937999	0.990528	0.438969	48.9196	51.9717	10.802151	0.735292	0.0000%	$\times 1.15$
VMem ( Li et al., 2025c )	0.939695	0.991162	0.447630	54.4727	56.5060	12.768029	0.582628	0.0000%	$\times 1.47$

Table 1: Comparison of the Base, five memory-management baselines, and SMI on HY1.5. All methods use identical generation settings. Bold indicates the best reported value.

Panel of Figure 5 — Figure 5: Qualitative comparisons on HY1.5: (a) spatial consistency and (b) generation stability. Red boxes highlight corresponding regions in the initial and final frames.

Setting	Spatial Clustering	Within-cluster Sparsification	Reliability-Aware Filtering	Background Consistency $\uparrow$	Motion Smoothness $\uparrow$	Aesthetic Quality $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	Sparsity $\uparrow$	Latency $\downarrow$
	Modules			VBench			Reconstruction		Efficiency
Full	$\checkmark$	$\checkmark$	$\checkmark$	0.942890	0.991999	0.469357	13.309393	0.554519	83.6842%	$\times 1.38$
(a)	$\checkmark$	$\checkmark$		0.941184	0.991616	0.455922	13.232422	0.561628	81.5789%	$\times 1.36$
(b)		$\checkmark$	$\checkmark$	0.939966	0.991568	0.450583	11.050455	0.675472	80.5263%	$\times 1.38$
(c)			$\checkmark$	0.939622	0.991600	0.457186	12.985179	0.567009	35.7368%	$\times 1.32$
(d)				0.937700	0.991267	0.446470	12.812125	0.581249	0.0000%	$\mathbf{\times 1.27}$

Table 3: Component ablation on HY1.5. A checkmark denotes an enabled module. Latency is relative to the Base. Bold indicates the best value.

Variant	Background Consistency $\uparrow$	Motion Smoothness $\uparrow$	Aesthetic Quality $\uparrow$	Camera-Constrained Consistency $\uparrow$	Overall Visual Quality $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	Sparsity $\uparrow$	Latency $\downarrow$
	VBench			GPT-5.6-sol evaluation		Reconstruction		Efficiency
SMI (full)	0.942890	0.991999	0.469357	67.0200	65.3677	13.309393	0.554519	83.6842%	$\mathbf{\times 1.38}$
w/o fine-tuning	0.941856	0.991978	0.451266	42.8380	44.7440	9.396166	0.714904	86.3444%	$\times 1.42$

Table 4: Effect of operation-oriented fine-tuning on HY1.5. Both variants use Qwen3.5-4B. Latency is relative to the Base; bold indicates the better value.

Figure 7: Qualitative ablations on HY1.5. (1) Component ablation: SMI (full) is compared with variants without (a) reliability-aware filtering, (b) spatial clustering, (c) spatial clustering and within-cluster sparsification, and (d) all three components. The red box highlights an inconsistent poolside region upon revisiting. (2) Operation-oriented training: SMI with the fine-tuned Qwen3.5-4B is compared with its counterpart without fine-tuning on two scenes.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: Action-aware retrieval examples. ‘Current’ denotes the current observation, and A–G denote historical candidates. Green borders mark the selected candidates.

Figure 9: Additional action-aware retrieval examples. ‘Current’ denotes the current observation, and A–G denote historical candidates. Green borders mark the selected candidates.

Panel of Figure 10 — Figure 10: Spatial clustering: add to an existing cluster (top) or create a new cluster (bottom).

Method	Adjacent transition	Long-range identity	Cumulative geometry	Overall scene consistency	Camera-gated scene	Structural integrity	Artifact cleanliness	Detail clarity	Overall visual quality
Base (FoV)	63.5462	51.8905	45.5081	53.2088	51.6053	55.9933	48.0538	53.7048	53.1538
FramePack	56.9633	42.8667	35.3834	44.5193	43.2800	49.8634	40.5100	47.8134	46.6474
Deep Forcing	44.2133	39.6633	28.2400	37.9464	37.6760	37.5733	25.1633	36.4701	33.6297
MoC	61.4733	49.6100	42.3133	50.7506	48.9196	54.4600	46.8767	53.3933	51.9717
VMem	66.0933	55.7800	48.6767	56.5863	54.4727	59.2067	51.3800	57.4433	56.5060
MemFlow	56.7033	41.1300	34.8067	43.4457	43.3160	48.9834	39.9500	45.8667	45.6500

Appendix

Table 5: Complete GPT-5.6-sol evaluation on HY1.5. All scores are on a 0–100 scale; higher is better.

Method	Adjacent transition	Long-range identity	Cumulative geometry	Overall scene consistency	Camera-gated scene	Structural integrity	Artifact cleanliness	Detail clarity	Overall visual quality
Base (FoV)	47.8335	31.0978	29.1101	34.8049	34.6524	39.6402	34.8039	37.8779	37.8363
FramePack	51.6066	30.6701	28.8400	35.4547	34.7984	39.7833	34.9867	38.0200	37.9917
Deep Forcing	61.6470	44.8030	43.2600	48.6450	46.8560	50.8130	49.6300	52.1630	50.7300
MoC	59.8633	45.9233	43.0867	48.2164	45.5013	48.9634	47.7200	52.3930	49.2763
VMem	58.9533	41.0967	37.2434	44.6153	43.8800	45.6400	43.5534	47.3770	45.3613
MemFlow	54.3633	36.1333	31.5034	39.5520	38.8487	40.2234	39.1734	42.1770	40.2990

Appendix

Table 6: Complete GPT-5.6-sol evaluation on Wan2.2. All scores are on a 0–100 scale; higher is better.


	Base	VMem	FramePack	SMI (ours)
Scene consistency (%)	11.00	27.00	7.50	52.50
Generation stability (%)	11.33	27.83	2.00	57.83

Appendix

Table 7: Best-choice preference shares (%) on 15 scenes. Bold indicates the highest share in each row.

Operation	HY1.5	Wan2.2
Spatial clustering	12,652	12,538
Within-cluster sparsification	7,156	7,135
Action-aware retrieval	24,623	6,852
Reliability-aware filtering	19,480	4,411
Total	63,911	30,936

Appendix

Table 8: Composition of the operation-oriented training datasets.

Operation	Label	HY1.5	Wan2.2
Spatial clustering	Belongs	9,041 (71.46%)	8,972 (71.56%)
Spatial clustering	Does not belong	3,611 (28.54%)	3,566 (28.44%)
Within-cluster sparsification	Keep	2,045 (28.58%)	2,051 (28.75%)
Within-cluster sparsification	Remove	5,111 (71.42%)	5,084 (71.25%)
Reliability-aware filtering	Filtered ( $\leq 6$ )	4,783 (24.55%)	1,711 (38.79%)
Reliability-aware filtering	Retained ( $>6$ )	14,697 (75.45%)	2,700 (61.21%)

Appendix

Table 9: Label composition of the spatial clustering, within-cluster sparsification, and reliability-aware filtering training data.

Motion	HY1.5	Wan2.2
mostly_static	0.75%	2.01%
move_backward	9.36%	7.41%
move_forward	12.50%	7.68%
move_left	10.20%	7.59%
move_right	9.81%	7.41%
tilt_down	2.70%	4.25%

Appendix

Table 10: Motion composition of the action-aware retrieval training data.

Panel of Figure 13 — Figure 13: Additional qualitative results on HY1.5.

Panel of Figure 14 — Figure 14: More qualitative results on HY1.5. Red boxes mark spatial inconsistencies.

Panel of Figure 15 — Figure 15: More qualitative results on HY1.5. Red boxes mark spatial inconsistencies.

Panel of Figure 16 — Figure 16: More qualitative results on HY1.5. Red boxes mark spatial inconsistencies.

Panel of Figure 17 — Figure 17: Additional qualitative results on Wan2.2.

Panel of Figure 18 — Figure 18: More qualitative results on Wan2.2.

Memory-Augmented Video Generation

Memory-augmented video generation gives video models persistent context to preserve identities, scenes, and events across long sequences.

See all 110 papers

LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video
Jun 4, 2026Shiqiang Lang, Jing Liu, Haoyang He +6Visual Spatial Reasoning Multimodal Large Language Models
Beijing University of Posts and Telecommunications · Zhongguancun Academy · Institute of Automation, Chinese Academy of Sciences +2
Long-horizon Video Generation with Slot Memory Addressed By Region Tokens
Oct 13, 2026Eren Cetin, Julius Erbach, Vukasin Bozic +6Long-Horizon Video Generation Memory-Augmented Video Generation
ETH Zurich · Google
DecMem: Towards Minute-Long Consistent World Generation with Decoupled Memory
May 29, 2026Zhenhao Yang, Xiaoshi Wu, Zhengyao Lv +5Long-Horizon Video Generation Memory-Augmented Video Generation
1The University of Hong Kong · *Work done during an internship at Kling Team, Kuaishou Tech. · 2Kling Team, Kuaishou Technology

Jun 4, 2026J/K move · Enter open · S save

LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video

Shiqiang Lang, Jing Liu, Haoyang He, Peiwen Sun+5

Beijing University of Posts and Telecommunications · Zhongguancun Academy · Institute of Automation, Chinese Academy of Sciences+2

Visual Spatial Reasoning Multimodal Large Language Models Temporal Video Understanding

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have advanced image and video understanding and can increasingly handle longer visual inputs. Long-horizon tasks such as autonomous driving and robotic navigation require more than recognizing the current view, as models must remember and retrieve previously observed spatial layouts, routes, viewpoint changes, and object states. To evaluate this capability, we introduce LongSpace-Bench, a room-tour video benchmark for long-horizon spatial memory, covering scene perception, spatial relations, and spatial memory. In this work, we further propose LongSpace, a memory framework for long-video spatial reasoning. LongSpace models long videos as sequential chunks, incorporates 3D structural cues into early decoder layers, and constructs layer-aware memory for question-guided retrieval. Experiments on multiple spatial reasoning benchmarks show that LongSpace improves long-video spatial understanding, further demonstrating explicit spatial memory as a key capability for long-horizon video MLLMs.

Abstract

Memory-Augmented Video Generation

Explore similar work

LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video

Long-horizon Video Generation with Slot Memory Addressed By Region Tokens

DecMem: Towards Minute-Long Consistent World Generation with Decoupled Memory