Interactive video generation has significant potential for scene simulation and video creation. However, existing methods often struggle with maintaining scene consistency during long video generation under dynamic camera control due to limited contextual information. To address this challenge, we propose MemCam, a memory-augmented interactive video generation approach that treats previously generated frames as external memory and leverages them as contextual conditioning to achieve controllable camera viewpoints with high scene consistency. To enable longer and more relevant context, we design a context compression module that encodes memory frames into compact representations and employs co-visibility-based selection to dynamically retrieve the most relevant historical frames, thereby reducing computational overhead while enriching contextual information. Experiments on interactive video generation tasks show that MemCam significantly outperforms existing baseline methods as well as open-source state-of-the-art approaches in terms of scene consistency, particularly in long video scenarios with large camera rotations.
Figures & tables
Fig. 1: Methodology. (Left) Overview of MemCam: the Context Compressor encodes historical frames selected via co-visibility into compact representations, which are concatenated with the noisy prediction sequence and fed into the DiT Block. (Right) Illustration of co-visibility computation between predicted and historical camera FOVs.
Fig. 2: Qualitative Comparison Results. (a) and (b) are evaluated on the Context-as-Memory dataset, and (c) and (d) on RealEstate10K. MemCam achieves superior performance in scene memory retention and overall generation quality. In contrast, other methods exhibit varying degrees of scene inconsistency due to insufficient utilization of contextual information.
Context-as-Memory Dataset
Methods
90° Round-trip
360° Round-trip
PSNR ↑
SSIM ↑
LPIPS ↓
FVD ↓
PSNR ↑
SSIM ↑
LPIPS ↓
FVD ↓
I2V
15.81
0.452
0.470
528.51
9.75
0.332
0.603
988.82
DFoT
16.76
0.474
0.393
683.59
8.94
0.252
0.613
1188.34
GF
16.57
0.486
0.348
557.66
10.07
0.402
0.565
852.05
MemCam (Ours)
17.83
0.506
0.357
215.71
14.81
0.423
0.504
167.87
TABLE I: Quantitative Comparison Results. We evaluate on two datasets with 90° and 360° round-trip benchmarks. ↑ means higher is better, ↓ means lower is better. Best results are in bold , second best are underlined .
90° Round-trip
Methods
PSNR ↑
SSIM ↑
LPIPS ↓
FVD ↓
Recent
17.80
0.498
0.375
231.75
Random
17.36
0.502
0.400
237.87
TopK
17.39
0.497
0.394
256.88
Ours
17.83
0.506
0.357
215.71
360° Round-trip
TABLE II: Ablation on Context Selection Strategy.
Methods
PSNR ↑
SSIM ↑
LPIPS ↓
FVD ↓
s/frame ↓
None-19
14.69
0.417
0.513
184.42
4.45
None-38
14.76
0.424
0.494
179.29
8.82
None-76
14.88
0.430
0.489
164.37
22.15
Ours-19
13.88
0.398
0.534
203.30
2.14
Ours-38
14.61
0.407
0.527
182.99
2.81
Ours-76
14.81
0.423
0.504
167.87
4.47
TABLE III: Ablation on Context Compression Module.
Spatially consistent long-horizon video generation aims to maintain temporal and spatial consistency along predefined camera trajectories. Existing methods mostly entangle memory modeling with video generation, leading to inconsistent content during scene revisits and diminished generative capacity when exploring novel regions, even trained on extensive annotated data. To address these limitations, we propose a decoupled framework that separates memory conditioning from generation. Our approach significantly reduces training costs while simultaneously enhancing spatial consistency and preserving the generative capacity for novel scene exploration. Specifically, we employ a lightweight, independent memory branch to learn precise spatial consistency from historical observation. We first introduce a hybrid memory representation to capture complementary temporal and spatial cues from generated frames, then leverage a per-frame cross-attention mechanism to ensure each frame is conditioned exclusively on the most spatially relevant historical information, which is injected into the generative model to ensure spatial consistency. When generating new scenes, a camera-aware gating mechanism is proposed to mediate the interaction between memory and generation modules, enabling memory conditioning only when meaningful historical references exist. Compared with the existing method, our method is highly data-efficient, yet the experiments demonstrate that our approach achieves state-of-the-art performance in terms of both visual quality and spatial consistency.
Yanjun Guo, Zhengqiang Zhang, Pengfei Wang +3
The Hong Kong Polytechnic University · OPPO Research Institute
Despite remarkable progress in video generation, maintaining long-term scene consistency upon revisiting previously explored areas remains challenging. Existing solutions rely either on explicitly constructing 3D geometry, which suffers from error accumulation and scale ambiguity, or on naive camera Field-of-View (FoV) retrieval, which typically fails under complex occlusions. To overcome these limitations, we propose I3DM, a novel implicit 3D-aware memory mechanism for consistent video scene generation that bypasses explicit 3D reconstruction. At the core of our approach is a 3D-aware memory retrieval strategy, which leverages the intermediate features of a pre-trained Feed-Forward Novel View Synthesis (FF-NVS) model to score view relevance, enabling robust retrieval even in highly occluded scenarios. Furthermore, to fully utilize the retrieved historical frames, we introduce a 3D-aligned memory injection module. This module implicitly warps historical content to the target view and adaptively conditions the generation on reliable warping regions, leading to improved revisit consistency and accurate camera control. Extensive experiments demonstrate that our method outperforms state-of-the-art approaches, achieving superior revisit consistency, generation fidelity, and camera control precision.
Jia Li, Han Yan, Yihang Chen +6
Monash University · Vertex Lab · Shanghai Jiao Tong University
Consistent video generation under editing operations requires persistence: when edits modify scene appearance or layout, subsequent generations should remain coherent across time and viewpoints. However, existing memory designs struggle to maintain long-term consistency after such modifications, as stored contexts may become outdated or invalid. To address this, we propose PermaVid, a novel framework built upon a multi-modal context memory that disentangles spatial context into semantic appearance and geometric structure, together with an edit-aware memory update and retrieval strategy that keeps memory evolution aligned with subsequent observations. Specifically, we develop two complementary memory banks: an RGB context memory that captures appearance-aware observations while implicitly encoding geometry, and a depth context memory that preserves geometry-only structure disentangled from semantics. Building on this design, we introduce a memory-guided video generation model that performs multi-modal feature fusion under reference conditions drawn from mixed-modality memory contexts. Experiments demonstrate that our method maintains strong long-term semantic and structural consistency after edits, significantly outperforming state-of-the-art methods.
Shuai Yang, Bingjie Gao, Ziwei Liu +3
Shanghai Jiao Tong University · S-Lab, Nanyang Technological University · Shanghai Innovation Institute +2