Despite remarkable progress in video generation, maintaining long-term scene consistency upon revisiting previously explored areas remains challenging. Existing solutions rely either on explicitly constructing 3D geometry, which suffers from error accumulation and scale ambiguity, or on naive camera Field-of-View (FoV) retrieval, which typically fails under complex occlusions. To overcome these limitations, we propose I3DM, a novel implicit 3D-aware memory mechanism for consistent video scene generation that bypasses explicit 3D reconstruction. At the core of our approach is a 3D-aware memory retrieval strategy, which leverages the intermediate features of a pre-trained Feed-Forward Novel View Synthesis (FF-NVS) model to score view relevance, enabling robust retrieval even in highly occluded scenarios. Furthermore, to fully utilize the retrieved historical frames, we introduce a 3D-aligned memory injection module. This module implicitly warps historical content to the target view and adaptively conditions the generation on reliable warping regions, leading to improved revisit consistency and accurate camera control. Extensive experiments demonstrate that our method outperforms state-of-the-art approaches, achieving superior revisit consistency, generation fidelity, and camera control precision.
Figures & tables
Figure 2: Limitations of existing memory mechanisms. Top: Explicit geometry-based methods (e.g., Gen3C ( Ren et al. 2025 ) ) suffer from scale ambiguity, causing inaccurate camera navigation (e.g., colliding with the wall) and revisit inconsistencies. Bottom: Implicit FoV-based methods (e.g., WorldPlay ( Sun et al. 2025 ) ) fail under occlusions, as FoV overlap ignores actual visibility. Retrieving irrelevant historical frames causes repeated content and inconsistent revisits. Frames with matching colors denote the same viewpoint and should be strictly consistent. Red circles and boxes indicate inconsistencies and repeated content.
Figure 3: Overview of the proposed I3DM framework. Left: 3D-aware Memory Retrieval. For each historical frame in the memory bank, we first extract 3D-aware intermediate features using a pre-trained NVS model. A lightweight scoring network then evaluates their spatial relevance to the target view to select the most relevant frames. Right: 3D-aligned Memory Injection. The retrieved frames are processed by an Adaptive NVS Module to align them with the target view. These aligned results are then used to condition the Wan-DiT backbone for consistent video scene generation.
Figure 4: 3D-aware Memory Retrieval Module. We first set the last frame as an anchor. For each historical candidate frame, we extract implicit 3D-aware features from the lth layer of the frozen LVSM. Based on these features, a Scoring CNN then predicts a spatial uncertainty map for the target view. Notably, the full Transformer is executed only during training to provide supervision; during inference, the process terminates at the lth layer to efficiently deduce the score map.
Method
Visual Quality
Camera Control
Revisit Consistency
FID ↓
FVD ↓
Rerr∘↓
Terr↓
PSNR ↑
SSIM ↑
LPIPS ↓
Re10K Dataset
ViewCrafter
58.63
–
71.116
0.7867
9.29
0.270
0.6797
Voyager
31.90
377.86
17.801
0.3471
14.75
0.499
0.4708
Gen3C
29.05
292.56
8.183
0.1450
15.23
0.553
0.4397
WorldWarp
21.90
181.95
13.003
0.2161
14.77
0.592
0.4828
Table 1: Comparisons on Re10K (top) and T&T (bottom).
Method
FID ↓
FVD ↓
Rerr∘↓
Terr↓
PSNR ↑
SSIM ↑
LPIPS ↓
Temporal
23.67
189.39
5.617
0.1277
13.78
0.522
0.4660
Random
18.50
141.88
2.075
0.0566
19.25
0.687
0.2027
FoV-based
17.63
134.69
1.895
0.0510
22.72
0.781
0.1084
Geo-based
17.79
133.66
2.027
0.0511
20.94
0.738
0.1346
I3D-TopK
17.59
134.13
2.046
0.0498
22.33
0.769
0.1138
I3D-Ours
17.55
131.66
1.991
0.0505
24.73
0.828
0.0756
Table 2: Ablation studies of memory retrievals (top) and memory injection mechanisms (bottom) on Re10K Dataset.
Figure 5: Qualitative comparison on the Re10K (left) and T&T (right) datasets. The frames with matching colors should remain strictly consistent; red circles and arrows highlight visual inconsistencies and inaccurate camera motion, respectively.
Figure 6: Ablation of memory retrieval strategies. Frames with the same color should remain strictly consistent. Red circles highlight visual inconsistencies. Temporal and random strategies fail to maintain scene consistency. Geometry-, FoV- and I3D-TopK-based approaches improve revisit consistency but still exhibit artifacts, while our full I3D-based retrieval strategy robustly maintains strict consistency.
Figure 7: Ablation of memory injection mechanisms. Removing memory or spatial alignment compromises revisit consistency, while freezing the NVS module causes inaccurate camera motion and navigation failures (e.g., failing to enter the room). Jointly fine-tuning the NVS module ensures both consistent generation and accurate camera control.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Frames
Bank size
Retrieval time
Total time
Ratio
77
4
0.00
1.24
0.0%
229
44
0.04
1.32
3.1%
457
104
0.10
1.36
7.3%
609
144
0.14
1.39
9.9%
837
204
0.19
1.47
13.2%
989
244
0.23
1.50
15.5%
Appendix
Table 1: Per-frame runtime (sec) as generated video length increases. Bank size denotes the number of frames in the memory bank for retrieval before generating the current clip.
Figure 1: Our method alleviates temporal drifting by memory mechanism.
Figure 2: Visualization of confidence maps and greedy maximum-coverage selection. Given the anchor frame and target Plücker rays, our network predicts a spatial confidence map for each candidate (middle row). The selected candidates provide complementary marginal gains that collectively cover the target view (bottom row).
Figure 3: Visual comparison of the frozen and adaptive NVS modules. The purple dashed boxes indicate warping errors introduced by the frozen NVS module, which causes inaccurate camera motion in the generated results (highlighted by red dashed boxes). The adaptive NVS module suppresses these unreliable warped regions, allowing the model to fall back on its internal generative priors to produce convincing results with accurate camera motion.
Layer
Visual Quality
Revisit Consistency
Per-clip Time Cost
FID ↓
FVD ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Retrieval ↓
Generation ↓
L0
18.655
148.764
17.183
0.634
0.2764
0.1853
56.7241
L4
18.098
133.384
24.609
0.823
0.0786
1.3481
58.3389
L6
17.553
131.657
24.732
0.828
0.0756
1.9739
59.0871
L12
17.656
134.054
24.771
0.828
0.0761
3.8737
61.4180
L24
17.599
128.981
24.823
0.830
0.0742
7.6167
64.9117
Appendix
Table 2: Ablation studies of different feature layers for memory retrieval on the Re10K dataset. Time costs are averaged per video clip and reported in seconds (s).
Figure 4: Replacing the FoV-based retrieval of Matrix-Game-3.0 ( Wang et al. 2026 ) with our 3D-aware retrieval strategy improves revisit consistency while retaining its original frame-concatenation injection.
Figure 5: Qualitative comparison of different memory retrieval strategies. (Top) The camera trajectory illustrates the memory bank initialization (blue) and the target sequence to be generated (orange), alongside the ground truth (GT) frames. (Bottom) For each strategy, the left panel shows the four retrieved historical frames, while the right panel displays the corresponding target views synthesized by the pre-trained NVS model. Red dashed circles highlight artifacts caused by suboptimal retrieval.
Figure 6: Qualitative comparison on an outdoor RealEstate10K scene. Black dashed arrows link corresponding frames that should remain consistent; red dashed circles highlight visual inconsistencies, red dashed arrows indicate inaccurate camera motion, and red dashed boxes denote repetitive generated content.
Figure 7: Qualitative comparison on an indoor RealEstate10K scene. Black dashed arrows link corresponding frames that should remain consistent; red dashed circles highlight visual inconsistencies.
Figure 8: Qualitative comparison on the out-of-distribution scenes. Red dashed boxes denote the generated artifacts.
Figure 9: Our visual results on the RealEstate10K dataset. Black dashed arrows link corresponding frames that should remain consistent.
Generating coherent videos grounded in a unified underlying world remains a major challenge, particularly under large viewpoint changes, where existing methods often struggle to maintain stable representations of individual objects. To this end, we introduce Totem, an object-centric 3D memory framework for world-consistent video generation. Totem first represents each selected object as a persistent 3D memory aligned to a shared world frame, assigning it a dedicated state. Based on these memories, we design a dual-stream 3D conditioning architecture that combines object memories with scene geometry for target-view synthesis. Moreover, these object memories constrain the model's attention with 3D structural information, thereby preserving object surfaces and relative spatial layouts. Extensive experiments on world-generation and object-centric benchmarks demonstrate that Totem generates videos depicting a more object-consistent 3D world under challenging camera trajectories, while supporting versatile applications.
Zichen Wan, Tianyu Huang, Fan Li +3
Harbin Institute of Technology · Huawei Technologies Ltd.
Scene-consistent video generation aims to create videos that explore 3D scenes based on a camera trajectory. Previous methods rely on video generation models with external memory for consistency, or iterative 3D reconstruction and inpainting, which accumulate errors during inference due to incorrect intermediary outputs, non-differentiable processes, and separate models. To overcome these limitations, we introduce ``geometry-as-context". It iteratively completes the following steps using an autoregressive camera-controlled video generation model: (1) estimates the geometry of the current view necessary for 3D reconstruction, and (2) simulates and restores novel view images rendered by the 3D scene. Under this multi-task framework, we develop the camera gated attention module to enhance the model's capability to effectively leverage camera poses. During the training phase, text contexts are utilized to ascertain whether geometric or RGB images should be generated. To ensure that the model can generate RGB-only outputs during inference, the geometry context is randomly dropped from the interleaved text-image-geometry training sequence. The method has been tested on scene video generation with one-direction and forth-and-back trajectories. The results show its superiority over previous approaches in maintaining scene consistency and camera control.
JiaKui Hu, Jialun Liu, Liying Yang +7
Institute of Medical Technology, Peking University · TeleAI · MUST
Interactive video generation has significant potential for scene simulation and video creation. However, existing methods often struggle with maintaining scene consistency during long video generation under dynamic camera control due to limited contextual information. To address this challenge, we propose MemCam, a memory-augmented interactive video generation approach that treats previously generated frames as external memory and leverages them as contextual conditioning to achieve controllable camera viewpoints with high scene consistency. To enable longer and more relevant context, we design a context compression module that encodes memory frames into compact representations and employs co-visibility-based selection to dynamically retrieve the most relevant historical frames, thereby reducing computational overhead while enriching contextual information. Experiments on interactive video generation tasks show that MemCam significantly outperforms existing baseline methods as well as open-source state-of-the-art approaches in terms of scene consistency, particularly in long video scenarios with large camera rotations.