Generative view synthesis aims to synthesize novel-view videos from reference observations along user-defined camera trajectories. Maintaining global scene consistency throughout this process remains challenging for camera-conditioned interactive video generators. Existing approaches typically select reference frames based solely on camera poses, overlooking whether the content remains semantically relevant or even visible. To address these limitations, we draw inspiration from semantic and episodic memory in human cognition, in which semantic memory captures generalized knowledge across experiences, whereas episodic memory preserves information tied to specific past events and contexts. Motivated by the complementary roles of semantic and episodic memory, we introduce SE-Mem, a Semantic-Episodic Memory framework that organizes historical observations in a structured 3D representation and augments its elements with semantic importance scores. A two-stage memory retrieval strategy is designed to identify the most informative frames to provide contextual guidance. Taking into account both semantic significance and geometric visibility, SE-Mem preserves semantic coherence and fine-grained visual details during novel view synthesis. Extensive evaluations show that our approach improves visual quality and achieves competitive geometric consistency across multiple benchmarks.
Figures & tables
Figure 1: Scene-consistent generative view synthesis with SE-Mem. Given a single reference image and a user-defined camera trajectory, SE-Mem synthesizes a sequence of novel views while preserving previously observed scene content. Its Semantic-Episodic Memory combines structured 3D observations with semantic importance to retrieve relevant historical views, helping maintain scene consistency when the camera revisits earlier viewpoints in both indoor and outdoor environments.
Figure 2: Overview of SE-Mem. Given a sequence of target camera poses {ctgt} , SE-Mem maintains a Semantic-Episodic Memory that combines 3D episodic observations with semantic importance scores. Historical frames are processed to obtain 3D point representations and importance maps, which are integrated into a voxelized memory representation. During synthesis, a two-stage retrieval strategy selects reference frames by considering target-view visibility and voxel importance. The retrieved frames and their corresponding camera conditions are then provided to a camera-conditioned video generator to synthesize novel views, while newly generated observations are incrementally incorporated into the memory.
Figure 3: Visualization of importance maps. Given an input scene (left), our hierarchical importance scoring module (Eq. ( 4 )) produces an importance map (right), where brighter regions indicate higher importance values. This map guides our SE-Mem to prioritize semantically informative scene elements during both memory construction and retrieval.
RealEstate10K
Tanks-and-Temples
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Rdist↓
Tdist↓
PSNR ↑
SSIM ↑
LPIPS ↓
Rdist↓
Tdist↓
MotionCtrl [ 32 ]
12.54
0.353
0.565
0.813
0.742
11.76
0.430
0.656
1.280
0.796
ViewCrafter [ 40 ]
16.57
0.583
0.447
0.474
0.623
12.84
0.461
0.598
1.039
0.820
GEN3C [ 21 ]
18.21
0.664
0.372
0.505
0.368
14.27
0.490
0.541
0.554
0.701
SEVA [ 43 ]
18.09
0.641
0.349
0.467
0.293
14.50
0.522
0.539
0.601
0.605
VMem [ 13 ]
17.98
0.632
0.356
0.592
0.481
14.73
0.514
0.513
0.715
0.472
Table 1: Quantitative comparison for the General Synthesis task on RealEstate10K and Tanks-and-Temples datasets. Best results are highlighted in bold, and second-best results are underlined.
Figure 4: Qualitative results on the Viewpoint Revisiting task. Our method maintains scene consistency when returning to previously observed viewpoints, successfully recovering the original environment with reduced perceptual drift and improved visual fidelity.
RealEstate10K
Tanks-and-Temples
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Rdist↓
Tdist↓
PSNR ↑
SSIM ↑
LPIPS ↓
Rdist↓
Tdist↓
MotionCtrl [ 32 ]
9.78
0.314
0.660
1.407
0.920
8.15
0.265
0.792
2.854
1.826
SEVA [ 43 ]
13.20
0.501
0.459
0.967
0.516
11.09
0.329
0.636
1.973
0.921
GEN3C [ 21 ]
13.09
0.486
0.472
0.871
0.461
11.51
0.325
0.649
1.564
1.387
ViewCrafter [ 40 ]
13.38
0.473
0.434
0.912
0.654
11.35
0.309
0.672
1.746
0.710
VMem [ 13 ]
13.64
0.525
0.418
1.032
0.547
12.10
0.342
0.627
1.529
0.852
Table 2: Quantitative results on the Viewpoint Revisiting task on RealEstate10K and Tanks-and-Temples datasets. Best results are highlighted in bold, and second-best results are underlined.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Rdist↓
Tdist↓
MotionCtrl [ 32 ]
9.52
0.274
0.735
2.090
1.082
ViewCrafter [ 40 ]
12.01
0.353
0.621
2.577
0.760
GEN3C [ 21 ]
12.41
0.340
0.587
2.147
0.585
SEVA [ 43 ]
12.14
0.326
0.609
1.905
0.748
VMem [ 13 ]
12.28
0.359
0.550
1.792
0.632
SE-Mem (Ours)
12.35
0.376
0.522
2.086
0.517
Table 3: Quantitative comparison under the Viewpoint Revisiting setting on the DL3DV dataset. Best results are highlighted in bold, and second-best results are underlined.
Figure 5: Qualitative results on DL3DV under the Viewpoint Revisiting setting. Representative examples show that SE-Mem preserves scene structure and previously observed content when the camera returns to earlier viewpoints across diverse indoor and outdoor scenes.
Figure 6: Qualitative synthesis results on in-the-wild images from internet sources. SE-Mem preserves scene structure across the illustrated camera trajectories without domain-specific fine-tuning.
Backbone
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Rdist↓
Tdist↓
SEVA
Baseline
13.20
0.501
0.459
0.967
0.516
+ SE-Mem
14.22
0.549
0.387
0.853
0.499
ViewCrafter
Baseline
13.38
0.473
0.434
0.912
0.654
+ SE-Mem
13.90
0.490
0.414
0.727
0.536
MVGenMaster
Baseline
11.29
0.342
0.604
1.327
1.108
+ SE-Mem
13.25
0.472
0.479
1.056
0.942
Table 4: Effect of SE-Mem on different camera-conditioned backbones.
Variant
PSNR ↑
SSIM ↑
LPIPS ↓
Rdist↓
Tdist↓
(i) Sliding Window
13.05
0.496
0.472
1.120
0.637
(ii) FOV
13.70
0.516
0.430
0.904
0.488
(iii) Ours (w/o semantic)
14.01
0.531
0.413
0.879
0.502
(iv) Ours (w/ semantic)
14.22
0.549
0.387
0.853
0.499
Table 5: Ablation study of memory retrieval strategy and semantic components on RealEstate10K.
Components
PSNR ↑
SSIM ↑
LPIPS ↓
Rdist↓
Tdist↓
Distinctiveness
13.71
0.526
0.421
0.921
0.561
Distinctiveness + Object
14.15
0.542
0.392
0.824
0.478
Distinctiveness + Object + Local
14.22
0.549
0.387
0.853
0.499
Table 6: Ablation study of different importance scoring components on the viewpoint revisiting task.
N
PSNR ↑
SSIM ↑
LPIPS ↓
Rdist↓
Tdist↓
N=0
13.97
0.534
0.391
0.885
0.508
N=1
14.22
0.549
0.387
0.853
0.499
N=2
13.74
0.518
0.412
0.942
0.524
Table 7: Ablation study on the number of complementary frames N under the Viewpoint Revisiting setting on RealEstate10K. The total context window size is fixed at K=4 .
Method
Peak GPU Memory
Time (s)
SEVA
17.84 GB
175
VMem
18.20 GB
612
Ours
18.15 GB
347
Table 8: Memory and time cost under the viewpoint revisiting setting. All measurements are conducted on a single NVIDIA A100 GPU.
Camera-conditioned video generation uses camera poses and scene geometry to create temporally coherent videos from specified viewpoints and trajectories.
Novel view synthesis from sparse images must reconcile faithful reconstruction of observed regions with plausible completion of unseen content, while maintaining world consistency across viewpoints. Existing geometry-based methods preserve observed scene structure but often struggle to complete unseen regions, whereas video generative models offer rich appearance priors but accumulate inconsistencies during sequential view generation. We propose GeoVerse, a framework that synthesizes world-consistent novel views by performing generation within the geometric latent space of a pretrained 3D foundation model and injecting appearance priors from a video generative model. Specifically, GeoVerse extracts multilevel features from Wan2.2 VACE and injects them into the geometric latent diffusion model via a ControlNet-style adapter, incorporating video-learned appearance priors to enhance structural completion. To enforce cross-view coherence, a global spatial memory continuously aggregates observed and synthesized content, reprojecting target-aligned guidance to anchor subsequent predictions to a shared scene representation. Extensive experiments across diverse datasets demonstrate improved visual quality and geometric consistency, with a 2.23 dB higher PSNR on DL3DV and 32.4% lower ATE on Mip-NeRF360 compared to GLD.
Kerui Ren, Tao Lu, Linning Xu +5
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · The Chinese University of Hong Kong +3
Interactive video generation has significant potential for scene simulation and video creation. However, existing methods often struggle with maintaining scene consistency during long video generation under dynamic camera control due to limited contextual information. To address this challenge, we propose MemCam, a memory-augmented interactive video generation approach that treats previously generated frames as external memory and leverages them as contextual conditioning to achieve controllable camera viewpoints with high scene consistency. To enable longer and more relevant context, we design a context compression module that encodes memory frames into compact representations and employs co-visibility-based selection to dynamically retrieve the most relevant historical frames, thereby reducing computational overhead while enriching contextual information. Experiments on interactive video generation tasks show that MemCam significantly outperforms existing baseline methods as well as open-source state-of-the-art approaches in terms of scene consistency, particularly in long video scenarios with large camera rotations.
Despite remarkable progress in video generation, maintaining long-term scene consistency upon revisiting previously explored areas remains challenging. Existing solutions rely either on explicitly constructing 3D geometry, which suffers from error accumulation and scale ambiguity, or on naive camera Field-of-View (FoV) retrieval, which typically fails under complex occlusions. To overcome these limitations, we propose I3DM, a novel implicit 3D-aware memory mechanism for consistent video scene generation that bypasses explicit 3D reconstruction. At the core of our approach is a 3D-aware memory retrieval strategy, which leverages the intermediate features of a pre-trained Feed-Forward Novel View Synthesis (FF-NVS) model to score view relevance, enabling robust retrieval even in highly occluded scenarios. Furthermore, to fully utilize the retrieved historical frames, we introduce a 3D-aligned memory injection module. This module implicitly warps historical content to the target view and adaptively conditions the generation on reliable warping regions, leading to improved revisit consistency and accurate camera control. Extensive experiments demonstrate that our method outperforms state-of-the-art approaches, achieving superior revisit consistency, generation fidelity, and camera control precision.
Jia Li, Han Yan, Yihang Chen +6
Monash University · Vertex Lab · Shanghai Jiao Tong University