SE-Mem: Semantic-Episodic Memory for Scene-Consistent Generative View Synthesis

Changsong Lei; Yihan Li; Yu-Hui Wen; Huankun Sheng; Ming Li; Yeying Fan; Minnan Luo; Yong-Jin Liu

cs.CVOct 31, 2026

SE-Mem: Semantic-Episodic Memory for Scene-Consistent Generative View Synthesis

Authors: Changsong Lei, Yihan Li, Yu-Hui Wen, Huankun Sheng, Ming Li, Yeying Fan, Minnan Luo, Yong-Jin Liu

Organizations: Tsinghua University · Beijing Jiaotong University · Xi’an Jiaotong University

Abstract

Generative view synthesis aims to synthesize novel-view videos from reference observations along user-defined camera trajectories. Maintaining global scene consistency throughout this process remains challenging for camera-conditioned interactive video generators. Existing approaches typically select reference frames based solely on camera poses, overlooking whether the content remains semantically relevant or even visible. To address these limitations, we draw inspiration from semantic and episodic memory in human cognition, in which semantic memory captures generalized knowledge across experiences, whereas episodic memory preserves information tied to specific past events and contexts. Motivated by the complementary roles of semantic and episodic memory, we introduce SE-Mem, a Semantic-Episodic Memory framework that organizes historical observations in a structured 3D representation and augments its elements with semantic importance scores. A two-stage memory retrieval strategy is designed to identify the most informative frames to provide contextual guidance. Taking into account both semantic significance and geometric visibility, SE-Mem preserves semantic coherence and fine-grained visual details during novel view synthesis. Extensive evaluations show that our approach improves visual quality and achieves competitive geometric consistency across multiple benchmarks.

Figures & tables

Figure 1: Scene-consistent generative view synthesis with SE-Mem. Given a single reference image and a user-defined camera trajectory, SE-Mem synthesizes a sequence of novel views while preserving previously observed scene content. Its Semantic-Episodic Memory combines structured 3D observations with semantic importance to retrieve relevant historical views, helping maintain scene consistency when the camera revisits earlier viewpoints in both indoor and outdoor environments.

Figure 2: Overview of SE-Mem. Given a sequence of target camera poses $\{c_{tgt}\}$ , SE-Mem maintains a Semantic-Episodic Memory that combines 3D episodic observations with semantic importance scores. Historical frames are processed to obtain 3D point representations and importance maps, which are integrated into a voxelized memory representation. During synthesis, a two-stage retrieval strategy selects reference frames by considering target-view visibility and voxel importance. The retrieved frames and their corresponding camera conditions are then provided to a camera-conditioned video generator to synthesize novel views, while newly generated observations are incrementally incorporated into the memory.

Figure 3: Visualization of importance maps. Given an input scene (left), our hierarchical importance scoring module (Eq. ( 4 )) produces an importance map (right), where brighter regions indicate higher importance values. This map guides our SE-Mem to prioritize semantically informative scene elements during both memory construction and retrieval.

	RealEstate10K					Tanks-and-Temples
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	$R_{dist}\downarrow$	$T_{dist}\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	$R_{dist}\downarrow$	$T_{dist}\downarrow$
MotionCtrl [ 32 ]	12.54	0.353	0.565	0.813	0.742	11.76	0.430	0.656	1.280	0.796
ViewCrafter [ 40 ]	16.57	0.583	0.447	0.474	0.623	12.84	0.461	0.598	1.039	0.820
GEN3C [ 21 ]	18.21	0.664	0.372	0.505	0.368	14.27	0.490	0.541	0.554	0.701
SEVA [ 43 ]	18.09	0.641	0.349	0.467	0.293	14.50	0.522	0.539	0.601	0.605
VMem [ 13 ]	17.98	0.632	0.356	0.592	0.481	14.73	0.514	0.513	0.715	0.472

Table 1: Quantitative comparison for the General Synthesis task on RealEstate10K and Tanks-and-Temples datasets. Best results are highlighted in bold, and second-best results are underlined.

Figure 4: Qualitative results on the Viewpoint Revisiting task. Our method maintains scene consistency when returning to previously observed viewpoints, successfully recovering the original environment with reduced perceptual drift and improved visual fidelity.

	RealEstate10K					Tanks-and-Temples
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	$R_{dist}\downarrow$	$T_{dist}\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	$R_{dist}\downarrow$	$T_{dist}\downarrow$
MotionCtrl [ 32 ]	9.78	0.314	0.660	1.407	0.920	8.15	0.265	0.792	2.854	1.826
SEVA [ 43 ]	13.20	0.501	0.459	0.967	0.516	11.09	0.329	0.636	1.973	0.921
GEN3C [ 21 ]	13.09	0.486	0.472	0.871	0.461	11.51	0.325	0.649	1.564	1.387
ViewCrafter [ 40 ]	13.38	0.473	0.434	0.912	0.654	11.35	0.309	0.672	1.746	0.710
VMem [ 13 ]	13.64	0.525	0.418	1.032	0.547	12.10	0.342	0.627	1.529	0.852

Table 2: Quantitative results on the Viewpoint Revisiting task on RealEstate10K and Tanks-and-Temples datasets. Best results are highlighted in bold, and second-best results are underlined.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	$R_{\mathrm{dist}}\downarrow$	$T_{\mathrm{dist}}\downarrow$
MotionCtrl [ 32 ]	9.52	0.274	0.735	2.090	1.082
ViewCrafter [ 40 ]	12.01	0.353	0.621	2.577	0.760
GEN3C [ 21 ]	12.41	0.340	0.587	2.147	0.585
SEVA [ 43 ]	12.14	0.326	0.609	1.905	0.748
VMem [ 13 ]	12.28	0.359	0.550	1.792	0.632
SE-Mem (Ours)	12.35	0.376	0.522	2.086	0.517

Table 3: Quantitative comparison under the Viewpoint Revisiting setting on the DL3DV dataset. Best results are highlighted in bold, and second-best results are underlined.

Figure 5: Qualitative results on DL3DV under the Viewpoint Revisiting setting. Representative examples show that SE-Mem preserves scene structure and previously observed content when the camera returns to earlier viewpoints across diverse indoor and outdoor scenes.

Figure 6: Qualitative synthesis results on in-the-wild images from internet sources. SE-Mem preserves scene structure across the illustrated camera trajectories without domain-specific fine-tuning.

Backbone	Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	$R_{dist}\downarrow$	$T_{dist}\downarrow$
SEVA	Baseline	13.20	0.501	0.459	0.967	0.516
	+ SE-Mem	14.22	0.549	0.387	0.853	0.499
ViewCrafter	Baseline	13.38	0.473	0.434	0.912	0.654
	+ SE-Mem	13.90	0.490	0.414	0.727	0.536
MVGenMaster	Baseline	11.29	0.342	0.604	1.327	1.108
	+ SE-Mem	13.25	0.472	0.479	1.056	0.942

Table 4: Effect of SE-Mem on different camera-conditioned backbones.

Variant	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	$R_{dist}\downarrow$	$T_{dist}\downarrow$
(i) Sliding Window	13.05	0.496	0.472	1.120	0.637
(ii) FOV	13.70	0.516	0.430	0.904	0.488
(iii) Ours (w/o semantic)	14.01	0.531	0.413	0.879	0.502
(iv) Ours (w/ semantic)	14.22	0.549	0.387	0.853	0.499

Table 5: Ablation study of memory retrieval strategy and semantic components on RealEstate10K.

Components	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	$R_{dist}\downarrow$	$T_{dist}\downarrow$
Distinctiveness	13.71	0.526	0.421	0.921	0.561
Distinctiveness + Object	14.15	0.542	0.392	0.824	0.478
Distinctiveness + Object + Local	14.22	0.549	0.387	0.853	0.499

Table 6: Ablation study of different importance scoring components on the viewpoint revisiting task.

$N$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	$R_{\mathrm{dist}}\downarrow$	$T_{\mathrm{dist}}\downarrow$
$N=0$	13.97	0.534	0.391	0.885	0.508
$N=1$	14.22	0.549	0.387	0.853	0.499
$N=2$	13.74	0.518	0.412	0.942	0.524

Table 7: Ablation study on the number of complementary frames

N

under the Viewpoint Revisiting setting on RealEstate10K. The total context window size is fixed at

K=4

Method	Peak GPU Memory	Time (s)
SEVA	17.84 GB	175
VMem	18.20 GB	612
Ours	18.15 GB	347

Table 8: Memory and time cost under the viewpoint revisiting setting. All measurements are conducted on a single NVIDIA A100 GPU.

Camera-Conditioned Video Generation

Camera-conditioned video generation uses camera poses and scene geometry to create temporally coherent videos from specified viewpoints and trajectories.

See all 38 papers

GeoVerse: World-Consistent Novel View Synthesis in Geometric Latent Space
Sep 28, 2026Kerui Ren, Tao Lu, Linning Xu +5Multi-View Consistency Novel View Synthesis
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · The Chinese University of Hong Kong +3
MemCam: Memory-Augmented Camera Control for Consistent Video Generation
Mar 27, 2026Xinhang Gao, Junlin Guan, Shuhan Luo +3Video Generation Camera-Controlled Video Generation
Guilin University of Electronic Technology, China
I3DM: Implicit 3D-aware Memory Retrieval and Injection for Consistent Video Scene Generation
Mar 24, 2026Jia Li, Han Yan, Yihang Chen +63D-Aware Video Generation Memory-Augmented Video Generation
Monash University · Vertex Lab · Shanghai Jiao Tong University

Sep 28, 2026J/K move · Enter open · S save

GeoVerse: World-Consistent Novel View Synthesis in Geometric Latent Space

Kerui Ren, Tao Lu, Linning Xu, Changjian Jiang+4

Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · The Chinese University of Hong Kong+3

Multi-View Consistency Novel View Synthesis Sparse-View 3D Reconstruction 3D Diffusion Models

Open paper AskPDF arXiv

Novel view synthesis from sparse images must reconcile faithful reconstruction of observed regions with plausible completion of unseen content, while maintaining world consistency across viewpoints. Existing geometry-based methods preserve observed scene structure but often struggle to complete unseen regions, whereas video generative models offer rich appearance priors but accumulate inconsistencies during sequential view generation. We propose GeoVerse, a framework that synthesizes world-consistent novel views by performing generation within the geometric latent space of a pretrained 3D foundation model and injecting appearance priors from a video generative model. Specifically, GeoVerse extracts multilevel features from Wan2.2 VACE and injects them into the geometric latent diffusion model via a ControlNet-style adapter, incorporating video-learned appearance priors to enhance structural completion. To enforce cross-view coherence, a global spatial memory continuously aggregates observed and synthesized content, reprojecting target-aligned guidance to anchor subsequent predictions to a shared scene representation. Extensive experiments across diverse datasets demonstrate improved visual quality and geometric consistency, with a 2.23 dB higher PSNR on DL3DV and 32.4% lower ATE on Mip-NeRF360 compared to GLD.

Abstract

Camera-Conditioned Video Generation

Explore similar work

GeoVerse: World-Consistent Novel View Synthesis in Geometric Latent Space

MemCam: Memory-Augmented Camera Control for Consistent Video Generation

I3DM: Implicit 3D-aware Memory Retrieval and Injection for Consistent Video Scene Generation