Demiurge: Forging Navigable 4D Worlds via Autoregressive Video Rerendering

Hanxin Zhu; Jianqi Ma; Cong Wang; Zhihao Hu; Junyao Gao; Tianyu He; Junyan Lin; Jiayin Xiao

cs.CVOct 10, 2026

Demiurge: Forging Navigable 4D Worlds via Autoregressive Video Rerendering

Authors: Hanxin Zhu, Jianqi Ma, Cong Wang, Zhihao Hu, Junyao Gao, Tianyu He, Junyan Lin, Jiayin Xiao,

Organizations: USTC · Tencent · CASIA · Tongji University ·

Abstract

Recent advances in video diffusion enable novel-view video synthesis from a single input video, supporting world modeling, embodied intelligence, and robotic simulation. However, existing methods process camera trajectories independently, lacking the cross-trajectory consistency needed for coherent, navigable 4D worlds. We introduce Demiurge, an autoregressive video rerendering pipeline built on a dynamic 4D point-cloud memory cache. It promotes cross-trajectory consistency through four components: (1) an appearance video diffusion model that synthesizes photorealistic novel-view videos conditioned on the 4D memory; (2) a memory-consistent video depth completion model that aligns known-region depths with cached geometry and completes missing regions, mitigating scale ambiguity and discontinuities during memory updates; (3) memory-aware dual-stream warping that provides high-fidelity visual guidance and occlusion cues for novel-view synthesis; and (4) geometry-aware video refinement that enhances perceptual quality. Extensive experiments demonstrate state-of-the-art performance and three key capabilities: coherent spatiotemporal lattice construction over dense multi-view trajectories; large-parallax video rerendering under challenging camera motions, including 180-degree rotations with no input-view overlap; and geometry-faithful outputs that can be directly optimized into high-quality 3D Gaussian Splatting representations with minimal artifacts. Project page: https://hanxinzhu-lab.github.io/demiurge_project_page/.

Figures & tables

Figure 1: Comparison of our proposed method with existing methods. (a) Existing methods independently synthesize videos along different target trajectories, resulting in cross-trajectory inconsistencies and only isolated 2D cross-view videos. (b) In contrast, Demiurge autoregressively generates each trajectory with a shared 4D memory, maintaining cross-trajectory consistency and thereby enabling coherent 4D world construction.

Figure 2: Overview of Demiurge. Demiurge performs autoregressive video rerendering with an evolving 4D point-cloud memory through a generate–recover–update cycle . By integrating dual-stream warping, appearance generation, depth completion, and optional refinement, it enables coherent 4D world construction, large-parallax rerendering, and geometry-faithful 3D/4D reconstruction.

Method	FVD $\downarrow$	Q-Align-Video $\uparrow$		Q-Align-Image $\uparrow$		P-MSE $\downarrow$	R-Err $\downarrow$	T-Err $\downarrow$	View Sync $\uparrow$
Method	FVD $\downarrow$	Quality	Aesthetics	Quality	Aesthetics	P-MSE $\downarrow$	R-Err $\downarrow$	T-Err $\downarrow$	View Sync $\uparrow$
TrajCrafter Yu et al. (2025b)	597.652	4.262	3.242	3.950	3.035	0.007	4.106	2.403	21.426
InspatioWorld Team et al. (2026)	369.577	4.413	3.366	4.196	3.135	0.017	7.119	2.159	40.607
ReCamMaster Bai et al. (2025)	478.897	4.189	3.162	3.956	2.966	0.046	4.248	2.934	39.624
Gen3C Ren et al. (2025)	363.400	4.277	3.266	4.060	3.074	0.008	6.645	2.647	46.562
Voyager Huang et al. (2025)	839.415	4.081	3.141	3.774	2.950	0.031	10.803	3.715	22.106

Table 1: Quantitative comparison of generated novel-view videos. Demiurge achieves the best perceptual quality, cross-trajectory consistency, and camera controllability. Red, orange, and yellow denote the best, second-best, and third-best results, respectively.

Method	FVD $\downarrow$	Q-Align-Video $\uparrow$		Q-Align-Image $\uparrow$		PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Average $\downarrow$
Method	FVD $\downarrow$	Quality	Aesthetics	Quality	Aesthetics	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Average $\downarrow$
TrajCrafter Yu et al. (2025b)	1258.756	3.430	2.703	2.837	2.481	20.153	0.671	0.368	0.127
InspatioWorld Team et al. (2026)	1141.013	3.590	2.747	3.098	2.5213	21.767	0.706	0.296	0.102
ReCamMaster Bai et al. (2025)	1318.239	3.343	2.547	2.810	2.322	22.212	0.752	0.327	0.099
Gen3C Ren et al. (2025)	1143.709	3.390	2.623	2.865	2.399	22.161	0.727	0.282	0.096
Voyager Huang et al. (2025)	1677.240	2.970	2.385	2.263	2.174	16.116	0.568	0.552	0.207

Table 2: Quantitative comparison of 3DGS reconstructed from generated videos. Average is the geometric mean of

\mathrm{MSE}=10^{-\mathrm{PSNR}/10}

\sqrt{1-\mathrm{SSIM}}

, and LPIPS. Red, orange, and yellow denote the best, second-best, and third-best results, respectively.

Figure 3: Qualitative comparison. Demiurge better preserves appearance and geometry across camera trajectories, reducing hallucinations, content drift, and structural distortions.

Variant	Components				Generated Videos				Reconstructed 3DGS
Variant	NM	Depth Module	DSW	Ref.	FVD $\downarrow$	Q-Align-V $\uparrow$	P-MSE $\downarrow$	View Sync $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Avg. $\downarrow$
w/o Noise Monitor	–	VDC.	✓	✓	339.238	4.460	0.0013	62.575	24.217	0.757	0.239	0.076
w/ Monocular Depth	✓	M. Est.	✓	✓	388.597	4.447	0.0020	54.226	24.132	0.749	0.255	0.079
w/o Dual-stream Warping	✓	VDC.	–	✓	383.525	4.412	0.0049	51.498	23.886	0.744	0.254	0.081
w/o Refinement	✓	VDC.	✓	–	347.922	4.521	0.0042	60.680	24.325	0.756	0.239	0.076
Full model	✓	VDC.	✓	✓	334.518	4.549	0.0010	64.893	24.360	0.769	0.237	0.075

Table 3: Component ablation of Demiurge. NM, VDC, DSW, and Ref. denote noise monitoring, video depth completion, dual-stream warping, and geometry-aware refinement, respectively. M. Est. directly uses monocular video depth estimation.

Figure 4: Qualitative component ablation. NM suppresses projection artifacts, VDC improves geometric consistency, dual-stream warping preserves both appearance and occlusion geometry, and refinement removes residual visual artifacts.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Coherent spatiotemporal lattice generation (Case 1). Rows correspond to different camera trajectories and columns to temporally aligned frames; each sample shows the generated RGB view and its recovered depth.

Figure 6: Coherent spatiotemporal lattice generation (Case 2). Rows correspond to different camera trajectories and columns to temporally aligned frames; each sample shows the generated RGB view and its recovered depth.

Figure 7: Large-parallax video rerendering. Demiurge progressively rotates the camera by up to $180^{\circ}$ rightward (a) and leftward (b) , while maintaining consistent scene content across eight successive trajectories. Sampled frames in each row are ordered chronologically.

Figure 8: Geometry-faithful 3DGS reconstruction (Case 1). At each time step, we show the reconstructed 3DGS (top) and its multi-view RGB (bottom left) and depth renderings (bottom right).

Figure 9: Geometry-faithful 3DGS reconstruction (Case 2). At each time step, we show the reconstructed 3DGS (top) and its multi-view RGB (bottom left) and depth renderings (bottom right).

Figure 10: Additional comparison with TrajCrafter. Demiurge better preserves object appearance and scene geometry across different camera trajectories, with less content drift and viewpoint-dependent distortion.

Figure 11: Additional comparison with InSpatioWorld. Demiurge maintains more consistent foreground details and background layout across trajectories, whereas InSpatioWorld introduces noticeable content changes.

Figure 12: Additional comparison with ReCamMaster. Demiurge preserves object identity and spatial structure under changing viewpoints, while ReCamMaster exhibits cross-trajectory appearance and geometry inconsistencies.

Figure 13: Additional comparison with GEN3C. Demiurge produces more consistent object appearance, dynamics, and spatial relationships across multiple target trajectories.

Figure 14: Additional comparison with Spatia. Demiurge follows the prescribed camera trajectories while maintaining coherent scene content, whereas Spatia exhibits weaker viewpoint variation and cross-trajectory consistency.

Figure 15: Additional comparison with Voyager. Demiurge better preserves the identity, scale, and geometry of the dynamic foreground object across substantial camera motions.

Figure 16: Additional ablation of the noise monitor. The noise monitor filters unreliable cache projections, reducing boundary artifacts and stabilizing the recovered geometry over time.

Figure 17: Additional ablation of video depth completion. Cache-consistent depth completion produces more coherent object geometry and temporal structure than direct monocular depth estimation.

Figure 18: Additional ablation of geometry-aware refinement. Refinement suppresses residual appearance artifacts and texture flicker while preserving the recovered geometry.

Figure 19: Additional ablation of dual-stream warping. The appearance stream preserves visual details, while the geometry stream provides reliable occlusion reasoning; combining them yields consistent appearance and geometry.

Figure 20: Compatibility with GEN3C (Case 1, RGB). Replacing our appearance generator with GEN3C still enables coherent RGB generation across twelve target trajectories. Frames in each row are arranged chronologically.

Figure 21: Compatibility with GEN3C (Case 1, depth). Our depth completion module recovers temporally coherent geometry from the GEN3C-generated videos for subsequent 4D memory updates.

Figure 22: Compatibility with GEN3C (Case 2, RGB). The integrated framework preserves the dynamic subject and scene appearance across twelve target trajectories.

Figure 23: Compatibility with GEN3C (Case 2, depth). Consistent depth recovery across trajectories demonstrates that the remaining Demiurge pipeline can operate with an existing appearance generator.

Figure 24: Additional visualization (Case 1). Generated RGB frames and recovered depths across ten target camera trajectories. Each row corresponds to one trajectory, with sampled frames arranged chronologically from left to right.

Figure 25: Additional visualization (Case 2). Generated RGB frames and recovered depths across ten target camera trajectories. Each row corresponds to one trajectory, with sampled frames arranged chronologically from left to right.

Figure 26: Additional visualization (Case 3). Generated RGB frames and recovered depths across ten target camera trajectories. Each row corresponds to one trajectory, with sampled frames arranged chronologically from left to right.

Figure 27: Additional visualization (Case 4). Generated RGB frames and recovered depths across ten target camera trajectories. Each row corresponds to one trajectory, with sampled frames arranged chronologically from left to right.

Figure 28: Additional visualization (Case 5). Generated RGB frames and recovered depths across ten target camera trajectories. Each row corresponds to one trajectory, with sampled frames arranged chronologically from left to right.

Figure 29: Additional visualization (Case 6). Generated RGB frames and recovered depths across ten target camera trajectories. Each row corresponds to one trajectory, with sampled frames arranged chronologically from left to right.

Figure 30: Additional visualization (Case 7). Generated RGB frames and recovered depths across ten target camera trajectories. Each row corresponds to one trajectory, with sampled frames arranged chronologically from left to right.

Figure 31: Additional visualization (Case 8). Generated RGB frames and recovered depths across ten target camera trajectories. Each row corresponds to one trajectory, with sampled frames arranged chronologically from left to right.

Figure 32: Additional visualization (Case 9). Generated RGB frames and recovered depths across ten target camera trajectories. Each row corresponds to one trajectory, with sampled frames arranged chronologically from left to right.

Dynamic Novel View Synthesis

Dynamic novel view synthesis creates time-consistent images or videos from unseen camera positions in scenes that change over time.

See all 40 papers

Holo4D: Holistic 4D Reconstruction as Geometric Control for Video Diffusion
Oct 18, 2026Yushi Lan, Zeren Jiang, Koichi Namekata +3Camera-Controlled Video Generation Dynamic Novel View Synthesis
University of Oxford · Nanyang Technological University
Vista4D: Video Reshooting with 4D Point Clouds
Apr 23, 2026Kuan Heng Lin, Zhizheng Liu, Pablo Salamanca +9Camera-Controlled Video Generation 4D Reconstruction
1Eyeline Labs · 3Columbia University · 4UCLA +3
Streaming4D: Accelerate 4D World Models via Block-wise Video Generation and Incremental Reconstruction
Sep 1, 2026Xiaoyan Liu, Jiaxin Liu, Kangrui Li +14D Reconstruction 3D Reconstruction
The Chinese University of Hong Kong · The Hong Kong Polytechnic University · The University of New South Wales +1

Oct 18, 2026J/K move · Enter open · S save

Holo4D: Holistic 4D Reconstruction as Geometric Control for Video Diffusion

Yushi Lan, Zeren Jiang, Koichi Namekata, Xingang Pan+2

University of Oxford · Nanyang Technological University

Camera-Controlled Video Generation Dynamic Novel View Synthesis Trajectory-Guided Video Generation Geometric Consistency in Video Generation

Open paper AskPDF arXiv

Synthesizing a monocular video from a new camera trajectory requires preserving the observed scene geometry and motion while completing regions that become newly visible. Feed-forward 4D reconstructors recover the structure of the observed scene, whereas video diffusion models provide a generative prior for completing unseen content. We introduce Holo4D, a video reshooting framework that uses reconstructed 3D trajectories as a holistic source-view 4D scaffold. We project these trajectories into the requested views and encode their geometry and source appearance to guide a pretrained video diffusion model. Together with the source video and target cameras, the scaffold grounds generation in reconstructed observations while leaving unsupported regions to the generative prior. Experiments demonstrate the value of trajectory-based conditioning for camera control and novel-view synthesis. The framework accommodates different 4D reconstructors without retraining the video generator, allowing it to benefit from future advances in source reconstruction. More details can be found in our project page.

Abstract

Dynamic Novel View Synthesis

Explore similar work

Holo4D: Holistic 4D Reconstruction as Geometric Control for Video Diffusion

Vista4D: Video Reshooting with 4D Point Clouds

Streaming4D: Accelerate 4D World Models via Block-wise Video Generation and Incremental Reconstruction