Recent advances in video diffusion enable novel-view video synthesis from a single input video, supporting world modeling, embodied intelligence, and robotic simulation. However, existing methods process camera trajectories independently, lacking the cross-trajectory consistency needed for coherent, navigable 4D worlds. We introduce Demiurge, an autoregressive video rerendering pipeline built on a dynamic 4D point-cloud memory cache. It promotes cross-trajectory consistency through four components: (1) an appearance video diffusion model that synthesizes photorealistic novel-view videos conditioned on the 4D memory; (2) a memory-consistent video depth completion model that aligns known-region depths with cached geometry and completes missing regions, mitigating scale ambiguity and discontinuities during memory updates; (3) memory-aware dual-stream warping that provides high-fidelity visual guidance and occlusion cues for novel-view synthesis; and (4) geometry-aware video refinement that enhances perceptual quality. Extensive experiments demonstrate state-of-the-art performance and three key capabilities: coherent spatiotemporal lattice construction over dense multi-view trajectories; large-parallax video rerendering under challenging camera motions, including 180-degree rotations with no input-view overlap; and geometry-faithful outputs that can be directly optimized into high-quality 3D Gaussian Splatting representations with minimal artifacts. Project page: https://hanxinzhu-lab.github.io/demiurge_project_page/.
Figures & tables
Figure 1: Comparison of our proposed method with existing methods. (a) Existing methods independently synthesize videos along different target trajectories, resulting in cross-trajectory inconsistencies and only isolated 2D cross-view videos. (b) In contrast, Demiurge autoregressively generates each trajectory with a shared 4D memory, maintaining cross-trajectory consistency and thereby enabling coherent 4D world construction.
Figure 2: Overview of Demiurge. Demiurge performs autoregressive video rerendering with an evolving 4D point-cloud memory through a generate–recover–update cycle . By integrating dual-stream warping, appearance generation, depth completion, and optional refinement, it enables coherent 4D world construction, large-parallax rerendering, and geometry-faithful 3D/4D reconstruction.
Method
FVD ↓
Q-Align-Video ↑
Q-Align-Image ↑
P-MSE ↓
R-Err ↓
T-Err ↓
View Sync ↑
Quality
Aesthetics
Quality
Aesthetics
TrajCrafter Yu et al. (2025b)
597.652
4.262
3.242
3.950
3.035
0.007
4.106
2.403
21.426
InspatioWorld Team et al. (2026)
369.577
4.413
3.366
4.196
3.135
0.017
7.119
2.159
40.607
ReCamMaster Bai et al. (2025)
478.897
4.189
3.162
3.956
2.966
0.046
4.248
2.934
39.624
Gen3C Ren et al. (2025)
363.400
4.277
3.266
4.060
3.074
0.008
6.645
2.647
46.562
Voyager Huang et al. (2025)
839.415
4.081
3.141
3.774
2.950
0.031
10.803
3.715
22.106
Table 1: Quantitative comparison of generated novel-view videos. Demiurge achieves the best perceptual quality, cross-trajectory consistency, and camera controllability. Red, orange, and yellow denote the best, second-best, and third-best results, respectively.
Method
FVD ↓
Q-Align-Video ↑
Q-Align-Image ↑
PSNR ↑
SSIM ↑
LPIPS ↓
Average ↓
Quality
Aesthetics
Quality
Aesthetics
TrajCrafter Yu et al. (2025b)
1258.756
3.430
2.703
2.837
2.481
20.153
0.671
0.368
0.127
InspatioWorld Team et al. (2026)
1141.013
3.590
2.747
3.098
2.5213
21.767
0.706
0.296
0.102
ReCamMaster Bai et al. (2025)
1318.239
3.343
2.547
2.810
2.322
22.212
0.752
0.327
0.099
Gen3C Ren et al. (2025)
1143.709
3.390
2.623
2.865
2.399
22.161
0.727
0.282
0.096
Voyager Huang et al. (2025)
1677.240
2.970
2.385
2.263
2.174
16.116
0.568
0.552
0.207
Table 2: Quantitative comparison of 3DGS reconstructed from generated videos. Average is the geometric mean of MSE=10−PSNR/10 , 1−SSIM , and LPIPS. Red, orange, and yellow denote the best, second-best, and third-best results, respectively.
Figure 3: Qualitative comparison. Demiurge better preserves appearance and geometry across camera trajectories, reducing hallucinations, content drift, and structural distortions.
Variant
Components
Generated Videos
Reconstructed 3DGS
NM
Depth Module
DSW
Ref.
FVD ↓
Q-Align-V ↑
P-MSE ↓
View Sync ↑
PSNR ↑
SSIM ↑
LPIPS ↓
Avg. ↓
w/o Noise Monitor
–
VDC.
✓
✓
339.238
4.460
0.0013
62.575
24.217
0.757
0.239
0.076
w/ Monocular Depth
✓
M. Est.
✓
✓
388.597
4.447
0.0020
54.226
24.132
0.749
0.255
0.079
w/o Dual-stream Warping
✓
VDC.
–
✓
383.525
4.412
0.0049
51.498
23.886
0.744
0.254
0.081
w/o Refinement
✓
VDC.
✓
–
347.922
4.521
0.0042
60.680
24.325
0.756
0.239
0.076
Full model
✓
VDC.
✓
✓
334.518
4.549
0.0010
64.893
24.360
0.769
0.237
0.075
Table 3: Component ablation of Demiurge. NM, VDC, DSW, and Ref. denote noise monitoring, video depth completion, dual-stream warping, and geometry-aware refinement, respectively. M. Est. directly uses monocular video depth estimation.
Figure 4: Qualitative component ablation. NM suppresses projection artifacts, VDC improves geometric consistency, dual-stream warping preserves both appearance and occlusion geometry, and refinement removes residual visual artifacts.
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Coherent spatiotemporal lattice generation (Case 1). Rows correspond to different camera trajectories and columns to temporally aligned frames; each sample shows the generated RGB view and its recovered depth.
Figure 6: Coherent spatiotemporal lattice generation (Case 2). Rows correspond to different camera trajectories and columns to temporally aligned frames; each sample shows the generated RGB view and its recovered depth.
Figure 7: Large-parallax video rerendering. Demiurge progressively rotates the camera by up to 180∘ rightward (a) and leftward (b) , while maintaining consistent scene content across eight successive trajectories. Sampled frames in each row are ordered chronologically.
Figure 8: Geometry-faithful 3DGS reconstruction (Case 1). At each time step, we show the reconstructed 3DGS (top) and its multi-view RGB (bottom left) and depth renderings (bottom right).
Figure 9: Geometry-faithful 3DGS reconstruction (Case 2). At each time step, we show the reconstructed 3DGS (top) and its multi-view RGB (bottom left) and depth renderings (bottom right).
Figure 10: Additional comparison with TrajCrafter. Demiurge better preserves object appearance and scene geometry across different camera trajectories, with less content drift and viewpoint-dependent distortion.
Figure 11: Additional comparison with InSpatioWorld. Demiurge maintains more consistent foreground details and background layout across trajectories, whereas InSpatioWorld introduces noticeable content changes.
Figure 12: Additional comparison with ReCamMaster. Demiurge preserves object identity and spatial structure under changing viewpoints, while ReCamMaster exhibits cross-trajectory appearance and geometry inconsistencies.
Figure 13: Additional comparison with GEN3C. Demiurge produces more consistent object appearance, dynamics, and spatial relationships across multiple target trajectories.
Figure 14: Additional comparison with Spatia. Demiurge follows the prescribed camera trajectories while maintaining coherent scene content, whereas Spatia exhibits weaker viewpoint variation and cross-trajectory consistency.
Figure 15: Additional comparison with Voyager. Demiurge better preserves the identity, scale, and geometry of the dynamic foreground object across substantial camera motions.
Figure 16: Additional ablation of the noise monitor. The noise monitor filters unreliable cache projections, reducing boundary artifacts and stabilizing the recovered geometry over time.
Figure 17: Additional ablation of video depth completion. Cache-consistent depth completion produces more coherent object geometry and temporal structure than direct monocular depth estimation.
Figure 18: Additional ablation of geometry-aware refinement. Refinement suppresses residual appearance artifacts and texture flicker while preserving the recovered geometry.
Figure 19: Additional ablation of dual-stream warping. The appearance stream preserves visual details, while the geometry stream provides reliable occlusion reasoning; combining them yields consistent appearance and geometry.
Figure 20: Compatibility with GEN3C (Case 1, RGB). Replacing our appearance generator with GEN3C still enables coherent RGB generation across twelve target trajectories. Frames in each row are arranged chronologically.
Figure 21: Compatibility with GEN3C (Case 1, depth). Our depth completion module recovers temporally coherent geometry from the GEN3C-generated videos for subsequent 4D memory updates.
Figure 22: Compatibility with GEN3C (Case 2, RGB). The integrated framework preserves the dynamic subject and scene appearance across twelve target trajectories.
Figure 23: Compatibility with GEN3C (Case 2, depth). Consistent depth recovery across trajectories demonstrates that the remaining Demiurge pipeline can operate with an existing appearance generator.
Figure 24: Additional visualization (Case 1). Generated RGB frames and recovered depths across ten target camera trajectories. Each row corresponds to one trajectory, with sampled frames arranged chronologically from left to right.
Figure 25: Additional visualization (Case 2). Generated RGB frames and recovered depths across ten target camera trajectories. Each row corresponds to one trajectory, with sampled frames arranged chronologically from left to right.
Figure 26: Additional visualization (Case 3). Generated RGB frames and recovered depths across ten target camera trajectories. Each row corresponds to one trajectory, with sampled frames arranged chronologically from left to right.
Figure 27: Additional visualization (Case 4). Generated RGB frames and recovered depths across ten target camera trajectories. Each row corresponds to one trajectory, with sampled frames arranged chronologically from left to right.
Figure 28: Additional visualization (Case 5). Generated RGB frames and recovered depths across ten target camera trajectories. Each row corresponds to one trajectory, with sampled frames arranged chronologically from left to right.
Figure 29: Additional visualization (Case 6). Generated RGB frames and recovered depths across ten target camera trajectories. Each row corresponds to one trajectory, with sampled frames arranged chronologically from left to right.
Figure 30: Additional visualization (Case 7). Generated RGB frames and recovered depths across ten target camera trajectories. Each row corresponds to one trajectory, with sampled frames arranged chronologically from left to right.
Figure 31: Additional visualization (Case 8). Generated RGB frames and recovered depths across ten target camera trajectories. Each row corresponds to one trajectory, with sampled frames arranged chronologically from left to right.
Figure 32: Additional visualization (Case 9). Generated RGB frames and recovered depths across ten target camera trajectories. Each row corresponds to one trajectory, with sampled frames arranged chronologically from left to right.
Synthesizing a monocular video from a new camera trajectory requires preserving the observed scene geometry and motion while completing regions that become newly visible. Feed-forward 4D reconstructors recover the structure of the observed scene, whereas video diffusion models provide a generative prior for completing unseen content. We introduce Holo4D, a video reshooting framework that uses reconstructed 3D trajectories as a holistic source-view 4D scaffold. We project these trajectories into the requested views and encode their geometry and source appearance to guide a pretrained video diffusion model. Together with the source video and target cameras, the scaffold grounds generation in reconstructed observations while leaving unsupported regions to the generative prior. Experiments demonstrate the value of trajectory-based conditioning for camera control and novel-view synthesis. The framework accommodates different 4D reconstructors without retraining the video generator, allowing it to benefit from future advances in source reconstruction. More details can be found in our project page.
Yushi Lan, Zeren Jiang, Koichi Namekata +3
University of Oxford · Nanyang Technological University
We present Vista4D, a robust and flexible video reshooting framework that grounds the input video and target cameras in a 4D point cloud. Specifically, given an input video, our method re-synthesizes the scene with the same dynamics from a different camera trajectory and viewpoint. Existing video reshooting methods often struggle with depth estimation artifacts of real-world dynamic videos, while also failing to preserve content appearance and failing to maintain precise camera control for challenging new trajectories. We build a 4D-grounded point cloud representation with static pixel segmentation and 4D reconstruction to explicitly preserve seen content and provide rich camera signals, and we train with reconstructed multiview dynamic data for robustness against point cloud artifacts during real-world inference. Our results demonstrate improved 4D consistency, camera control, and visual quality compared to state-of-the-art baselines under a variety of videos and camera paths. Moreover, our method generalizes to real-world applications such as dynamic scene expansion and 4D scene recomposition. See our project page for results, code, and models: https://eyeline-labs.github.io/Vista4D
Current 4D generation paradigms are often bottlenecked by a sequential decoupling design: video is generated first, followed by 3D reconstruction, leading to high interaction latency. This limits applications in interactive real-time scenarios. To this end, we propose Streaming4D, a tightly coupled synchronous pipeline that integrates block-wise autoregressive video generation with incremental 3D reconstruction. Unlike traditional frame-by-frame emission and delayed geometry recovery, Streaming4D generates temporal video blocks and immediately triggers reconstruction for each completed block, enabling parallel execution between synthesis and geometric updates. This approach allows the world representation to evolve online with the video stream, reducing feedback latency while preserving geometric fidelity. We instantiate Streaming4D using a Self-Forcing-style autoregressive generator and an incremental reconstruction backend. Experiments show consistent runtime improvements across resolutions on a single RTX 4090 (1.24× speedup), while maintaining high-quality 4D geometry and multi-view consistency.
Xiaoyan Liu, Jiaxin Liu, Kangrui Li +1
The Chinese University of Hong Kong · The Hong Kong Polytechnic University · The University of New South Wales +1