Reconstructing 4D scenes from casually captured monocular video is vital for applications in virtual reality (VR) and embodied AI. Recent advances in 4D reconstruction and novel view synthesis have substantially propelled this capability. However, existing reconstruction methods generally cannot recover regions beyond visible camera limits. Consequently, we introduce a new paradigm that achieves 4D scene synthesis by combining visible-region reconstruction from monocular input with invisible-region generation beyond observable camera boundaries. We present Panoramic Alignment for Strategic Exploitation of Generative Priors (PASTEL). Specifically, PASTEL proposes panoramic scene alignment, a novel representation that reformulates the intractable 3D "invisible region" exploration into a tractable 2D directional trajectory planning. This is achieved by reducing the viewpoint planning from 6-DoF search to a 2D directional search with explicit visibility boundaries. By operating within this panoramic space, our method strategically identifies camera trajectories that maximize exploration beyond observable boundaries while minimizing viewpoint deviation. Experimental results show that PASTEL can not only extrapolate plausible scene content beyond the observable boundaries of input monocular videos, but also substantially boost monocular 4D reconstruction performance. PASTEL outperforms the previous state-of-the-art method by 0.9dB in full-image PSNR on the DyCheck IPhone dataset.
Figures & tables
Figure 1: Overview of PASTEL . (a) Given monocular videos, PASTEL proposes panoramic alignment to reformulate scene exploration. (b) PASTEL shows seamless extension of rendering beyond observable boundaries in orange boxed areas. PASTEL also preserves high fidelity and consistency in yellow boxed areas.
Figure 2: Architecture overview of PASTEL . (a) Given a monocular video with predicted camera poses and depth, PASTEL designs panoramic representation alignment by projecting all video frames into a unified panoramic space. PASTEL then devises chosen cameras serving as the beginning of trajectories. (b) Along the designated trajectory, both static and dynamic components are utilized to achieve a comprehensive view expansion, which feeds into the diffusion prior for refinement. Finally, we compare warped and rendered images from Gaussian Splatting to identify unreliable and invisible regions, which are used to strategically supervise the 4D scene.
Figure 3: Qualitative comparison on DyCheck [ 23 ] . The yellow boxed areas highlight synthesis beyond covisible regions. Our method achieves superior overall quality and consistency with the ground truth, excelling in both reconstruction fidelity and the ability to induce details in areas beyond covisible regions.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
mPSNR ↑
mSSIM ↑
mLPIPS ↓
Rec. (h)
FPS
4D GS [ 43 ]
15.71
0.450
0.398
16.54
0.594
0.347
1.2
44
Shape-of-Motion [ 40 ]
16.79
0.510
0.391
17.32
0.598
0.296
2.0
40
MoSca [ 20 ]
17.33
0.572
0.355
19.32
0.706
0.264
1.3
38
USPLAT4D [ 10 ]
17.63
0.583
0.341
19.63
0.716
0.250
-
-
Cat4D [ 44 ]
15.91
0.427
0.398
17.39
0.607
0.341
-
-
TrajectoryCrafter [ 50 ]
12.71
0.324
0.519
14.24
0.417
0.519
-
-
Table 1: Comparison with reconstruction-based and generation-based methods on DyCheck [ 23 ] . “Rec.” denotes reconstruction time in hours. “FPS” denotes rendering speed. Our model significantly outperforms all baselines in PSNR, SSIM, and LPIPS, with comparable time cost.
Figure 4: Qualitative comparison on Kubric-4D [ 9 ] . The red boxed areas highlight synthesis in previously unobserved regions under large camera displacements. Our method achieves superior overall quality and consistency with the ground truth.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
mPSNR ↑
mSSIM ↑
mLPIPS ↓
GCD [ 38 ]
18.59
0.555
0.383
18.68
0.560
0.312
TrajectoryCrafter [ 50 ]
20.93
0.730
0.257
23.44
0.781
0.186
CogNVS [ 4 ]
22.63
0.760
0.232
24.85
0.799
0.162
PASTEL (Ours)
22.75
0.733
0.212
24.25
0.781
0.149
Table 2: Comparison with generation-based methods on Kubric-4D [ 9 ] . Our model achieves the best full-image PSNR, LPIPS under large camera displacements, while remaining competitive on the remaining metrics.
Figure 5: Reconstruction on monocular videos with complex motion and heavy occlusions. PASTEL captures highly dynamic scenes with irregular motion, self-occlusion, and dis-occlusion. Compared with MoSca [ 20 ] , PASTEL complements geometry of dynamic content in unseen and occluded regions while improving static backgrounds at the same time.
Components
PSNR ↑
SSIM ↑
LPIPS ↓
mPSNR ↑
mSSIM ↑
mLPIPS ↓
Full model
18.54
0.582
0.335
19.75
0.739
0.247
w/o panorama (linear)
15.73
0.497
0.370
19.42
0.728
0.249
Cartesian-TopK
17.31
0.548
0.352
19.55
0.733
0.251
w/o trajectory alignment
17.87
0.572
0.342
19.57
0.736
0.247
w/o view expansion
17.44
0.564
0.357
19.45
0.733
0.250
w/o strategic supervision
14.09
0.306
0.548
14.80
0.506
0.445
Table 3: Ablation study on different components of the system. Panoramic representation, trajectory alignment, and view expansion are crucial for views beyond the camera range. Strategic supervision markedly improves performance.
Figure 6: Visual comparison of ablations. As exhibited in the red-boxed areas, panoramic representation facilitates the effective extension of views beyond camera boundaries. Trajectory alignment and view expansion contribute to the consistent synthesis of extended views of red-boxed areas. Strategic supervision is pivotal in harmonizing the generative content and the original monocular video, culminating in coherent 4D renderings in the orange-boxed areas.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Method
PSNR ↑
LPIPS ↓
Method
PSNR ↑
LPIPS ↓
D-NeRF [ 31 ]
21.49
0.232
CTNeRF [ 26 ]
26.13
0.082
NR-NeRF [ 37 ]
19.69
0.323
DynPoint [ 59 ]
26.53
0.068
TiNeuVox [ 7 ]
19.74
0.285
D-NPC [ 18 ]
25.64
0.109
HyperNeRF [ 28 ]
17.60
0.367
RoDynRF [ 24 ]
25.89
0.067
NSFF [ 22 ]
24.33
0.199
Casual-FVS [ 19 ]
24.57
0.081
DynNeRF [ 8 ]
26.10
0.082
GaussianMarbles [ 32 ]
22.32
0.129
Appendix
Table 4: Quantitative comparison with reconstruction-based methods on Nvidia [ 23 ] . The best results are highlighted in bold , while the second best are underlined .
Figure 7: Qualitative comparison on NSFF [ 48 ] . Our method achieves better overall quality and consistency with the ground truth. Please zoom in for clearer visualization.
Figure 8: Reconstruction on real-world monocular videos. The first three columns exhibit novel view synthesis with different timestamps. The last two columns compare the whole scene with the previous state-of-the-art method, Mosca [ 20 ] . PASTEL can be effectively applied to the reconstruction of any monocular video, achieving both superior overall quality and effective synthesis of extended views.
Figure 9: Qualitative comparison with Cartesian-TopK baseline. Our panoramic trajectory planning provides more structured and consistent view expansions than unconstrained Cartesian sampling.
Trajectory Directions ( M )
Interpolation Steps ( L )
SSIM Threshold ( ϵ )
M
PSNR
SSIM
LPIPS
L
PSNR
SSIM
LPIPS
ϵ
PSNR
SSIM
LPIPS
2
17.70
0.573
0.355
12
17.88
0.575
0.352
0.1
18.29
0.569
0.350
8
18.54
0.582
0.335
49
18.54
0.582
0.335
0.3
18.54
0.582
0.335
32
17.98
0.578
0.342
196
17.61
0.573
0.357
0.5
18.06
0.560
0.358
Appendix
Table 5: Ablations on trajectory directions M , interpolation steps L , and SSIM threshold ϵ . Gray rows indicate the default settings used in our final model.
Figure 10: Ablations on trajectory directions M , interpolation steps L , and SSIM threshold ϵ . Trajectory directions M and interpolation steps L has significant influence on red-boxed regions beyond observable boundary. SSIM threshold ϵ influences detailed textures on orange-boxed regions.
Generative prior
PSNR ↑
SSIM ↑
LPIPS ↓
TrajectoryCrafter [ 50 ]
18.54
0.582
0.335
ReCamMaster [ 1 ]
17.95
0.551
0.361
ViewCrafter [ 51 ]
17.22
0.513
0.408
Appendix
Table 6: Ablation study on generative priors. We compared three recently open-sourced camera-controlled video generation methods. TrajectoryCrafter performs best in our architecture.
Table 7: Ablation study on robustness to estimation errors. We simulate inaccuracy by adding noise with standard deviation (std) to the original depth (left) and camera pose (right).
Figure 11: Scene-specific adaptation of the generative prior. Fine-tuning on observed regions shows only slight improvement of color consistency in red-boxed unseen regions.