UniFusion: Sparse-View 4D Reconstruction via Unified Spatio-temporal Depth Alignment

Yongzhe Lyu; Shaofei Wang; Yixin Chen; Siyuan Huang

cs.CVSep 5, 2026

UniFusion: Sparse-View 4D Reconstruction via Unified Spatio-temporal Depth Alignment

Authors: Yongzhe Lyu, Shaofei Wang, Yixin Chen, Siyuan Huang

Organizations: State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China

Abstract

In this paper, we address the challenging problem of 4D reconstruction from sparse-view videos. This setup usually relies on monocular depth estimation to provide priors for the reconstruction model. A key challenge arises from limited cross-view overlap and temporal variation, making monocular depth predictions inconsistent across views and time. Existing methods align spatial and temporal dimensions in separate stages, requiring foreground segmentation masks while failing to leverage temporal cues for cross-view alignment. Contrary to these methods, we propose a unified spatial-temporal depth alignment framework that jointly resolves cross-view and cross-time inconsistencies without distinguishing foreground/background. Our method represents depth maps across views and time as a set of spatio-temporal neural fields. This representation not only yields fast convergence, but also captures spatio-temporal correlation among depth maps implicitly, without dependence on external segmentation/tracking models. We also propose a multi-view depth-order loss while leveraging the classic scale-and-shift-invariant loss to further improve the final depth quality. The aligned depths initialize and supervise Gaussian splatting models for 4D reconstruction. Experiments on Ego-Exo4D and EgoHuman demonstrate that our improved depth alignment substantially benefits dynamic Gaussian-splatting-based reconstruction methods for novel-time/view synthesis and geometry accuracy/consistency.

Figures & tables

Figure 1: Sparse-view 4D Reconstruction via spatially and temporally aligned monocular depths. Left: given 4-view video streams with minimal overlap, we leverage a unified spatio-temporal depth alignment framework to produce multi-view consistent (top) and temporally coherent (bottom) depth maps. Right : the aligned depth maps serve as both initialization and supervision for 4D reconstruction, substantially improving the quality over state-of-the-art methods, achieving reasonable novel view synthesis even when shifted $\sim 45^{\circ}$ from the training view.

Panel of Figure 2 — Figure 2: Overview of our method. Given monocular depths (estimated from RGB images) and sparse SfM points from sparse synchronized cameras ( left ), we align them jointly across views and time via per-view spatio-temporal neural fields ( Stage 1 ). Each view maintains spatial and depth encodings shared across time, while a per-view deformation MLP predicts depth offsets along ray directions. Temporal variation is captured through low-rank residual weight conditioning ( inset ): each layer’s weights decompose into a shared base plus a compact residual parameterized by per-frame temporal coefficients. The aligned depths initialize and supervise 4D Gaussian splatting for novel-view/time synthesis ( Stage 2 ).

Figure 3: Qualitative analysis of held-out frames of training view on EgoExo4D. Our aligned depth enhances the geometry quality of both MonoFusion and deformable 2DGS, removing ghosting artifacts that are clearly visible in the held-out frames (top two rows), while substantially improving the overall depth quality of deformable 2DGS(bottom two rows).

	Full image				Dynamic region
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	AbsRel $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
SOM †	14.73	0.535	0.482	0.843	15.63	0.559	0.450
Dyn3D-GS †	24.28	0.692	0.539	0.612	24.61	0.673	0.384
MV-SOM-DS †	28.37	0.906	0.079	0.398	28.23	0.931	0.063
MV-SOM †	26.91	0.890	0.138	0.474	27.31	0.919	0.078
MonoFusion †	30.43	0.927	0.061	0.290	29.71	0.947	0.017

Table 1: Quantitative comparison on Ego-Exo4D. Baseline results (marked

\dagger

) are copied from MonoFusion [ 72 ] . “X + Ours” denotes backend X initialized and supervised with our aligned depths.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
SOM	13.97	0.652	0.645
MV-SOM	13.36	0.588	0.564
SplatFields	17.74	0.675	0.571
MonoFusion	14.15	0.436	0.628
MonoFusion + Ours	20.41	0.612	0.492
	(+6.26)	(+.176)	(-.136)

Table 2: Quantitative comparison on EgoHuman. We evaluate novel view synthesis on two sequences from the EgoHuman dataset, substantially outperforming all baselines.

Figure 4: Qualitative analysis of novel view synthesis on EgoExo4D. We show extreme novel view synthesis results ( $\sim 45^{\circ}$ from training view), further confirming that our aligned depth consistently benefits the baseline models.

Figure 5: Qualitative Results on the EgoHumans dataset. On the more challenging EgoHumans dataset, our aligned depth helps MonoFusion to recover from catastrophic failure, while significantly improving the novel view rendering quality of deformable 2DGS.

	MonoFusion			Def-GS
Configuration	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
(a) Per-frame MAtCha	30.14	0.943	0.070	32.60	0.956	0.071
(b) + time embedding	28.53	0.935	0.088	32.60	0.947	0.071
(c) + residual weights	30.90	0.948	0.067	33.61	0.960	0.055
(d) + $\mathcal{L}_{\text{order}}$	31.52	0.953	0.065	33.90	0.961	0.055
(e) + $\mathcal{L}_{\text{ssi}}$ (full)	31.63	0.955	0.064	34.66	0.963	0.049

Table 3: Progressive ablation of alignment components on Ego-Exo4D on two Gaussian splatting backends.

Figure 6: Limitations. While our improved depth consistently enhanced the geometry quality of the different baselines, we note that the naive deformable 2DGS can sometimes become the bottleneck and fail to capture the fast/complex motion. In such a case, MonoFusion, when combined with proper depth initialization/supervision from our method, can produce more reasonable foreground geometry and rendering.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Padding Ratio	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
$1.0\times$	25.94	0.924	0.074
$1.1\times$	27.20	0.943	0.060
$1.2\times$	28.25	0.955	0.049
$1.4\times$	29.48	0.958	0.040
MonoFusion $\dagger$	29.71	0.947	0.017

Appendix

Table 4: Sensitivity analysis of the foreground padding ratio. We evaluate the impact of different padding scales on photometric metrics for dynamic objects. The ratio

1.4\times

is selected as it yields results most consistent with the baseline values reported in MonoFusion [ 72 ] .

Figure 7: Qualitative Analysis of temporal encoding . We unproject depth maps from one view to demonstrate the effectiveness of our temporal encoding.

Figure 8: Qualitative Analysis depth order loss . For better understanding, we use depth unprojections from 2 opposed views to highlight multi-view consistency.

Method	Reproj. Consistency $\downarrow$	Flow-warped Depth $\downarrow$	Order Violation $\downarrow$
MAtCha	0.585	0.034	9.75%
MonoFusion	0.648	0.016	28.10%
Ours	0.536	0.015	4.84%

Appendix

Table 5: Direct evaluation of depth alignment quality on Ego-Exo4D. Our method improves cross-view reprojection consistency, temporal depth stability, and cross-view depth-order consistency without relying on Gaussian splatting optimization.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
MonoFusion	16.83	0.846	0.514
MonoFusion + Ours	18.49	0.894	0.440
Def-2DGS	16.82	0.825	0.476
Def-2DGS + Ours	19.15	0.906	0.331

Appendix

Table 6: Experiment on an additional Harmony4D scene.

	Def-GS			MonoFusion
Configuration	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
(a) Per-frame MAtCha	20.93	0.830	0.305	19.54	0.566	0.516
(b) + residual weights (ours)	21.80	0.804	0.309	19.59	0.590	0.514
(c) + $\mathcal{L}_{\text{order}}$	22.15	0.815	0.306	19.94	0.583	0.506
(d) + $\mathcal{L}_{\text{ssi}}$ (full)	22.76	0.856	0.277	20.41	0.612	0.492

Appendix

Table 7: Progressive ablation of alignment components on EgoHuman on two Gaussian splatting backends.

$R$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
4	33.76	0.969	0.047
8	34.66	0.963	0.049
16	34.51	0.966	0.045

Appendix

Table 8: Ablation on residual rank

R

on Ego-Exo4D (Def-2DGS + Ours).

Loss type	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Hinge (ours)	34.66	0.963	0.049
Symmetric $L_{1}$	34.44	0.966	0.047

Appendix

Table 9: Ablation on

\mathcal{L}_{\text{order}}

formulation on Ego-Exo4D (Def-2DGS + Ours).

#Cams	Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
3	Def-2DGS	15.14	0.485	0.630
3	Def-2DGS + Ours	18.91	0.731	0.417
6	Def-2DGS	18.42	0.645	0.468
6	Def-2DGS + Ours	21.79	0.842	0.271

Appendix

Table 10: Evaluation under different camera baselines on EgoHuman using Def-2DGS as the backend. The 3- and 6-camera settings roughly correspond to

120^{\circ}

and

60^{\circ}

camera spacing, respectively.

Figure 9: More qualitative results of novel view synthesis on EgoExo4D.

Figure 10: We initialize and supervise Deformable 2DGS on EgoHuman using depth priors predicted by VGGT.

Sparse-View 4D Reconstruction

See all 13 papers

D^2-4DGS: Dual-Depth Guided Sparse-Camera 4D Gaussian Splatting
Aug 3, 2026Jijian Zhao3D Gaussian Splatting Sparse-View 4D Reconstruction
Huazhong University of Science and Technology
Depth Anything V4: Dynamic 4D Scene Reconstruction via Riemannian Flow Matching on 4D Gaussian Splatting
Aug 18, 2026Jiaming Fan, Jian Lu, Jinling Jia +14D Reconstruction Dynamic Scene Reconstruction
College of Artificial Intelligence, Nanjing University of Posts and Telecommunications · College of Automation, Nanjing University of Posts and Telecommunications · Institute of Ad-vanced Technology for Carbon Neutrality, Nanjing University of Posts and Telecommunications
Sparse-View 4D Gaussian Splatting via Spatiotemporal Priors and Generative Assistance
Oct 3, 2026Shengqi Wang, Zhengxian Yang, Kaiwen Tian +6Sparse-View 4D Reconstruction Dynamic Scene Reconstruction
Department of Automation, Tsinghua University, Beijing, China · JD.com, Beijing, China · Beijing MEET YUAN Co., Ltd, Beijing, China

Aug 3, 2026J/K move · Enter open · S save

D^2-4DGS: Dual-Depth Guided Sparse-Camera 4D Gaussian Splatting

Jijian Zhao

Huazhong University of Science and Technology

3D Gaussian Splatting Sparse-View 4D Reconstruction Dynamic Scene Reconstruction Dynamic Novel View Synthesis

Open paper AskPDF arXiv

Dynamic 4D Gaussian Splatting has emerged as an efficient representation for dynamic novel view synthesis through explicit scene modeling and real-time rendering. However, existing methods typically require dense multi-view videos for sufficient geometric constraints, making capture expensive and limiting sparse-camera deployment. Reducing input views lowers acquisition cost but weakens geometry supervision, often causing missing structures and floating Gaussians. Depth priors provide geometric cues, yet no single source offers both dense coverage and reliable geometry. Monocular depth provides dense structure but is scale-ambiguous and locally biased, whereas multi-view geometric depth provides incomplete anchors consistent with the reconstruction coordinate system. To exploit their complementarity, we propose D

^2

-4DGS, a sparse-camera dynamic 4D Gaussian Splatting framework guided by dual-source depth priors. We align monocular estimates with valid multi-view geometric depths and verify their consistency to identify reliable geometric anchors. These verified anchors support consistency-aware pruning and depth supervision, while verified geometric depths and aligned mono-only estimates provide candidate geometry for densification in under-reconstructed regions. Finally, RGB-D joint optimization improves appearance fidelity and geometric consistency under sparse-view supervision. Across all nine dataset--view settings, D

^2

-4DGS achieves the highest PSNR, improving by 1.33 dB on average over the best competing method in each setting.

Abstract

Sparse-View 4D Reconstruction

Explore similar work

D^2-4DGS: Dual-Depth Guided Sparse-Camera 4D Gaussian Splatting

Depth Anything V4: Dynamic 4D Scene Reconstruction via Riemannian Flow Matching on 4D Gaussian Splatting

Sparse-View 4D Gaussian Splatting via Spatiotemporal Priors and Generative Assistance