SparseCam4D: Spatio-Temporally Consistent 4D Reconstruction from Sparse Cameras

Weihong Pan; Xiaoyu Zhang; Zhuang Zhang; Zhichao Ye; Nan Wang; Haomin Liu; Guofeng Zhang

Figure 2: Spatio-temporal inconsistency. Real cameras (grey) capture consistent content of multi-view dynamic scene, while generative results (orange) include additional observations at different poses and time. Inconsistencies across poses at the same time are referred to as spatial inconsistencies , and inconsistencies across time at the same pose are referred to as temporal inconsistencies .

Figure 3: Method overview. Given a generated frame at temporal index

t

and pose index

s

, each 4D Gaussian at

c=(x,y,z)

is projected onto the planes of the Spatio-Temporal Distortion Field to obtain deformation features, which are then decoded by a small MLP to produce the deformation values. We use separate photometric losses for real and generated frames, and additionally introduce regularization terms on pose, feature plane, and spatial smoothness to enhance optimization stability.

Method	Technicolor			Neural 3D Video			Nvidia Dynamic Scenes
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
HyperReel [ 1 ]	14.14	0.453	0.616	15.63	0.582	0.500	19.88	0.528	0.396
4DGaussians [ 38 ]	16.20	0.505	0.552	17.40	0.673	0.320	16.81	0.372	0.516
4D-Rotor [ 5 ]	14.85	0.426	0.581	18.20	0.708	0.357	19.38	0.508	0.389
RealTime4DGS [ 43 ]	16.53	0.510	0.542	17.31	0.649	0.442	17.91	0.479	0.426
MonoFusion* [ 35 ]	17.97	0.578	0.352	18.43	0.738	0.270	20.22	0.590	0.192

Table 1: Qualitative comparisons on Technicolor [ 26 ] , Neural 3D Video [ 13 ] , and Nvidia Dynamic Scenes [ 46 ] Datasets. The first and second best performances are highlighted in red and yellow. Our method shows superior performance compared to all baseline methods across all metrics. Note that MonoFusion ∗ is our reproduced version.

Figure 4: Qualitative Comparisons of different methods on Technicolor [ 26 ] , Neural 3D Video [ 13 ] , and Nvidia Dynamic Scenes [ 46 ] Datasets . We conduct comparisons with representative dynamic scene reconstruction methods: MonoFusion [ 35 ] , 4DGS [ 38 ] , 4D-Rotor [ 5 ] , and Realtime4DGS [ 43 ] . MonoFusion ∗ is our reproduced version. Our method significantly outperforms other baselines, producing visually reliable results with sharper details. Please zoom in for more details. Additional qualitative comparisons are included in the supplementary material.

Setting	Train		Jumping
Setting	LPIPS $\downarrow$	SSIM $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$
w/o distortion field	0.608	0.426	0.319	0.674
w/o time axis	0.458	0.480	0.279	0.738
w/o pose axis	0.469	0.462	0.268	0.733
Ours	0.264	0.656	0.170	0.793

Table 2: Ablation studies on STDF. We randomly select one representative scene from Technicolor [ 26 ] and Nvidia Dynamic Scenes [ 46 ] to ablate Spatio-Temporal Distortion Field.

Figure 5: Spatio-Temporal Consistency. Rendering results (top) and space-time slices (bottom) constructed by concatenating the red pixel locations across all time steps, demonstrate that direct reconstruction from diffusion observations leads to severe blur and temporal instability(e.g., the moving hand at the bottom right).

Setting	Train		Jumping
Setting	LPIPS $\downarrow$	SSIM $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$
w/o pose optimization	0.336	0.569	0.217	0.754
w/o $\mathcal{L}_{lpips}$	0.285	0.646	0.185	0.782
w/o $\mathcal{L}_{TV}$	0.309	0.616	0.191	0.768
w/o $\mathcal{L}_{smooth}$	0.308	0.619	0.191	0.770
Ours	0.264	0.656	0.170	0.793

Table 3: Ablation studies on pose optimization and loss components. We randomly select one representative scene from Technicolor [ 26 ] and Nvidia Dynamic Scenes [ 46 ] Datasets to ablate pose optimization and loss components.

Figure 6: Visualization of the STDF. Spatio-Temporal Distortion Field output is rendered as a per-primitive attribute, with brighter regions indicating higher distortions (left). The corresponding areas in the input generated image (right) align with regions exhibiting noticeable deformation (red box).

Cook Spinach	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
ViewCrafter (w/o STDF)	21.42	0.775	0.302
ViewCrafter (w STDF)	23.93	0.832	0.232
ReCamMaster (w/o STDF)	21.97	0.756	0.315
ReCamMaster (w STDF)	23.61	0.806	0.247

Table 4: Ablation study with alternative video diffusion models (VDMs) on Cook Spinach.

Figure 7: Spatio-Temporal consistency on ReCamMaster.

Figure 8: Visualization of selected training cameras. The non-red frustums denote the cameras used for training, corresponding to the images shown at the bottom and on the right respectively. The remaining red frustums indicate the cameras used for evaluation.

Figure 9: Reliability of PSNR vs. LPIPS in Ablation Experiments. We find that directly using generated-view reconstruction introduces severe oversmoothness that, paradoxically, favors PSNR computation, preventing PSNR from accurately reflecting changes in reconstruction quality. In contrast, LPIPS effectively captures the variations in reconstruction quality under these ablation settings.

	3 views			6 views			9 views
	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
4DGaussian	16.72	0.435	0.546	19.86	0.581	0.448	21.41	0.601	0.430
Ours	21.56	0.656	0.264	23.58	0.687	0.243	24.72	0.751	0.205

Table 5: Performance with different number of camera views. We set 4DGaussian as the baseline model for dynamic scene reconstruction and evaluate the rendering quality with different number of training views.

Figure 10: Visualization of the point-cloud rendering condition, generation prior, and final results.

	Coffee Martini	Playground	Average
RealTime4DGS	11.48fps	11.51fps	11.50fps
Ours	11.82fps	11.97fps	11.90fps

Table 6: Efficiency Comparison with RealTime4DGS.

Figure 11: Visualization of one failure case. From left to right are the source image and point-cloud–rendering condition provided to ViewCrafter, the corresponding generated image, and a Gaussian rendering near that generated image. Since ViewCrafter is primarily trained on scene-centric data with few human subjects, this example suffers from an out-of-domain issue. In addition, the quality of the point-cloud–rendering condition is relatively low. These factors jointly lead to low-quality generated images, which in turn degrade the Gaussian reconstruction quality on the human body.

Figure 12: Visualization of Spatio-Temporal Distortion Field

Figure 13: Additional qualitative comparison results on Technicolor Dataset. We highlight the geometric completeness of background structures, bottom decorations, and the red bridge across different viewpoints, as well as the temporal consistency of dynamic regions such as human faces, the ‘Happy Birthday’ text, and the toy train.

Figure 14: Additional qualitative comparison results on Neural 3D Video Dataset (part I).

Figure 15: Additional qualitative comparison results on Neural 3D Video Dataset (part II).

Figure 16: Additional qualitative comparison results on Neural 3D Video Dataset (part III).

Figure 17: Additional qualitative comparison results on Neural 3D Video Dataset (part IV).

Figure 18: Additional qualitative comparison results on Nvidia Dynamic Scenes Dataset (part I). In the top panel, we emphasize the geometric completeness of the left archway and the right escalator across different viewpoints, as well as the temporal consistency of dynamic regions such as the human body, balloon, and the blue tether. In the bottom panel, we highlight the fine details of background grass, shoes, and umbrella textures, together with accurate motion fitting under fast movement in the leftmost column.

Figure 19: Additional qualitative comparison results on Nvidia Dynamic Scenes Dataset (part II).

	Fabien	Theater	Train	Birthday	Painter	Average
	Technicolor - PSNR
HyperReel	13.62	14.24	15.44	12.28	15.13	14.14
4DGaussians	18.31	17.53	16.72	11.97	16.48	16.20
4DRotor	15.21	16.00	14.01	12.42	16.62	14.85
RealTime4DGS	13.00	16.27	16.59	18.45	18.76	16.53
MonoFusion	17.86	19.48	15.12	16.47	20.92	17.97

Table 7: PSNR, SSIM, and LPIPS of our framework on Technicolor, with cam00, cam09 and cam15 served as training views.

	coffee martini	cook spinach	cut roasted beef	flame salmon	flame steak	sear steak	Average
	Neural 3D Video - PSNR
HyperReel	14.07	16.44	17.04	10.64	17.57	17.28	15.63
4DGaussians	14.49	17.94	18.34	14.83	19.53	18.57	17.40
4DRotor	15.86	19.65	18.51	16.10	19.69	19.05	18.20
RealTime4DGS	15.07	17.95	17.04	14.65	19.57	18.86	17.31
MonoFusion	15.59	19.71	19.15	15.42	20.02	19.91	18.43

Table 8: PSNR, SSIM, and LPIPS of our framework on Neural 3D Video, with cam01, cam05 and cam10 served as training views.

	jumping	balloon1	balloon3	playground	skating	umbrella	Average
	Nvidia Dynamic Scenes - PSNR
HyperReel	24.45	15.46	22.58	15.41	21.05	24.74	19.88
4DGaussians	17.38	16.96	14.56	12.75	19.70	19.52	16.81
4DRotor	21.90	14.55	20.33	10.95	26.28	22.29	19.38
RealTime4DGS	17.85	18.36	19.07	13.96	18.52	19.71	17.91
MonoFusion	20.72	20.31	19.81	16.86	23.55	20.07	20.22

Table 9: PSNR, SSIM, and LPIPS of our framework on Nvidia Dynamic Scenes, with cam01, cam06 and cam10 as training views.

	Fabien	Theater	Train	Birthday	Painter	Average
	Technicolor - PSNR
cam01	25.581	21.231	23.456	19.562	26.322	23.23
cam02	21.886	19.203	21.202	18.227	25.736	21.251
cam03	18.844	18.5	19.468	17.692	27.939	20.488
cam04	28.676	23.089	23.59	20.991	27.423	24.754
cam05	28.162	23.422	23.833	20.837	27.249	24.701

Table 10: PSNR, SSIM, and LPIPS of our framework on Technicolor. Three cameras (cam00, cam09 and cam15) are served as training views for scene Fabien , Theater and Train , while only two cameras (cam00 and cam15) are training views for scene Birthday and Painter .

	coffee martini	cook spinach	cut roasted beef	flame salmon	flame steak	sear steak	Average
	Neural 3D Video - PSNR
cam00	21.654	28.114	26.901	22.323	27.558	27.744	25.715
cam02	25.236	25.926	25.206	24.866	25.472	25.365	25.345
cam03	-	25.194	24.231	-	25.489	25.891	25.201
cam04	25.477	31.417	-	25.741	31.034	30.641	28.862
cam06	20.31	27.966	26.145	21.048	27.624	27.609	25.117

Table 11: PSNR, SSIM, and LPIPS of our framework on Neural 3D Video. Three cameras (cam01, cam05 and cam10) are training views.

	jumping	balloon1	balloon3	playground	skating	umbrella	Average
	Nvidia Dynamic Scenes - PSNR
cam02	25.867	26.076	28.318	19.736	32.098	26.946	26.507
cam03	23.858	25.564	27.681	19.201	26.698	27.409	25.069
cam04	25.41	25.337	26.352	18.441	30.907	27.033	25.58
cam05	26.698	26.248	27.455	20.261	32.017	28.36	26.84
cam07	24.477	25.839	27.519	19.161	29.468	22.384	24.808

Table 12: PSNR, SSIM, and LPIPS of our framework on Nvidia Dynamic Scenes. Three cameras (cam01, cam06 and cam10) are training views.

SparseCam4D: Spatio-Temporally Consistent 4D Reconstruction from Sparse Cameras

Abstract

Figures & tables

Sparse-View 4D Reconstruction

4D Reconstruction from Sparse Dynamic Cameras

4C4D: 4 Camera 4D Gaussian Splatting

Stitch4D: Sparse Multi-Location 4D Urban Reconstruction via Spatio-Temporal Interpolation

Abstract

Sparse-View 4D Reconstruction

Explore similar work

4D Reconstruction from Sparse Dynamic Cameras

4C4D: 4 Camera 4D Gaussian Splatting

Stitch4D: Sparse Multi-Location 4D Urban Reconstruction via Spatio-Temporal Interpolation