A Compact Explicit 4D Representation for Dynamic Scenes

Di Yang; Zhihao Li; Yanhai Xiong; Yufei Wang

cs.CVOct 1, 2026

A Compact Explicit 4D Representation for Dynamic Scenes

Authors: Di Yang, Zhihao Li, Yanhai Xiong, Yufei Wang

Organizations: College of William & Mary · SparcAI Inc.

Abstract

A compact dynamic-scene representation must retain both the surfaces seen over time and the appearance needed to render them from new viewpoints. We present Sparc4D, a feed-forward autoencoder that encodes a monocular video with known cameras into a sparse 4D scene state. Static features are shared across the clip, while spatially anchored temporal slots compress time-varying features. A sparse decoder produces 2D Gaussian surfels, while stored source pixels preserve fine texture through geometric re-projection. The state includes one full source frame and dynamic-region pixels sampled every fourth frame, alongside learned features and sparse occupancy. For a 32-frame MultiCamVideo clip, it averages 0.95M 32-bit-equivalent values on random windows and 0.92M on the first-32 protocol. On first-32, Sparc4D reaches 21.70,dB, compared with 20.40,dB for MoVieS. On randomly placed windows, their PSNR scores are comparable. With stored texture disabled, temporal slots compress the time-varying feature state by a median $4.0\times$ and reduce the mean state from 1.04M to 0.42M values, with essentially unchanged target-view reconstruction quality. Without fine-tuning on real data, Sparc4D transfers to DyCheck and Neu3D, where stored texture improves LPIPS while slightly reducing PSNR.

Figures & tables

Figure 1 : Overview of Sparc4D. (a) A posed video is encoded into a compact per-clip state (dashed box) and decoded at a queried time $t$ and camera $c$ . Images are real MultiCamVideo inputs and complete-model outputs; the point cloud shows input observations. (b) Temporal slots compress per-frame dynamic features. (c) Stored pixels are re-projected through decoded geometry and change colour, not geometry.

	Trained		Full image				Co-visible			Uniform fill
Method	on MCV	#Floats	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	AbsRel $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
MoVieS	–	719M	17.26	0.512	0.366	0.615 (0.272)	18.55	0.530	0.296	16.43	0.498	0.411
4DGT	–	26.9M	14.35	0.403	0.550	0.640 (0.370)	15.14	0.411	0.538	14.49	0.412	0.546
C4G	–	0.92M	13.64	0.369	0.596	0.543 (0.450)	14.76	0.394	0.566	14.21	0.393	0.572
C4G	✓	0.92M	16.51	0.466	0.584	0.356 (0.346)	17.14	0.470	0.561	15.65	0.444	0.564
Sparc4D	✓	0.95M	17.29	0.516	0.371	0.254 (0.164)	18.94	0.539	0.291	16.59	0.508	0.405

Table 1 : Novel-view synthesis on MultiCamVideo (404 random-window test clips, 9 target cameras

\times

32 frames,

256^{2}

). Best and second-best results are shown in bold and italics, respectively. Sparc4D denotes the complete method, including stored texture. #Floats uses the materialized-state convention [ 5 ] for the baselines and the ledger in Eq. 5 for our encoded state; shared network weights are excluded. C4G is shown with released weights and after a 30,000-step fine-tuning run on the same MCV training split; the final Sparc4D model is independently trained for the same number of steps. AbsRel uses DA3 pseudo ground truth, also our input depth; parentheses give median-scale-aligned error. Co-visible metrics use the shared visibility mask. Uniform fill assigns the same colour outside that mask for all methods.

Figure 2 : Qualitative comparison, first-32 protocol. Two fixed test scenes at frame 16, using the median-distance target camera. Orange and blue boxes mark dynamic and static $64^{2}$ crops, enlarged below at identical coordinates for all methods. Sparc4D is the complete model; baseline images use released weights. Camera distance is shown in metres.

Figure 3 : Qualitative comparison, random-window protocol. Same layout and method settings as Fig. 2 ; target cameras require at least 60% observed pixels. These examples show sharper retained texture as well as missing dynamic support: storing colour cannot reconstruct a surface absent from the queried-time geometry.

			DyCheck			Neu3D (9 targets)			Neu3D cam00
Method	Training data	#Floats	mPSNR $\uparrow$	mSSIM $\uparrow$	mLPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$
MoVieS	8 public datasets ( $\sim$ 174K scenes)	719M	13.95	0.345	0.488	15.61	0.445	0.304	17.01
4DGT	$\sim$ 300 h real monocular video	26.9M	11.84	0.277	0.599	17.33	0.530	0.281	19.51
C4G	Spring, Kubric, RE10K	0.92M	13.52	0.351	0.634	11.78	0.285	0.588	13.41
C4G (MCV fine-tuned)	+ MultiCamVideo	0.92M	13.34	0.340	0.747	15.75	0.443	0.579	16.21
Sparc4D	MultiCamVideo (synthetic)	0.95M	13.89	0.319	0.544	16.45	0.462	0.296	18.44

Table 3 : Zero-shot generalization to real videos. Best and second-best results are shown in bold and italics. None of the methods is fine-tuned on DyCheck or Neu3D. DyCheck uses official co-visibility evaluation; Neu3D reports nine target cameras and cam00 separately. #Floats repeats the MCV reference size for context rather than a measured real-data average.

	DyCheck		Neu3D
Configuration	mPSNR $\uparrow$	mLPIPS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$
w/o stored texture	14.23	0.587	16.60	0.306
w/o dynamic texture	14.07	0.554	16.53	0.303
Sparc4D	13.89	0.544	16.45	0.296

Table 4: Stored texture on real videos. All rows use the same model weights and inputs. DyCheck uses official masked metrics; Neu3D uses full-image metrics over nine target cameras.

Configuration	#Floats	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	$\Delta$ PSNR
Geometry only	76.2K	16.76	0.492	0.475	–
+ appearance code	0.87M	16.94	0.500	0.448	$+0.18\pm 0.04$
+ larger code	3.52M	16.99	0.502	0.442	$+0.06\pm 0.01$
+ motion mask	3.58M	16.96	0.502	0.444	$-0.04\pm 0.01$
Final-backbone config. †	0.42M	16.95	0.501	0.430	$+0.01\pm 0.03$

Table 5: Learned-architecture ablation (404 random-window clips). All rows use 30k total optimization steps: a shared 20k geometry-only initialization followed by an independent 10k branch.

\Delta

PSNR gives paired per-clip differences to the corresponding parent configuration (mean

\pm

95% CI). All rows omit stored texture.

Configuration	Size	PSNR $\uparrow$	LPIPS $\downarrow$	Dyn. LPIPS $\downarrow$
w/o stored texture	0.42M	17.03	0.421	0.302
w/o dynamic texture	0.61M	17.29	0.379	0.278
Dynamic texture interval: 1	2.08M	17.35	0.369	0.260
Dynamic texture interval: 2	1.33M	17.32	0.370	0.261
Dynamic texture interval: 8	0.76M	17.25	0.373	0.266
Sparc4D (interval: 4)	0.95M	17.29	0.371	0.263

Table 6: Stored-texture ablation (404 random-window clips). All rows use the same 30k-step backbone. Interval is the spacing between stored dynamic-texture frames; the first full source frame is retained except in “w/o stored texture.” Dynamic LPIPS uses region crops.

	Storage $\downarrow$		Reconstruction
Representation	Time-varying features	Total state	PSNR $\uparrow$	Dyn. PSNR $\uparrow$	Dyn. LPIPS $\downarrow$
Per-frame	871.9K	1.04M	17.02	16.09	0.300
Temporal slots	249.4K	0.42M	17.03	16.09	0.302

Table 7: Temporal compression ablation on random windows. Both variants use the same trained weights and omit stored texture. “Per-frame” bypasses temporal-slot compression for dynamic latent and appearance features; dynamic occupancy is unchanged. Sizes are mean 32-bit-equivalent values per 32-frame clip.

	Dynamic		Static
Method	PSNR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$
4DGT	16.19	0.345	18.56
C4G	14.32	0.434	15.97
MoVieS	19.02	0.150	21.79
w/o stored texture	18.16	0.219	22.13
w/o dynamic texture	18.73	0.180	25.18

Table 8: Static and dynamic regions , first-32 protocol. Dynamic-region LPIPS is computed on the bounding box of the region.

Explore similar work

Reconstructing the Dynamic World: A Representation-Centric View of 4D Scene Reconstruction
Sep 30, 2026Ziren Gong, Guo Chen, Yongjia Li +13Neural Radiance Fields Scene Understanding
Department of Computer Science and Engineering, University of Bologna, Italy · Wangxuan Institute of Computer Technology, Peking University, China · Department of Computing, The Hong Kong Polytechnic University, Hong Kong +8
Learning Global Motion with Compact Gaussians for Feed-Forward 4D Reconstruction
May 29, 2026Mungyeom Kim, Minkyeong Jeon, Honggyu An +104D Reconstruction Feed-Forward 3D Gaussian
KAIST AI · ETH Zürich, ETH AI Center · Sony AI +1
Velox: Learning Representations of 4D Geometry and Appearance
May 6, 2026Anagh Malik, Dorian Chan, Xiaoming Zhao +3Four-Dimensional Point Clouds
Apple · University of Toronto

Sep 30, 2026, cs.CVJ/K move · Enter open · S save

Reconstructing the Dynamic World: A Representation-Centric View of 4D Scene Reconstruction

Ziren Gong, Guo Chen, Yongjia Li, Yihua Shao+12

Department of Computer Science and Engineering, University of Bologna, Italy · Wangxuan Institute of Computer Technology, Peking University, China · Department of Computing, The Hong Kong Polytechnic University, Hong Kong+8

Neural Radiance Fields Scene Understanding Representation Quality

Open paper AskPDF arXiv

4D scene reconstruction aims to recover the evolving geometry, appearance, and motion of dynamic environments from visual observations. Despite substantial progress in neural scene representations, reconstructing dynamic scenes remains challenging due to non-rigid motion, occlusions, temporal inconsistencies, and the trade-offs between reconstruction fidelity and computational efficiency. Recent advances in Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have introduced diverse approaches to representing and reconstructing dynamic scenes, yet their relationships, underlying design choices, and evaluation protocols remain fragmented. In this paper, we present a unified perspective on 4D scene reconstruction, organizing existing methods around their scene representations, temporal modeling strategies, reconstruction pipelines, and optimization objectives. Through this framework, we examine how different design choices affect geometric fidelity, appearance consistency, motion representation, and computational efficiency. We further consolidate commonly used datasets and evaluation metrics, identify limitations in current experimental practices, and discuss open challenges in reconstructing complex, dynamic real-world environments. By connecting methodological developments with their underlying assumptions and evaluation evidence, this work provides a structured foundation for understanding existing approaches and identifying future research directions. An evolving collection of relevant papers and resources is available at https://github.com/ZiyangYan/Awesome-4D-Scene-Reconstruction.

Abstract

Explore similar work

Reconstructing the Dynamic World: A Representation-Centric View of 4D Scene Reconstruction

Learning Global Motion with Compact Gaussians for Feed-Forward 4D Reconstruction

Velox: Learning Representations of 4D Geometry and Appearance