3D Gaussian Splatting has demonstrated remarkable real-time rendering capabilities and superior visual quality in novel view synthesis for static scenes. Building upon these advantages, researchers have progressively extended 3D Gaussians to dynamic scene reconstruction. Deformation field-based methods have emerged as a promising approach among various techniques. These methods maintain 3D Gaussian attributes in a canonical field and employ the deformation field to transform this field across temporal sequences. Nevertheless, these approaches frequently encounter challenges such as suboptimal rendering speeds, significant dependence on initial point clouds, and vulnerability to local optima in dim scenes. To overcome these limitations, we present FRoG, an efficient and robust framework for high-quality dynamic scene reconstruction. FRoG integrates per-Gaussian embedding with a coarse-to-fine temporal embedding strategy, accelerating rendering through the early fusion of temporal embeddings. Moreover, to enhance robustness against sparse initializations, we introduce a novel depth- and error-guided sampling strategy. This strategy populates the canonical field with new 3D Gaussians at low-deviation initial positions, significantly reducing the optimization burden on the deformation field and improving detail reconstruction in both static and dynamic regions. Furthermore, by modulating opacity variations, we mitigate the local optima problem in dim scenes, improving color fidelity. Comprehensive experimental results validate that our method achieves accelerated rendering speeds while maintaining state-of-the-art visual quality.
Figures & tables
Fig. 1 : Overview of FRoG. We begin by performing early fusion on the coarse and fine temporal embeddings through the Hadamard product, aiming to enhance rendering speed while preserving high rendering quality. During rendering, we apply an aggressive opacity reduction strategy to prevent the model from converging to local optima. To further enhance the rendering quality, we compute the error between the rendered image and the ground truth, and sample in the canonical field based on the depth map, thereby complementing the details in regions where the initial point cloud is sparse.
Fig. 2 : Visualization of the Learned Canonical Fields. Direct renderings of the undeformed canonical fields from scenes in the N3DV ( flame_salmon , cook_spinach ) and HyperNeRF ( chicken , peel-banana ) datasets. Each canonical rendering is paired with a corresponding ground truth frame. The high visual similarity across both multiview and monocular datasets validates our assumption that the learned canonical field provides a stable and coherent representation of the scene, forming a solid basis for our sampling strategy.
Fig. 3 : Comparison of depth computation methods. The visual results on the chicken scene clearly show that our chosen median depth (c) produces a cleaner depth map with fewer floaters compared to mean depth (d). Its structure aligns better with the pseudo-GT depth (b), which was generated by the SOTA model Depth Anything [ 64 ] .
Fig. 4 : Observation of 3DGs in dim scenes .
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Storage ↓
Training Time ↓
FPS ↑
Mix Voxels [ 30 ]
30.30
0.918
0.127
512 MB
1h 28m
1.01
K-Planes [ 19 ]
30.86
0.939
0.096
309 MB
1h 33m
0.15
D3DGS [ 48 ]
28.27
0.917
0.156
75MB
2h 17m
20.29
ST-GS † [ 28 ]
28.94
0.920
0.078
49 MB
4h 25m
124
Swift4D [ 55 ]
30.05
0.931
0.055
116MB
48m
138.00
4DGS [ 65 ]
30.30
0.933
0.069
3.6GB
7h 43m
54.36
TABLE I : Quantitative comparison on the N3DV dataset. Storage, training time, and FPS are calculated on flame_salmon_frag1 . † For a fairer comparison under our setting, ST-GS was evaluated using a single first-frame COLMAP initialization.
Fig. 5 : Qualitative comparisons on the N3DV dataset.
Fig. 6 : Qualitative comparisons on the HyperNeRF dataset.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Storage ↓
Training Time ↓
FPS ↑
Nerfies 1,2 [ 13 ]
22.23
-
0.170
-
∼ hours
< 1
HyperNeRF 1,2 [ 12 ]
22.29
0.598
0.153
15 MB
32h 45m
< 1
TiNeuVox 1,2 [ 16 ]
24.20
0.616
0.393
48 MB
30m
1
Ours 1,2
26.31
0.721
0.199
37 MB
57m
146.50
DN-4DGS 2 [ 54 ]
25.59
0.691
-
68 MB
1h 12m
20
Ours 2
26.31
0.721
0.199
51 MB
1h 28m
110.45
TABLE II : Quantitative comparison on the HyperNeRF dataset. Storage, training time, and FPS are tested across all four scenes, based on the NVIDIA RTX A6000. 1 Storage, training time, and FPS are tested only on Broom . 2 Storage, training time, and FPS are tested using an NVIDIA RTX 3090. For a fair comparison, we also report results using the specific settings of other methods.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Storage ↓
Training Time ↓
FPS ↑
DyNeRF 1,2 [ 15 ]
31.80
-
0.140
0.6MB
-
0.02
HyperReel 1,2 [ 67 ]
32.32
0.899
0.118
289 MB
2h 45m
0.91
Ours 1,2
33.28
0.907
0.105
43 MB
1h 51m
109.10
4DGS [ 65 ]
29.54
0.873
0.109
-
-
-
4DGaussians [ 53 ]
29.62
0.844
0.176
85 MB
27m
49.81
E-D3DGS [ 26 ]
33.08
0.902
0.110
55 MB
2h 35m
71.72
TABLE III : Quantitative comparison on the Technicolor dataset. Storage, training time, and FPS are tested on all five scenes and are calculated on an A6000. For storage, training time, and FPS, 1 are tested only on Painter , and 2 are measured using a 3090.
Fig. 7 : Qualitative comparisons on the Technicolor dataset.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Storage ↓
Training Time ↓
FPS ↑
Coarse
30.44
0.930
0.059
57 MB
1h 51m
102.95
Fine
30.38
0.927
0.062
59 MB
1h 49m
103.35
Concat
30.64
0.932
0.056
58 MB
1h 56m
92.03
Add
30.55
0.930
0.057
59 MB
1h 53m
92.57
Attention
30.74
0.935
0.053
61 MB
2h 5m
69.62
Dual
30.79
0.937
0.051
60 MB
2h 22m
49.60
TABLE IV : Comprehensive ablation study on the N3DV dataset. The top section compares various temporal embedding fusion strategies, while the bottom section demonstrates the incremental performance gains provided by the Canonical Field Sampling and Aggressive Opacity Reduction strategies.
Fig. 8 : Ablation studies on embedding fusion .
Fig. 9 : Ablation studies of Canonical Field Sampling. The top row shows results on the dynamic broom scene, and the bottom row shows results on the static background of the salmon scene. Our method (c) shows clear improvements in both cases.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
w/o Sampling
25.96
0.717
0.194
Inverse Deform
26.08
0.718
0.192
Ours
26.31
0.721
0.187
TABLE V : Comparison of different sampling strategies on the HyperNeRF dataset.
Fig. 10 : Ablation studies on adaptive opacity reduction .
k value
PSNR ↑
SSIM ↑
LPIPS ↓
0
30.86
0.939
0.052
1
30.89
0.940
0.052
5
30.93
0.940
0.051
10
31.01
0.941
0.050
20
30.80
0.935
0.055
TABLE VI : Sensitivity analysis of the parameter k in our aggressive opacity reduction strategy. The experiment was conducted on the N3DV dataset. k=0 corresponds to disabling the strategy. Our chosen value is highlighted in bold.
Fig. 11 : Illustration of limitations on challenging monocular scenes. Failure cases show blurry reconstructions and artifacts in regions with significant view information loss. (a) The Broom scene. (b) The 3D Printer scene.
Dynamic scene reconstruction and novel view synthesis are fundamental to next-generation visual intelligence applications such as virtual reality, robotics, and digital twins. However, high-fidelity reconstruction of complex, time-varying scenes from arbitrary viewpoints remains a significant challenge. Existing dynamic 3DGS methods suffer from computational inefficiency, since they model all Gaussians as dynamic components. While recent decomposition-based approaches address this issue, they still struggle with degraded reconstruction quality and prolonged training time. To mitigate these limitations, we propose a novel dynamic reconstruction framework built upon an efficient static-dynamic decomposition strategy using a Feed-Forward Gaussian Splatting encoder and an optical flow model. By eliminating redundant computations on static regions, our method achieves state-of-the-art performance, outperforming existing baselines across rendering quality, training and rendering speed, and storage efficiency. Notably, on the Neural 3D dataset, our framework requires only 10 minutes for training and achieves a rendering speed of over 700 FPS on a single NVIDIA RTX 5090 GPU at resolution of 1352x1014. Furthermore, our decomposition strategy eliminates the need for COLMAP preprocessing and enables deterministic initialization, thereby enhancing both efficiency and reproducibility.
Youngtae Han, Sung-hwan Han, Youngmin Yi
Department of Artificial Intelligence Engineering, Sogang University
Dynamic scene reconstruction via 3D Gaussian Splatting (3DGS) has emerged as a compelling approach for representing evolving environments, yet understanding trade-offs between methodologies remains crucial. This paper presents a comprehensive analysis of dynamic 3DGS methods, categorizing them into two paradigms: structure-guided methods employing auxiliary representations (deformation fields, canonical spaces, grids) to model temporal changes, and gaussian-centric methods encoding dynamics directly into primitives via continuous functions or 4D representations. We evaluate representative methods from both paradigms on the D-NeRF benchmark. Our findings reveal that structure-guided methods achieve superior reconstruction fidelity and compact model sizes, while gaussian-centric approaches demonstrate significantly higher rendering speeds enabling real-time performance, though with greater quality variability and potentially substantial storage overhead. This analysis highlights a fundamental trade-off between reconstruction quality/compactness versus rendering speed, providing insights to guide future research and application development in dynamic scene reconstruction.
Recent advancements in 3D Gaussian Splatting (3DGS) have demonstrated its potential for efficient and photorealistic 3D reconstructions, which is crucial for diverse applications such as robotics and immersive media. However, current Gaussian-based methods for dynamic scene reconstruction struggle with large inter-frame displacements, leading to artifacts and temporal inconsistencies under fast object motions. To address this, we introduce TrackerSplat, a novel method that integrates advanced point tracking methods to enhance the robustness and scalability of 3DGS for dynamic scene reconstruction. TrackerSplat utilizes off-the-shelf point tracking models to extract pixel trajectories and triangulate per-view pixel trajectories onto 3D Gaussians to guide the relocation, rotation, and scaling of Gaussians before training. This strategy effectively handles large displacements between frames, dramatically reducing the fading and recoloring artifacts prevalent in prior methods. By accurately positioning Gaussians prior to gradient-based optimization, TrackerSplat overcomes the quality degradation associated with large frame gaps when processing multiple adjacent frames in parallel across multiple devices, thereby boosting reconstruction throughput while preserving rendering quality. Experiments on real-world datasets confirm the robustness of TrackerSplat in challenging scenarios with significant displacements, achieving superior throughput under parallel settings and maintaining visual quality compared to baselines. The code is available at https://github.com/yindaheng98/TrackerSplat.
Daheng Yin, Isaac Ding, Yili Jin +2
Simon Fraser University, Burnaby, Canada · McGill University, Montreal, Canada · Nankai University, Tianjin, China