3D Gaussian Splatting has demonstrated remarkable real-time rendering capabilities and superior visual quality in novel view synthesis for static scenes. Building upon these advantages, researchers have progressively extended 3D Gaussians to dynamic scene reconstruction. Deformation field-based methods have emerged as a promising approach among various techniques. These methods maintain 3D Gaussian attributes in a canonical field and employ the deformation field to transform this field across temporal sequences. Nevertheless, these approaches frequently encounter challenges such as suboptimal rendering speeds, significant dependence on initial point clouds, and vulnerability to local optima in dim scenes. To overcome these limitations, we present FRoG, an efficient and robust framework for high-quality dynamic scene reconstruction. FRoG integrates per-Gaussian embedding with a coarse-to-fine temporal embedding strategy, accelerating rendering through the early fusion of temporal embeddings. Moreover, to enhance robustness against sparse initializations, we introduce a novel depth- and error-guided sampling strategy. This strategy populates the canonical field with new 3D Gaussians at low-deviation initial positions, significantly reducing the optimization burden on the deformation field and improving detail reconstruction in both static and dynamic regions. Furthermore, by modulating opacity variations, we mitigate the local optima problem in dim scenes, improving color fidelity. Comprehensive experimental results validate that our method achieves accelerated rendering speeds while maintaining state-of-the-art visual quality.
Figures & tables
Fig. 1 : Overview of FRoG. We begin by performing early fusion on the coarse and fine temporal embeddings through the Hadamard product, aiming to enhance rendering speed while preserving high rendering quality. During rendering, we apply an aggressive opacity reduction strategy to prevent the model from converging to local optima. To further enhance the rendering quality, we compute the error between the rendered image and the ground truth, and sample in the canonical field based on the depth map, thereby complementing the details in regions where the initial point cloud is sparse.
Fig. 2 : Visualization of the Learned Canonical Fields. Direct renderings of the undeformed canonical fields from scenes in the N3DV ( flame_salmon , cook_spinach ) and HyperNeRF ( chicken , peel-banana ) datasets. Each canonical rendering is paired with a corresponding ground truth frame. The high visual similarity across both multiview and monocular datasets validates our assumption that the learned canonical field provides a stable and coherent representation of the scene, forming a solid basis for our sampling strategy.
Fig. 3 : Comparison of depth computation methods. The visual results on the chicken scene clearly show that our chosen median depth (c) produces a cleaner depth map with fewer floaters compared to mean depth (d). Its structure aligns better with the pseudo-GT depth (b), which was generated by the SOTA model Depth Anything [ 64 ] .
Fig. 4 : Observation of 3DGs in dim scenes .
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Storage ↓
Training Time ↓
FPS ↑
Mix Voxels [ 30 ]
30.30
0.918
0.127
512 MB
1h 28m
1.01
K-Planes [ 19 ]
30.86
0.939
0.096
309 MB
1h 33m
0.15
D3DGS [ 48 ]
28.27
0.917
0.156
75MB
2h 17m
20.29
ST-GS † [ 28 ]
28.94
0.920
0.078
49 MB
4h 25m
124
Swift4D [ 55 ]
30.05
0.931
0.055
116MB
48m
138.00
4DGS [ 65 ]
30.30
0.933
0.069
3.6GB
7h 43m
54.36
TABLE I : Quantitative comparison on the N3DV dataset. Storage, training time, and FPS are calculated on flame_salmon_frag1 . † For a fairer comparison under our setting, ST-GS was evaluated using a single first-frame COLMAP initialization.
Fig. 5 : Qualitative comparisons on the N3DV dataset.
Fig. 6 : Qualitative comparisons on the HyperNeRF dataset.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Storage ↓
Training Time ↓
FPS ↑
Nerfies 1,2 [ 13 ]
22.23
-
0.170
-
∼ hours
< 1
HyperNeRF 1,2 [ 12 ]
22.29
0.598
0.153
15 MB
32h 45m
< 1
TiNeuVox 1,2 [ 16 ]
24.20
0.616
0.393
48 MB
30m
1
Ours 1,2
26.31
0.721
0.199
37 MB
57m
146.50
DN-4DGS 2 [ 54 ]
25.59
0.691
-
68 MB
1h 12m
20
Ours 2
26.31
0.721
0.199
51 MB
1h 28m
110.45
TABLE II : Quantitative comparison on the HyperNeRF dataset. Storage, training time, and FPS are tested across all four scenes, based on the NVIDIA RTX A6000. 1 Storage, training time, and FPS are tested only on Broom . 2 Storage, training time, and FPS are tested using an NVIDIA RTX 3090. For a fair comparison, we also report results using the specific settings of other methods.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Storage ↓
Training Time ↓
FPS ↑
DyNeRF 1,2 [ 15 ]
31.80
-
0.140
0.6MB
-
0.02
HyperReel 1,2 [ 67 ]
32.32
0.899
0.118
289 MB
2h 45m
0.91
Ours 1,2
33.28
0.907
0.105
43 MB
1h 51m
109.10
4DGS [ 65 ]
29.54
0.873
0.109
-
-
-
4DGaussians [ 53 ]
29.62
0.844
0.176
85 MB
27m
49.81
E-D3DGS [ 26 ]
33.08
0.902
0.110
55 MB
2h 35m
71.72
TABLE III : Quantitative comparison on the Technicolor dataset. Storage, training time, and FPS are tested on all five scenes and are calculated on an A6000. For storage, training time, and FPS, 1 are tested only on Painter , and 2 are measured using a 3090.
Fig. 7 : Qualitative comparisons on the Technicolor dataset.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Storage ↓
Training Time ↓
FPS ↑
Coarse
30.44
0.930
0.059
57 MB
1h 51m
102.95
Fine
30.38
0.927
0.062
59 MB
1h 49m
103.35
Concat
30.64
0.932
0.056
58 MB
1h 56m
92.03
Add
30.55
0.930
0.057
59 MB
1h 53m
92.57
Attention
30.74
0.935
0.053
61 MB
2h 5m
69.62
Dual
30.79
0.937
0.051
60 MB
2h 22m
49.60
TABLE IV : Comprehensive ablation study on the N3DV dataset. The top section compares various temporal embedding fusion strategies, while the bottom section demonstrates the incremental performance gains provided by the Canonical Field Sampling and Aggressive Opacity Reduction strategies.
Fig. 8 : Ablation studies on embedding fusion .
Fig. 9 : Ablation studies of Canonical Field Sampling. The top row shows results on the dynamic broom scene, and the bottom row shows results on the static background of the salmon scene. Our method (c) shows clear improvements in both cases.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
w/o Sampling
25.96
0.717
0.194
Inverse Deform
26.08
0.718
0.192
Ours
26.31
0.721
0.187
TABLE V : Comparison of different sampling strategies on the HyperNeRF dataset.
Fig. 10 : Ablation studies on adaptive opacity reduction .
k value
PSNR ↑
SSIM ↑
LPIPS ↓
0
30.86
0.939
0.052
1
30.89
0.940
0.052
5
30.93
0.940
0.051
10
31.01
0.941
0.050
20
30.80
0.935
0.055
TABLE VI : Sensitivity analysis of the parameter k in our aggressive opacity reduction strategy. The experiment was conducted on the N3DV dataset. k=0 corresponds to disabling the strategy. Our chosen value is highlighted in bold.
Fig. 11 : Illustration of limitations on challenging monocular scenes. Failure cases show blurry reconstructions and artifacts in regions with significant view information loss. (a) The Broom scene. (b) The 3D Printer scene.