Organizations: School of Computer Science and Technology, University of Science and Technology of China, Hefei 230026, China · School of Artificial Intelligence and Data Science, University of Science and Technology of China, Hefei 230026, China · The Chinese University of Hong Kong, Hong Kong 999077, China · Institute of Advanced Technology, University of Science and Technology of China, Hefei 230026, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei 230088, China · School of information Science and Technology, University of Science and Technology of China, Hefei 230026, China
Reconstructing static 3D scene from monocular video with dynamic objects is important for numerous applications such as virtual reality and autonomous driving. Current approaches typically rely on background for static scene reconstruction, limiting the ability to recover regions occluded by dynamic objects. In this paper, we propose GA-GS, a Generation-Assisted Gaussian Splatting method for Static Scene Reconstruction. The key innovation of our work lies in leveraging generation to assist in reconstructing occluded regions. We employ a motion-aware module to segment and remove dynamic regions, and thenuse a diffusion model to inpaint the occluded areas, providing pseudo-ground-truth supervision. To balance contributions from real background and generated region, we introduce a learnable authenticity scalar for each Gaussian primitive, which dynamically modulates opacity during splatting for authenticity-aware rendering and supervision. Since no existing dataset provides ground-truth static scene of video with dynamic objects, we construct a dataset named Trajectory-Match, using a fixed-path robot to record each scene with/without dynamic objects, enabling quantitative evaluation in reconstruction of occluded regions. Extensive experiments on both the DAVIS and our dataset show that GA-GS achieves state-of-the-art performance in static scene reconstruction, especially in challenging scenarios with large-scale, persistent occlusions.
Figures & tables
Fig. 1: Comparison between the previous pipeline (a) and our proposed GA-GS (b). Previous methods supervise 3D Gaussian primitives solely based on background regions after dynamic object removal. In contrast, our GA-GS leverages a diffusion model to generate occluded content for auxiliary supervision and introduces authenticity-driven rendering to balance real and generated information.
Fig. 2: An overview of our GA-GS pipeline. We use VGGT [ 13 ] to obtain accurate camera poses and per-pixel 3D positions, Then we employ a motion-aware SAM-based module to segment moving regions, and use a diffusion model to inpaint occlusions, providing pseudo-ground-truth supervision. In the opacity blending stage, the parameter θ is used to control the opacity of each Gaussian primitive, and the image space mask is applied to constrain the final loss.
Fig. 3: Visualization on the DAVIS dataset. Since ground truth for the static background is unavailable, the first row only shows the input containing dynamic objects. Compared to the baselines, our method achieves better visual results in both background reconstruction and occlusion region recovery.
Blackswan
Camel
Car-shadow
Dog
Horse-jump
Motocross-jump
Parkour
Soapbox
Average
WildGaussians [ 11 ]
18.95
19.19
21.45
19.74
18.79
7.91
18.89
20.55
18.18
Robust3DGS [ 12 ]
19.58
21.31
29.31
22.48
20.87
13.83
21.29
22.55
21.40
DAS3R [ 28 ]
23.90
27.27
29.13
28.63
25.09
17.09
28.09
26.41
25.70
GA-GS (Ours)
25.75
27.69
31.95
29.15
25.67
20.42
28.74
29.72
27.39
TABLE I: Quantitative results on DAVIS dataset. Eight scenes from dataset are selected for evaluation. PSNR is computed over background regions, as defined by the ground-truth dynamic masks. Bold values indicate best method in average.
Fig. 4: Data acquisition process of the Trajectory-Match dataset. (a) We employ a robot-mounted camera platform to ensure precise and repeatable camera trajectories. (b) For each scene, we first capture a dynamic sequence containing moving objects such as pedestrians or vehicles. (c) We then record a corresponding static sequence along the same trajectory after removing dynamic elements, which serves as ground-truth background. This paired acquisition setup enables direct quantitative evaluation of reconstruction performance in occluded regions.
Fig. 5: Visualizations on Trajectory-Match Dataset. The second row presents the ground truth of recorded static scene, serving as a reference for comparing the reconstructions of GA-GS and the baseline in the third and fourth rows.
Metrics
Indoor
Outdoor
Street
Average
Indoor occluded
Outdoor occluded
Street occluded
Average occluded
PSNR ↑
19.47
20.10
21.55
20.37
15.82
15.07
15.29
15.39
WildGaussians
SSIM ↑
0.66
0.65
0.77
0.69
-
-
-
-
[ 11 ]
LPIPS ↓
0.27
0.42
0.37
0.35
-
-
-
-
PSNR ↑
19.45
18.89
20.61
19.65
16.11
15.73
14.98
15.61
Robust3DGS
SSIM ↑
0.61
0.58
0.70
0.63
-
-
-
-
[ 12 ]
LPIPS ↓
0.26
0.31
0.24
0.27
-
-
-
-
TABLE II: Comparison of static scene reconstruction performance on Trajectory-Match Dataset. Results are provided for full scene static reconstruction (indoor, outdoor, street, average) and dynamic occluded region reconstruction (indoor-occluded, outdoor-occluded, street-occluded, average-occluded). Bold values highlight the best-performing method in average.
w/o both
GA-GS w/o θ
GA-GS w/o generation
GA-GS
Indoor
24.32
24.68
24.97
25.34
Outdoor
23.72
24.07
24.13
24.67
Street
24.18
24.53
25.19
25.62
Average
24.07
24.43
24.76
25.21
Indoor-occl.
23.52
23.89
24.43
24.91
Outdoor-occl.
23.05
23.45
23.90
24.36
TABLE III: Quantitative results of ablation experiments on authenticity parameter and generation module. ’Occl.’ refers to occluded regions, θ denotes authenticity parameter. The column “w/o both” corresponds to removing both generation θ , i.e., using masked observations for training.
Method
Pose
Indoor
Outdoor
Street
Avg
DAS3R
COLMAP
21.45
20.07
22.31
21.28
VGGT
20.78
19.31
21.54
20.54
Ours
COLMAP
25.61
25.03
25.88
25.51
VGGT
25.34
24.67
25.62
25.21
TABLE IV: Impact of pose and initialization source. We compare COLMAP and VGGT for both our method and DAS3R.
Setting
Full PSNR
Full SSIM
Full LPIPS
Occl. PSNR
(0.5, 0.5)
24.46
0.84
0.13
23.92
(0.7, 0.3)
24.81
0.85
0.12
24.29
(0.8, 0.2)
25.03
0.85
0.11
24.56
(0.9, 0.1)
25.21
0.86
0.11
24.80
(0.95, 0.05)
25.08
0.85
0.12
24.48
TABLE V: Sensitivity to authenticity initialization. We vary the initial authenticity values assigned to Gaussian primitives from real and generated regions, respectively.
Tsinghua University, Shenzhen International Graduate School Shenzhen, Guangdong, China · Huawei Technologies Ltd Shenzhen, Guangdong, China · Harbin Institute of Technology, Shenzhen Shenzhen, Guangdong, China +1