Federated 3D Gaussian Splatting for Large-Scale Scene Reconstruction at Wireless Edge
Authors: Guanlin Wu, Chao Hu, Pu Chen, Juyong Zhang, Han Hu, Shuguang Cui, Jie Xu
Organizations: School of Science and Engineering, the Shenzhen Future Network of Intelligence Institute, and the Guangdong Provincial Key Laboratory of Future Networks of Intelligence, The Chinese University of Hong Kong (Shenzhen), Longgang, Shenzhen 518172, China · Department of Computer Science, University of Liverpool, Liverpool, L697ZX, UK · School of Mathematical Science, University of Science and Technology of China, Hefei 230026, China · School of Information and Electronics, Beijing Institute of Technology, Beijing 100081, China
Three-dimensional (3D) Gaussian splatting (3D-GS) has emerged as a promising technique for large-scale scene reconstruction due to its high rendering efficiency and fidelity. However, the training of large-scale 3D-GS models at wireless edge faces various technical challenges including the limited communication, computation, and graphics processing unit (GPU) memory resources at edge devices, the structural inconsistency issue across local models hindering their effective aggregation, as well as privacy leakage risks associated with raw visual content and camera parameters. To address these challenges, this paper proposes a novel resource-efficient federated learning framework for efficiently training 3D-GS models of large scenes under severe resource constraints. First, we propose an on-device model lightweighting mechanism that adaptively selects and prunes Gaussian points to balance the rendering quality and training efficiency. In this mechanism, we quantitatively evaluate the importance of different Gaussian points at each device to facilitate the pruning, and use a novel importance-to-latency ratio criterion to determine the number of pruned Gaussian points under GPU memory and computation/communication latency constraints. Furthermore, we develop a 3D-GS model recovery mechanism that restores structural consistency across local 3D-GS models without accessing private camera parameters, enabling their effective aggregation towards a global model. Finally, extensive experiments show that our approach significantly accelerates convergence, maintains high rendering quality, and reduces training latency compared to state-of-the-art federated 3D-GS baselines.
Figures & tables
Fig. 1: Illustration of the proposed federated 3D-GS learning framework. At each communication round, the edge server broadcasts the global 3D-GS model to distributed devices; each device then performs importance-to-latency ratio-based point selection, local lightweighted training, and uploads the updated local model, position gradients, and binary mask to the server for recovery and aggregation.
Fig. 2: Distribution and range of importance scores for Gaussian points in an example from experiments with 5 devices and a total of 3,222,794 Gaussian points.
Uk ( ×106 )
Dk
Predicted [GB]
Measured [GB]
Rel. err. [%]
0.756472
320
5.309
5.265
0.84
0.934897
320
5.392
5.329
1.18
1.132375
320
5.484
5.426
1.07
TABLE I: Validation of the Analytical Peak-Memory Model in ( 19 ) Against the Measured Peak GPU Memory During Local 3D-GS Training.
Tsplat(s)
Trender(s)
Measured
Estimated
Measured
Estimated
RTX 3090
1.5530
2.4728
41.3355
43.8591
RTX 3060
4.1076
5.9211
72.3471
75.9602
TABLE II: Estimation and Measurement of Latency for Two Stages (Splatting and Rendering) for 100 Local Epochs.
Fig. 3: Comparison of training loss convergence versus latency on (a) the Building scene, (b) the Rubble scene, and (c) the Polytech scene.
Fig. 4: Comparison of different training designs in terms of PSNR, SSIM, and LPIPS w.r.t. total latency in the Building scene.
Fig. 5: Comparison of different training designs in terms of PSNR, SSIM, and LPIPS w.r.t. total latency in the Rubble scene.
Fig. 6: Comparison of different training designs in terms of PSNR, SSIM, and LPIPS w.r.t. total latency in the Polytech scene.
Scenes
Building
Rubble
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Fed3D-GS
17.21
0.567
0.375
19.18
0.552
0.429
Proposed
17.91
0.554
0.363
20.74
0.628
0.438
TABLE III: Comparison of Reconstruction Quality With Fed3D-GS.
Method
Building
Rubble
Latency
Points
Memory
Latency
Points
Memory
3D-GS
9.27
4330717
7.18
7.71
3154812
6.59
Fed3D-GS
1.83
1967079
5.93
1.69
1001575
4.68
Proposed
1.51
1358773
5.69
1.39
1260379
4.81
TABLE IV: Comparison of Total Latency (in Hours), Number of Gaussian Points in local model, and Peak GPU Memory Consumption (in GB) for One Device after 30,000 Training Epochs.
Fig. 7: Rendering quality and training loss w.r.t. total latency on the Building scene under moderate/severe camera heterogeneity.
Setting
PSNR ↑
SSIM ↑
LPIPS ↓
Homogeneous (ours)
17.91
0.554
0.363
Moderate heterogeneity
16.92
0.525
0.466
Severe heterogeneity
16.88
0.518
0.471
TABLE V: Reconstruction Quality Under Device Heterogeneity on the Building Scene.
Communication format
PSNR ↑
SSIM ↑
LPIPS ↓
FP32 (ours)
17.91
0.554
0.363
FP16
17.19
0.536
0.448
INT8
11.03
0.290
0.766
TABLE VI: Round-Wise Communication-Precision Ablation on the Building Scene.
Fig. 8: Comparison of total latency required to first reach a PSNR of 17 dB in the Building scene and 19 dB in the Rubble scene. Each bar is decomposed into communication latency, local training latency, and retraining latency (if applicable). Schemes (a)–(d) correspond to benchmark methods: (a) Fed3D-GS, (b) full model training design, (c) full memory load with recovery design, and (d) lightweighting only design, while (e) denotes the proposed design.
Fig. 9: Comparison of the proposed method and Fed3D-GS (with different numbers of local training epochs) in terms of total latency w.r.t. varying server computation resource.
Fig. 10: Qualitative comparison of novel view rendering among 3D-GS, Fed3D-GS, the lightweighting only design, and the proposed method. Our proposed design efficiently suppresses visual artifacts and achieves better reconstruction quality compared with the other benchmark schemes. The first row shows the results on the building scene, while the second and third rows show the results on the rubble scene.