Continual learning of 3D Gaussian fields aims to maintain faithful reconstructions of the evolving world. However, directly fine-tuning an existing Gaussian field on new observations can cause catastrophic forgetting of previously reconstructed content. To address this challenge, existing methods employ selective local optimization, replay, or parameter regularization, yet still struggle to balance adaptation to scene changes with preservation of still-valid content. Our key insight is to view 3D Gaussian primitives themselves as self-regulating continual learners. Based on this insight, we introduce GaussLearner, which equips each Gaussian with a closed continual-learning loop connecting observation, self-regulation, and adaptation at the primitive level. At each novel state, GaussLearner first attributes pixel-wise inconsistencies in current observations to their contributing Gaussians. Each Gaussian then self-regulates through a learning weight that balances current observations against cross-state reliability. This weight governs both Gaussian parameter adaptation and optimizer memory refresh. As current inconsistencies are resolved, the adaptation outcome renews each Gaussian's reliability for future states. The resulting field is a globally coupled population of locally self-regulating learners. Across scenes from CL-Splats, WAT, and our CL-3D dataset, GaussLearner improves average PSNR by 3.41 dB over the strongest baseline. On CL-3D, it achieves the best reconstruction quality on both changed and unchanged views.
Figures & tables
Figure 2: Comparison of 3D Gaussian continual learning paradigms. (a) Existing methods treat Gaussians as update targets. Replay and regularization may retain obsolete content or hinder adaptation, while selective local optimization can propagate 2D detection errors into 3D updates. (b) GaussLearner instead views each Gaussian primitive as a continual learner equipped with a closed learning loop.
Figure 3: Framework of GaussLearner . Each Gaussian acts as a stateful continual learner with scene parameters and cross-state reliability. At each scene state, pixel-wise inconsistencies in current observations are attributed to their contributing Gaussians. Each Gaussian then self-regulates through a per-Gaussian learning weight that balances current observations against cross-state reliability. This weight governs both parameter adaptation and optimizer-memory refresh. Cross-state reliability is updated at state boundaries using resolved pressure and final support. The resulting field is a globally coupled population of locally self-regulating continual learners.
Figure 4: Qualitative comparisons on CL-3D. From top to bottom are ClubRoom, Fridge, OfficeHub, and StudyNook. GaussLearner adapts to revised content while preserving still-valid regions, with less forgetting and fewer reconstruction artifacts.
ClubRoom
FloorLounge
Fridge
OfficeHub
StudyNook
ApartmentRoom
Method
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
3DGS [ 17 ] (TOG ’23)
19.00
0.732
20.02
0.837
20.16
0.762
25.28
0.831
9.75
0.605
16.48
0.655
CLNeRF [ 7 ] (ICCV ’23)
22.33
0.833
25.51
0.920
24.95
0.831
26.23
0.881
21.49
0.809
20.87
0.769
3DGS + EWC [ 20 ]
19.88
0.793
22.90
0.933
24.61
0.856
24.96
0.856
20.64
0.863
17.21
0.762
3DGS + MAS [ 2 ]
20.13
0.796
24.25
0.940
24.51
0.856
25.08
0.867
20.67
0.863
17.15
0.764
CL-Splats [ 1 ] (ICCV ’25)
20.40
0.827
28.14
0.941
21.91
0.773
20.62
0.818
23.39
0.877
20.37
0.760
Table 1: Novel-view synthesis results on CL-3D. For each scene, results are averaged equally across evaluated states. Higher is better for PSNR and SSIM. Best results are bold, and second-best results are underlined.
Figure 5: Comparison of adaptation and preservation. We report PSNR on changed and unchanged test views across all six scenes in the CL-3D dataset. Changed views measure adaptation to current observations, while unchanged views measure preservation of still-valid historical content.
Figure 6: Qualitative comparisons on WAT and CL-Splats. From top to bottom are Grill, Car, Cupboard, and Desk. Red boxes and enlarged crops highlight how GaussLearner incorporates revised scene content while preserving surrounding content that remains valid.
Cupboard
Desk
Room
Shelf
Method
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
3DGS [ 17 ]
25.69
0.918
28.32
0.875
20.93
0.725
26.52
0.914
CLNeRF [ 7 ]
30.21
0.916
29.78
0.860
29.92
0.937
22.48
0.867
3DGS + EWC [ 20 ]
30.59
0.954
27.68
0.884
21.05
0.731
29.64
0.939
3DGS + MAS [ 2 ]
30.89
0.956
27.51
0.883
21.09
0.732
29.24
0.940
CL-Splats [ 1 ]
29.13
0.940
24.17
0.831
26.82
0.909
26.78
0.906
Table 2: Novel-view synthesis results on CL-Splats. State 1 results are reported. Best results are bold, and second-best results are underlined.
Breville
Car
Grill
Mac
Method
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
3DGS [ 17 ]
30.60
0.942
27.80
0.871
27.46
0.866
31.02
0.949
CLNeRF [ 7 ]
29.20
0.870
22.54
0.590
24.13
0.675
28.46
0.917
3DGS + EWC [ 20 ]
30.35
0.942
27.27
0.851
27.28
0.869
31.33
0.952
3DGS + MAS [ 2 ]
30.19
0.939
27.19
0.847
26.96
0.864
31.30
0.951
CL-Splats [ 1 ]
22.52
0.837
22.70
0.769
19.08
0.721
25.87
0.904
Table 3: Novel-view synthesis results on WAT. For each scene, results are averaged equally across states 1–4. Best results are bold, and second-best results are underlined.
Figure 7: Within-state self-regulation. Mean pressure P and learning weight W over normalized update progress. Shading denotes standard error.
Changed
Unchanged
Variant
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
View-Wise Sum
26.10
0.901
21.86
0.857
W=P
26.02
0.901
21.90
0.859
Pre-Adam Gradient Scaling
25.90
0.897
21.75
0.851
Frozen Reliability
26.06
0.902
21.87
0.860
Ungated Reliability Renewal
26.05
0.902
21.88
0.860
Table 4: Ablating Observation Aggregation, Self-Regulation, Parameter Adaptation, and Reliability Renewal. Results are equally averaged over StudyNook, Fridge, and ClubRoom.
Variant
Changed PSNR ↑
Unchanged PSNR ↑
γˉ
ρm
ρv
Dense Adam
25.71
21.95
No boundary refresh
Sparse Adam [ 23 ]
25.71
21.37
No boundary refresh
Uniform Refresh γ=0.90
25.68
21.98
0.900
0.900
0.900
γ=0.95
25.67
21.97
0.950
0.950
0.950
GaussLearner
26.44
22.18
0.944
0.816
0.829
Table 5: Ablating Optimizer Memory Refresh for Subsequent Adaptation. Results are equally averaged over StudyNook, Fridge, and ClubRoom. γˉ is the mean assigned boundary-retention factor; ρm and ρv are the geometric-mean RMS retention of the first- and square-root second-moment states.
Current feed-forward 3D reconstruction methods predict pixel aligned Gaussian primitives, resulting in highly redundant representations. A natural solution is to prune the redundant Gaussians, but naive pruning introduces severe artifacts and often requires inference time fine-tuning, breaking the feed-forward paradigm. Based on previous works, high frequency regions require more Gaussian primitives, while low frequency regions can be represented with significantly fewer primitives. Motivated by this, we propose a novel approach to explicitly control the number of Gaussians by leveraging local texture information. Our approach achieves this through three key components: (1) texture estimation to capture spatial variation in scene detail, (2) texture-aware pruning that removes redundant Gaussians from low frequency regions, and (3) an adaptive Gaussian head that predicts the modified attributes of the retained primitives without breaking the feed-forward paradigm. Experiments on RE10K, ACID, DL3DV, Tanks and Temples, and DTU demonstrate the effectiveness of our approach, while ablation studies validate the contributions of its key components.
Badrinath Singhal, Srihari K G, Sreehari Iyer +2
Vision and AI Lab, Indian Institute of Science, Bangalore · Samsung R & D Institute India - Bangalore
Streaming 3D reconstruction demands both speed and temporal fidelity, goals that existing methods undermine by updating every Gaussian every frame, even in static regions. We present DecoGS, a method for efficient online training of 3D Gaussians from streaming videos. Unlike prior methods that update the entire scene indiscriminately, DecoGS introduces an adaptive mechanism that selectively focuses optimization on spatiotemporal regions exhibiting motion or photometric changes. This targeted training strategy eliminates redundant updates that cause flickering and drift in nominally static regions, while enabling fast, high-fidelity scene updates. The pipeline further integrates region-aware Gaussian management through gradient gating and efficient visibility filtering to maintain temporal coherence and a compact memory footprint. On N3DV and MeetRoom, DecoGS achieves 34.55 and 31.60 dB PSNR respectively, outperforming all streaming and offline baselines, while rendering at 261 FPS with 70× lower temporal flicker than the best prior method, requiring no large-scale pretraining.
3D Gaussian Splatting (3DGS) has become a vital tool in novel view synthesis. 3D Gaussian functions with attributes like color, opacity, and scale are stacked together to explicitly model a radiance field. However, discrete 3D Gaussians usually produce poor depth with few geometry details but lots of floaters, which hinders the effective modeling of smooth radiance fields. To address this issue, we propose to learn a radiance field within a continuous view-dependent color field parameterized by neural networks. With continuous constraints in color, the bias on low-frequency signals of neural networks can highly encourage the geometry modeling to contribute to high-frequency color variations on images, getting rid of solely relying on the color attribute itself. Moreover, we impose a constraint to improve the Lipschitz continuity of the neural color field, making large changes of each Gaussian's color also match with large changes of its position in terms of a ratio. To this end, we additionally introduce novel self-learning strategies to improve the learning of the neural view-dependent color field without any external knowledge or priors, aiming for better generalization on unseen scenes. Our numerical and visual comparisons on widely used benchmarks justify our idea and show better ability of high fidelity geometry recovery in 3DGS. Project page: https://github.com/yangyixi21/Neural-Color-Fields-3DGS.
Yixiao Yang, Yu-Shen Liu, Zhizhong Han
School of Software, Tsinghua University, Beijing, China · Department of Computer Science, Wayne State University, Detroit, USA