GaussLearner: 3D Gaussian Primitives as Self-Regulating Continual Learners
Organizations: Zhejiang University · Chengdu Minto Tech
Abstract
Continual learning of 3D Gaussian fields aims to maintain faithful reconstructions of the evolving world. However, directly fine-tuning an existing Gaussian field on new observations can cause catastrophic forgetting of previously reconstructed content. To address this challenge, existing methods employ selective local optimization, replay, or parameter regularization, yet still struggle to balance adaptation to scene changes with preservation of still-valid content. Our key insight is to view 3D Gaussian primitives themselves as self-regulating continual learners. Based on this insight, we introduce GaussLearner, which equips each Gaussian with a closed continual-learning loop connecting observation, self-regulation, and adaptation at the primitive level. At each novel state, GaussLearner first attributes pixel-wise inconsistencies in current observations to their contributing Gaussians. Each Gaussian then self-regulates through a learning weight that balances current observations against cross-state reliability. This weight governs both Gaussian parameter adaptation and optimizer memory refresh. As current inconsistencies are resolved, the adaptation outcome renews each Gaussian's reliability for future states. The resulting field is a globally coupled population of locally self-regulating learners. Across scenes from CL-Splats, WAT, and our CL-3D dataset, GaussLearner improves average PSNR by 3.41 dB over the strongest baseline. On CL-3D, it achieves the best reconstruction quality on both changed and unchanged views.
Figures & tables
| ClubRoom | FloorLounge | Fridge | OfficeHub | StudyNook | ApartmentRoom | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM |
| 3DGS [ 17 ] (TOG ’23) | 19.00 | 0.732 | 20.02 | 0.837 | 20.16 | 0.762 | 25.28 | 0.831 | 9.75 | 0.605 | 16.48 | 0.655 |
| CLNeRF [ 7 ] (ICCV ’23) | 22.33 | 0.833 | 25.51 | 0.920 | 24.95 | 0.831 | 26.23 | 0.881 | 21.49 | 0.809 | 20.87 | 0.769 |
| 3DGS + EWC [ 20 ] | 19.88 | 0.793 | 22.90 | 0.933 | 24.61 | 0.856 | 24.96 | 0.856 | 20.64 | 0.863 | 17.21 | 0.762 |
| 3DGS + MAS [ 2 ] | 20.13 | 0.796 | 24.25 | 0.940 | 24.51 | 0.856 | 25.08 | 0.867 | 20.67 | 0.863 | 17.15 | 0.764 |
| CL-Splats [ 1 ] (ICCV ’25) | 20.40 | 0.827 | 28.14 | 0.941 | 21.91 | 0.773 | 20.62 | 0.818 | 23.39 | 0.877 | 20.37 | 0.760 |
| Cupboard | Desk | Room | Shelf | |||||
|---|---|---|---|---|---|---|---|---|
| Method | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM |
| 3DGS [ 17 ] | 25.69 | 0.918 | 28.32 | 0.875 | 20.93 | 0.725 | 26.52 | 0.914 |
| CLNeRF [ 7 ] | 30.21 | 0.916 | 29.78 | 0.860 | 29.92 | 0.937 | 22.48 | 0.867 |
| 3DGS + EWC [ 20 ] | 30.59 | 0.954 | 27.68 | 0.884 | 21.05 | 0.731 | 29.64 | 0.939 |
| 3DGS + MAS [ 2 ] | 30.89 | 0.956 | 27.51 | 0.883 | 21.09 | 0.732 | 29.24 | 0.940 |
| CL-Splats [ 1 ] | 29.13 | 0.940 | 24.17 | 0.831 | 26.82 | 0.909 | 26.78 | 0.906 |
| Breville | Car | Grill | Mac | |||||
|---|---|---|---|---|---|---|---|---|
| Method | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM |
| 3DGS [ 17 ] | 30.60 | 0.942 | 27.80 | 0.871 | 27.46 | 0.866 | 31.02 | 0.949 |
| CLNeRF [ 7 ] | 29.20 | 0.870 | 22.54 | 0.590 | 24.13 | 0.675 | 28.46 | 0.917 |
| 3DGS + EWC [ 20 ] | 30.35 | 0.942 | 27.27 | 0.851 | 27.28 | 0.869 | 31.33 | 0.952 |
| 3DGS + MAS [ 2 ] | 30.19 | 0.939 | 27.19 | 0.847 | 26.96 | 0.864 | 31.30 | 0.951 |
| CL-Splats [ 1 ] | 22.52 | 0.837 | 22.70 | 0.769 | 19.08 | 0.721 | 25.87 | 0.904 |
| Changed | Unchanged | |||
|---|---|---|---|---|
| Variant | PSNR | SSIM | PSNR | SSIM |
| View-Wise Sum | 26.10 | 0.901 | 21.86 | 0.857 |
| 26.02 | 0.901 | 21.90 | 0.859 | |
| Pre-Adam Gradient Scaling | 25.90 | 0.897 | 21.75 | 0.851 |
| Frozen Reliability | 26.06 | 0.902 | 21.87 | 0.860 |
| Ungated Reliability Renewal | 26.05 | 0.902 | 21.88 | 0.860 |
| Variant | Changed PSNR | Unchanged PSNR | |||
|---|---|---|---|---|---|
| Dense Adam | 25.71 | 21.95 | No boundary refresh | ||
| Sparse Adam [ 23 ] | 25.71 | 21.37 | No boundary refresh | ||
| Uniform Refresh | 25.68 | 21.98 | 0.900 | 0.900 | 0.900 |
| 25.67 | 21.97 | 0.950 | 0.950 | 0.950 | |
| GaussLearner | 26.44 | 22.18 | 0.944 | 0.816 | 0.829 |
3D Gaussian Splatting
3D Gaussian Splatting represents scenes as optimizable 3D Gaussian primitives for fast, photorealistic rendering from new viewpoints.
See all 1,086 papers