Continual Learning as Shared-Manifold Continuation Under Compatible Shift
Abstract
Continual learning methods usually preserve old behavior by regularizing parameters, matching old outputs, or replaying previous examples. These strategies can reduce forgetting, but they do not directly specify how the latent representation should evolve. We study a narrower geometric alternative for the regime where old and new data should remain on the same latent support: continual learning as continuation of a shared manifold. We instantiate this view within Support-Preserving Manifold Assimilation (SPMA) and evaluate a geometry-preserving variant, SPMA-OG, that combines sparse replay, output distillation, relational geometry preservation, local smoothing, and chart-assignment regularization on old anchors. On representative compatible-shift CIFAR10 and Tiny-ImageNet runs, SPMA-OG improves over sparse replay baselines in old-task retention and representation-preservation metrics while remaining competitive on new-task accuracy. On a controlled synthetic atlas-manifold benchmark, it achieves near-perfect anchor-geometry preservation while also improving new-task accuracy over replay. These results provide evidence that geometry-aware anchor regularization is a useful inductive bias when continual learning should preserve a shared latent support rather than create a new one.
Figures & tables
| Method | Replay | CIFAR10-Shift Old | CIFAR10-Shift New | CIFAR10-Shift Harm. | Tiny-Shift Old | Tiny-Shift New | Tiny-Shift Harm. |
|---|---|---|---|---|---|---|---|
| Plain FT | 640 | 0.7151 | 0.7748 | 0.7438 | 0.0838 | 0.3320 | 0.1338 |
| Anchor CE | 256 | 0.7977 | 0.7773 | 0.7874 | 0.1983 | 0.3214 | 0.2453 |
| ER-512 | 512 | 0.7948 | 0.7755 | 0.7850 | 0.2218 | 0.3254 | 0.2638 |
| SPMA-OG | 512 | 0.8195 | 0.7906 | 0.8048 | 0.3059 | 0.3250 | 0.3152 |
| Method | CIFAR10-Shift CKA | CIFAR10-Shift Dist. Corr. | CIFAR10-Shift Support-In | Tiny CKA | Tiny Dist. Corr. | Tiny Support-In |
|---|---|---|---|---|---|---|
| Plain FT | 0.8485 | 0.8166 | 0.0000 | 0.4200 | 0.4625 | 0.6183 |
| Anchor CE | 0.8816 | 0.8476 | 0.0000 | 0.4330 | 0.3639 | 0.6702 |
| ER-512 | 0.8966 | 0.8670 | 0.0000 | 0.4439 | 0.3962 | 0.5907 |
| SPMA-OG | 0.9209 | 0.8970 | 0.7950 | 0.7525 | 0.7773 | 0.7279 |
| Method | Old After | New After | Harmonic Mean | Anchor CKA | Anchor Dist. Corr. |
|---|---|---|---|---|---|
| Plain FT | 0.5800 | 0.8994 | 0.7052 | 0.9563 | 0.9470 |
| Replay Anchor | 0.9213 | 0.8712 | 0.8956 | 0.9813 | 0.9759 |
| Old Geometry | 0.9169 | 0.8719 | 0.8938 | 0.9883 | 0.9834 |
| SPMA-OG | 0.9269 | 0.8875 | 0.9068 | 0.9994 | 0.9992 |
| Method | Replay | Old Before | Old After | New After | Harmonic Mean |
|---|---|---|---|---|---|
| Plain FT | 12800 | 0.3482 | 0.0838 | 0.3320 | 0.1338 |
| Anchor CE | 256 | 0.3482 | 0.1983 | 0.3214 | 0.2453 |
| ER-512 | 512 | 0.3482 | 0.2218 | 0.3254 | 0.2638 |
| SPMA-OG | 512 | 0.3482 | 0.3059 | 0.3250 | 0.3152 |
Continual Learning
Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.
See all 304 papers