Preventing Latent Rehearsal Decay in Online Continual SSL with SOLAR

Giacomo Cignoni; Simone Magistri; Andrew D. Bagdanov; Antonio Carta

cs.LGApr 12, 2026

Preventing Latent Rehearsal Decay in Online Continual SSL with SOLAR

Authors: Giacomo Cignoni, Simone Magistri, Andrew D. Bagdanov, Antonio Carta

Organizations: Department of Computer Science, University of Pisa, Pisa, Italy · Media Integration and Communication Center (MICC), University of Florence, Italy

Abstract

This paper explores Online Continual Self-Supervised Learning (OCSSL), a scenario in which models learn from continuous streams of unlabeled, non-stationary data, where methods typically employ replay and fast convergence is a central desideratum. We find that OCSSL requires particular attention to the stability-plasticity trade-off: stable methods (e.g. replay with Reservoir sampling) are able to converge faster compared to plastic ones (e.g. FIFO buffer), but incur in performance drops under certain conditions. We explain this collapse phenomenon with the Latent Rehearsal Decay hypothesis, which attributes it to latent space degradation under excessive stability of replay. We introduce two metrics (Overlap and Deviation) that diagnose latent degradation and correlate with accuracy declines. Building on these insights, we propose SOLAR, which leverages efficient online proxies of Deviation to guide buffer management and incorporates an explicit Overlap loss, allowing SOLAR to adaptively managing plasticity. Experiments demonstrate that SOLAR achieves state-of-the-art performance on OCSSL vision benchmarks, with both high convergence speed and final performance.

Figures & tables

Figure 1: Motivation and overview. ( a ) Reservoir buffer, traditionally considered stability-focused, converges faster than FIFO but leads to degraded latent structure, manifesting as a phenomenon we call Latent Rehearsal Decay . ( b ) SSL methods can be viewed as solving an instance-discrimination problem: each sample and its augmentations form a hyperball in the embedding space. The spread of each hyperball ( Deviation ) and their pairwise Overlap are metrics that reveal the deterioration in feature space quality caused by Latent Rehearsal Decay. ( c ) Our proposed SOLAR method improves both Deviation and Overlap metrics and yields better overall performance.

Figure 2: Impact of training length. (a) Reports the Final Accuracy for Reservoir and FIFO, on CIFAR-100, trained on an OCSSL stream with varying training length, which is expressed as number of minibatch passes (i.e. training steps) per incoming stream minibatch. Reservoir converges faster for shorter schedules, but suffers unusual performance drops for longer schedules. (b) Gives the average per-task probing accuracy on past and future tasks relative to the current task (CIFAR-100). Past task accuracy does not decrease and both FIFO and Reservoir achieve similar results, excluding final Reservoir drop (explained later), implying that forgetting has little impact.

Figure 3: Latent Rehearsal Decay . Accuracy and metrics calculated on the entire training stream of ImageNet100. Both FIFO and Reservoir have an initial Convergence phase , where Accuracy increases and both metrics improve. After a certain point, Reservoir transitions into the Latent Rehearsal Decay phase , where the sudden drop in accuracy is preceded by a sharp decrease in Deviation and increase in Overlap. See Appendix C.2 for further analysis.

Figure 4: Overview of SOLAR. ( a ) The deviation-aware buffer tracks average representation, loss, and extraction count for each sample; ( b ) Low-deviation representations are discarded; ( c ) Sampling is inversely proportional to extraction counts; ( d ) Sampled images are augmented and used for SSL training with $\mathcal{L}_{\textit{SSL}}$ ; ( e ) The top- $K$ samples with highest loss are extracted to compute Overlap loss $\mathcal{L}_{\text{overlap}}$ ( f ).

			CIFAR-100 (20 exps)		ImageNet100 (20 exps)		CLEAR100 (11 exps)
Method	Buffer	Distill.	Final Acc.	Avg. Acc.	Final Acc.	Avg. Acc.	Final Acc.	Avg. Acc.
ER	Reservoir	✗	$44.4\pm{\scriptstyle 1.0}$	$39.6\pm{\scriptstyle 0.3}$	$40.5\pm{\scriptstyle 1.5}$	$39.3\pm{\scriptstyle 0.4}$	$47.1\pm{\scriptstyle 0.2}$	$35.3\pm{\scriptstyle 0.4}$
ER	FIFO	✗	$\underline{49.0\pm{\scriptstyle 0.3}}$	$39.3\pm{\scriptstyle 0.2}$	$48.7\pm{\scriptstyle 1.0}$	$38.9\pm{\scriptstyle 0.3}$	$45.3\pm{\scriptstyle 1.8}$	$34.9\pm{\scriptstyle 0.5}$
MinRed	MinRed	✗	$46.5\pm{\scriptstyle 0.3}$	$\underline{40.9\pm{\scriptstyle 0.3}}$	$48.0\pm{\scriptstyle 0.3}$	$\underline{42.5\pm{\scriptstyle 0.2}}$	$\underline{51.3\pm{\scriptstyle 0.1}}$	$39.7\pm{\scriptstyle 0.1}$
LARS	LARS	✗	$47.0\pm{\scriptstyle 0.7}$	$39.5\pm{\scriptstyle 0.2}$	$43.2\pm{\scriptstyle 2.8}$	$39.2\pm{\scriptstyle 0.2}$	$44.7\pm{\scriptstyle 0.9}$	$34.1\pm{\scriptstyle 0.1}$
PER	PER	✗	$48.5\pm{\scriptstyle 0.2}$	$39.1\pm{\scriptstyle 0.2}$	$48.8\pm{\scriptstyle 0.6}$	$38.8\pm{\scriptstyle 0.2}$	$46.0\pm{\scriptstyle 0.5}$	$35.0\pm{\scriptstyle 0.2}$

Table 1: Results on streaming online CIFAR-100 (20 experiences), ImageNet-100 (20 experiences), and CLEAR100 (11 experiences). Best in bold , second best underlined . We report mean and std over 3 runs.

Figure 5: Performance at different training lengths. SOLAR outperforms competitors independently of training length.

Figure 6: Impact of buffer size.

Figure 7: Training curves across buffer sizes

|\mathcal{M}|

on CIFAR-100. Smaller buffer corresponds to stronger Latent Rehearsal Decay. SOLAR beats FIFO and Reservoir across the training stream.

		CIFAR-100	ImageNet100
Buffer	$\mathcal{L}_{\text{overlap}}$	Final/Avg.	Final/Avg.
Reservoir	✗	$44.4/39.6$	$40.5/39.3$
Reservoir	✓	$46.3/40.3$	$35.8/37.6$
FIFO	✗	$49.0/39.3$	$48.7/38.9$
FIFO	✓	$46.9/38.4$	$45.4/37.9$
Deviation-Aware	✗	$47.7/41.4$	$46.9/\mathbf{42.8}$

Table 2: Ablation on buffer type and Overlap Loss

\mathcal{L}_{\text{overlap}}

Method	Final Acc.	Avg. Acc.
pretrained	$36.42$	$-$
Reservoir	$42.67$	$40.45$
FIFO	$43.91$	$40.77$
SOLAR	$\mathbf{44.11}$	$\mathbf{41.73}$

Table 3: iNaturalist results.

Panel of Figure 8 — Figure 8: Probing accuracy of each task at the end of each experience in the OCSSL stream (CIFAR-100) for Reservoir (top) and FIFO (bottom). The vertical dashed line indicates the point when the corresponding task just ended in the training stream.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Reports the Final Accuracy for Reservoir and FIFO, on ImageNet100, trained on an OCSSL stream with varying training length, which is expressed as number of minibatch passes (i.e. training steps) per incoming stream minibatch. Reservoir converges faster for shorter schedules, but suffers unusual performance drops for longer schedules.

Figure 10: The Figure shows Deviation, Overlap probing accuracy, and smoothed training SSL loss during OCSSL training on ImageNet100. We observe loss instabilities in correspondence with degradation of our metrics, especially for Reservoir.

Figure 11: Accuracy and Latent Rehearsal Decay metrics on CIFAR-100. Both FIFO and Reservoir have an initial Convergence phase , in which Accuracy increases and both metrics improve. After a certain point, Reservoir transitions into the Latent Rehearsal Decay phase , in which the sudden drop in accuracy is preceded by a sharp decrease in Deviation and increase in Overlap.

Figure 12: Plot showing no evidence of feature collapse for Reservoir on the CIFAR-100 test set expressed using the metrics based on analyzing SVD decompositions of feature representations ( Li et al., 2022 ) .

Figure 13: Uniformity loss, a measure of SSL collapse ( Wang & Isola, 2020 ) , calculated on past, future and current experiences for Reservoir, FIFO and SOLAR on ImageNet100.

Figure 14: Probing accuracy, Deviation and Overlap metrics calculated on the entire training stream and only on buffer samples (training on ImageNet100), for both Reservoir and FIFO. We observe similar curves for metrics on the buffer and on the entire training data.

Figure 15: Online versus true Overlap calculated for SOLAR on ImageNet100, with

|\mathcal{M}|=2000

. Online Overlap estimation is a good proxy for the true Overlap . Online Overlap closely matches the true Overlap calculated on the buffer only, demonstrating that the slight mismatch is only due to the buffer not being representative enough of the entire stream.

Figure 16: Ablation on

\omega

, the hyperparameter controlling overlap loss

\mathcal{M}_{\text{overlap}}

strength. The mean and standard deviation of Final and Average Accuracy for ImageNet100 are reported.

Table 4: Ablation study on effect of extraction criterion (left) and use of EMA to update buffer metrics (right).

Method	Update Policy	Extract Policy	Computational Scalability
MIR	Reservoir	Max loss interference	✗
GSS	Max gradient diversity	Random	✗
PER	FIFO	Soft priority on highest loss	✓
LARS	Reservoir + lowest-loss elimination	Random	✓
Deviation-Aware	Lowest loss	Soft priority on lowest $e_{i}^{\mathcal{M}}$	✓

Appendix

Table 5: Comparison of components of methods prioritizing “hard” samples in memory replay.

Figure 17: Dynamic regularization versus fixed distillation. These training curves on ImageNet100 show that CLA-E converges slowly when compared to SOLAR, while CLA-R, despite fast convergence, suffers from Latent Rehearsal Decay.

Figure 18: Changing buffer sizes on ImageNet100. We note that MinRed performance is greatly impacted by the reduced buffer size.

Method	Time ( s )
Reservoir	2146
FIFO	2133
MinRed	2061
LARS	2150
PER	2215
SCALE	4288

Appendix

Table 6: Training times (in seconds) of compared methods calculated on a single task on ImageNet100.

			CIFAR-100 (20 exps)		ImageNet100 (20 exps)
Method	Buffer	Distill.	Final Acc.	Avg. Acc.	Final Acc.	Avg. Acc.
ER	Reservoir	✗	$45.16$	$42.03$	$47.90$	$43.02$
ER	FIFO	✗	$\underline{46.35}$	$41.81$	$\mathbf{49.38}$	$42.12$
MinRed	MinRed	✗	$43.92$	$\underline{43.05}$	$45.54$	$\underline{43.30}$
SCALE	PSA	✓	$32.15$	$27.28$	$36.45$	$31.48$
SOLAR	Deviation-Aware	✗	$\mathbf{46.62}$	$\mathbf{43.35}$	$\underline{49.10}$	$\mathbf{43.53}$

Appendix

Table 7: Results on streaming online CIFAR-100 (20 experiences) and ImageNet-100 (20 experiences), with SimCLR backbone. Best in bold , second best underlined .

Figure 19: Probing accuracies during training on CIFAR100 with SimCLR.

Figure 20: Figure reports metrics based on analyzing SVD decompositions of feature representations ( Li et al., 2022 ) for Reservoir and FIFO. Features are extracted at the beginning, middle and end of OCSSL training on CIFAR100 (each row of plots shows metrics for each training checkpoint). We include lines for features SVD on current, past and future experiences, relative to the training checkpoint experience.

Online Continual Learning

Online continual learning adapts models to sequential data streams while preserving useful knowledge and limiting forgetting.

See all 106 papers

From Generic to Dedicated: A Novel Optimizer for Online Continual Learning
Oct 20, 2026Yongyi Wu, Zheng Wang, Sen LinOnline Continual Learning Catastrophic Forgetting
University of Houston Houston, USA
CLIMB: Centroid-Based Hierarchical Memory for Online Continual Self-Supervised Learning
Jun 30, 2026Julien Lefebvre, Stefan Duffner, Mathieu LefortSelf-Supervised Learning Replay-Based Continual Learning
Universite Claude Bernard Lyon 1, CNRS, INSA Lyon, LIRIS, UMR 5205, 69622 Villeurbanne, France · INSA Lyon, CNRS, Universite Claude Bernard Lyon 1, LIRIS, UMR 5205, 69621 Villeurbanne, France · Universite Claude Bernard Lyon 1 / Univ. Rennes, Inria, CNRS, LIRIS UMR 5205 / IRISA UMR 6074, France
MANGO: Meta-Adaptive Network Gradient Optimization for Online Continual Learning
May 18, 2026Ankita Awasthi, Marco Apolinario, Kaushik RoyClass-Incremental Learning Meta-Learning
Purdue University, USA · TU Delft, Netherlands

Oct 20, 2026J/K move · Enter open · S save

From Generic to Dedicated: A Novel Optimizer for Online Continual Learning

Yongyi Wu, Zheng Wang, Sen Lin

University of Houston Houston, USA

Online Continual Learning Catastrophic Forgetting

Open paper AskPDF arXiv

Online Continual Learning (OCL) requires models to learn from non-stationary data streams under a strict single-pass constraint, making them highly susceptible to catastrophic forgetting. While existing studies have explored various strategies from data, architecture, and optimization perspectives, the optimizer often directly uses generic approaches such as SGD and AdamW. In this work, we reveal a new link between optimization dynamics and OCL needs by recasting Polyak-Ruppert averaging as the engine of "plasticity" and Primal averaging as the anchor of "stability". This inspires our novel optimizer, SPIN (Stability-Plasticity INterpolation), which explicitly decouples plasticity and stability by interpolating between a fast-moving "plasticity" sequence and an adaptive "stability" sequence. We theoretically demonstrate that SPIN implicitly employs an inverse Hessian approximation, providing crucial Tikhonov regularization to damp noisy, single-sample Hessian estimates and mitigate forgetting. More importantly, SPIN can be seamlessly integrated into existing OCL methods by taking resulted gradients as input and replacing standard optimizers. Extensive experiments show that using SPIN instead of standard optimizers yields consistent and significant performance gains across multiple OCL approaches, including the challenging rehearsal-free setting and under strict memory constraints.

Abstract

Online Continual Learning

Explore similar work

From Generic to Dedicated: A Novel Optimizer for Online Continual Learning

CLIMB: Centroid-Based Hierarchical Memory for Online Continual Self-Supervised Learning

MANGO: Meta-Adaptive Network Gradient Optimization for Online Continual Learning