From Generic to Dedicated: A Novel Optimizer for Online Continual Learning

Yongyi Wu; Zheng Wang; Sen Lin

cs.LGOct 20, 2026

From Generic to Dedicated: A Novel Optimizer for Online Continual Learning

Authors: Yongyi Wu, Zheng Wang, Sen Lin

Organizations: University of Houston Houston, USA

Abstract

Online Continual Learning (OCL) requires models to learn from non-stationary data streams under a strict single-pass constraint, making them highly susceptible to catastrophic forgetting. While existing studies have explored various strategies from data, architecture, and optimization perspectives, the optimizer often directly uses generic approaches such as SGD and AdamW. In this work, we reveal a new link between optimization dynamics and OCL needs by recasting Polyak-Ruppert averaging as the engine of "plasticity" and Primal averaging as the anchor of "stability". This inspires our novel optimizer, SPIN (Stability-Plasticity INterpolation), which explicitly decouples plasticity and stability by interpolating between a fast-moving "plasticity" sequence and an adaptive "stability" sequence. We theoretically demonstrate that SPIN implicitly employs an inverse Hessian approximation, providing crucial Tikhonov regularization to damp noisy, single-sample Hessian estimates and mitigate forgetting. More importantly, SPIN can be seamlessly integrated into existing OCL methods by taking resulted gradients as input and replacing standard optimizers. Extensive experiments show that using SPIN instead of standard optimizers yields consistent and significant performance gains across multiple OCL approaches, including the challenging rehearsal-free setting and under strict memory constraints.

Figures & tables

Figure 1: The comparison of plasticity (Learning Accuracy) and stability (Average Forgetting) of Polyak-Ruppert averaging and Primal averaging on split CIFAR-100. Learning Accuracy for the

j

-th task is the accuracy evaluated on the test set of task

j

immediately after the network has been trained sequentially from task

1

up to task

j

Method Family	Hessian Approximation
Regularization (EWC) [ 27 ]	$\approx(\sum_{i=1}^{j-1}H_{i})^{-1}$
Meta-CL (La-MAML) [ 19 ]	$\approx(H_{M}^{j})^{-1}$
SPIN	$\approx\gamma(I+\rho_{t}H_{t})^{-1}$

Table 1: A Unified View of CL methods as Hessian Approximation.

Figure 2: Empirical validation of SPIN. (a) Verification of the quasi-steady state assumption. (b) Stability of the update L2 norm compared to a baseline optimizer.

		Split CIFAR-100 (10 tasks, M=2k)						Split Tiny-ImageNet (20 tasks, M=5k)
Baseline		SGD	SPIN-SGD	RMSprop	Adam	AdamW	SPIN-AdamW	SGD	SPIN-SGD	RMSprop	Adam	AdamW	SPIN-AdamW
ER	(2019)	$27.94$	33.31	$26.38$	$31.02$	$33.07$	35.92	$21.68$	25.24	$19.25$	$23.38$	$23.90$	25.94
DER++	(2020)	$26.10$	35.90	$27.17$	$30.84$	$31.11$	36.06	$18.95$	27.27	$23.37$	$26.34$	$26.69$	29.79
COPE	(2021)	$30.81$	34.18	$31.12$	$32.90$	$33.55$	35.63	$18.31$	20.43	$18.01$	$19.80$	$20.77$	21.66
SCR	(2021)	$29.82$	31.53	$28.07$	$30.52$	$31.32$	32.86	$17.26$	18.77	$17.11$	$18.30$	$18.74$	20.39
ER-ACE	(2022)	$28.54$	36.14	$27.65$	$32.43$	$34.77$	37.02	$24.95$	28.11	$23.75$	$28.10$	$28.52$	30.42

Table 2: Comparison of final Average Accuracy (Acc. %) using different optimizers on Split CIFAR-100 (M=2k) and Split Tiny-ImageNet (M=5k) . Our SPIN is benchmarked against 4 standard optimizers across 11 SOTA OCL methods. The best results are highlighted using bold font.

		Split CIFAR-100 (10 tasks, M=2k)						Split Tiny-ImageNet (20 tasks, M=5k)
Baseline		SGD	SPIN-SGD	RMSprop	Adam	AdamW	SPIN-AdamW	SGD	SPIN-SGD	RMSprop	Adam	AdamW	SPIN-AdamW
ER	(2019)	$36.86$	46.18	$34.54$	$40.68$	$41.50$	48.08	$28.36$	34.38	$28.14$	$32.47$	$32.81$	37.33
DER++	(2020)	$35.16$	45.49	$33.68$	$37.99$	$38.86$	46.26	$21.20$	35.39	$30.50$	$36.49$	$36.87$	40.08
COPE	(2021)	$37.76$	44.62	$34.09$	$39.48$	$42.01$	47.38	$26.49$	26.80	$24.44$	$28.54$	$27.99$	31.01
SCR	(2021)	$42.55$	45.70	$37.74$	$41.24$	$42.44$	48.20	$24.03$	26.01	$23.15$	$26.19$	$26.24$	28.30
ER-ACE	(2022)	$38.38$	45.91	$38.48$	$41.26$	$42.05$	46.61	$32.20$	37.25	$31.56$	$36.55$	$37.54$	40.03

Table 3: Comparison of Averaged Anytime Accuracy (AAA %) using different optimizers on Split CIFAR-100 (M=2k) and Split Tiny-ImageNet (M=5k) . Our SPIN is benchmarked against 4 standard optimizers across 11 SOTA OCL methods. The best results are highlighted using bold font.

Table 6

Figure 3: Task-wise Learning Accuracy and Relative Forgetting of the ER method (

M=2000

), comparing the performance of AdamW and SPIN.

Table 8

Table 10

Panel of Figure 5 — Figure 5: Comparison between parameter-space interpolation and gradient-space averaging in OCL.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Trajectories of Polyak-Ruppert Averaging and Primal Averaging. Polyak-Ruppert Averaging computes the gradient at the current, fast-moving model iterate $\theta_{t}$ : $g_{t}=\nabla\mathcal{L}(\theta_{t})$ . The final model is the average of this noisy sequence: $\bar{\theta}_{T}=\frac{1}{T}\sum_{t=1}^{T}\theta_{t}$ . Primal Averaging computes the gradient at the averaged iterate over the current trajectory: $g_{t}=\nabla\mathcal{L}(\bar{\theta}_{t})$ , where $\bar{\theta}_{t}=\frac{1}{t}\sum_{i=1}^{t}\theta_{i}$ . This stable gradient is then used to guide the update.

Figure 7: The pipeline of SPIN. The gradient $\nabla\mathcal{L}$ at the model interpolation is obtained from the upstream design of OCL algorithms.

Method	SGD	SPIN-SGD	AdamW	SPIN-AdamW
ER	0.01	0.01	0.0005	0.0005
DER++	0.03	0.03	0.0005	0.0005
COPE	0.03	0.03	0.0005	0.0005
SCR	0.05	0.05	0.005	0.005
ER-ACE	0.01	0.01	0.0005	0.0005
OCM	0.05	0.05	0.001	0.001

Appendix

Table 11: Learning rates for each OCL method and optimizer.

Method	HP	Values
ER	optimizer	[SGD, AdamW, SPIN, RMSProp, Adam]
	lr	[0.1, 0.3, 0.05, 0.01, 0.005, 0.001, 0.0005, 0.0001]
	weight decay	[1e-4]
	momentum (for SGD)	[0.9]
	aug. strat.	[partial, full]
DER++	optimizer	[SGD, AdamW, SPIN, RMSProp, Adam]

Appendix

Table 12: Hyperparameter for part of baseline methods.

Metric	AdamW	SPIN
Acc (%)	25.89	26.72
AAA (%)	33.65	34.98

Appendix

Table 13: Optimization performance on the ViT architecture.

Figure 8: Analysis of hyperparameter robustness and comparisons with different baselines.

Online Continual Learning

Online continual learning adapts models to sequential data streams while preserving useful knowledge and limiting forgetting.

See all 106 papers

MANGO: Meta-Adaptive Network Gradient Optimization for Online Continual Learning
May 18, 2026Ankita Awasthi, Marco Apolinario, Kaushik RoyClass-Incremental Learning Meta-Learning
Purdue University, USA · TU Delft, Netherlands
Natural Gradient Descent for Online Continual Learning
Mar 21, 2026Joe Khawand, David ColliauxKronecker-Factored Approximation Online Continual Learning
Ecole Polytechnique & Télécom Paris Paris, France · Sony Computer Science Laboratories Paris, France
Fast and Slow Variational Continual Learning
Jun 22, 2026Subarnaduti Paul, Yohan Jung, Mohammad Emtiyaz Khan +3Continual Learning for LLMs Deep Learning Optimization
Faculty of Mathematics and Computer Science, University of Bremen, Bremen, Germany · Department of Computer Science & AI, Jeonbuk National University, Republic of Korea · Department of Computer Science, TU Darmstadt, Darmstadt, Germany +3

May 18, 2026J/K move · Enter open · S save

MANGO: Meta-Adaptive Network Gradient Optimization for Online Continual Learning

Ankita Awasthi, Marco Apolinario, Kaushik Roy

Purdue University, USA · TU Delft, Netherlands

Class-Incremental Learning Meta-Learning Continual Learning Replay-Based Continual Learning Online Continual Learning Incremental Learning+1

Open paper AskPDF arXiv

In Online Continual Learning (OCL), a neural network sequentially learns from a non-stationary data stream in a single-pass with access only to a limited memory replay buffer. This contrasts sharply with off-line continual learning where training is multiple epoch dependent on large datasets. The main challenge faced by OCL is to overcome catastrophic forgetting of past tasks (stability) while learning new ones efficiently (plasticity). Existing methods counter forgetting via replay-based rehearsal, output level distillation, fixed regularization, or meta-learning on the current data. However, these methods have limitations: rehearsal introduces a stored sample bias; distillation operates on output-distributions without modulating parameter updates; fixed-regularization penalizes parameters irrespective of sensitivity; stream-only meta-learning lacks a feedback controlled parameter update. We propose Meta-Adaptive Network Gradient Optimization (MANGO), an OCL framework that balances stability-plasticity via gradient-gating and meta-learned regularization. Gradient-gating scales parameter updates based on sensitivity, preventing destructive updates. Meta-learned regularization adapts stability coefficients, evaluating the effect of parameter update on replay. In MANGO, replay acts as both a training signal and a forgetting evaluator. We evaluated our method on three standard OCL benchmark datasets. MANGO outperforms strong baselines, achieving state-of-the-art results with consistent performance across replay sizes. In domain incremental learning on CLEAR-10 and class incremental learning on CIFAR-100 and Tiny-ImageNet, it achieves highest accuracy among all baselines and achieves positive Backward Transfer, overcoming forgetting on CLEAR-10.

Abstract

Online Continual Learning

Explore similar work

MANGO: Meta-Adaptive Network Gradient Optimization for Online Continual Learning

Natural Gradient Descent for Online Continual Learning

Fast and Slow Variational Continual Learning