Natural Gradient Descent for Online Continual Learning

Joe Khawand; David Colliaux

cs.LGMar 21, 2026

Natural Gradient Descent for Online Continual Learning

Authors: Joe Khawand, David Colliaux

Organizations: Ecole Polytechnique & Télécom Paris Paris, France · Sony Computer Science Laboratories Paris, France

Abstract

Online Continual Learning (OCL) for image classification represents a challenging subset of Continual Learning, focusing on classifying images from a stream without assuming data independence and identical distribution (i.i.d). The primary challenge in this context is to prevent catastrophic forgetting, where the model's performance on previous tasks deteriorates as it learns new ones. Although various strategies have been proposed to address this issue, achieving rapid convergence remains a significant challenge in the online setting. In this work, we introduce a novel approach to training OCL models that utilizes the Natural Gradient Descent optimizer, incorporating an approximation of the Fisher Information Matrix (FIM) through Kronecker Factored Approximate Curvature (KFAC). This method demonstrates substantial improvements in performance across all OCL methods, particularly when combined with existing OCL tricks, on datasets such as Split CIFAR-100, CORE50, and Split miniImageNet.

Figures & tables

Figure 1: Average End Accuracy on Split-CIFAR-100 Krizhevsky et al. (2009) for 3 buffer sizes (1k-5k-10k) using ER Rolnick et al. (2019) in combination with OCL tricks Mai et al. (2022) .

Dataset	#Task	#Train/task	#Class	Setting
Split CIFAR-100 Krizhevsky et al. (2009)	20	2500	100	OCI
Split MiniImageNet Chaudhry et al. (2019)	20	2500	100	OCI
NS-MiniImageNet Mai et al. (2022)	10	5000	100	ODI
CORE50-NC Lomonaco and Maltoni (2017)	9	12000 $\sim$ 24000	50	OCI
CORE50-NI Lomonaco and Maltoni (2017)	8	15000	50	ODI

Table 1: Summary of the datasets used.

Panel of Table 2 — Table 2: End Average Accuracy of methods and tricks with or without NGD-KFAC for the OCI setting on Split CIFAR-100.

Panel of Table 3 — Table 3: Average End Accuracy for NonStationary-MiniImageNet (Noise, Occlusion, Blur) and CORe50-NI in the ODI setting.

Panel of Table 4 — Table 4: End Average Accuracy of methods and tricks with or without NGD-KFAC for the OCI setting on Split MiniImageNet.

Panel of Table 5 — Table 5: End Average Accuracy of methods and tricks with or without NGD-KFAC for the OCI setting on CORe50-NC.

Figure 2: Average End Accuracy by trick, averaged on all datasets in the OCI setting with regards to the damping parameter $\epsilon$ .

Kronecker-Factored Approximation

See all 14 papers

MANGO: Meta-Adaptive Network Gradient Optimization for Online Continual Learning
May 18, 2026Ankita Awasthi, Marco Apolinario, Kaushik RoyClass-Incremental Learning Meta-Learning
Purdue University, USA · TU Delft, Netherlands
CLIMB: Centroid-Based Hierarchical Memory for Online Continual Self-Supervised Learning
Jun 30, 2026Julien Lefebvre, Stefan Duffner, Mathieu LefortSelf-Supervised Learning Replay-Based Continual Learning
Universite Claude Bernard Lyon 1, CNRS, INSA Lyon, LIRIS, UMR 5205, 69622 Villeurbanne, France · INSA Lyon, CNRS, Universite Claude Bernard Lyon 1, LIRIS, UMR 5205, 69621 Villeurbanne, France · Universite Claude Bernard Lyon 1 / Univ. Rennes, Inria, CNRS, LIRIS UMR 5205 / IRISA UMR 6074, France
A Faster Path to Continual Learning
Apr 13, 2026Wei Li, Hangjie Yuan, Zixiang Zhao +3Continual Learning Neural Network Optimization
College of Computer Science, Sichuan University, China · College of Computer Science and Technology, Zhejiang University, China · Photogrammetry and Remote Sensing Lab, ETH Zürich, Switzerland +3

May 18, 2026J/K move · Enter open · S save

MANGO: Meta-Adaptive Network Gradient Optimization for Online Continual Learning

Ankita Awasthi, Marco Apolinario, Kaushik Roy

Purdue University, USA · TU Delft, Netherlands

Class-Incremental Learning Meta-Learning Continual Learning Replay-Based Continual Learning Online Continual Learning Incremental Learning+1

Open paper AskPDF arXiv

In Online Continual Learning (OCL), a neural network sequentially learns from a non-stationary data stream in a single-pass with access only to a limited memory replay buffer. This contrasts sharply with off-line continual learning where training is multiple epoch dependent on large datasets. The main challenge faced by OCL is to overcome catastrophic forgetting of past tasks (stability) while learning new ones efficiently (plasticity). Existing methods counter forgetting via replay-based rehearsal, output level distillation, fixed regularization, or meta-learning on the current data. However, these methods have limitations: rehearsal introduces a stored sample bias; distillation operates on output-distributions without modulating parameter updates; fixed-regularization penalizes parameters irrespective of sensitivity; stream-only meta-learning lacks a feedback controlled parameter update. We propose Meta-Adaptive Network Gradient Optimization (MANGO), an OCL framework that balances stability-plasticity via gradient-gating and meta-learned regularization. Gradient-gating scales parameter updates based on sensitivity, preventing destructive updates. Meta-learned regularization adapts stability coefficients, evaluating the effect of parameter update on replay. In MANGO, replay acts as both a training signal and a forgetting evaluator. We evaluated our method on three standard OCL benchmark datasets. MANGO outperforms strong baselines, achieving state-of-the-art results with consistent performance across replay sizes. In domain incremental learning on CLEAR-10 and class incremental learning on CIFAR-100 and Tiny-ImageNet, it achieves highest accuracy among all baselines and achieves positive Backward Transfer, overcoming forgetting on CLEAR-10.

Abstract

Kronecker-Factored Approximation

Explore similar work

MANGO: Meta-Adaptive Network Gradient Optimization for Online Continual Learning

CLIMB: Centroid-Based Hierarchical Memory for Online Continual Self-Supervised Learning

A Faster Path to Continual Learning