A Faster Path to Continual Learning

Wei Li; Hangjie Yuan; Zixiang Zhao; Borui Kang; Ziwei Liu; Tao Feng

cs.LGApr 13, 2026

A Faster Path to Continual Learning

Authors: Wei Li, Hangjie Yuan, Zixiang Zhao, Borui Kang, Ziwei Liu, Tao Feng

Organizations: College of Computer Science, Sichuan University, China · College of Computer Science and Technology, Zhejiang University, China · Photogrammetry and Remote Sensing Lab, ETH Zürich, Switzerland · School of Computer Science, Nanjing University, China ·

Abstract

Continual Learning (CL) aims to train neural networks on a dynamic stream of tasks without forgetting previously learned knowledge. Among optimization-based approaches, C-Flat has emerged as a promising solution due to its plug-and-play nature and its ability to encourage uniformly low-loss regions for both new and old tasks. However, C-Flat requires three additional gradient computations per iteration, imposing substantial overhead on the optimization process. In this work, we propose C-Flat Turbo, a faster yet stronger optimizer that significantly reduces the training cost. We show that the gradients associated with first-order flatness contain direction-invariant components relative to the proxy-model gradients, enabling us to skip redundant gradient computations in the perturbed ascent steps. Moreover, we observe that these flatness-promoting gradients progressively stabilize across tasks, which motivates a linear scheduling strategy with an adaptive trigger to allocate larger turbo steps for later tasks. Experiments show that C-Flat Turbo is 1.0 $\times$ to 1.25 $\times$ faster than C-Flat across a wide range of CL methods, while achieving comparable or even improved accuracy.

Figures & tables

Figure 1: Brief illustration of C-Flat [ 6 ] . (a) SGD optimizes along the negative direction of the gradients,

g=\nabla L(f(\theta^{T}))

. (b) SAM [ 14 ] computes the gradients

g_{s}

at an adversarially perturbed position

\theta+\rho\cdot g/\|g\|

, and then updates the original model parameters. (c) C-Flat [ 6 ] further calculates the first-order flatness gradient

g_{f}

, based on the perturbed parameters

\theta+\rho\cdot(g_{s}-g)/\|(g_{s}-g)\|

Figure 2: Schematic illustration of C-Flat Turbo. Left: LookSAM [ 36 ] decomposes the SAM gradients into two components: one parallel to $\boldsymbol{g}$ that reduces the empirical loss, and an orthogonal component $\boldsymbol{g}_{\mathrm{vs}}$ that guides convergence toward a common low-loss region. Empirical results show that $\boldsymbol{g}_{\mathrm{vs}}$ varies significantly more slowly than $\boldsymbol{g}$ . Right: C-Flat Turbo investigates the latent invariance of $\boldsymbol{g}_{\mathrm{vf}}$ , the flatness component orthogonal to the gradients at the perturbed model $\boldsymbol{\theta}+\boldsymbol{\epsilon}_{1}^{*}$ , which exhibits even slower variation than $\boldsymbol{g}_{\mathrm{vs}}$ in LookSAM.

Panel of Figure 3 — Figure 3: Left: Distributions of gradient correction ratios for $\boldsymbol{g}_{s}-\boldsymbol{g}$ and $\boldsymbol{g}_{f}$ across training epochs. A larger portion of data near the distribution tails indicates increasingly pronounced differences. Right: L2-norm distances between gradients and their counterparts from five steps earlier. Changes along the sharpness-related direction ( $\boldsymbol{g}_{vs}$ ) and flatness-related direction ( $\boldsymbol{g}_{vf}$ ) evolve more slowly than those along the empirical gradient directions ( $\boldsymbol{g}$ and $\boldsymbol{g}_{0}$ ).

Panel of Figure 4 — Figure 4: Left: Gradient dynamics across tasks in CL. Both sharpness- and flatness-related gradients fluctuate substantially in early stages but progressively stabilize as training proceeds. Right: Q–Q plots of $\|\boldsymbol{g}\|^{2}$ and $\|\boldsymbol{g}_{0}\|^{2}$ , showing that both statistics gradually approach a normal distribution over the course of learning.

Model	Method	CIFAR100 B0_Inc10		CUB B0_Inc10		IN-R B0_Inc20		ObjNet B0_Inc10		Img/s $\uparrow$
Model	Method	Avg $\uparrow$	Last $\uparrow$	Avg $\uparrow$	Last $\uparrow$	Avg $\uparrow$	Last $\uparrow$	Avg $\uparrow$	Last $\uparrow$	Img/s $\uparrow$
Typical	iCaRL [ 42 ]	77.83	66.64	82.91	74.00	72.13	61.62	48.06	28.20	73.35 (100%)
	+C-Flat [ 6 ]	79.72	67.15	83.47	74.81	72.92	62.35	49.59	29.03	19.72 ( 26.9% )
	+C-Flat Turbo	79.82	68.54	84.00	75.12	73.11	62.38	50.49	29.30	45.89 ( 62.6% )
	MEMO [ 70 ]	82.26	73.89	86.66	79.90	70.96	61.05	56.22	38.32	135.14 (100%)
	+C-Flat [ 6 ]	82.61	75.49	87.03	80.73	71.69	62.93	56.50	39.45	46.11 ( 34.1% )

Table 1: Accuracy and training speeds using five state-of-the-art methods, with a pre-trained ViT-B/16-IN1K backbone. Red and green denote the baseline and the efficient optimizer, respectively. Bolded indicates the best result.

Panel of Figure 5 — Figure 5: (a) Sensitivity analysis of $k$ . (b) - (d) Evolution of loss, sharpness and flatness on EASE w/ and w/o C-Flat Turbo.

Method	ResNet-18			ResNet-34
Method	Avg $\uparrow$	Last $\uparrow$	Img/s $\uparrow$	Avg $\uparrow$	Last $\uparrow$	Img/s $\uparrow$
iCaRL [ 42 ]	59.13 ±0.30	41.23 ±0.91	2333.3 (100%)	58.80 ±1.04	41.26 ±0.99	1250.4 (100%)
+C-Flat [ 6 ]	59.45 ±0.18	42.47 ±0.06	686.3 ( 29.4% )	59.55 ±0.93	42.09 ±0.61	359.8 ( 28.8% )
+C-Flat Turbo	59.84 ±0.05	42.84 ±0.18	1750.1 ( 75.0% )	59.75 ±0.55	42.34 ±0.54	960.6 ( 76.8% )
MEMO [ 70 ]	48.63 ±0.78	29.19 ±0.89	2413.8 (100%)	68.49 ±1.74	57.05 ±1.46	1873.2 (100%)
+C-Flat [ 6 ]	49.98 ±0.61	30.76 ±0.57	886.1 ( 36.7% )	69.00 ±1.39	59.29 ±0.73	569.1 ( 30.4% )

Table 2: Accuracy results with ResNet-18 and ResNet-34 trained from scratch. Bolded indicates the best result.

Method	L2P [ 57 ]			EASE [ 67 ]
Method	Avg $\uparrow$	Last $\uparrow$	Img/s $\uparrow$	Avg $\uparrow$	Last $\uparrow$	Img/s $\uparrow$
Vanilla	87.92 ±1.30	83.66 ±1.49	110.29 (100%)	91.16 ±0.71	87.49 ±0.32	166.67 (100%)
+SAM [ 14 ]	88.18 ±1.39	83.84 ±1.39	56.60 ( 51.3% )	91.36 ±0.82	87.61 ±0.27	86.71 ( 52.0% )
+LookSAM [ 36 ]	88.35 ±1.39	83.80 ±1.20	88.76 ( 80.5% )	90.89 ±0.86	86.99 ±0.28	132.74 ( 79.6% )
+C-Flat [ 6 ]	88.62 ±0.88	84.04 ±0.60	28.63 ( 30.0% )	91.60 ±1.18	87.69 ±0.50	44.25 ( 26.5% )
+C-Flat Turbo	89.57 ±0.36	84.48 ±0.09	65.50 (59.4%)	91.75 ±0.80	87.74 ±0.20	102.74 (61.6%)

Table 3: Accuracy and training speeds of L2P and EASE across various optimizers. Task orders are shuffled for evaluation on dynamic sequences. Red and green denote the baseline and the efficient optimizer. Bolded indicates the best result.

Panel of Figure 6 — Figure 6: Accuracy and training speed comparison of different Turbo scheduling strategies. The upper part shows results on MEMO, while the lower part shows results on EASE.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Methods	Epochs	LR	BS	Tasks	Exemplar	$\rho$	$\lambda$	$k$	$\beta$
iCaRL	20	1e-3	32	10	20	0.1	0.2	5	0.8
MEMO	20	1e-3	32	10	20	0.1	0.2	5	0.8
L2P	5	2e-3	16	10	-	0.02	0.2	5	0.8
Ranpac	5	1e-2	16	10	-	0.05	0.2	5	0.8
EASE	5	2.5e-3	16	10	-	0.05	0.2	5	0.8

Appendix

Table 4: Hyperparameter settings for CIFAR100.

Optimizer	C-Flat Turbo
$k_{0}$	2	2	2	5	5	5	10	10	10
$c$	5	10	20	5	10	20	5	10	20
Avg	92.19	92.18	92.05	92.08	92.08	92.01	92.07	92.04	92.04
Last	87.67	87.59	87.45	87.62	87.54	87.42	87.51	87.59	87.46
Img/s	99.17	99.91	63.19	104.86	102.74	99.05	104.64	106.89	107.39

Appendix

Table 5: Sensitivity analysis of the scheduler hyperparameters in C-Flat Turbo on CIFAR100.

$m$	0.1	0.2	0.5	1	2	5
Avg	91.98	92.07	92.05	92.06	91.86	91.94
Last	87.43	87.48	87.39	87.50	87.07	87.16
Img/s	75.56	105.43	97.69	102.74	154.52	135.18

Appendix

Table 6: Sensitivity analysis of the trigger threshold hyperparameter in C-Flat Turbo on CIFAR100.

Method	Optimizer	Backbone	Trainable / total params	Memory
EASE	C-Flat	ViT-Base-16	1.19M / 86.99M	2.14GB
	C-Flat Turbo	ViT-Base-16	1.19M / 86.99M	2.15GB
	C-Flat	ViT-Large-16	3.17M / 306.47M	5.34GB
	C-Flat Turbo	ViT-Large-16	3.17M / 306.47M	5.37GB
iCaRL	C-Flat	ResNet-18	11.17M / 11.17M	1.55GB
	C-Flat Turbo	ResNet-18	11.17M / 11.17M	1.66GB

Appendix

Table 7: Memory usage of different architectures. EASE uses a frozen backbone, while iCaRL updates the full backbone.

	L2P			EASE
Method	Avg	Last	Img/s	Avg	Last	Img/s
+SS-SAM	89.50	84.45	77.42 (70.20%)	90.90	87.87	91.65 (54.99%)
+AE-SAM	89.70	84.48	80.86 (73.32%)	92.03	87.37	112.58 (67.54%)
+C-Flat Turbo	89.78	84.69	65.50 (59.4%)	92.36	87.96	102.74 (61.6%)

Appendix

Table 8: Comparison with SS-SAM and AE-SAM on L2P (left) and EASE (right).

Method	Clipart	Infograph	Painting	Quickdraw	Real	Sketch	Avg
DUCT	74.21	60.52	67.73	63.50	69.51	68.63	67.35
+C-Flat	74.93	60.88	68.10	63.30	70.22	69.41	67.81
+C-Flat Turbo	74.89	60.91	68.01	63.75	70.82	69.50	67.98

Appendix

Table 9: Results on DomainNet under the DIL setting.

Method	CIL		TIL
Method	Avg	Last	Avg	Last
DualPrompt	85.77	89.10	97.79	99.40
+C-Flat	85.85	87.80	97.67	99.30
+C-Flat Turbo	86.01	87.00	97.94	99.40

Appendix

Table 10: Results on CIFAR100 under the CIL and TIL settings.

Method	reuse	scheduler	trigger	T1	T2	T3	T4	T5	T6	T7	T8	T9	T10	Avg	Img/s
EASE	$\times$	$\times$	$\times$	98.40	96.25	94.63	93.88	91.80	90.92	90.47	88.09	87.58	87.17	91.92	166.67
+C-Flat	$\times$	$\times$	$\times$	98.50	96.45	94.87	94.08	91.94	91.05	90.64	88.44	87.93	87.58	92.15	44.25
+C-Flat Turbo	$\checkmark$	$\times$	$\times$	98.50	96.37	94.77	93.94	91.92	91.05	90.67	88.28	87.81	87.45	92.08	67.20
	$\checkmark$	$\checkmark$	$\times$	98.40	96.31	94.74	93.89	91.90	91.00	90.70	88.25	87.75	87.40	92.03	74.63
	$\checkmark$	$\checkmark$	$\checkmark$	98.50	96.60	95.07	94.15	92.08	91.27	90.73	88.52	88.00	87.57	92.25	102.74

Appendix

Table 11: Per-task accuracy and ablation study results for EASE trained on the 10-split CIFAR100 dataset.

Panel of Figure 7 — Figure 7: Visualization of L2-norm distances of the gradients every 5 steps across 10 tasks.

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 294 papers

Fast and Slow Variational Continual Learning
Jun 22, 2026Subarnaduti Paul, Yohan Jung, Mohammad Emtiyaz Khan +3Continual Learning for LLMs Deep Learning Optimization
Faculty of Mathematics and Computer Science, University of Bremen, Bremen, Germany · Department of Computer Science & AI, Jeonbuk National University, Republic of Korea · Department of Computer Science, TU Darmstadt, Darmstadt, Germany +3
MANGO: Meta-Adaptive Network Gradient Optimization for Online Continual Learning
May 18, 2026Ankita Awasthi, Marco Apolinario, Kaushik RoyClass-Incremental Learning Meta-Learning
Purdue University, USA · TU Delft, Netherlands
Learning in the Null Space: Small Singular Values for Continual Learning
Feb 25, 2026Cuong Anh Pham, Praneeth Vepakomma, Samuel HorváthContinual Learning Orthogonal Representation Learning
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE · Massachusetts Institute of Technology (MIT), USA

Jun 22, 2026J/K move · Enter open · S save

Fast and Slow Variational Continual Learning

Subarnaduti Paul, Yohan Jung, Mohammad Emtiyaz Khan, Siddharth Swaroop+2

Faculty of Mathematics and Computer Science, University of Bremen, Bremen, Germany · Department of Computer Science & AI, Jeonbuk National University, Republic of Korea · Department of Computer Science, TU Darmstadt, Darmstadt, Germany+3

Continual Learning for LLMs Deep Learning Optimization Continual Learning Domain-Incremental Learning

Open paper AskPDF arXiv

Continual learning remains a major challenge for modern deep networks, partly because commonly used optimizers lack inherent mechanisms for continual adaptation. One such natural mechanism is fast and slow adaptation to balance stability and plasticity. This mechanism has deep roots in neuroscience and biology, but there is no consensus on how to best incorporate it in commonly used optimizers. Here, we show that this can be easily done via the VCL framework, where past posteriors are used as priors in the future. Our key idea is to incorporate slow adaptation via merging of past posteriors to slow down the drift in the knowledge as learning progresses. The merged posterior is then used as the prior in the VCL update to implement the fast-weight updates. These steps can be seamlessly implemented in the IVON optimizer, whose form and costs are nearly identical to that of Adam. We call this new optimizer the Continual IVON (CoVON) optimizer and show that it not only consistently improves over existing VCL optimizers, but also performs better than other weight-regularization strategies across domain-incremental learning, continual pre-training, and fine-tuning of large language models.

Abstract

Continual Learning

Explore similar work

Fast and Slow Variational Continual Learning

MANGO: Meta-Adaptive Network Gradient Optimization for Online Continual Learning

Learning in the Null Space: Small Singular Values for Continual Learning