Elastic Weight Consolidation Done Right for Continual Learning

Xuan Liu; Xiaobin Chang

cs.LGMar 19, 2026

Elastic Weight Consolidation Done Right for Continual Learning

Authors: Xuan Liu, Xiaobin Chang

Organizations: School of Artificial Intelligence, Sun Yat-sen University, China · Key Laboratory of Intelligent Assessment Technology for Sustainable Tourism, Ministry of Culture and Tourism, Sun Yat-sen University · Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China

Abstract

Weight regularization methods in continual learning (CL) alleviate catastrophic forgetting by assessing and penalizing changes to important model weights. Elastic Weight Consolidation (EWC) is a foundational and widely used approach within this framework that estimates weight importance based on gradients. However, it has consistently shown suboptimal performance. In this paper, we conduct a systematic analysis of importance estimation in EWC from a gradient-based perspective. For the first time, we find that EWC's reliance on the Fisher Information Matrix (FIM) results in gradient vanishing and inaccurate importance estimation in certain scenarios. Our analysis also reveals that Memory Aware Synapses (MAS), a variant of EWC, imposes unnecessary constraints on parameters irrelevant to prior tasks, termed the redundant protection. Consequently, both EWC and its variants exhibit fundamental misalignments in estimating weight importance, leading to inferior performance. To tackle these issues, we propose the Logits Reversal (LR) operation, a simple yet effective modification that rectifies EWC's importance estimation. Specifically, reversing the logit values during the calculation of FIM can effectively prevent both gradient vanishing and redundant protection. Extensive experiments across various CL tasks and datasets show that the proposed method significantly outperforms existing EWC and its variants. Therefore, we refer to it as EWC Done Right (EWC-DR). Code is available at https://github.com/scarlet0703/EWC-DR.

Figures & tables

Figure 1: Overview of the EWC learning process. After training on task $t-1$ , model weights $\theta^{t-1}$ are obtained. The gradient of the cross-entropy loss $\mathcal{L}_{CE}$ is computed using the dataset $\mathcal{D}^{t-1}$ with the model $f(\cdot,\theta^{t-1})$ . These gradients estimate the weight importance matrix $\Omega^{t-1}$ , but they are not used to update the weights. When learning a new task $t$ , a regularization term $\mathcal{L}_{reg}$ based on $\Omega^{t-1}$ is added into the total loss to constrain changes and preserve important parameters in $\theta^{t-1}$ .

Panel of Figure 2 — Figure 2: A Case Study of Gradient Vanishing.

Panel of Figure 3 — Figure 3: A Case Study of Redundant Protection.

method	Big start						Equally split
	T=5		T=10		T=20		T=5		T=10		T=20
	$A_{last}$	$A_{avg}$	$A_{last}$	$A_{avg}$	$A_{last}$	$A_{avg}$	$A_{last}$	$A_{avg}$	$A_{last}$	$A_{avg}$	$A_{last}$	$A_{avg}$
EWC	14.61	32.82	11.53	27.02	6.20	19.61	17.80	39.02	10.62	27.90	5.48	18.13
online EWC	29.70	45.65	26.36	40.36	17.63	31.71	26.13	43.97	17.45	33.04	11.52	25.49
SI	19.26	37.68	11.33	27.45	5.62	17.62	13.10	29.84	9.91	23.58	7.98	20.72
MAS	35.37	48.32	29.09	43.24	19.98	34.77	29.45	43.68	23.41	37.84	13.24	27.39

Table 1: EFCIL performance on CIFAR-100. Reported results represent means across three independent trials.

method	Big start						Equally split
	T=5		T=10		T=20		T=5		T=10		T=20
	$A_{last}$	$A_{avg}$	$A_{last}$	$A_{avg}$	$A_{last}$	$A_{avg}$	$A_{last}$	$A_{avg}$	$A_{last}$	$A_{avg}$	$A_{last}$	$A_{avg}$
EWC	11.44	26.57	5.76	15.52	3.73	9.86	20.78	42.37	10.60	28.75	5.28	18.74
online EWC	23.56	46.68	16.12	39.71	13.00	32.74	26.64	46.58	12.78	29.72	6.34	18.91
SI	9.19	24.56	4.80	13.66	2.93	8.30	17.83	40.19	9.10	26.44	4.77	16.77
MAS	21.06	42.59	13.72	31.17	7.86	22.44	45.33	62.99	24.36	44.16	11.47	26.83

Table 2: EFCIL performance on ImageNet-Subset. Reported results represent means across three independent trials.

method	Big start						Equally split
	T=5		T=10		T=20		T=5		T=10		T=20
	$A_{last}$	$A_{avg}$	$A_{last}$	$A_{avg}$	$A_{last}$	$A_{avg}$	$A_{last}$	$A_{avg}$	$A_{last}$	$A_{avg}$	$A_{last}$	$A_{avg}$
EWC	9.74	19.92	7.00	15.35	4.56	12.51	20.15	32.10	11.42	23.52	5.34	15.58
online EWC	27.02	39.63	20.03	31.76	18.52	31.96	24.01	35.41	16.77	27.63	10.95	19.44
SI	7.46	17.59	4.04	10.10	2.24	5.68	12.14	27.02	6.97	19.00	3.81	12.41
MAS	25.53	38.45	15.32	27.97	11.67	25.98	20.04	32.95	11.56	24.63	7.41	17.51

Table 3: EFCIL performance on Tiny-ImageNet. Reported results represent means across three independent trials.

Figure 4: CD diagram comparing the

A_{avg}

of different methods across all evaluated EFCIL settings. The CD is 1.438 with a significance level of 0.05.

Panel of Figure 5 — Figure 5: Importance matrices of the FC layer weights (ground truth at class 2) for EWCs .

EWC
Checkpoint	VQA2	NLVR2	SNLI-VE	VCR
Task 1: VQA2	67.89	–	–	–
Task 2: NLVR2	39.94 [41.17]	72.85	–	–
Task 3: SNLI-VE	40.83 [39.86]	64.95 [34.59]	74.60	–
Task 4: VCR	18.70 [72.46]	52.13 [ 90.66 ]	41.84 [79.39]	59.30

Table 4: Results of MCIT. The accuracy

a_{t,j}\uparrow

(%) with forgetting transfer

\mathcal{F}_{t,j}\downarrow

(%) in brackets are reported.

Figure 6: The incremental accuracy (

A_{t}

) and new task accuracy (

A_{\mathrm{new}}

) of EWC and EWC-DR across tasks in the MCIT scenario. Solid lines denote

A_{t}

, while dashed lines denote

A_{\mathrm{new}}

method	category	T=5	T=10	T=20
iCaRL-CNN †	Replay	51.07	48.66	44.43
iCaRL-NCM †	Replay	58.56	54.19	50.51
LWF-MC	Regularization	45.93	27.43	20.07
MUC	Regularization	49.42	30.19	21.27
GPM	Optimization	41.51	37.78	41.27
ADAM-NSCL	Optimization	22.67	12.72	8.79

Table A : EFCIL performance on CIFAR-100 compared against strong baselines. Reported results represent means across three independent trials.

method	Big start			Equally split
method	T=5	T=10	T=20	T=5	T=10	T=20
FeTrIL	66.3	65.2	61.5	60.4	52.1	43.2
PASS	63.5	61.8	58.1	63.4	52.2	41.8
SSRE	65.9	65.0	61.7	56.6	44.4	33.6
EWC	33.4	26.4	19.2	39.2	27.3	17.7
EWC-DR	63.9	62.1	53.5	61.5	49.6	35.3

Table B : EFCIL results on CIFAR-100 comparing EWC-DR with modern methods. Average incremental accuracy (

A_{avg}

) is reported using seed 1993.

Figure A : Sensitivity analysis of the average accuracy (

A_{avg}

) with respect to the regularization parameter

\lambda

on CIFAR-100 under the 10-task equally split EFCIL setting.

class	0	1	2 (GT)	3	4
EWC	0.003	0.013	0.219	0.136	0.018
MAS	14.44	8.53	25.48	10.26	18.08
EWC-DR	5.30	6.55	41.38	2.51	11.12

Table C : Statistics of the FC layer weight importance.

Regularization-Based Continual Learning

See all 13 papers

Revisiting Weight Regularization for Low-Rank Continual Learning
Feb 19, 2026Yaoyue Zheng, Yin Zhang, Joost van de Weijer +4Incremental Learning Elastic Weight Consolidation
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Shaanxi, China · Computer Vision Center, Barcelona, Spain · Universitat Aut`onoma de Barcelona, Barcelona, Spain +3
The Sequential Price of Continual Learning
Aug 30, 2026Zonghuan Xu, Xingjun MaContinual Learning Elastic Weight Consolidation
Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China
From Weights to Features: SAE-Guided Activation Regularization for LLM Continual Learning
Jun 25, 2026Evan Ning, Wei Xue, Dong Lou +1Continual Learning for LLMs Catastrophic Forgetting
1The Hong Kong University of Science and Technology

Feb 19, 2026J/K move · Enter open · S save

Revisiting Weight Regularization for Low-Rank Continual Learning

Yaoyue Zheng, Yin Zhang, Joost van de Weijer, Gido M van de Ven+3

State Key Laboratory of Human-Machine Hybrid Augmented Intelligence Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Shaanxi, China · Computer Vision Center, Barcelona, Spain · Universitat Aut`onoma de Barcelona, Barcelona, Spain+3

Incremental Learning Elastic Weight Consolidation Parameter-Efficient Fine-Tuning Low-Rank Adaptation

Open paper AskPDF arXiv

Continual Learning (CL) with large-scale pre-trained models (PTMs) has recently gained wide attention, shifting the focus from training from scratch to continually adapting PTMs. This has given rise to a promising paradigm: parameter-efficient continual learning (PECL), where task interference is typically mitigated by assigning a task-specific module during training, such as low-rank adapters. However, weight regularization techniques, such as Elastic Weight Consolidation (EWC)-a key strategy in CL-remain underexplored in this new paradigm. In this paper, we revisit weight regularization in low-rank CL as a new perspective for mitigating task interference in PECL. Unlike existing low-rank CL methods, we mitigate task interference by regularizing a shared low-rank update through EWC, thereby keeping the storage requirement and inference costs constant regardless of the number of tasks. Our proposed method EWC-LoRA leverages a low-rank representation to estimate parameter importance over the full-dimensional space. This design offers a practical, computational- and memory-efficient solution for CL with PTMs, and provides insights that may inform the broader application of regularization techniques within PECL. Extensive experiments on various benchmarks demonstrate the effectiveness of EWC-LoRA, achieving a stability-plasticity trade-off superior to existing low-rank CL approaches. These results indicate that, even under low-rank parameterizations, weight regularization remains an effective mechanism for mitigating task interference. Code is available at: https://github.com/yaoyz96/low-rank-cl.

Abstract

Regularization-Based Continual Learning

Explore similar work

Revisiting Weight Regularization for Low-Rank Continual Learning

The Sequential Price of Continual Learning

From Weights to Features: SAE-Guided Activation Regularization for LLM Continual Learning