Beyond Parameter Protection: Fisher-Shaped EWC for Replay-Free Continual Learning

Liying Wang; Simone Magistri; Joost van de Weijer

cs.LGOct 27, 2026

Beyond Parameter Protection: Fisher-Shaped EWC for Replay-Free Continual Learning

Authors: Liying Wang, Simone Magistri, Joost van de Weijer

Organizations: Department of Computer Science, Universitat Aut`onoma de Barcelona, Spain · Computer Vision Center, Barcelona, Spain · Media Integration and Communication Center, University of Florence, Italy

Abstract

In replay-free continual learning, Elastic Weight Consolidation (EWC) uses Fisher information matrix (FIM) to protect acquired knowledge by penalizing changes to historically important parameters. However, a local importance estimate at the retained solution does not in general characterize sensitivity after finite parameter displacement. This motivates controlling Fisher sensitivity over a neighborhood of potential future updates. We propose Fisher-Shaped EWC (FS-EWC), which extends the role of FIM from parameter protection to solution formation for future retention. During task learning, FS-EWC minimizes the worst-case empirical FIM trace within a prescribed neighborhood, explicitly favoring solutions that are less sensitive to bounded future displacement. At task completion, trajectory-based model merging constructs the retained solution from multiple optimization states rather than a single terminal iterate, after which FIM is estimated for subsequent consolidation. Experiments across multiple benchmarks show that FS-EWC improves performance over EWC, while controlled studies indicate a more favorable stability--plasticity balance and greater tolerance to parameter drift. These results suggest that effective knowledge retention requires both forming solutions with future retention in mind and protecting acquired knowledge.

Figures & tables

Figure 1: Beyond protection: shaping solutions for future retention. The surfaces schematically depict task- $t$ performance around learned solutions, while trajectories illustrate parameter updates during task- $t+1$ learning. Unlike finetuning, EWC penalizes subsequent drift using historical parameter importance. FS-EWC additionally penalizes the worst-case empirical FIM over a prescribed neighborhood during task- $t$ learning, favoring solutions less sensitive to bounded future drift while preserving historical protection. Complemented by model merging over task trajectory, our approach incorporates future retention into how solutions are learned and retained for subsequent protection.

Method	FGVC-Aircraft		Stanford Cars		CUB-200		Oxford Flowers		5Datasets
	FAA	AAA	FAA	AAA	FAA	AAA	FAA	AAA	FAA	AAA
Vanilla LoRA	39.85	38.42	31.74	41.67	58.31	69.61	52.92	66.49	66.90	68.85
InfLoRA	41.35	39.62	35.98	44.57	61.36	70.38	57.34	69.98	78.77	77.25
OLoRA	37.67	37.77	35.30	44.70	60.70	68.71	47.68	63.18	79.12	76.96
SDLoRA	25.96	26.63	34.89	43.28	62.01	69.52	49.05	64.59	74.42	74.10
CL-LoRA	29.35	26.54	34.01	41.86	78.31	81.01	97.83	98.34	67.05	61.97

Table 2: Fine-grained and heterogeneous vision performance . FAA and stage-averaged AAA are reported in percentages. 5Datasets denotes the cross-dataset continual learning stream in the following order: CUB-200

\rightarrow

Aircraft

\rightarrow

Cars

\rightarrow

Flowers

\rightarrow

Pets. The corresponding average-accuracy trajectories are provided in Fig. 6 .

Figure 3: Less forgetting despite larger realized drift. Realized-drift diagnostics on ImageNet-A with LoRA. Relative to the Task 1 solution, we report effective-weight displacement

d_{t}

, Task 1 loss change

D_{1,t}^{\mathrm{test}}

, Task 1 accuracy, and empirical-Fisher sensitivity averaged along the straight segment to each subsequent task endpoint.

	Training objective		Task 1 endpoint		Task 2 endpoint				Task 2update
Variant	Task 1	Task 2	$a_{11}\uparrow$	$\operatorname{tr}{\bm{F}}_{1}^{\mathrm{emp}}$	$a_{21}\uparrow$	$a_{22}\uparrow$	BWT $\uparrow$	$A_{2}\uparrow$	$d_{2}$
Vanilla LoRA	$\mathsf{C}$	$\mathsf{C}$	${87.08}_{\scriptscriptstyle(0.38)}$	${31.44}_{\scriptscriptstyle(4.24)}$	${82.41}_{\scriptscriptstyle(1.13)}$	${76.38}_{\scriptscriptstyle(0.40)}$	${-4.67}_{\scriptscriptstyle(1.36)}$	${79.39}_{\scriptscriptstyle(0.74)}$	${0.38}_{\scriptscriptstyle(0.02)}$
Fisher Shaping	$\mathsf{C}+\mathsf{F}$	$\mathsf{C}+\mathsf{F}$	${87.30}_{\scriptscriptstyle(0.57)}$	${21.93}_{\scriptscriptstyle(2.98)}$	${83.39}_{\scriptscriptstyle(0.33)}$	${80.84}_{\scriptscriptstyle(1.14)}$	${-3.91}_{\scriptscriptstyle(0.65)}$	${82.12}_{\scriptscriptstyle(0.59)}$	${0.50}_{\scriptscriptstyle(0.01)}$
Task-1 only	$\mathsf{C}+\mathsf{F}$	$\mathsf{C}$	${87.30}_{\scriptscriptstyle(0.57)}$	${21.93}_{\scriptscriptstyle(2.98)}$	${86.54}_{\scriptscriptstyle(0.82)}$	${72.96}_{\scriptscriptstyle(1.38)}$	${-0.76}_{\scriptscriptstyle(1.36)}$	${79.75}_{\scriptscriptstyle(1.10)}$	${0.36}_{\scriptscriptstyle(0.01)}$
EWCLoRA	$\mathsf{C}$	$\mathsf{C}+\mathsf{P}$	${87.08}_{\scriptscriptstyle(0.38)}$	${31.44}_{\scriptscriptstyle(4.24)}$	${82.95}_{\scriptscriptstyle(0.76)}$	${76.25}_{\scriptscriptstyle(0.60)}$	${-4.13}_{\scriptscriptstyle(1.00)}$	${79.60}_{\scriptscriptstyle(0.67)}$	${0.28}_{\scriptscriptstyle(0.01)}$
FS-EWCLoRA	$\mathsf{C}+\mathsf{F}$	$\mathsf{C}+\mathsf{F}+\mathsf{P}$	${87.30}_{\scriptscriptstyle(0.57)}$	${21.93}_{\scriptscriptstyle(2.98)}$	${83.50}_{\scriptscriptstyle(0.50)}$	${79.92}_{\scriptscriptstyle(1.04)}$	${-3.80}_{\scriptscriptstyle(1.00)}$	${81.71}_{\scriptscriptstyle(0.49)}$	${0.43}_{\scriptscriptstyle(0.01)}$

Table 4: Two-task diagnostics of solution shaping and historical protection with LoRA on ImageNet-A.

\mathsf{C}

\mathsf{F}

, and

\mathsf{P}

denote current-task fitting, Fisher shaping, and historical EWC protection, respectively.

Method	Diagonal			Kronecker-factored
Method	Batched	Per-example	Model-sampled	Batched	Per-example	Model-sampled
EWC	51.96 / 66.30	50.60 / 66.60	50.76 / 66.60	53.74 / 67.37	52.75 / 67.55	52.71 / 67.33
FS-EWC	59.29 / 70.69	59.67/ 70.88	59.62 / 70.86	59.82 / 71.00	59.95 / 71.16	59.62 / 71.12

Table 5: Performance under different Fisher structures and computations. Under each structure, empirical FIM is computed using either batch gradients (Batch) or per-sample gradients (Per-sample), while Model denotes model-sampled FIM. Each entry reports FAA / AAA (%).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
${\bm{\theta}}$ , ${\bm{\theta}}_{t}$	Candidate parameters and the retained task- $t$ anchor.
${\bm{F}}_{t}^{\mathrm{emp}}$ , $\widehat{{\bm{F}}}_{t}$ , $\varepsilon_{F}$	Empirical FIM and the normalized and damped current-task FIM, Fisher-normalization stabilizer.
${\bm{\Lambda}}_{t}$ , $\Omega_{t-1}$	Consolidated importance and the inherited EWC penalty.
$\bm{\Delta}$ , $\bm{\epsilon}$	Future parameter displacement and sensitivity-search perturbation.
$D_{t}({\bm{\theta}},\bm{\Delta})$	Task-loss change; positive values indicate deterioration.
$\mathcal{R}_{\mathrm{F},t}$ , $\mathcal{R}_{\mathrm{F},t}^{\max}$	Fisher trace and its maximum over ${\mathbb{U}}_{r}({\bm{\theta}})$ .

Appendix

Table 7: Key notation used in FS-EWC.

Dataset	Classes	Tasks	Task partition
Aircraft	100	10	$10\times 10$
Cars	196	10	$16+9\times 20$
CUB-200	200	10	$20\times 10$
Flowers	102	10	$12+9\times 10$
5Datasets	635	5	$200/100/196/102/37$

Appendix

Table 8: Task partitions for the fine-grained and heterogeneous vision benchmarks.

Order	Task sequence
1	DBpedia $\rightarrow$ Amazon $\rightarrow$ Yahoo $\rightarrow$ AG News
2	DBpedia $\rightarrow$ Amazon $\rightarrow$ AG News $\rightarrow$ Yahoo
3	Yahoo $\rightarrow$ Amazon $\rightarrow$ AG News $\rightarrow$ DBpedia

Appendix

Table 9: Task orders for continual text classification.

Figure 6: Stage-wise average-accuracy trajectories on the fine-grained and heterogeneous vision benchmarks reported in Table 3 .

Figure 7: Task-wise accuracy trajectories for Llama-3.2-1B and Llama-3.2-3B across three continual text-classification task orders. Each curve starts when the corresponding dataset is learned and tracks its accuracy after subsequent tasks.

Model	Variant	Training objective		Task 1 endpoint		Task 2 endpoint			Task 2 update
Model	Variant	Task 1	Task 2	$a_{11}\uparrow$	$\mathcal{F}_{1}$	$a_{21}\uparrow$	$a_{22}\uparrow$	BWT $\uparrow$	$d_{2}$
LoRA	Vanilla	$\mathsf{C}$	$\mathsf{C}$	90.73	19.885	84.85	89.59	-5.87	1.560
	Shaping only	$\mathsf{C}+\mathsf{F}$	$\mathsf{C}+\mathsf{F}$	93.34	10.946	84.80	93.81	-8.55	3.131
	EWC	$\mathsf{C}$	$\mathsf{C}+\mathsf{P}$	90.73	19.885	87.95	83.45	-2.78	0.0106
	FS-EWC	$\mathsf{C}+\mathsf{F}$	$\mathsf{C}+\mathsf{F}+\mathsf{P}$	93.34	10.946	89.10	88.33	-4.24	0.0204
	Task-1 shaping	$\mathsf{C}+\mathsf{F}$	$\mathsf{C}+\mathsf{P}$	93.34	10.946	91.82	82.57	-1.52	0.392

Appendix

Table 10: Two-task solution-shaping diagnostics under LoRA adaptation and full-parameter fine-tuning on ImageNet-R with ViT-B/16.

\mathsf{C}

\mathsf{F}

, and

\mathsf{P}

denote current-task fitting, Fisher shaping, and historical EWC protection, respectively.

\mathcal{F}_{1}

denotes the Task-1 Fisher statistic reported under each parameterization, and

d_{2}

denotes the Task-2 parameter displacement in the corresponding trainable parameter space. All variants use endpoint references without trajectory averaging.

Method	Metric	Diagonal			Kronecker-factored
Method	Metric	Batched	Per-example	Model-sampled	Batched	Per-example	Model-sampled
EWC	FAA	${51.96}_{\scriptscriptstyle(1.15)}$	${50.60}_{\scriptscriptstyle(1.07)}$	${50.76}_{\scriptscriptstyle(1.14)}$	${53.74}_{\scriptscriptstyle(0.47)}$	${52.75}_{\scriptscriptstyle(0.78)}$	${52.71}_{\scriptscriptstyle(1.17)}$
EWC	AAA	${66.30}_{\scriptscriptstyle(0.53)}$	${66.60}_{\scriptscriptstyle(0.97)}$	${66.60}_{\scriptscriptstyle(1.09)}$	${67.37}_{\scriptscriptstyle(0.18)}$	${67.55}_{\scriptscriptstyle(0.94)}$	${67.33}_{\scriptscriptstyle(0.89)}$
FS-EWC	FAA	${59.29}_{\scriptscriptstyle(0.55)}$	${59.67}_{\scriptscriptstyle(1.07)}$	${59.62}_{\scriptscriptstyle(0.88)}$	${59.82}_{\scriptscriptstyle(0.91)}$	${59.95}_{\scriptscriptstyle(0.93)}$	${59.62}_{\scriptscriptstyle(0.82)}$
FS-EWC	AAA	${70.69}_{\scriptscriptstyle(0.67)}$	${70.88}_{\scriptscriptstyle(0.50)}$	${70.86}_{\scriptscriptstyle(0.51)}$	${71.00}_{\scriptscriptstyle(0.78)}$	${71.16}_{\scriptscriptstyle(0.47)}$	${71.12}_{\scriptscriptstyle(0.46)}$

Appendix

Table 11: Multi-seed results for the Fisher-complementarity study in Table 5 . FAA and AAA are reported as mean

\pm

sample standard deviation (%) over seeds 0, 1, and 2.

Representation	Method	Historical regularization coefficient $\lambda$
Representation	Method	$0$	$2\times 10^{2}$	$2\times 10^{3}$	$2\times 10^{4}$	$2\times 10^{5}$	$2\times 10^{6}$
Diagonal	EWC	49.51 / 64.95	49.24 / 64.66	49.37 / 64.66	49.70 / 65.01	50.56 / 65.55	52.07 / 66.38
Diagonal	FS-EWC	58.53 / 69.82	58.79 / 69.65	58.39 / 69.75	58.59 / 69.66	58.39 / 69.73	58.66 / 69.94
Kronecker- factored	EWC	49.31 / 64.92	49.31 / 64.89	49.70 / 65.15	50.23 / 66.15	53.32 / 67.33	NaN
Kronecker- factored	FS-EWC	58.85 / 69.80	58.39 / 69.78	58.66 / 69.80	58.46 / 69.81	58.79 / 70.14	58.92 / 70.57

Appendix

Table 12: Sensitivity to the historical regularization coefficient

\lambda

under batched empirical Fisher. Each entry reports FAA / AAA (%) for seed 2.

Retained state	Checkpoints $\mathcal{K}_{t}$	Metric	Seed 0	Seed 1	Seed 2	Mean $\pm$ SD
Endpoint	$\{50\}$	FAA	58.20	59.97	59.51	${59.23}_{\scriptscriptstyle(0.92)}$
		AAA	70.48	71.00	70.41	${70.63}_{\scriptscriptstyle(0.33)}$
		BWT	$-7.29$	$-8.66$	$-8.29$	${-8.08}_{\scriptscriptstyle(0.71)}$
Late-stage	$\{30,40,50\}$	FAA	58.53	59.84	59.91	${59.43}_{\scriptscriptstyle(0.78)}$
		AAA	70.51	70.98	70.35	${70.61}_{\scriptscriptstyle(0.33)}$
		BWT	$-7.01$	$-8.35$	$-7.68$	${-7.68}_{\scriptscriptstyle(0.67)}$

Appendix

Table 13: Multi-seed results for retained-state construction on ImageNet-A. Each retained state is constructed by uniformly averaging the LoRA factors at checkpoints

\mathcal{K}_{t}

, with

\omega=1

; the current-task Fisher is estimated after merging. Checkpoint

0

denotes the state before the first optimizer update of each task, while checkpoint

50

denotes the terminal iterate.

Elastic Weight Consolidation

Elastic Weight Consolidation (EWC) protects parameters important to earlier tasks, helping neural networks learn sequentially without catastrophic forgetting.

See all 17 papers

Elastic Weight Consolidation Done Right for Continual Learning
Mar 19, 2026Xuan Liu, Xiaobin ChangRegularization-Based Continual Learning Continual Learning
School of Artificial Intelligence, Sun Yat-sen University, China · Key Laboratory of Intelligent Assessment Technology for Sustainable Tourism, Ministry of Culture and Tourism, Sun Yat-sen University · Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China
The Sequential Price of Continual Learning
Aug 30, 2026Zonghuan Xu, Xingjun MaContinual Learning Elastic Weight Consolidation
Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China
Layers Matter: Why Continual Learning Regularization Should Be Layer-Adaptive
Aug 16, 2026Brian B. Moser, Ahmed Anwar, Tobias Christian Nauen +5Regularization-Based Continual Learning Continual Learning
German Research Center for Artificial Intelligence (DFKI) · RPTU University Kaiserslautern-Landau

Mar 19, 2026J/K move · Enter open · S save

Elastic Weight Consolidation Done Right for Continual Learning

Xuan Liu, Xiaobin Chang

School of Artificial Intelligence, Sun Yat-sen University, China · Key Laboratory of Intelligent Assessment Technology for Sustainable Tourism, Ministry of Culture and Tourism, Sun Yat-sen University · Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China

Regularization-Based Continual Learning Continual Learning Elastic Weight Consolidation Catastrophic Forgetting Fisher Information

Open paper AskPDF arXiv

Weight regularization methods in continual learning (CL) alleviate catastrophic forgetting by assessing and penalizing changes to important model weights. Elastic Weight Consolidation (EWC) is a foundational and widely used approach within this framework that estimates weight importance based on gradients. However, it has consistently shown suboptimal performance. In this paper, we conduct a systematic analysis of importance estimation in EWC from a gradient-based perspective. For the first time, we find that EWC's reliance on the Fisher Information Matrix (FIM) results in gradient vanishing and inaccurate importance estimation in certain scenarios. Our analysis also reveals that Memory Aware Synapses (MAS), a variant of EWC, imposes unnecessary constraints on parameters irrelevant to prior tasks, termed the redundant protection. Consequently, both EWC and its variants exhibit fundamental misalignments in estimating weight importance, leading to inferior performance. To tackle these issues, we propose the Logits Reversal (LR) operation, a simple yet effective modification that rectifies EWC's importance estimation. Specifically, reversing the logit values during the calculation of FIM can effectively prevent both gradient vanishing and redundant protection. Extensive experiments across various CL tasks and datasets show that the proposed method significantly outperforms existing EWC and its variants. Therefore, we refer to it as EWC Done Right (EWC-DR). Code is available at https://github.com/scarlet0703/EWC-DR.

Abstract

Elastic Weight Consolidation

Explore similar work

Elastic Weight Consolidation Done Right for Continual Learning

The Sequential Price of Continual Learning

Layers Matter: Why Continual Learning Regularization Should Be Layer-Adaptive