cs.LGNov 11, 2026

Improved Elastic Weight Consolidation as an Optimization Constraint for Continual Learning

Authors: Davide Wiest

Organizations: Department of Computer Science, TU Darmstadt Darmstadt, Germany

Abstract

Regularization-based continual learning reduces catastrophic forgetting by penalizing changes to parameters that mattered for earlier tasks. Elastic Weight Consolidation (EWC), a standard instance, weights its quadratic penalty by the empirical Fisher, usually justified as approximating loss curvature. We instead derive the penalty from what training on a new task should preserve. In this formulation, Continual Learning as an Optimization Constraint (CLOC), new training must keep chosen functionals of learned behavior unchanged, and the local approximation of this constraint is the stored penalty. We show that EWC's empirical Fisher is exactly the first-order form of this constraint when the preserved functional is each old sample's loss. The derivation exposes a flaw: EWC weights each sample by the scale of its loss gradient, so samples the model fits well barely contribute, although later training can still forget them. Improved EWC (IEWC) instead penalizes how far each old output moves off its loss level set, rather than how much its loss changes, which removes this weighting for any differentiable loss and output geometry. Across classification, segmentation, language, forecasting, and diffusion, IEWC variants match or improve on EWC and outperform EWC-DR, a classification-specific correction. In a held-out evaluation with validation-selected hyperparameters, removing this weighting alone raises ImageNet-R accuracy over EWC by 2.00 percentage points (95% CI 0.93-3.07). More broadly, CLOC turns the design of continual-learning penalties into a choice of what to preserve, from which a principled penalty follows by derivation.

Figures & tables

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Elastic Weight Consolidation Done Right for Continual Learning

    Mar 19, 2026Xuan Liu, Xiaobin ChangRegularization-Based Continual LearningContinual Learning

  2. Beyond Parameter Protection: Fisher-Shaped EWC for Replay-Free Continual Learning

    Oct 27, 2026Liying Wang, Simone Magistri, Joost van de WeijerElastic Weight ConsolidationRegularization-Based Continual Learning

  3. The Sequential Price of Continual Learning

    Aug 30, 2026Zonghuan Xu, Xingjun MaContinual LearningElastic Weight Consolidation