Improved Elastic Weight Consolidation as an Optimization Constraint for Continual Learning
Organizations: Department of Computer Science, TU Darmstadt Darmstadt, Germany
Abstract
Regularization-based continual learning reduces catastrophic forgetting by penalizing changes to parameters that mattered for earlier tasks. Elastic Weight Consolidation (EWC), a standard instance, weights its quadratic penalty by the empirical Fisher, usually justified as approximating loss curvature. We instead derive the penalty from what training on a new task should preserve. In this formulation, Continual Learning as an Optimization Constraint (CLOC), new training must keep chosen functionals of learned behavior unchanged, and the local approximation of this constraint is the stored penalty. We show that EWC's empirical Fisher is exactly the first-order form of this constraint when the preserved functional is each old sample's loss. The derivation exposes a flaw: EWC weights each sample by the scale of its loss gradient, so samples the model fits well barely contribute, although later training can still forget them. Improved EWC (IEWC) instead penalizes how far each old output moves off its loss level set, rather than how much its loss changes, which removes this weighting for any differentiable loss and output geometry. Across classification, segmentation, language, forecasting, and diffusion, IEWC variants match or improve on EWC and outperform EWC-DR, a classification-specific correction. In a held-out evaluation with validation-selected hyperparameters, removing this weighting alone raises ImageNet-R accuracy over EWC by 2.00 percentage points (95% CI 0.93-3.07). More broadly, CLOC turns the design of continual-learning penalties into a choice of what to preserve, from which a principled penalty follows by derivation.
Figures & tables
| Symbol | Meaning |
|---|---|
| Output Jacobian, output-loss gradient, and . | |
| Mean from scalar preservation. | |
| Squared dual norm of the output-loss gradient. | |
| Mean from level-set preservation. |
| Task | Benchmark | Model | Metric | Seq. | EWC (OPG) | EWC-DR | IEWC | IEWC+RDW |
|---|---|---|---|---|---|---|---|---|
| Classification | ImageNet-R | ViT-B/16 | Acc. | .2896 | .5981 | .5687 | .6332 | .6250 |
| Fgt. | .7066 | .3595 | .3933 | .3142 | .3285 | |||
| Language | TRACE | Qwen2.5-3B | Score | .5760 | .5972 | .5972 | .5938 | .6272 |
| Fgt. | .1139 | .0762 | .0781 | .0820 | .0046 | |||
| Forecasting | Weather, ECL, Traffic | PatchTST | MSE | .2733 | .2388 | – | .2418 | .2383 |
| Fgt. | .0535 | .0140 | – | .0191 | .0147 |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Flip rate | Corrupt/clean | Corrupt trace share | Leading-vector alignment | Top-5 overlap | |||
|---|---|---|---|---|---|---|---|
| ratio | OPG | IEF | OPG | IEF | OPG | IEF | |
| 81.26 | 0.783 | 0.054 | 0.426 | 1.000 | 0.902 | 0.996 | |
| 36.67 | 0.789 | 0.104 | 0.720 | 0.999 | 0.923 | 0.997 | |
| 11.23 | 0.736 | 0.202 | 0.875 | 0.998 | 0.930 | 0.990 | |
| IEF minus OPG | 5% flips | 10% flips | 20% flips |
|---|---|---|---|
| Corrupt trace share | [ ] | [ ] | [ ] |
| Leading-vector alignment | [ ] | [ ] | [ ] |
| Top-5 overlap | [ ] | [ ] | [ ] |
| Domain | Data splits | Validation runs | Test runs |
|---|---|---|---|
| ImageNet-R | Stratified 80%/20% split of the original 24,000-example train set for train/test; the original 6,000-example test set is used only for validation | 3 class orders | 10 paired class orders |
| TRACE | Distributed train/eval/test files for C-STANCE, ScienceQA, and NumGLUE-cm; prompt/answer overlaps are zero | 3 task orders | 6 task orders 2 optimizer seeds |
| Domain | Method | Selected | Final performance | Forgetting |
|---|---|---|---|---|
| ImageNet-R | Sequential | – | 0.2808 0.0438 | 0.7018 0.0467 |
| Offline/joint | – | 0.8199 0.0029 | 0.0000 0.0000 | |
| OPG, uniform | 1.0000 | 0.6416 0.0301 | 0.2835 0.0337 | |
| IEF, uniform | 1.0000 | 0.6616 0.0168 | 0.2566 0.0177 | |
| OPG, RDW | 1.0000 | 0.6445 0.0278 | 0.2803 0.0310 | |
| IEF, RDW | 1.0000 | 0.6617 0.0185 | 0.2565 0.0196 |
| Domain | Type | Comparison | Mean paired difference | 95% CI | Exact | Holm |
|---|---|---|---|---|---|---|
| ImageNet-R | Primary | IEF uniform OPG uniform | 0.0200 | [0.0093, 0.0307] | 0.0078 | – |
| Secondary | IEF, RDW OPG, RDW | 0.0171 | [0.0092, 0.0251] | 0.0039 | 0.0156 | |
| Secondary | OPG, RDW OPG, uniform | 0.0030 | [-0.0024, 0.0084] | 0.2637 | 0.5273 | |
| Secondary | IEF, RDW IEF, uniform | [-0.0017, 0.0018] | 0.8867 | 0.8867 | ||
| Secondary | Sampled model Fisher OPG, uniform | 0.0061 | [-0.0021, 0.0144] | 0.1270 | 0.3809 | |
| Secondary | MAS OPG, uniform | -0.1814 | [-0.1986, -0.1643] | 0.0020 | 0.0098 |
| Task | Metric | Seq. | EWC (OPG) | EWC-DR | IEWC | IEWC+RDW |
|---|---|---|---|---|---|---|
| Classification | Final avg. acc. | 0.2896(.0405) | 0.5981(.0337) | 0.5687(.0263) | 0.6332(.0232) | 0.6250(.0263) |
| Avg. forgetting | 0.7066(.0466) | 0.3595(.0378) | 0.3933(.0286) | 0.3142(.0250) | 0.3285(.0278) | |
| NLP | Final avg. score | 0.5760(.0027) | 0.5972(.0020) | 0.5972(.0059) | 0.5938(.0034) | 0.6272(.0080) |
| Avg. forgetting | 0.1139(.0111) | 0.0762(.0034) | 0.0781(.0109) | 0.0820(.0122) | 0.0046(.0079) | |
| Forecasting | Final avg. MSE | 0.2733(.0065) | 0.2388(.0021) | – | 0.2418(.0038) | 0.2383(.0007) |
| MSE forgetting | 0.0535(.0081) | 0.0140(.0020) | – | 0.0191(.0047) | 0.0147(.0021) |
| Task type | Dataset, sequence, and model | Selected values |
|---|---|---|
| Classification | ImageNet-R; ten 20-class distributions; pretrained ViT-B/16 with LoRA rank 4. | OPG: ; DR: ; U-IEWC: ; weighted IEWC: . |
| NLP | TRACE, C-STANCE FOMC ScienceQA; pretrained Qwen2.5-3B with LoRA rank 8. | OPG: ; DR: ; U-IEWC: ; weighted IEWC: . |
| Forecasting | Weather ECL Traffic; PatchTST trained from scratch. | OPG: ; U-IEWC: ; weighted IEWC: . |
| Segmentation | VOC 15–5, base then novel classes; pretrained ConvNeXtV2-tiny FPN with full fine-tuning. | OPG: ; DR: ; U-IEWC: ; weighted IEWC: . |
| Diffusion | MNIST DDPM, labels ; conditional DDPM trained from scratch. | OPG: ; U-IEWC: ; weighted IEWC: ; output ablation: . |
| Method | |||
|---|---|---|---|
| OPG-EWC | 0.5617 0.0062 | 0.5132 0.0112 | 0.2773 0.0320 |
| EWC-DR | 0.5932 0.0296 | 0.5310 0.0226 | 0.4598 0.0095 |
| IEWC | 0.5978 0.0260 | 0.5642 0.0215 | 0.5322 0.0239 |
| Method | Noise | Entries | Tail mass ratio | Profile L1 | Log-scale RMSE |
|---|---|---|---|---|---|
| OPG-EWC | 0.1 | 512 | 0.0288 | 0.2610 | 1.4660 |
| OPG-EWC | 0.25 | 512 | 0.0244 | 0.2737 | 1.5337 |
| IEWC | 0.1 | 512 | 1.0833 | 0.0350 | 0.0350 |
| IEWC | 0.25 | 512 | 1.1003 | 0.0154 | 0.0415 |
| Symbol | Meaning |
|---|---|
| Parameter gradient of the selected scalar functional; IEWC uses the old-sample loss. | |
| Mean from scalar-loss preservation; only the likelihood-score case is the observed-label empirical-Fisher algebra. | |
| Fisher information averaging score outer products over labels sampled from the frozen predictive distribution. | |
| Squared dual norm of the output-loss gradient under the matched output metric. | |
| Level-set-normalized mean ; the undamped Euclidean case is iEF. | |
| Damping attenuation relative to the undamped summand. |