Layers Matter: Why Continual Learning Regularization Should Be Layer-Adaptive

Brian B. Moser; Ahmed Anwar; Tobias Christian Nauen; Shishir Muralidhara; Federico Raue; René Schuster; Stanislav Frolov; Andreas Dengel

cs.LGAug 16, 2026

Layers Matter: Why Continual Learning Regularization Should Be Layer-Adaptive

Authors: Brian B. Moser, Ahmed Anwar, Tobias Christian Nauen, Shishir Muralidhara, Federico Raue, René Schuster, Stanislav Frolov, Andreas Dengel

Organizations: German Research Center for Artificial Intelligence (DFKI) · RPTU University Kaiserslautern-Landau

Abstract

Continual learning regularizers like EWC fight forgetting by penalizing changes from previous-task parameters with per-parameter importance, typically diagonal Fisher values. Per-parameter looks more flexible than per-layer, but each layer's diagonal Fisher is a weak summary of its actual curvature, missing the top-eigenvalue information that controls forgetting. Adversarial bit-flip attacks and Hessian-spectrum studies show that this missing per-layer sensitivity spans orders of magnitude in neural networks. Under a block-diagonal Hessian assumption, the layer-level analogue of EWC's existing diagonal assumption, we prove three things. Forgetting decomposes as a sum of per-layer terms weighted by each layer's top Hessian eigenvalue. Diagonal-Fisher weights cannot recover this eigenvalue. For instance, two layers with identical Fisher averages can have top eigenvalues differing by a factor as large as the layer width. For the same level of forgetting, uniform regularization loses new-task performance by an amount scaling with the layer condition number. Our theoretical analysis leads to a simple recipe: protect early layers strongly, let deeper layers move. We apply this recipe to EWC and SLCA and show clear improvements in average performance and forgetting metrics.

Figures & tables

Figure 1: Measured per-layer Hessian top eigenvalue

s_{\ell}

on ResNet-50 (left) and ViT-B/16 (right), pretrained on ImageNet, evaluated at a 100-class cross-entropy minibatch.

y

-axis is

\log_{10}s_{\ell}

. ResNet-50:

\kappa=134.6

, with layer1 dominating. ViT-B/16:

\kappa=27.4

, with patch_embed dominating and the twelve transformer blocks within an order of magnitude of each other. Both profiles are heavy-tailed.

Panel of Figure 2 — Figure 2: Depth-weighted EWC on Split-CIFAR-100 (MediumCNN, 10 tasks, 3 seeds, mean $\pm$ std bands). Top, average accuracy across tasks (left) and forgetting (right). Bottom, final-task accuracy (left) and backward transfer (right). The dotted vertical line marks $\alpha=1$ (uniform EWC). The avg-acc peak at $c=200$ sits at $\alpha=2$ with non-overlapping seed bands over uniform.

Figure 3: SLCA depth-weighted sweep on Split-CIFAR-100 (ViT-B/16 + ImageNet-21k pretraining, 10 tasks). Per-block LR multiplier

\lambda_{l}=c\cdot\alpha^{l-L}

L=15

. Top: final-task accuracy (left) and forgetting (right). Bottom: average incremental accuracy (left) and BWT (right). The dotted vertical line is default SLCA (

\alpha=1

). The optimum is consistently away from

\alpha=1

\alpha\in\{1.08,1.18\}

, with best configuration

(c=1,\alpha=1.18)

giving

+0.7\%

final-task accuracy over default SLCA at

(c=1,\alpha=1)

		ImageNet-21k pretrain		MoCoV3 pretrain
Method	Memory-free	Last-acc	Inc-acc	Last-acc	Inc-acc
Joint training (upper bound)	–	93.22	—	89.11	—
GDumb [ 36 ]		81.92	89.46	69.72	80.95
DER++ [ 3 ]		84.50	91.49	63.64	79.55
BiC [ 54 ]		88.45	93.37	80.57	89.39
L2P [ 53 ]	$\checkmark$	82.76	88.48	—	—

Table 1: Class-incremental learning on Split-CIFAR-100 (10 tasks, ViT-B/16), two pretrainings: ImageNet-21k supervised and ImageNet-1k MoCoV3 [ 7 ] . Last-acc and Inc-acc as in [ 60 ] . Prior-method numbers reproduced under SLCA’s protocol; the SLCA rows report multi-seed mean

\pm

std over task-order seeds (Appendix G ). Bold = best in column among non-upper-bound rows.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	$L$	$\kappa$	mean adjacent $\rho$	$\kappa^{\mathrm{eff}}$
ResNet-18	6	55.1	0.975	9.4
ResNet-34	6	75.4	0.967	12.9
ResNet-50	6	195.3	1.211 †	27.7 †
ResNet-101	6	181.4	1.337 †	23.6 †
ViT-B/16	14	31.5	0.867	2.6
ViT-L/16	26	22.3	0.764	1.11

Appendix

Table 2: Effective condition number

\kappa^{\mathrm{eff}}=\kappa/(1+\rho(L-1))

evaluated at the measured mean adjacent

\rho

per backbone. † : measured

\rho\geq 1

, outside the domain of Assumption 10 ; the entry extrapolates the monotone closed form.

Backbone	blockdiag/full	$s_{\ell}$ -bound/full	bound holds
MediumCNN	$1.56\pm 2.34$	$2463\pm 3661$	3/3
ResNet-18	$0.71\pm 0.16$	$780\pm 399$	3/3
ResNet-50	$0.50\pm 0.08$	$954\pm 514$	3/3

Appendix

Table 3: Tightness probes on the task-2 displacement (Split-CIFAR-100, 3 seeds per backbone, mean

\pm

std). “blockdiag/full” is the block-diagonal quadratic divided by the full quadratic; “

s_{\ell}

-bound/full” is the upper bound

\tfrac{1}{2}\sum_{\ell}s_{\ell}\left\lVert\Delta\theta^{(\ell)}\right\rVert_{2}^{2}

divided by the full quadratic; “bound holds” counts runs where

|\text{full}-\text{blockdiag}|

is within the measured coupling bound of Proposition 11 .

ResNet-50			ViT-B/16
block	$d_{\ell}$	$s_{\ell}$	block	$d_{\ell}$	$s_{\ell}$
stem	9,536	39.05	patch_embed	590,592	78.21
layer1	215,808	3,167.48	block0	7,087,872	10.70
layer2	1,219,584	519.36	block1	7,087,872	2.85
layer3	7,098,368	139.32	block2-11	$\sim$ 7M each	3–7
layer4	14,964,736	57.73	head	76,900	6.61

Appendix

Table 4: Measured per-layer top Hessian eigenvalue

s_{\ell}

on a 100-class CIFAR-100-style cross-entropy minibatch.

Backbone	$L$	Params (M)	$\kappa$	Most-sensitive	Least-sensitive
ResNet-18	6	11.2	55.1	layer1	stem
ResNet-34	6	21.3	75.4	layer3	stem
ResNet-50	6	23.7	195.3	layer1	fc
ResNet-101	6	42.7	181.4	layer1	layer4
ViT-B/16	14	85.7	31.5	patch_embed	block1
ViT-L/16	26	303.2	22.3	block9	block0

Appendix

Table 5: Per-layer Hessian condition number

\kappa=\max_{\ell}s_{\ell}/\min_{\ell}s_{\ell}

measured across six ImageNet-pretrained backbones, with means taken over 5 calibration batches. Full per-layer numbers are in Appendix E .

L

is the number of architectural blocks used in the partition.

Figure 4: Per-layer

s_{\ell}

(log scale) on ViT-B/16 under three pretrainings, measured with the same 14-block partition, 100-class calibration head, and a fixed calibration seed.

\kappa

: 31.5 (IN1k-sup), 14.6 (IN21k), 27.4 (MoCoV3). IN1k-sup and MoCoV3 peak at the patch embedding; IN21k peaks at block10.

Panel of Figure 5 — Figure 5: Depth-weighted EWC on Split-CIFAR-10 (SmallCNN, 5 tasks, 3 seeds, mean $\pm$ std bands). Same panel layout as Figure 2 .

Panel of Figure 6 — Figure 6: Depth-weighted EWC on Split-CIFAR-100 with a deeper backbone (ResNet-18, 6 architectural blocks, 10 tasks, 3 seeds). Same panel layout as Figure 2 .

Figure 7: SLCA depth-weighted sweep on Split-CIFAR-100 (ViT-B/16 + MoCoV3, 10 tasks). Same parametrization and panel layout as Figure 3 .

Pretraining	Schedule	$n$	Last-acc	Inc-acc	Forgetting
ImageNet-21k	uniform ( $c=1$ , $\alpha=1$ )	5	$91.35\pm 0.23$	$94.40\pm 0.69$	$5.83\pm 0.42$
ImageNet-21k	geometric, ours ( $c=1$ , $\alpha=1.18$ )	5	$91.80\pm 0.23$	$94.51\pm 0.66$	$4.37\pm 0.49$
ImageNet-21k	measured $s_{\ell}$ ( $c=1$ )	3	$90.96\pm 0.18$	$94.45\pm 0.21$	$6.61\pm 0.28$
MoCoV3	uniform ( $c=1$ , $\alpha=1$ )	3	$84.44\pm 0.29$	$90.12\pm 0.32$	$9.33\pm 0.44$
MoCoV3	geometric, ours ( $c=0.5$ , $\alpha=0.92$ )	3	$84.26\pm 0.48$	$90.08\pm 0.32$	$10.09\pm 0.62$
MoCoV3	measured $s_{\ell}$ ( $c=0.5$ )	3	$76.60\pm 0.71$	$86.68\pm 0.22$	$20.41\pm 0.85$

Appendix

Table 6: Multi-seed SLCA on Split-CIFAR-100 (10 tasks, ViT-B/16), mean

\pm

std over

n

task-order seeds, all values in %. Last-acc and Inc-acc as in Table 1 . Geometric and measured-

s_{\ell}

rows use each pretraining’s best

(c,\alpha)

and best

c

, respectively.

Panel of Figure 8 — Figure 8: TUNA orthogonality sweep on ImageNet-R B0-Inc20 (ViT-B/16). Per-layer orthogonality weight $\lambda_{j}=c/\alpha^{j}$ . Top, final average accuracy (left) and forgetting (right). Bottom, average incremental accuracy (left) and BWT (right). The optimum is visibly above $\alpha=1$ , suggesting that on this benchmark late-block orthogonality should be relaxed relative to early blocks.

Panel of Figure 9 — Figure 9: TUNA orthogonality sweep on CIFAR-100 B0-Inc10 (ViT-B/16, 10 tasks). Per-layer orthogonality weight $\lambda_{j}=c/\alpha^{j}$ . Same panel layout as Figure 8 .

Figure 10: Best avg-acc per schedule family on MediumCNN/Split-CIFAR-100 (3-seed mean ± std). Bars: best over

\alpha

within each family. Dashed line and shaded band: uniform EWC (

\alpha=1

). All three monotone-decreasing schedules consistently beat uniform; the differences between schedule shapes are within or near the seed-to-seed std bands. Linear and step are not run at

c=20{,}000

since the rank ordering is already clear at smaller

c

ResNet-50 / Split-CIFAR-100
$c$	uniform ( $\alpha=1$ )	measured- $s$ schedule
50	—	$0.330\pm 0.117$
200	$\mathbf{0.453\pm 0.042}$	$0.312\pm 0.058$
1000	$0.407\pm 0.036$	$0.278\pm 0.054$
5000	$0.366\pm 0.013$	$0.195\pm 0.027$
ResNet-18 / Split-CIFAR-100

Appendix

Table 7: Direct measured-

s

schedule vs. uniform EWC on Split-CIFAR-100 (3 seeds, mean

\pm

std). Best

c

per backbone in bold.

Figure 11: The interpolated family

\lambda_{\ell}\propto(s_{\ell}/s_{\mathrm{med}})^{\beta}

on Split-CIFAR-100 (left: ResNet-18, right: ResNet-50; 3 seeds, mean

\pm

std, one curve per

c

). At the best

c

per cell, ResNet-18 peaks at

\beta^{*}=0.5

(

0.590\pm 0.014

0.557\pm 0.021

uniform); ResNet-50 is best at

\beta=0

(

0.441\pm 0.015

) and decreases monotonically to

0.330\pm 0.120

\beta=1

Arm	ResNet-18	ResNet-50
uniform ( $\alpha=1$ )	$0.555\pm 0.008$	$0.453\pm 0.034$
static measured	$0.575\pm 0.009$	$0.330\pm 0.095$
online $\beta=1$	$0.570\pm 0.011$	$\mathbf{0.462\pm 0.056}$
online $\beta=0.5$	$\mathbf{0.589\pm 0.008}$	$0.460\pm 0.033$

Appendix

Table 8: Per-task re-measured schedules on Split-CIFAR-100: avg-acc at the best

c

per arm (3 seeds, mean

\pm

std). Best arm per backbone in bold.

Figure 12: Per-backbone forgetting swing on Split-CIFAR-10/CIFAR-100 (forg at

\alpha=2^{3}

minus forg at

\alpha=2^{-2}

, max across

c

) vs measured

\kappa

. Backbones with measured

\kappa

(ResNet-18, ResNet-50) are plotted at their true

\kappa

; SmallCNN/MediumCNN are trained from scratch and are placed on the left margin since their

s_{\ell}

profile was not measured. The trend is monotone in

\kappa

on the two pretrained backbones and consistent in magnitude with the bound’s

(\kappa-1)/\kappa

scaling.

Figure 13: Per-layer operator-norm

s_{\ell}

(blue) and trace-divided-by-width

s_{\ell}^{\mathrm{tr}}=\tr(H^{(\ell,\ell)})/d_{\ell}

(orange), each normalised by its median. Both signals identify the same dominant block ( layer1 on ResNet-50, patch_embed on ViT-B/16), but the operator norm has a sharper peak. The trace proxy is a reasonable cheap surrogate for the qualitative ordering, but understates the dynamic range of

\kappa

Figure 14: Per-layer

\log_{10}s_{\ell}

on BERT-base, evaluated on a random-token calibration batch with a

10

-class linear head on CLS. Embeddings dominate (

s_{\text{embed}}=70

), with the twelve transformer blocks in

[8,26]

and the head at

s_{\text{head}}=21

. Measured

\kappa=9.15

Method	Avg-acc	Forgetting
SeqFT	$0.624\pm 0.010$	$0.158\pm 0.011$
Uniform EWC ( $c=10000$ )	$0.728\pm 0.002$	$0.027\pm 0.006$
Measured- $s$ EWC ( $c=10000$ )	$0.722\pm 0.005$	$0.035\pm 0.004$

Appendix

Table 9: Five-dataset text continual learning with BERT-base (3 seeds, mean

\pm

std). SeqFT is sequential fine-tuning without regularization; both EWC arms are shown at

c=10000

Figure 15: Per-layer

s_{\ell}

on ResNet-50 measured with calibration batches drawn from each of the

10

Split-CIFAR-100 tasks separately. Each line is one architectural block; the

x

-axis is task index. The per-block ordering ( layer1 on top, then layer3 and fc , then layer2 and layer4 , with stem at the bottom) is preserved across all

10

tasks. Absolute magnitudes vary by a factor of

\sim 2

within each block, but the rank is invariant.

Experiment	# jobs	Wallclock/job	GPU-h (approx.)
EWC sweep, SmallCNN/CIFAR-10 (3 seeds, 6 $\times$ 4 grid)	72	5–10 min	10
EWC sweep, MediumCNN/CIFAR-100 (3 seeds, 6 $\times$ 4 grid)	72	10–20 min	20
EWC sweep, ResNet-18/CIFAR-100 (3 seeds, 6 $\times$ 4 grid)	72	20–40 min	40
EWC sweep, ResNet-50/CIFAR-100 (3 seeds, 6 $\times$ 4 grid)	72	30–60 min	60
EWC measured- $s$ , ResNet-18/CIFAR-100 (3 seeds, 4 $c$ , Appendix K )	12	20–40 min	7
EWC measured- $s$ , ResNet-50/CIFAR-100 (3 seeds, 4 $c$ , Appendix K )	12	30–60 min	10

Appendix

Table 10: Per-experiment compute summary. EWC and SLCA depth-weighted sweeps use 3 seeds where indicated. “Wallclock per job” is the typical end-to-end time on a single allocation; “Total” aggregates across the full grid.

Figure 16: Per-layer

\log_{10}|s_{\ell}|

at 11 checkpoints during task 1 of Split-CIFAR-100 (one panel per backbone; mean over 3 seeds with seed band; open markers flag negative signed-Rayleigh curvature; dashed line:

\kappa(t)

Method	MediumCNN	ResNet-18	ResNet-50
ER ( $M=500$ )	$0.441\pm 0.012$	$0.658\pm 0.009$	$0.620\pm 0.027$
ER ( $M=2000$ )	$0.443\pm 0.014$	$0.725\pm 0.012$	$0.684\pm 0.031$
ER ( $M=6200$ )	$0.493\pm 0.002$	$0.775\pm 0.012$	$0.720\pm 0.011$
EWC uniform (best $c$ )	$0.572\pm 0.004$	$0.555\pm 0.009$	$0.453\pm 0.042$
EWC geometric (best $\alpha$ , $c$ )	$0.598\pm 0.023$	$0.542\pm 0.038$	$0.431\pm 0.023$
ER+EWC ( $M=500$ , $c=200$ , $\alpha=2$ )	$0.574\pm 0.005$	$0.593\pm 0.044$	$0.505\pm 0.017$

Appendix

Table 11: Average accuracy on Split-CIFAR-100 (3 seeds, mean

\pm

std): ER at three memory budgets, the EWC arms at their best configuration, and the composition of the two families. Best

c

200

throughout; the best geometric

\alpha

2

on MediumCNN and ResNet-50 and

4

on ResNet-18.

Regularization-Based Continual Learning

See all 13 papers

Learning to Forget: Continual Learning with Adaptive Weight Decay
Apr 29, 2026Aditya A. Ramesh, Alex Lewandowski, Jürgen SchmidhuberDeep Learning Optimization Weight Decay
The Swiss AI Lab, IDSIA USI-SUPSI, Lugano, Switzerland · University of Alberta, Alberta Machine Intelligence Institute, Edmonton, Canada · Center of Excellence for Generative AI, KAUST, Thuwal, Saudi Arabia
KAN-CL: Per-Knot Importance Regularization for Continual Learning with Kolmogorov-Arnold Networks
May 12, 2026Minjong CheonRegularization-Based Continual Learning Continual Learning
Sejong University Department of Computer Science and Engineering
The Sequential Price of Continual Learning
Aug 30, 2026Zonghuan Xu, Xingjun MaContinual Learning Elastic Weight Consolidation
Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China

Apr 29, 2026J/K move · Enter open · S save

Learning to Forget: Continual Learning with Adaptive Weight Decay

Aditya A. Ramesh, Alex Lewandowski, Jürgen Schmidhuber

The Swiss AI Lab, IDSIA USI-SUPSI, Lugano, Switzerland · University of Alberta, Alberta Machine Intelligence Institute, Edmonton, Canada · Center of Excellence for Generative AI, KAUST, Thuwal, Saudi Arabia

Deep Learning Optimization Weight Decay Continual Learning Online Continual Learning Online Learning Catastrophic Forgetting

Open paper AskPDF arXiv

Continual learning agents with finite capacity must balance acquiring new knowledge with retaining the old. This requires controlled forgetting of knowledge that is no longer needed, freeing up capacity to learn. Weight decay, viewed as a mechanism for forgetting, can serve this role by gradually discarding information stored in the weights. However, a fixed scalar weight decay drives this forgetting uniformly over time and uniformly across all parameters, even when some encode stable knowledge while others track rapidly changing targets. We introduce Forgetting through Adaptive Decay (FADE), which adapts per-parameter weight decay rates online via approximate meta-gradient descent. We derive FADE for the online linear setting and apply it to the final layer of neural networks. Our empirical analysis shows that FADE automatically discovers distinct decay rates for different parameters, complements step-size adaptation, and consistently improves over fixed weight decay across online tracking and streaming classification problems.

Abstract

Regularization-Based Continual Learning

Explore similar work

Learning to Forget: Continual Learning with Adaptive Weight Decay

KAN-CL: Per-Knot Importance Regularization for Continual Learning with Kolmogorov-Arnold Networks

The Sequential Price of Continual Learning