Learning without Overwriting: Structural Knowledge Fusion for Continual Learning

Kanishk Bakshi; Amitansu Pattanaik; Kathiravan Srinivasan

cs.LGOct 18, 2026

Learning without Overwriting: Structural Knowledge Fusion for Continual Learning

Authors: Kanishk Bakshi, Amitansu Pattanaik, Kathiravan Srinivasan

Organizations: School of Computer Science and Engineering, Vellore Institute of Technology, Vellore, Tamil Nadu 632014, India · Defence Geoinformatics Research Establishment, Defence Research and Development Organisation, New Delhi 110054, India

Abstract

Catastrophic forgetting in sequential deep learning arises from a fixed-capacity architectural constraint: parameter updates encoding new tasks perturb weight subspaces supporting prior representations. We introduce the Knowledge Fusion Network (KFN), which mitigates the stability-plasticity trade-off in class-incremental learning through novelty-gated structural expansion rather than parameter negotiation within a shared capacity budget. KFN decouples task acquisition from knowledge consolidation: each incoming task is learned in an isolated lightweight specialist network (2.8M parameters) and integrated into the global model via a Structural Fusion Module that allocates dedicated channel capacity conditioned on a quantitative novelty score. Dual structural isolation--channel-space separation with temporally phased gradient unfreezing--protects consolidated knowledge. On ten-task Split CIFAR-100, KFN achieves 11.94% minimum task accuracy and 71.16% task-10 accuracy, whereas shared-parameter baselines degrade severely: LwF to 9.07% average accuracy (0.00% min), Replay++ to 8.91% (85.08% forgetting), and Replay to 16.21% (73.44% forgetting); KFN's forgetting (57.92%) is 15.5 pp lower than Replay. Furthermore, KFN prevents logit-scale drift, maintaining near-symmetric task performance. Across CIFAR-100 and EuroSAT satellite imagery, KFN's class-incremental retention (72.7% and 42.5%) substantially exceeds the best baselines (29.6% and 0.94%). Crucially, KFN achieves parameter-efficient linear scaling, adding 25% of the base parameter count per step and requiring 25.1% of naive backbone duplication over ten tasks (28.1M vs 112.0M). With a 37.1% peak GPU memory reduction over full retraining, KFN establishes novelty-gated expansion as a practical path beyond fixed-capacity architectures.

Figures & tables

Method	$\mathcal{A}_{A}^{\mathrm{init}}$	$\mathcal{A}_{A}^{\mathrm{final}}$	$\mathcal{A}_{B}^{\mathrm{final}}$	$\bar{\mathcal{A}}$	Forgetting
KFN (ours)	$\mathbf{75.76\pm 1.80}$	$\mathbf{9.96\pm 1.53}$	$\mathbf{94.50\pm 0.26}$	$\mathbf{52.23}$	$\mathbf{65.80\pm 2.05}$
ER	$69.31\pm 1.64$	$6.60\pm 0.85$	$96.81\pm 0.35$	$51.70$	$62.71\pm 1.62$
LwF	$\mathbf{94.02\pm 0.17}$	$3.00\pm 0.36$	$96.19\pm 0.11$	$49.60$	$91.02\pm 0.48$
EWC	$40.94\pm 9.60$	$0.00\pm 0.00$	$\mathbf{97.44\pm 0.15}$	$48.72$	$40.94\pm 9.60$
MoE	$63.30\pm 2.46$	$1.93\pm 2.63$	$97.01\pm 0.16$	$49.47$	$61.37\pm 1.92$

TABLE I: Two-task class-incremental learning on CIFAR-10. Each task comprises five classes. KFN is the only method that simultaneously achieves high accuracy on both the retained task and the newly learned task. ALL RESULTS ARE REPORTED AS MEAN

\pm

STANDARD DEVIATION OVER 5 RANDOM SEEDS. ALL 5 prespecified EWC seeds are reported. Forgetting is measured as

\mathcal{A}_{A}^{\mathrm{init}}-\mathcal{A}_{A}^{\mathrm{final}}

. Bold: best; underline : second best.

Method	$\bar{\mathcal{A}}$ $\uparrow$	Plasticity $\uparrow$	$\mathcal{F}$ $\downarrow$	$\sigma(\mathcal{A})$ $\downarrow$	Min $\mathcal{A}$ $\uparrow$
KFN (ours)	22.29	74.41	$\mathbf{57.92}$	$\mathbf{16.73}$	$\mathbf{11.94}$
LwF	9.07	$\underline{83.89}$	83.14	26.56	0.00
Replay++	8.91	$\mathbf{85.48}$	85.08	26.73	0.00
Replay	$\underline{16.21}$	82.31	73.44	$\underline{21.66}$	$\underline{2.44}$

TABLE II: Ten-task class-incremental learning on Split CIFAR-100. Each task comprises ten classes presented sequentially.

\bar{\mathcal{A}}

: average final accuracy across all ten tasks. Plasticity: mean accuracy on each task immediately after learning it (diagonal of the accuracy matrix).

\mathcal{F}

: average forgetting.

\sigma(\mathcal{A})

: standard deviation of final per-task accuracies, measuring performance uniformity. Min

\mathcal{A}

: worst single-task accuracy after all ten tasks. All methods use identical backbone architectures (11.2M parameters). Exploratory single-run disclosure: due to the computational cost of ten sequential tasks per method, each row reports one run per method under an identical seed and class ordering; these values are not seed-averaged and should be read as exploratory rather than as mean

\pm

std estimates (cf. Section III-F ). EWC and MoE are omitted from the ten-task evaluation; given their failure modes on the two-task protocol (complete forgetting and severe plasticity deficit, respectively), their degradation at scale is predictable. Bold: best; underline : second best.

Fig. 3: Task-wise accuracy progression. Heatmap illustrating the evolution of per-task accuracy across the 10-task sequential learning process. KFN demonstrates a controlled, graded forgetting profile compared to the catastrophic interference observed in baselines.

Fig. 4: Effect of modern augmentations on CIL performance. The bar chart illustrates the asymmetric impact of advanced regularizers on shared-parameter methods. While LwF $++$ achieves high Task B accuracy at the cost of Task A collapse, and Replay $+$ boosts Task B by collapsing Task A, KFN achieves optimal, symmetric performance across both domains without requiring loss-level augmentation.

		Task-aware		Class-incremental
Method	Tier	$\mathcal{A}_{A}$	$\mathcal{A}_{B}$	$\mathcal{A}_{A}$	$\mathcal{A}_{B}$	Avg CIL	CIL Ret.
KFN (ours)	—	$\mathbf{59.39\pm 1.18}$	$67.57\pm 0.72$	$\mathbf{43.24\pm 3.90}$	$61.43\pm 2.76$	$52.34$	$\mathbf{72.7\%}$
LwF	Vanilla	$\mathbf{74.20\pm 0.42}$	$68.80\pm 0.28$	$7.49\pm 0.50$	$68.78\pm 0.28$	$38.14$	$10.1\%$
LwF $++$	Heavily aided	$71.34\pm 0.19$	$73.21\pm 0.73$	$0.94\pm 0.77$	$73.21\pm 0.72$	$37.07$	$1.3\%$
Replay	Vanilla	$51.54\pm 0.31$	$75.69\pm 0.56$	$\underline{15.24\pm 0.96}$	$75.10\pm 0.50$	$\mathbf{45.17}$	$\underline{29.6\%}$
Replay $+$	Intermediate	$44.05\pm 1.40$	$\mathbf{76.34\pm 0.70}$	$6.06\pm 1.19$	$\mathbf{75.96\pm 0.55}$	$41.01$	$13.8\%$

TABLE III: Two-task CIFAR-100 under task-aware and class-incremental evaluation. The tier column indicates the augmentation level (see text). CIL Retention measures the fraction of initial task-aware accuracy preserved under CIL evaluation:

\mathcal{A}_{A}^{\mathrm{CIL}}/\mathcal{A}_{A}^{\mathrm{init,TA}}\times 100

. ALL RESULTS ARE REPORTED AS MEAN

\pm

STANDARD DEVIATION OVER 5 RANDOM SEEDS. CIL Retention is computed as the ratio of means,

\mathcal{A}_{A}^{\mathrm{CIL}}/\mathcal{A}_{A}^{\mathrm{init,TA}}\times 100

, applied to the displayed mean accuracies. Bold: best; underline : second best.

Variant	AvgAcc	Plasticity	F	MinAcc	Params
FULL (complete KFN)	$\mathbf{41.00\pm 0.67}$	$71.04\pm 2.80$	$\mathbf{37.55\pm 3.69}$	$\mathbf{33.62\pm 1.89}$	14,147,346
REPLAY_ONLY	$40.50\pm 0.21$	$\mathbf{75.62\pm 0.43}$	$43.91\pm 0.42$	$24.76\pm 1.28$	13,987,729
FUSION_ONLY	$16.39\pm 0.11$	$81.67\pm 0.14$	$81.60\pm 0.19$	$0.00\pm 0.00$	13,987,729
NO_GRADUAL	$40.92\pm 0.78$	$69.56\pm 3.11$	$35.80\pm 4.27$	$33.28\pm 4.32$	14,147,346

TABLE IV: Ablation study on structural isolation mechanisms. Evaluated under controlled training settings. FULL: complete KFN. REPLAY_ONLY: no structural expansion (shared bottleneck). FUSION_ONLY: structural expansion without replay-anchored distillation. NO_GRADUAL: structural expansion with replay, but without phased unfreezing. MinAcc is the worst single-task accuracy after all five tasks.

Fig. 5: Mechanisms of structural isolation. Channel-space isolation dedicates separate channel groups to distinct tasks, while gradient isolation (phased unfreezing) and representation anchoring control optimization dynamics. Removing any single component significantly degrades retention.

Task	Role	Ref.	Overlap	Expected Gate
$T_{0}$	Base task	—	—	base
$T_{1}$	100% overlap with $T_{0}$	$T_{0}$	1.00	reuse
$T_{2}$	50% overlap with $T_{0}$	$T_{0}$	0.50	intermediate
$T_{3}$	0% overlap with $T_{0}$	$T_{0}$	0.00	expand
$T_{4}$	100% overlap with $T_{3}$	$T_{3}$	1.00	reuse

TABLE V: Controlled task protocol. Each task’s overlap with its designated reference is fixed by construction, yielding a known ground-truth gate decision.

Configuration	$\bar{A}$ $\uparrow$	Min $A$ $\uparrow$	$F$ $\downarrow$	BWT	Exp	Gate Acc	Params (M)
Always reuse	98.30 $\pm$ 1.67	95.24 $\pm$ 5.05	0.74 $\pm$ 0.32	$-$ 0.37 $\pm$ 0.31	0.0	0.67	0.115
Always expand	97.41 $\pm$ 2.32	92.48 $\pm$ 6.87	1.57 $\pm$ 0.58	$-$ 0.95 $\pm$ 0.31	4.0	0.33	0.138
Adaptive $\tau{=}0.30$	98.07 $\pm$ 1.77	95.40 $\pm$ 3.91	1.27 $\pm$ 0.90	$-$ 0.92 $\pm$ 0.80	2.0	1.00	0.127
Adaptive $\tau{=}0.50$	98.07 $\pm$ 1.77	95.40 $\pm$ 3.91	1.27 $\pm$ 0.90	$-$ 0.92 $\pm$ 0.80	2.0	1.00	0.127
Adaptive $\tau{=}0.70$	98.19 $\pm$ 1.53	95.24 $\pm$ 4.23	0.89 $\pm$ 0.33	$-$ 0.54 $\pm$ 0.30	1.8	1.00	0.126

TABLE VI: Controlled capacity-reuse experiment. All metrics are mean

\pm

std over 5 seeds. Gate Acc measures the fraction of three preregistered gate checks (

T_{1}{\to}\text{reuse}

T_{3}{\to}\text{expand}

T_{4}{\to}\text{reuse}

) that are satisfied. Exp: number of expansion events per sequence (out of 4 possible).

Fig. 6: Domain robustness of the stability-plasticity balance. Side-by-side comparison of Task A and Task B class-incremental accuracies on CIFAR-100 and EuroSAT. While shared-parameter baselines exhibit severe, domain-dependent asymmetry (favoring Task B across both domains while collapsing on Task A), KFN maintains a stable, symmetric predictive balance across both visual distributions.

Method	$\mathcal{A}_{A}$	$\mathcal{A}_{B}$	$\mathcal{A}_{A}$	$\mathcal{A}_{B}$	Avg CIL	CIL Ret.
	Task-aware		Class-incremental
KFN (ours)	$84.62\pm 1.77$	$98.25\pm 0.41$	$\mathbf{35.94\pm 6.50}$	$96.10\pm 1.36$	$\mathbf{66.02}$	$\mathbf{42.5\%}$
LwF ++	$\mathbf{97.80\pm 0.30}$	$97.86\pm 0.57$	$0.94\pm 1.10$	$97.86\pm 0.57$	$49.40$	$1.0\%$
Replay ++	$98.54\pm 0.26$	$\mathbf{99.22\pm 0.30}$	$0.00\pm 0.00$	$\mathbf{99.22\pm 0.30}$	$\underline{49.61}$	$0.0\%$

TABLE VII: Two-task class-incremental learning on EuroSAT. Each task comprises five land-use classes. CIL Retention measures the preservation of initial knowledge under unified evaluation:

\mathcal{A}_{A}^{\mathrm{CIL}}/\mathcal{A}_{A}^{\mathrm{init,TA}}\times 100

. ALL RESULTS ARE REPORTED AS MEAN

\pm

STANDARD DEVIATION OVER 5 RANDOM SEEDS. CIL Retention is computed as the ratio of means,

\mathcal{A}_{A}^{\mathrm{CIL}}/\mathcal{A}_{A}^{\mathrm{init,TA}}\times 100

, applied to the displayed mean accuracies. Bold: best; underline : second best.

Fig. 7: Parameter-efficient linear scaling. Parameter growth across the 10-task CIFAR-100 sequence. KFN exhibits linear parameter growth under task-dissimilar streams ( $\approx$ 28.1M final, growing at $\approx$ 2.8M per task—4 $\times$ more efficient than naive backbone duplication, which would add 11.2M per task to reach 112M total).

Method	Total Params	$\Delta P/T$	Avg Acc/M $\uparrow$	Min $\mathcal{A}$ /M $\uparrow$	Plasticity/M $\uparrow$
KFN (ours)	28.1M	2.8M	0.79	0.42	2.65
LwF ++	11.2M	0	$\mathbf{0.81}$	0.00	7.49
Replay ++	11.2M	0	0.80	0.00	$\mathbf{7.63}$

TABLE VIII: Parameter scaling and efficiency. Evaluated on the 10-task Split CIFAR-100 benchmark. Acc/M measures average raw parameter efficiency. Min

\mathcal{A}

/M measures parameter efficiency for the worst-served task.

\Delta P/T

is the per-task parameter overhead.

KFN phase breakdown
Method	Time (s)	Peak Memory (MB)	Speedup
Full Retrain	1032.1	1260.1	$1.00\times$
KFN (ours)	1129.3	792.6	0.92 $\times$
Phase 2 (Specialist)	5.6%	—	—
Phase 3 (Integration)	84.2%	—	—
Phase 4 (Bias Correction)	0.1%	—	—

TABLE IX: Computational and memory overhead during incremental learning. Metrics are measured during the integration of Task B on the CIFAR-10 protocol. Phase 2 (specialist training), Phase 3 (structural fusion), and Phase 4 (bias correction) compose the full KFN pipeline.

Class-Incremental Learning

Class-incremental learning studies how models can add new categories over time while retaining the ability to recognize previously learned classes.

See all 154 papers

Interference and Retention in Continual Learning
Jul 10, 2026Julius StörkContinual Learning Online Continual Learning
VARTA Microbattery GmbH, Ellwangen, Germany
KAN-CL: Per-Knot Importance Regularization for Continual Learning with Kolmogorov-Arnold Networks
May 12, 2026Minjong CheonRegularization-Based Continual Learning Continual Learning
Sejong University Department of Computer Science and Engineering
SPARS: A Stability-Plasticity-Aware Region Selection Framework for Continual Learning
Oct 14, 2026Teif Aldaajani, Huda Hakami, Maha A. ThafarRegularization-Based Continual Learning Incremental Learning
Department of Computer Science, College of Computers and Information Technology, Taif University, Taif, 21944, Saudi Arabia .

Jul 10, 2026J/K move · Enter open · S save

Interference and Retention in Continual Learning

Julius Störk

VARTA Microbattery GmbH, Ellwangen, Germany

Continual Learning Online Continual Learning Catastrophic Forgetting

Open paper AskPDF arXiv

Continual learning commonly relies on post-hoc mechanisms such as replay, elastic regularization, or distillation. This work argues that forgetting should instead be modeled directly as interference between tasks. In the frozen-feature regime, forgetting from learning a new task is exactly the interference energy induced on the old task. In deep networks, the same quantity is recovered through path-averaged curvature with minimal additional forward passes. When task supports are disjoint, forgetting can be eliminated structurally and when task supports overlap in conflicting directions, a non-zero distortion floor is unavoidable. The same geometry optimally merges models through task-aware orthogonalization. From this analysis we derive Interference-Gated Functional Allocation (IGFA), a replay-free, Fisher-free method that shares directions when tasks align and protects them when they conflict. Across benchmarks, IGFA achieves lossless retention when tasks are structurally separable and moves unavoidable cost from irreversible forgetting into deferred but recoverable plasticity when they are not. It matches the strongest replay-free structural baselines on dissimilar-task streams and improves on unconditional projection when similarity makes transfer worth preserving.

Abstract

Class-Incremental Learning

Explore similar work

Interference and Retention in Continual Learning

KAN-CL: Per-Knot Importance Regularization for Continual Learning with Kolmogorov-Arnold Networks

SPARS: A Stability-Plasticity-Aware Region Selection Framework for Continual Learning