Context Channel Capacity: An Information-Theoretic Framework for Understanding Catastrophic Forgetting

Ran Cheng

cs.LGMar 8, 2026

Context Channel Capacity: An Information-Theoretic Framework for Understanding Catastrophic Forgetting

Authors: Ran Cheng

Abstract

Catastrophic forgetting remains a central challenge in continual learning (CL), yet lacks a unified information-theoretic explanation for why some architectures forget catastrophically while others do not. We introduce Context Channel Capacity ( $C_\mathrm{ctx}$ ), the mutual information between a CL architecture's context signal and its generated parameters, and prove that zero forgetting requires $C_\mathrm{ctx} \geq H(T)$ , where $H(T)$ is the task identity entropy. We establish an Impossibility Triangle -- zero forgetting, online learning, and finite parameters cannot be simultaneously satisfied by sequential state-based learners -- and show that conditional regeneration architectures (HyperNetworks) bypass this triangle by redefining parameters as function values rather than states. We validate this framework across 8 CL methods on Split-MNIST (1,130+ experiments over 86 days, 4 seeds each), showing that $C_\mathrm{ctx}$ perfectly predicts forgetting behavior: methods with $C_\mathrm{ctx} = 0$ (NaiveSGD, EWC, SI, LwF, CFlow) exhibit catastrophic forgetting (6--97%), while methods with $C_\mathrm{ctx} \approx 1$ (HyperNetwork) achieve zero forgetting (98.8% ACC). We further propose Wrong-Context Probing (P5), a practical diagnostic protocol for measuring $C_\mathrm{ctx}$ , and extend the framework to CIFAR-10 via a novel Gradient Context Encoder that closes the oracle gap from 23.3pp to 0.7pp. A systematic taxonomy of 15+ closed research directions -- including the Hebbian null result (frozen random features outperform learned features), CFlow's $θ_0$ -memorizer phenomenon, and the $S_N$ symmetry barrier to column specialization -- provides the community with precisely diagnosed negative results. Our central design principle: architecture over algorithm -- the context pathway must be structurally unbypassable.

Figures & tables

Paradigm	Method	ACC (%) $\uparrow$	Fgt (%) $\downarrow$	P5 $\Delta$	$\hat{C}_{\mathrm{ctx}}$
State Protection ( $C_{\mathrm{ctx}}=0$ )	NaiveSGD	$18.7\pm 0.3$	$97.1\pm 0.4$	$0.0$	$0.000$
	EWC	$18.9\pm 0.1$	$97.6\pm 0.6$	$0.0$	$0.000$
	SI	$16.4\pm 3.7$	$97.1\pm 0.4$	$0.0$	$0.000$
	LwF	$24.2\pm 0.6$	$54.8\pm 5.5$	$0.0$	$0.000$
Replay ( $C_{\mathrm{ctx}}=0$ )	Experience Replay	$85.9\pm 1.3$	$12.5\pm 1.7$	$0.0$	$0.000$
State Transform. ( $C_{\mathrm{ctx}}\approx 0$ )	CFlow (ODE)	$92.4\pm 0.7$	$6.1\pm 1.1$	$0.0$	$0.000$

Table 1: Split-MNIST results across 8 continual learning methods (4 seeds each). ACC: mean accuracy after all 5 tasks. Fgt: mean forgetting. P5

\Delta

: accuracy drop under Wrong-Context Probing (§ ).

\hat{C}_{\mathrm{ctx}}

: empirical context channel capacity proxy. Methods are grouped by their effective

C_{\mathrm{ctx}}

regime. The table reveals a stark dichotomy: all methods with

C_{\mathrm{ctx}}=0

exhibit substantial forgetting, while methods with

C_{\mathrm{ctx}}\gg 0

achieve zero forgetting.

Method	Zero Forgetting	Online Learning	Bounded $\|\theta\|$	Escape Mechanism
EWC / SI / LwF	$\approx$	✓	✓	Approximate zero forgetting
Experience Replay	✓	$\times$	✓	Breaks causal constraint (revisits data)
Progressive Net	✓	✓	$\times$	Parameters grow linearly with $K$
HyperNet	✓	✓	✓ ∗	$\theta_{k}=g(c_{k})$ : params are function values
∗ Meta-learned parameters $(V,g,\theta_{\mathrm{base}})$ are fixed after joint training; $\theta_{k}$ is regenerated per task.

Table 2: How existing CL methods “escape” the Impossibility Triangle. Each method relaxes one of the three properties. HyperNetworks are unique in satisfying all three—by redefining

\theta

as a function value rather than a state (§ 2.6 ).

Panel of Figure 1 — Figure 1: DND neuron overlap and template similarity analysis. No emergent task specialization is observed: neurons are shared across tasks, and templates fail to differentiate.

Panel of Figure 2 — Figure 2: Evidence for the $S_{N}$ symmetry barrier. (a) HSPC-T fails to produce column specialization. (b) Metabolic pruning’s $\chi$ statistic is uncorrelated with column importance.

Figure 3: CFlow probing results. P5 (wrong context) causes zero accuracy change, while P6 (random $\theta_{0}$ ) causes a $-40$ pp drop. This confirms CFlow is a “ $\theta_{0}$ memorizer”—the context pathway is structurally dead.

Direction	Expts	ACC	Root Cause
Spatial Energy	18	$\pm 0$ pp	Distance-weighted $L_{1}$ $=$ regularization; hurts under capacity pressure
Riverbed Effect	12	$<0$ pp	Uniform suppression $-\Sigma a_{i}\log\pi_{i}$ , not selective routing
SPC-TC (ISTA)	82	$-0.6$ pp	Information bottleneck; learned dict $\approx$ random dict
CCD Theory	20	N/A	Qualitatively suggestive, quantitatively fails (3/5 predictions wrong)
OrthoCorr	8	$+0.3$ %	Conflicts with adaptive threshold; cannot help when $k_{\mathrm{eff}}\geq D$
Metabolic Pruning	16	3/5 fail	$\chi$ has zero discriminative power ( $\rho=0.018$ )

Table 3: Summary of closed research directions. All failures reduce to

C_{\mathrm{ctx}}=0

or a structural barrier to context utilization. Total: 200+ experiments across these directions.

Family	Method	Architecture	Key Hyperparameters
Ctx-free	NaiveSGD	MLP(784 $\to$ 256 $\to$ 128 $\to$ 10)	lr $=10^{-3}$ , Adam
	EWC	same MLP	$\lambda=400$
	SI	same MLP	$c=1.0$
	LwF	same MLP	$\alpha=1.0$ , $T=2.0$
	Exp. Replay	same MLP	buffer $=200$ /task
Ctx-implicit	CFlow	ConvTaskNet(4 filters) + FlowNet	$\theta_{\mathrm{dim}}=4842$ , $c_{\mathrm{dim}}=32$

Table 4: Architecture and hyperparameter details for all methods evaluated.

Method	Normal ACC	P5 ACC	P5b ACC	P6 ACC	P7 ACC	$\hat{C}_{\mathrm{ctx}}$
CFlow	$92.4$	$92.4$	—	$52.4$	—	$0.000$
HyperNet Oracle	$98.8$	$1.2$	$10.4$	$49.7$	—	$0.976$
HyperNet Learned	$98.9$	$3.7$	$19.5$	—	$53.1$	$0.952$

Table 5: Complete probing results for context-conditional methods (4 seeds). P5

=

wrong context, P5b

=

random context, P6

=

random

\theta_{\mathrm{base}}

, P7

=

zero context.

Method	ACC (%)	Fgt (%)	P5 $\Delta$	Context Source
HyperNet Oracle	$77.7\pm 0.9$	$0.0$	$-77.0$	One-hot task ID
HyperNet Learned (batch)	$54.4\pm 2.1$	—	$-0.5$	Batch pixel mean/var
HyperNet + GradCtx	$77.0\pm 0.9$	$0.0$	$-77.0$	$\nabla_{\theta}\mathcal{L}$
NestedCapsule + GradCtx	$\mathbf{78.5\pm 0.6}$	$0.0$	$-78.5$	$\nabla_{\theta}\mathcal{L}$ + routing
NestedCapsule Oracle	$79.5\pm 0.9$	$0.0$	—	One-hot task ID

Table 6: CIFAR-10 extension results (4 seeds). Batch statistics fail; gradient context restores performance; NestedCapsule achieves new SOTA.

Seed	$r_{\mathrm{eff}}$	Nominal rank	Utilization	$C_{\mathrm{ctx}}^{\mathrm{rank}}$ (bits, $b=1$ )
0	59.8	64	93.4%	59.8
1	58.9	64	92.0%	58.9
2	59.7	64	93.3%	59.7
3	59.7	64	93.3%	59.7
Mean	$59.5\pm 0.4$	64	93.0%	$59.5\pm 0.4$

Table 10

Component	Frozen ACC	Learned ACC	$\Delta$
DND feature extractor	81.95%	80.85%	$+1.10$
SPC-TC dictionary	$\approx$ learned	$\approx$ frozen	$\approx 0.0$
Gradient projection matrix	77.0%	—	(no learned variant)

Table 7: Instances of the “frozen

>

learned” phenomenon observed across the NeuralDrift project.

Framework	Info-theoretic	Arch. analysis	$C_{\mathrm{ctx}}$	Empirical	Neg. results
Achille and Soatto (2018)	✓	–	–	✓	–
Doan et al. (2021)	✓	–	–	✓	–
Taheri and Thrampoulidis (2025)	✓	–	–	–	–
Caraffa (2026)	✓	–	–	✓	–
Angelini and others (2023)	✓	–	–	✓	–
Ours	✓	✓	✓	✓	✓

Table 8: Comparison of theoretical CL frameworks. “Arch. analysis” = analyzes how architecture topology affects forgetting. “

C_{\mathrm{ctx}}

” = explicitly models context channel capacity. “Neg. results” = provides systematic negative results with root causes.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment	ACC (%)	Fgt (%)	Key Finding
TBTR 1024/k80	73.19	$-0.42$	Original result
TBTR-train	72.08	$-0.26$	Train data leak = 1.1pp
Frozen Random + TBTR	71.98	$-1.48$	Hebbian learning = 0 contribution
Shuffled Labels TBTR	33.81	39.07	TBTR uses real features
TBTR + Final Pass	80.85	1.30	Post-training fix = $+$ 8.8pp
Pure Random Proj + TBTR	81.95	$\sim$ 0	Beats DND by 1.1pp

Appendix

Table 9: DND diagnostic experiments on Split-MNIST (1024/k80 hidden configuration). “Frozen Random” replaces Hebbian learning with frozen random projections; “Shuffled Labels” randomizes TBTR labels; “Final Pass” adds a post-training template refresh over all tasks; “Pure Random Proj” uses a standalone random projection baseline.

Method	ACC (%)	Fgt (%)	P5 $\Delta$ (pp)
HyperNet Oracle	$90.3\pm 0.7$	$0.0$	$-75.3$
HyperNet Learned	$89.1\pm 0.8$	$0.0$	$-73.3$
CFlow v1	$73.0$	—	$0.0$
SGD + Replay	$66.5$	—	$0.0$

Appendix

Table 10: Permuted-MNIST results (4 seeds where available). Permuted-MNIST is harder than Split-MNIST because tasks share the same label space, making task identification more critical.

Method	ACC (%)	P5 $\Delta$ (pp)	Notes
HyperNet Oracle	$77.7$	—	One-hot task ID
HyperNet Learned (batch)	$54.4$	weak	$\cos>0.995$ across tasks
HyperNet Gradient Context	$77.0\pm 0.9$	$-77.0$	Oracle gap only 0.7pp
NestedCapsule Gradient Ctx	$78.5\pm 0.6$	—	New SOTA
NestedCapsule Oracle	$79.5\pm 0.9$	—	$+1.8$ pp over HyperNet Oracle
SGD + Replay	$\sim$ 46–63	$0.0$	Catastrophic forgetting

Appendix

Table 11: Split-CIFAR-10 results (4 seeds). Batch statistics become nearly identical across tasks (cosine similarity

>0.995

), causing the learned encoder to fail. Gradient context resolves this by using loss gradients as the context signal.

Direction	#Exps	Root Cause
Cosine threshold gating	$\sim$ 20	Gate values cluster near threshold; binary on/off, no graded routing
Output layer tuning (n/k/ $\eta$ )	$\sim$ 30	Output layer has $C_{\mathrm{ctx}}=0$ ; tuning parameters cannot create context
Myelination sweep	$\sim$ 25	Freezing weights preserves current state, does not create task routing
Split/Lock fast-slow	$\sim$ 15	Without context, fast and slow pathways converge to same representation
Overcapacity + ultra-sparse	$\sim$ 20	More capacity $\neq$ more specialization without symmetry breaking
Template repel	$\sim$ 15	Strength 0.1: zero effect. Strength 0.5: ACC drops 4pp (destructive)

Appendix

Table 12: Summary of closed research directions. Each direction was explored with multiple hyperparameter configurations and seeds before being closed. “Root cause” identifies the fundamental reason for failure.

Catastrophic Forgetting

Catastrophic forgetting is the loss of previously learned behavior when a model adapts to new tasks or changing data.

See all 351 papers

Apr 8, 2026

Information as Structural Alignment: A Dynamical Theory of Continual Learning

Catastrophic forgetting is not an engineering failure. It is a mathematical consequence of storing knowledge as global parameter superposition. Existing methods, such as regularization, replay, and frozen subnetworks, add external mechanisms to a shared-parameter substrate. None derives retention from the learning dynamics themselves. This paper introduces the Informational Buildup Framework (IBF), an alternative substrate for continual learning, based on the premise that information is the achievement of structural alignment rather than stored content. In IBF, two equations govern the dynamics: a Law of Motion that drives configuration toward higher coherence, and Modification Dynamics that persistently deform the coherence landscape in response to localized discrepancies. Memory, agency, and self-correction arise from these dynamics rather than being added as separate modules. We first demonstrate the full lifecycle in a transparent two-dimensional toy model, then validate across three domains: a controlled non-stationary world, chess evaluated independently by Stockfish, and Split-CIFAR-100 with a frozen ViT encoder. Across all three, IBF achieves replay-superior retention without storing raw data. We observe near-zero forgetting on CIFAR-100 (BT = -0.004), positive backward transfer in chess (+38.5 cp), and 43% less forgetting than replay in the controlled domain. In chess, the framework achieves a mean behavioral advantage of +88.9 +/- 2.8 cp under independent evaluation, exceeding MLP and replay baselines.

Radu Negulescu

The Informational Buildup Foundation

Continual Learning Catastrophic Forgetting Online Continual Learning

Ask PDF

Aug 16, 2026

Geometry of Forgetting: Representation Flux in Continual Learning

Catastrophic forgetting remains a fundamental obstacle to continual learning, where neural networks lose previously acquired knowledge while learning new tasks. Existing methods primarily mitigate forgetting through parameter regularization or experience replay, while the representation-space dynamics associated with forgetting remain less understood. We investigate latent representation evolution during sequential learning and introduce representation flux, a geometric measure of sample-level representation displacement across training. We show that representation flux is strongly associated with catastrophic forgetting across multiple benchmarks, with temporal analyses indicating that elevated flux can precede subsequent performance degradation. Representation displacement is also associated with confidence degradation, while complementary geometric properties provide additional information about sample-level forgetting. Motivated by these observations, we propose FlowLess-R, a representation-space regularization method that constrains replay representations relative to stored references while allowing continued learning. FlowLess-R is architecture-agnostic and integrates into replay-based methods through a representation-matching term. Experiments on SplitMNIST, SplitFashionMNIST, SplitCIFAR10, and SplitTinyImageNet show improved final average accuracy and reduced forgetting with ER, DER++, and ER-ACE. Our results identify representation flux as an informative geometric marker of forgetting and show that stabilizing latent representations provides a simple strategy for mitigating catastrophic forgetting.

Maksim A. Kazanskii

Catastrophic Forgetting Representation Drift Representation Learning Replay-Based Continual Learning

Ask PDF

Mar 4, 2026

Why Do Neural Networks Forget: A Study of Collapse in Continual Learning

Catastrophic forgetting is a major problem in continual learning, and lots of approaches arise to reduce it. However, most of them are evaluated through task accuracy, which ignores the internal model structure. Recent research suggests that structural collapse leads to loss of plasticity, as evidenced by changes in effective rank (eRank). This indicates a link to forgetting, since the networks lose the ability to expand their feature space to learn new tasks, which forces the network to overwrite existing representations. Therefore, in this study, we investigate the correlation between forgetting and collapse through the measurement of both weight and activation eRank. To be more specific, we evaluated four architectures, including MLP, ConvGRU, ResNet-18, and Bi-ConvGRU, in the split MNIST and Split CIFAR-100 benchmarks. Those models are trained through the SGD, Learning-without-Forgetting (LwF), and Experience Replay (ER) strategies separately. The results demonstrate that forgetting and collapse are strongly related, and different continual learning strategies help models preserve both capacity and performance in different efficiency.

Yunqin Zhu, Jun Jin

Department of Electrical and Computer Engineering, University of Alberta, Edmonton, Canada

Catastrophic Forgetting Incremental Learning Representation Collapse Neural Network Training Dynamics

Ask PDF

Abstract

Catastrophic Forgetting

Explore similar work

Information as Structural Alignment: A Dynamical Theory of Continual Learning

Geometry of Forgetting: Representation Flux in Continual Learning

Why Do Neural Networks Forget: A Study of Collapse in Continual Learning