Modular Continual Learning via Zero-Leakage Reconstruction Routing and Autonomous Task Discovery

Noureddine Kermiche

cs.LGApr 15, 2026

Modular Continual Learning via Zero-Leakage Reconstruction Routing and Autonomous Task Discovery

Authors: Noureddine Kermiche

Organizations: Western Digital Corporation, Irvine, CA, USA

Abstract

Catastrophic forgetting remains a primary hurdle in sequential task learning for artificial neural networks. We propose a silicon-native modular architecture that achieves structural parameter isolation using Task-Specific Experts and a distributed, outlier-based Gatekeeper. Moving beyond traditional sequential consolidation, our framework utilizes a Simultaneous Pipeline where Teacher learning, Student distillation, and Router manifold acquisition occur in parallel while raw data is present in a localized training session. This approach ensures computational efficiency and complies with privacy mandates like GDPR by deleting raw data as soon as a task is learned. We demonstrate that a Tight-Bottleneck Autoencoder (TB-AE) can effectively distinguish semantically crowded manifolds in high-dimensional latent spaces, overcoming the posterior collapse inherent to standard variational methods. By establishing strict topological boundaries, our TB-AE resolves latent space crowding in 4096-D LLM embeddings to provide a robust, unsupervised novelty signal. Furthermore, we validate an Autonomous Retrieval mechanism that confidently identifies returning manifolds, enabling stable lifelong learning without redundant module instantiation. Empirical results demonstrate that our ``Live Distillation'' approach acts as a natural regularizer, achieving strong retention across computer vision and natural language processing domains without suffering a student fidelity gap.

Figures & tables

Figure 1: Simultaneous Pipeline with Triple-Loss Objective and Automatic Routing during Inference.

Methodology	Task A Retention	CF Risk Profile	GDPR Compliance
Naive (Sequential)	19.40% (Catastrophic)	Severe	High (No buffering)
LwF [4]	79.80% (Drift)	Moderate	High (No buffering)
EWC [2]	84.00% (Degraded)	Moderate	High (Weight penalties only)
Experience Replay [3]	95.10% (High)	Low	Low (Violates Zero-Leakage)
Ours (Simultaneous)	99.42% (Strict Isolation)	Zero (Isolated)	High (Immediate Purge)

Table 1: Split-MNIST Comparative Benchmark (Task A Retention after Task B)

Router Loss	Input Feature	Routing Acc. (%)
VAE (ELBO)	Raw Pixels ( $x$ )	94.80
VAE (ELBO)	Backbone Latents ( $h_{backbone}$ )	86.70
VAE (ELBO)	Student Features ( $h_{student}$ )	61.20
TB-AE (MSE)	Raw Pixels ( $x$ )	96.10
TB-AE (MSE)	Backbone Latents ( $h_{backbone}$ )	88.40
TB-AE (MSE)	Student Features ( $h_{student}$ )	83.10

Table 2: Routing Ablation Matrix (Independent Tasks)

Bottleneck ( $k$ )	MSE Task A	MSE Task B	Discrimination Ratio
Narrow ( $k=4$ )	0.0674	0.2476	3.67x (Underfitting)
Tight ( $k=12$ )	0.0010	0.2109	203.78x (Optimal Separation)
Relaxed ( $k=32$ )	0.0012	0.2104	174.23x (Diminishing Returns)
Wide ( $k=64$ )	0.0011	0.2104	176.47x (Plateau)

Table 3: Ablation Study: Autoencoder Bottleneck (

k

) vs. Discrimination in 4096-D Space

Incoming Manifold	Router Evaluated	Recon. MSE	Autonomous Action
Task A (Returning)	Task A Router	0.0014	RECOGNIZED $\to$ Route to Expert A
Task B (Novel)	Task A Router	0.2105	NOVELTY $\to$ Spawn Expert B

Table 4: Autonomous Retrieval Signal-to-Noise (SNR) Evaluation

Explore similar work

Jul 26, 2026

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

Large language models generalize well to individual tasks but lack an inherent mechanism for learning them sequentially, leading to catastrophic forgetting. To mitigate this, LoRA-based continual learning methods allocate a separate low-rank adapter per task, yet existing approaches either require task identity at inference or sum all adapters indiscriminately, letting irrelevant branches distort the output. Recent gating-based solutions route inputs to the correct adapter but introduce trainable parameters that themselves need protection against forgetting. In this work, we observe that pooled token embeddings from a frozen LLM embedding layer already separate task distributions throughout the learning sequence. A Gaussian mixture model fitted on these embeddings, without any gradient-based training, is sufficient for task-agnostic adapter selection at test time. This eliminates the need for a learned gating module. On the adapter side, constraining each task's parameters to the principal subspace of the pretrained weights via SVD yields a compact latent-space parameterization. Within this subspace, orthogonal regularization directly controls inter-task interference. The resulting system, Latent-LoRA, is replay-free, requires no trainable routing component, and uses substantially fewer parameters per task. Experiments across five model scales and two established continual learning benchmarks show state-of-the-art performance with near-zero forgetting.

Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli +2

College of Health Solutions, Arizona State University, Phoenix, Arizona, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA +1

Continual Learning for LLMs Incremental Learning Parameter-Efficient Fine-Tuning

Ask PDF

Jun 5, 2026

Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning

Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge. Existing methods typically treat parameters uniformly, failing to distinguish between specific task knowledge and shared capabilities. We introduce Mixture of Sparse Experts for Task Agnostic Continual Learning (SETA), a framework that resolves the plasticity-stability conflict through adaptive sparse subspace decomposition into task-specific expert modules. Unlike standard updates, where tasks compete for the same parameters, SETA separates knowledge into unique experts, designed to isolate task-specific patterns, and shared experts, responsible for capturing common features. This structure is maintained through adaptive elastic anchoring and a routing-aware regularization that jointly protect shared knowledge at both the weight and routing levels and enable a unified gating network to automatically retrieve the correct expert combination during inference. Extensive experiments across diverse domain-specific benchmarks demonstrate that SETA achieves competitive or superior overall performance relative to state-of-the-art continual learning baselines, with particularly strong retention of early-task knowledge and improved backward transfer on LLaMA-2 7B and Qwen3-4B.

Fatema Siddika, Md Anwar Hossen, Tanwi Mallick +1

Iowa State University, Ames, USA · Argonne National Laboratory, USA

Continual Learning for LLMs Mixture-of-Experts Language Models Catastrophic Forgetting

Ask PDF

May 25, 2026

SAE-FD: Sparse Autoencoder Feature Distillation for Continual Learning of Large Language Models

Continual learning enables large language models to adapt to evolving tasks without retraining from scratch, yet catastrophic forgetting remains a central obstacle. Among continual learning methods, regularization-based approaches are widely used to constrain model updates and reduce forgetting, operating in weight space, gradient space, or output space. However, these dense representation spaces suffer from feature superposition, where multiple concepts are encoded in overlapping dimensions, making it difficult to selectively protect previously learned knowledge without impeding new-task learning. To address this issue, we propose \method (Sparse Autoencoder Feature Distillation), which anchors model representations in the sparse feature space of a pre-trained Sparse Autoencoder, where dense activations are decomposed into a sparse overcomplete basis that reduces representational entanglement, enabling more targeted regularization with less interference to new-task learning. Experiments on two continual learning benchmarks across three model architectures show that \method consistently outperforms existing regularization-based methods, achieving up to 52.70% average accuracy with only -0.46 backward transfer.

Mingxu Zhang, Yuhan Li, Lujundong Li +3

The Hong Kong University of Science and Technology (Guangzhou) · Nanjing University of Aeronautics and Astronautics · The 63rd Research Institute, National University of Defense Technology, Nanjing

Continual Learning for LLMs

Ask PDF