cs.LGMar 30, 2026

LACE: Loss-Adaptive Capacity Expansion for Continual Learning

Authors: Shivnath Tathe

Organizations: Independent Researcher, Pune, India

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Fixed representational capacity is a fundamental constraint in continual learning: practitioners must guess an appropriate model width before training, without knowing how many distinct concepts the data contains. We propose LACE (Loss-Adaptive Capacity Expansion), a simple online mechanism that expands a model's representational capacity during training by monitoring its own loss signal. When sustained loss deviation exceeds a threshold - indicating that the current capacity is insufficient for newly encountered data - LACE adds new dimensions to the projection layer and trains them jointly with existing parameters. Across synthetic and real-data experiments, LACE triggers expansions exclusively at domain boundaries (100% boundary precision, zero false positives), matches the accuracy of a large fixed-capacity model while starting from a fraction of its dimensions, and produces adapter dimensions that are collectively critical to performance (3% accuracy drop when all adapters removed). We further demonstrate unsupervised domain separation in GPT-2 activations via layer-wise clustering, showing a U-shaped separability curve across layers that motivates adaptive capacity allocation in deep networks. LACE requires no labels, no replay buffers, and no external controllers, making it suitable for on-device continual learning under resource constraints.

Figures & tables

ModelAccExpdfinald_{\text{final}}davgd_{\text{avg}}Precision
LACE (Dynamic)0.999973∼\sim 68100%
Fixed-Large0.999—8484—
Fixed-Small0.998—6464—
TABLE I: Baseline Comparison — 10 Domains
Fig. 1: Exp 1: Training loss and active dimensions for LACE vs fixed baselines over 10 sequential domains. Red dashed lines indicate expansion events.
Fig. 2: Exp 2: Per-domain accuracy over time for LACE (left) and Fixed-Large (right). Both models retain learned domains without forgetting.
ConditionAccuracyDrop
Baseline (all dims active)0.999—
Ablate dim 68 (most critical)0.9810.018
Ablate dim 700.9870.013
Ablate all adapter dims0.9690.030
TABLE II: Ablation Results
Fig. 3: Exp 3: Per-dimension accuracy drop (left) and collective ablation (right). Red bars indicate dimensions exceeding 1% individual drop threshold.
ConfigExpansionsPrecisionAccuracy
K=1K=1 (immediate)9100%0.999
K=3K=3 (confirmed)9100%1.000
TABLE III: Confirmation Window Comparison
Fig. 4: Exp 4: Loss curves and metric comparison for K=1K=1 vs K=3K=3 confirmation windows.
ModelFinal Accdfinald_{\text{final}}Expansions
LACE (Dynamic)0.6763830
Fixed-Large0.88448—
Fixed-Small0.4348—
TABLE IV: Capacity Wall — 50 Domains ( dbase=8d_{\text{base}}=8 , dmax=48d_{\text{max}}=48 )
Fig. 5: Exp 5: Accuracy over time across 50 domains. Fixed-Small plateaus early due to insufficient capacity. LACE grows adaptively and closes the gap to Fixed-Large.
Fig. 6: Exp 5: Active dimensions over time. LACE grows from d=8d=8 to d=38d=38 via 30 expansion events, staying well below Fixed-Large’s constant d=48d=48 .
ModelAccExpdfinald_{\text{final}}Precision
LACE (Dynamic)0.796234100%
Fixed-Large0.821—128—
Fixed-Small0.667—32—
TABLE V: Real-World Experiment — Wikipedia →\to Code →\to Chat
Fig. 7: Real-world experiment: Wikipedia →\to Code →\to Chat. LACE fires precisely at both domain boundaries (orange/purple vertical lines), expands from d=32d=32 to d=34d=34 , and outperforms Fixed-Small while approaching Fixed-Large accuracy.
Fig. 8: GPT-2 layer-wise domain separability. Purity drops in middle layers (3–7) where cross-token attention mixes representations, then recovers in deep layers (8–12).
Fig. 9: Toy experiment: Unsupervised K-Means activation clustering on 3-pattern synthetic data. Left: true patterns. Right: discovered clusters.
Fig. 10: Cluster center similarity matrix for toy K-Means experiment.
Fig. 11: GPT-2 activation space by layer, colored by true domain. Each subplot shows PCA projection of layer activations with cluster purity (p) and number of clusters (k).
Fig. 12: From-scratch training on 10 sequential domains: loss curve and active dimensions. Loss spikes at each domain boundary trigger expansions.
Fig. 13: Capacity wall experiment: training loss across 50 domains. Fixed-Small loss remains elevated throughout; LACE and Fixed-Large converge.
More in this topic

Online Continual Learning

Online continual learning adapts models to sequential data streams while preserving useful knowledge and limiting forgetting.

See all 106 papers

Explore similar work

CardsList
Mar 10, 2026

LCA: Local Classifier Alignment for Continual Learning

A fundamental requirement for intelligent systems is the ability to learn continuously under changing environments. However, models trained in this regime often suffer from catastrophic forgetting. Leveraging pre-trained models has recently emerged as a promising solution, since their generalized feature extractors enable faster and more robust adaptation. While some earlier works mitigate forgetting by fine-tuning only on the first task, this approach quickly deteriorates as the number of tasks grows and the data distributions diverge. More recent research instead seeks to consolidate task knowledge into a unified backbone, or adapting the backbone as new tasks arrive. However, such approaches may create a (potential) mismatch between task-specific classifiers and the adapted backbone. To address this issue, we propose a novel Local Classifier Alignment (LCA) loss to better align the classifier with backbone. Theoretically, we show that this LCA loss can enable the classifier to not only generalize well for all observed tasks, but also improve robustness. Furthermore, we develop a complete solution for continual learning, following the model merging approach and using LCA. Extensive experiments on several standard benchmarks demonstrate that our method often achieves leading performance, sometimes surpasses the state-of-the-art methods with a large margin.

Tung Tran, Danilo Vasconcellos Vargas, Khoat Than

Kyushu University, Fukuoka, Japan · Hanoi University of Science and Technology, Hanoi, Vietnam

Continual LearningIncremental Learning
AskPDF
May 12, 2026

Low-Rank Adapters Initialization via Gradient Surgery for Continual Learning

LoRA is widely adopted for continual fine-tuning of Large Language Models due to its parameter efficiency, modularity across tasks, and compatibility with replay strategies. However, LoRA-based continual learning remains vulnerable to catastrophic forgetting, whose severity depends on how successive task gradients interact: when consecutive task gradients conflict, standard adapter initializations channel updates into subspaces that overwrite previously learned directions. We propose SLICE, a gradient-surgery-based initialization for LoRA adapters in continual learning. SLICE accumulates gradients from both the current task and a replay buffer of prior tasks, reconciles them through a projection operator, and decomposes the result via truncated SVD to initialize the adapter weights. We evaluate SLICE on the TRACE benchmark and sequences of Super-NI tasks, including a set of adversarial Super-NI sequences that we construct by mining task pairs with maximally opposing gradients. Compared to vanilla LoRA, LoRA-GA, and LoRAM, SLICE consistently achieves a better stability-plasticity trade-off, improving Average Performance, Final Performance and Forgetting metrics while preserving General Performance and In Context Performance across both standard and adversarial continual learning sequences.

Joana Pasquali, Ramiro N. Barros, Arthur S. Bianchessi +7

MALTA, Machine Learning Theory and Applications Lab, PUCRS, Porto Alegre, Brazil · Kunumi Institute, Brazil

Gradient InterferenceCatastrophic ForgettingLow-Rank AdaptationContinual Learning for LLMsIncremental LearningContinual LoRA+1
AskPDF
Jun 26, 2026

When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning

Low-Rank Adaptation (LoRA) has become the standard tool for parameter-efficient fine-tuning of large pretrained models. When applied sequentially across tasks in Continual Learning (CL), the standard assumption is that each new task requires a dedicated low-rank adapter. In this work, we challenge this assumption empirically and structurally. We show that task-specific LoRA adapters in CL exhibit significant low-rank redundancy: the subspaces spanned by adapters trained on different tasks substantially overlap, and in many cases earlier adapters can faithfully represent later tasks. Building on this observation, we propose LiteLoRA, a plug-and-play gating mechanism that learns at train time whether to recruit a new adapter or reuse existing low-rank representations. Our method reduces the number of active adapters by 20-70% while matching or exceeding state-of-the-art performance on standard CL benchmarks, revealing that structural redundancy is pervasive and that selective learning is sufficient to achieve stability without sacrificing plasticity.

Tanguy Dieudonné, Giulia Lanzillotta, Enis Simsar +2

Department of Computer Science, ETH Zurich, Zurich, Switzerland.

Continual LearningLow-Rank AdaptationIncremental LearningParameter-Efficient Fine-TuningContinual LoRAAdapter-Based Continual Learning
AskPDF