LACE: Loss-Adaptive Capacity Expansion for Continual Learning

Shivnath Tathe

cs.LGMar 30, 2026

LACE: Loss-Adaptive Capacity Expansion for Continual Learning

Authors: Shivnath Tathe

Organizations: Independent Researcher, Pune, India

Abstract

Fixed representational capacity is a fundamental constraint in continual learning: practitioners must guess an appropriate model width before training, without knowing how many distinct concepts the data contains. We propose LACE (Loss-Adaptive Capacity Expansion), a simple online mechanism that expands a model's representational capacity during training by monitoring its own loss signal. When sustained loss deviation exceeds a threshold - indicating that the current capacity is insufficient for newly encountered data - LACE adds new dimensions to the projection layer and trains them jointly with existing parameters. Across synthetic and real-data experiments, LACE triggers expansions exclusively at domain boundaries (100% boundary precision, zero false positives), matches the accuracy of a large fixed-capacity model while starting from a fraction of its dimensions, and produces adapter dimensions that are collectively critical to performance (3% accuracy drop when all adapters removed). We further demonstrate unsupervised domain separation in GPT-2 activations via layer-wise clustering, showing a U-shaped separability curve across layers that motivates adaptive capacity allocation in deep networks. LACE requires no labels, no replay buffers, and no external controllers, making it suitable for on-device continual learning under resource constraints.

Figures & tables

Model	Acc	Exp	$d_{\text{final}}$	$d_{\text{avg}}$	Precision
LACE (Dynamic)	0.999	9	73	$\sim$ 68	100%
Fixed-Large	0.999	—	84	84	—
Fixed-Small	0.998	—	64	64	—

TABLE I: Baseline Comparison — 10 Domains

Fig. 1: Exp 1: Training loss and active dimensions for LACE vs fixed baselines over 10 sequential domains. Red dashed lines indicate expansion events.

Fig. 2: Exp 2: Per-domain accuracy over time for LACE (left) and Fixed-Large (right). Both models retain learned domains without forgetting.

Condition	Accuracy	Drop
Baseline (all dims active)	0.999	—
Ablate dim 68 (most critical)	0.981	0.018
Ablate dim 70	0.987	0.013
Ablate all adapter dims	0.969	0.030

TABLE II: Ablation Results

Fig. 3: Exp 3: Per-dimension accuracy drop (left) and collective ablation (right). Red bars indicate dimensions exceeding 1% individual drop threshold.

Config	Expansions	Precision	Accuracy
$K=1$ (immediate)	9	100%	0.999
$K=3$ (confirmed)	9	100%	1.000

TABLE III: Confirmation Window Comparison

Fig. 4: Exp 4: Loss curves and metric comparison for $K=1$ vs $K=3$ confirmation windows.

Model	Final Acc	$d_{\text{final}}$	Expansions
LACE (Dynamic)	0.676	38	30
Fixed-Large	0.884	48	—
Fixed-Small	0.434	8	—

TABLE IV: Capacity Wall — 50 Domains (

d_{\text{base}}=8

d_{\text{max}}=48

)

Fig. 5: Exp 5: Accuracy over time across 50 domains. Fixed-Small plateaus early due to insufficient capacity. LACE grows adaptively and closes the gap to Fixed-Large.

Fig. 6: Exp 5: Active dimensions over time. LACE grows from $d=8$ to $d=38$ via 30 expansion events, staying well below Fixed-Large’s constant $d=48$ .

Model	Acc	Exp	$d_{\text{final}}$	Precision
LACE (Dynamic)	0.796	2	34	100%
Fixed-Large	0.821	—	128	—
Fixed-Small	0.667	—	32	—

TABLE V: Real-World Experiment — Wikipedia

\to

Code

\to

Chat

Fig. 8: GPT-2 layer-wise domain separability. Purity drops in middle layers (3–7) where cross-token attention mixes representations, then recovers in deep layers (8–12).

Fig. 9: Toy experiment: Unsupervised K-Means activation clustering on 3-pattern synthetic data. Left: true patterns. Right: discovered clusters.

Fig. 10: Cluster center similarity matrix for toy K-Means experiment.

Fig. 11: GPT-2 activation space by layer, colored by true domain. Each subplot shows PCA projection of layer activations with cluster purity (p) and number of clusters (k).

Fig. 12: From-scratch training on 10 sequential domains: loss curve and active dimensions. Loss spikes at each domain boundary trigger expansions.

Fig. 13: Capacity wall experiment: training loss across 50 domains. Fixed-Small loss remains elevated throughout; LACE and Fixed-Large converge.

Online Continual Learning

Online continual learning adapts models to sequential data streams while preserving useful knowledge and limiting forgetting.

See all 106 papers

LCA: Local Classifier Alignment for Continual Learning
Mar 10, 2026Tung Tran, Danilo Vasconcellos Vargas, Khoat ThanContinual Learning Incremental Learning
Kyushu University, Fukuoka, Japan · Hanoi University of Science and Technology, Hanoi, Vietnam
Low-Rank Adapters Initialization via Gradient Surgery for Continual Learning
May 12, 2026Joana Pasquali, Ramiro N. Barros, Arthur S. Bianchessi +7Gradient Interference Catastrophic Forgetting
MALTA, Machine Learning Theory and Applications Lab, PUCRS, Porto Alegre, Brazil · Kunumi Institute, Brazil
When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning
Jun 26, 2026Tanguy Dieudonné, Giulia Lanzillotta, Enis Simsar +2Continual Learning Low-Rank Adaptation
Department of Computer Science, ETH Zurich, Zurich, Switzerland.

Mar 10, 2026J/K move · Enter open · S save

LCA: Local Classifier Alignment for Continual Learning

Tung Tran, Danilo Vasconcellos Vargas, Khoat Than

Kyushu University, Fukuoka, Japan · Hanoi University of Science and Technology, Hanoi, Vietnam

Continual Learning Incremental Learning

Open paper AskPDF arXiv

A fundamental requirement for intelligent systems is the ability to learn continuously under changing environments. However, models trained in this regime often suffer from catastrophic forgetting. Leveraging pre-trained models has recently emerged as a promising solution, since their generalized feature extractors enable faster and more robust adaptation. While some earlier works mitigate forgetting by fine-tuning only on the first task, this approach quickly deteriorates as the number of tasks grows and the data distributions diverge. More recent research instead seeks to consolidate task knowledge into a unified backbone, or adapting the backbone as new tasks arrive. However, such approaches may create a (potential) mismatch between task-specific classifiers and the adapted backbone. To address this issue, we propose a novel Local Classifier Alignment (LCA) loss to better align the classifier with backbone. Theoretically, we show that this LCA loss can enable the classifier to not only generalize well for all observed tasks, but also improve robustness. Furthermore, we develop a complete solution for continual learning, following the model merging approach and using LCA. Extensive experiments on several standard benchmarks demonstrate that our method often achieves leading performance, sometimes surpasses the state-of-the-art methods with a large margin.

Abstract

Online Continual Learning

Explore similar work

LCA: Local Classifier Alignment for Continual Learning

Low-Rank Adapters Initialization via Gradient Surgery for Continual Learning

When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning