Width Expansion as a Method for Class Incremental Learning
Authors: A. L. S. Conde, Y. Elkhatib, C. M. Ranieri
Organizations: Institute of Geosciences and Exact Sciences, São Paulo State University (UNESP), Av. 24-A, 1515, Rio Claro, 13506-692, SP, Brazil · School of Computing Science, University of Glasgow, Glasgow, G12 8QQ, Scotland, United Kingdom
Class Incremental Learning (Class-IL) requires models to learn new classes over time while preserving previously acquired knowledge without access to past data or task identity. This setting intensifies the stability-plasticity dilemma and makes catastrophic forgetting a central challenge. Existing approaches include regularization, knowledge distillation, replay, and architectural expansion. However, many expansion methods rely on explicit task identifiers or predefined growth strategies, limiting their applicability when task boundaries are unavailable at inference time. This work proposes a dynamic width expansion method that increases the number of neurons within existing layers according to a normalized loss criterion, without requiring task-specific information. An attention mechanism with persistent key-value memory is also incorporated to stabilize feature representations and reduce interference between previously learned and newly introduced classes. The approach is evaluated on Split MNIST and Split CIFAR-100 under the standard Class-IL protocol. Experiments compare fixed-capacity and dynamically expanding architectures, both with and without attention, combined with established continual learning methods including EWC, LwF, and A-GEM. Results show that progressive width expansion consistently improves performance over fixed architectures, particularly when combined with functional methods and A-GEM. The combination of width expansion and attention provides the most consistent gains. Overall, dynamic width expansion based on representational demand provides an effective and flexible strategy for Class-IL, although uncontrolled growth may increase overfitting and computational cost.
Figures & tables
Figure 1: Attention Mechanism
Figure 2: Example of Split MNIST and classification along the three types of incremental learning.
Figure 3: MLP model for MNIST
Figure 4: WE model for MNIST
Figure 5: MLP model with attention mechanism for MNIST
Figure 6: WE model with attention mechanism for MNIST
Figure 7: Sample images from the CIFAR-100 dataset illustrating the diversity of object categories and visual appearances.
Figure 8: Common feature extractor for CIFAR-100 models
Figure 9: CNN model for CIFAR-100
Figure 10: CNN model for CIFAR-100 with width dynamic layers
Figure 11: CNN model with attention mechanism for CIFAR-100
Figure 12: CNNWE model with attention mechanism for CIFAR-100
Strategy
MLP
WE
MLP + Attention
WE + Attention
Joint
97.76±0.12
-
-
-
None
19.62±0.07
-
-
-
EWC
19.60±0.09
19.41±0.10
18.67±3.06
19.75±0.17
SI
19.63±0.05
19.39±0.11
19.07±2.59
19.68±0.04
LwF
29.79±0.85
29.28±2.00
39.63±6.25
44.04±3.93
ER
88.80±0.73
83.44±0.93
56.74±33.12
87.74±0.60
Table 1: Results for Split MNIST Values represent Mean ± Standard Deviation
Figure 13: Comparison between LwF and A-GEM on Split-Mnist
Strategy
CNN
WE
CNN + Attention
WE + Attention
Joint
48.56±0.51
-
-
-
None
8.05±0.17
-
-
-
EWC
6.46±0.34
7.27±0.11
6.49±0.30
7.53±0.11
SI
5.82±0.16
8.43±0.08
5.89±0.17
8.30±0.09
LwF
12.58±0.30
14.20±0.34
15.13±0.56
17.91±0.90
LwM
10.64±0.26
12.52±0.38
10.39±0.85
12.18±0.86
Table 2: Results for Split CIFAR-100 without pretraining Values represent Mean ± Standard Deviation
Figure 14: Comparison between LwF and A-GEM on Split CIFAR-100
Strategy
CNN
WE
CNN + Attention
WE + Attention
Joint
44.09±0.40
-
-
-
None
8.04±0.06
-
-
-
EWC
7.58±0.24
7.64±0.17
7.83±0.14
7.86±0.15
SI
7.35±0.11
7.38±0.06
5.61±0.21
5.65±0.22
LwF
14.19±0.32
14.43±0.40
16.13±0.59
16.41±0.69
ER
31.72±0.26
31.60±0.56
33.38±0.57
33.28±0.38
Table 3: Results for Split CIFAR-100 with pretraining values represent Mean ± Standard Deviation
Figure 15: LwF comparison between with and without pretraining.
Figure 16: A-GEM comparison between with and without pretraining.
Class Incremental Learning (CIL) aims to learn new concepts consistently from a data stream without forgetting. Unlike typical CIL methods which need to learn a model from scratch, pre-trained model (PTM) can easily adapt to a new task with fine-tuning. However, existing PTM-based CIL methods fail to achieve a trade-off between performance and computational expenditure, i.e., they either adopt the same parameter space so that leading catastrophic forgetting, or expand a new branch for each task but adding more computational cost. To this end, we propose MetrIc Learning with Expandable Subspace (Miles) to harness the prior information within pre-trained knowledge, thereby orchestrating an efficient expansion of the parameter space through guided optimization. Specifically, it decouples the learnable modules with the pre-trained model, exploiting prior information from intermediate features of the backbone network to enable more flexible parameter expansion. Then, a central loss is adopted to guide the new category to cluster towards the corresponding prototype in the new task subspace while incorporating an auxiliary distance regularization term to maintain metric equilibrium across tasks. Extensive experiments on six benchmark datasets demonstrate that Miles achieves state-of-the-art performance in various CIL settings.
Kai Jiang, Zisong Lin, Hongyuan Zhang +2
National Key Laboratory of Radar Signal Processing, Xidian University, Xi’an 710071, China · The University of Hong Kong, Hong Kong SAR, China · Institute of Artificial Intelligence (TeleAI) of China Telecom, China
Class-Incremental Learning (CIL) aims to continuously learn new classes without forgetting previously acquired knowledge. Parameter-efficient fine-tuning with pre-trained models reduces parameter overhead but can suffer from cumulative interference and suboptimal alignment between inference samples and specialized modules. We propose Dynamic LoRA-Experts and Prototype-Ensemble Matching (DLEPEM), a two-stage rehearsal-free framework. DLEPEM allocates a task-specific LoRA-Expert for each incremental task to reduce cross-task interference, then combines frozen pre-trained-model prototypes with task-adaptive LoRA-Expert prototypes for reliable task-level discrimination. Experiments on standard CIL and Few-Shot CIL benchmarks demonstrate strong performance under the evaluated protocols.
Hongwei Zhao, Rui Liu, Yansong Liu
School of Computer Science and Engineering, Beihang University
The task of Long-tailed Class Incremental Learning (LT-CIL) addresses the sequential learning of new classes from datasets with imbalanced class distributions. This scenario intensifies the fundamental problem of catastrophic forgetting, inherent to continual learning, with the dual challenges of under-learning minority classes and overfitting majority classes. To tackle these combined issues, this paper proposes two main techniques. First, we introduce gradient consistency regularization, which leverages the moving average of gradients to suppress abrupt fluctuations and stabilize the training process. Second, we dynamically adjust the weight of the distillation loss by measuring the degree of class imbalance with normalized entropy. This adaptive weighting establishes an optimal balance between retaining old knowledge and acquiring new information. Experiments on the CIFAR-100-LT, ImageNetSubset-LT, and Food101-LT benchmarks show that our method achieves consistent accuracy improvements of up to 5.0%. Furthermore, we demonstrate dramatic gains in the challenging 'In-ordered' setting, where tasks progress from majority to minority classes, highlighting our method's robustness in mitigating forgetting under unfavorable learning dynamics. This enhanced performance is achieved without a significant increase in computational overhead, demonstrating the practicality of our framework.
Taigo Sakai, Kazuhiro Hotta
Meijo University, 1-501 Shiogamaguchi, Tempaku-ku, Nagoya 468-8502, Japan