Semantic-Guided Dynamic Sparsification for Pre-Trained Model-based Class-Incremental Learning

Ruiqi Liu; Boyu Diao; Zijia An; Runjie Shao; Feixiang Ge; Yongjun Xu

cs.CVJan 29, 2026

Semantic-Guided Dynamic Sparsification for Pre-Trained Model-based Class-Incremental Learning

Authors: Ruiqi Liu, Boyu Diao, Zijia An, Runjie Shao, Feixiang Ge, Yongjun Xu

Organizations: Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Abstract

Class-Incremental Learning (CIL) requires a model to continually learn new classes without forgetting old ones. A common and efficient solution freezes a pre-trained model and employs lightweight adapters, whose parameters are often forced to be orthogonal to prevent inter-task interference. However, we argue that this parameter-constraining method is detrimental to plasticity. To this end, we propose Semantic-Guided Dynamic Sparsification (SGDS), a novel method that proactively guides the activation space by governing the orientation and rank of its subspaces through targeted sparsification. Specifically, SGDS promotes knowledge transfer by encouraging similar classes to share a compact activation subspace, while simultaneously preventing interference by assigning non-overlapping activation subspaces to dissimilar classes. By sculpting class-specific sparse subspaces in the activation space, SGDS effectively mitigates interference without imposing rigid constraints on the parameter space. Extensive experiments on various benchmark datasets demonstrate the state-of-the-art performance of SGDS.

Figures & tables

Figure 1: Comparison between parameter-constraining and our activation-guiding methods. (a) Constraining adapter parameters (e.g., via orthogonality $\perp$ ) to reduce interference often harms model plasticity. (b) Our method guides the adapter’s input activations into orthogonal subspaces. This separation mitigates inter-task interference while preserving the model’s plasticity, as its parameters remain unconstrained.

Figure 2: An overview of SGDS. Operating at the input of each backbone adapter, SGDS proactively guides the activation space. The process begins by selecting a learning strategy based on semantic similarity: Knowledge Reuse (top path) to foster knowledge transfer, or New Subspace Allocation (bottom path) to prevent interference. This decision then guides the Semantic Exploration phase, which governs the subspace’s orientation by either aligning it with an existing subspace or creating a new orthogonal one. Subsequently, the Activation Compaction phase reduces the subspace’s rank by enforcing targeted sparsity, making it more compact. This entire process delivers a strategically structured activation stream to each adapter, freeing it from inter-task interference and thereby achieving a superior stability-plasticity balance.

Method	CIFAR-100		ImageNet-R		ImageNet-A		ObjectNet
Method	$\bar{\mathcal{A}}$ (%)	$\mathcal{A}_{T}$ (%)	$\bar{\mathcal{A}}$ (%)	$\mathcal{A}_{T}$ (%)	$\bar{\mathcal{A}}$ (%)	$\mathcal{A}_{T}$ (%)	$\bar{\mathcal{A}}$ (%)	$\mathcal{A}_{T}$ (%)
Finetune	38.90	20.17	32.32	22.78	24.28	14.51	19.14	8.73
L2P ( Wang et al. 2022c )	85.94	79.93	75.46	69.77	49.39	41.71	63.78	52.19
DualPrompt ( Wang et al. 2022b )	87.87	81.15	73.10	67.18	53.71	41.67	59.27	49.33
CODA-Prompt ( Smith et al. 2023 )	89.11	81.96	77.97	72.27	53.54	42.73	66.07	53.29
SLCA ( Zhang et al. 2023 )	92.49	88.55	81.17	77.00	68.66	58.74	72.55	61.30

Table 1: Average (

\bar{\mathcal{A}}

) and final task (

\mathcal{A}_{T}

) Top-1 accuracy comparison on four challenging class-incremental learning benchmarks. All methods use ViT-B/16-IN21K as the backbone and are evaluated in the exemplar-free setting. The specific task configurations (e.g., number of base classes and incremental steps) are detailed in the experimental setup section. The best performance in each column is highlighted in bold .

Panel of Figure 3 — Figure 3: Performance comparison with rehearsal-based methods on CIFAR-100 (left), ImageNet-R (center), and ObjectNet (right). The plots track the average accuracy as the model learns from a sequence of tasks. SGDS consistently outperforms strong baselines, and the performance gap widens over time, indicating superior knowledge retention.

Panel of Figure 4 — Figure 4: t-SNE visualizations of parameter and activation distributions on ImageNet-R. Colors represent tasks, and numbers denote layers. (a) The down-projection parameters ( $W_{\text{down}}$ ) cluster tightly by layer regardless of the task, indicating they learn a shared, task-agnostic function. (b) In contrast, the up-projection parameters ( $W_{\text{up}}$ ) are dispersed and form task-specific clusters, revealing their role in specializing for each task. (c) Similarly, the input activations are intermingled in early layers but become clearly separated by task in later layers.

SE	AC	$\bar{\mathcal{A}}$ (%)	$\mathcal{A}_{T}$ (%)
(Exploration)	(Compaction)	$\bar{\mathcal{A}}$ (%)	$\mathcal{A}_{T}$ (%)
✗	✗	83.42	79.27
✗	✓	84.53	79.23
✓	✗	84.66	79.35
✓	✓	85.41	80.52

Table 2: Ablation study on ImageNet-R. We analyze the contribution of the two core components of our proposed SGDS: Semantic Exploration (SE) and Activation Compaction (AC). The best performance in each column is highlighted in bold . The results confirm that both modules are crucial for optimal performance.

Parameter-Space Regularization
Strategy	$\bar{\mathcal{A}}$ (%)	$\mathcal{A}_{T}$ (%)
Regularize $W_{\text{down}}$ only	83.33	79.12
Regularize $W_{\text{up}}$ only	84.37	79.87
Regularize Both ( $W_{\text{down}}\&W_{\text{up}}$ )	83.22	78.55
Activation-Space Guidance (Ours)
SGDS on All Layers (1-12)	84.59	80.10

Table 3: Comparison of Activation Guidance and Parameter Constraints. We compare two distinct strategies for mitigating interference on ImageNet-R. The first group applies orthogonality regularization directly to the adapter’s parameter space. The second group implements SGDS by guiding the activation space at different network depths. The results show that guiding activations, particularly in later layers, yields superior performance.

Panel of Figure 5 — Figure 5: Hyperparameter Robustness on ImageNet-R. The plots show the average incremental accuracy as a function of (a) the compaction strength $\gamma$ and exploration balance $\beta$ , and (b) the sparsity ratio $k$ . SGDS demonstrates stable performance across a wide range of values for all key hyperparameters.

Panel of Figure 6 — Figure 6: t-SNE Visualization of Activation Frequencies on CIFAR-100. The visualization shows activation frequencies from the final network layer. The baseline model (a) shows mixed activations with no clear structure. In contrast, SGDS (b) organizes activations into distinct and compact clusters, grouping semantically similar classes (e.g., vehicles, natural objects) together.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 10 — Figure A1: Dynamic Adaptation of Learning Strategies in a Long Task Sequence (CIFAR-100, 20 Tasks). This figure illustrates how SGDS dynamically adjusts its learning strategy. (a) The left panel shows the per-task count of classes assigned to Knowledge Reuse versus New Subspace Allocation . Early tasks (e.g., 1-5) are dominated by New Subspace Allocation , demonstrating high plasticity as the model builds its initial semantic foundation. Later tasks show an increasing prevalence of Knowledge Reuse . (b) The right panel tracks the ratio of classes designated for Knowledge Reuse . The ratio steadily increases, indicating that as the model’s knowledge base expands, it increasingly and efficiently leverages existing subspaces. This automatic shift from a plasticity-dominant to a stability-focused strategy is crucial for SGDS’s strong performance in long-sequence CIL scenarios.

Figure A2: Performance stability across multiple random seeds on ImageNet-R (10-task setting). The plot displays the average accuracy as the number of learned tasks increases. All results shown are the mean performance calculated over five different random seeds, which vary the order of incoming classes. The consistent and superior performance of SGDS highlights its robustness against variations in the incremental learning sequence.

Panel of Figure 12 — Figure A3: Incremental accuracy trends of SGDS compared to SOTA CIL methods on four standard benchmarks: CIFAR-100, ImageNet-R, ImageNet-A, and ObjectNet. Across all scenarios, SGDS demonstrates consistently superior performance. This highlights its robust ability to mitigate catastrophic forgetting and maintain a clear advantage as the number of learned classes increases.

Class-Incremental Learning

Class-incremental learning studies how models can add new categories over time while retaining the ability to recognize previously learned classes.

See all 154 papers

From Isolation to Integration: Building an Adaptive Expert Forest for Pre-Trained Model-based Class-Incremental Learning
Feb 24, 2026Ruiqi Liu, Boyu Diao, Hangda Liu +3Class-Incremental Learning Incremental Learning
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Miles: Metric Learning with Expandable Subspace for Pre-Trained Model-Based Class-Incremental Learning
Jul 20, 2026Kai Jiang, Zisong Lin, Hongyuan Zhang +2Class-Incremental Learning Metric Learning
National Key Laboratory of Radar Signal Processing, Xidian University, Xi’an 710071, China · The University of Hong Kong, Hong Kong SAR, China · Institute of Artificial Intelligence (TeleAI) of China Telecom, China
Dynamic LoRA-Experts and Prototype-Ensemble Matching for Class-Incremental Learning
Sep 30, 2026Hongwei Zhao, Rui Liu, Yansong LiuClass-Incremental Learning Prototype-Based Classification
School of Computer Science and Engineering, Beihang University

Feb 24, 2026J/K move · Enter open · S save

From Isolation to Integration: Building an Adaptive Expert Forest for Pre-Trained Model-based Class-Incremental Learning

Ruiqi Liu, Boyu Diao, Hangda Liu, Zhulin An+2

Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Class-Incremental Learning Incremental Learning Mixture-of-Experts Inference

Open paper AskPDF arXiv

Class-Incremental Learning (CIL) requires models to learn new classes without forgetting old ones. A common method is to freeze a pre-trained model and train a new, lightweight adapter for each task. While this prevents forgetting, it treats the learned knowledge as a simple, unstructured collection and fails to use the relationships between tasks. To this end, we propose the Semantic-guided Adaptive Expert Forest (SAEF), a new method that organizes adapters into a structured hierarchy for better knowledge sharing. SAEF first groups tasks into conceptual clusters based on their semantic relationships. Then, within each cluster, it builds a balanced expert tree by creating new adapters from merging the adapters of similar tasks. At inference time, SAEF finds and activates a set of relevant experts from the forest for any given input. The final prediction is made by combining the outputs of these activated experts, weighted by how confident each expert is. Experiments on several benchmark datasets show that SAEF achieves SOTA performance.

Abstract

Class-Incremental Learning

Explore similar work

From Isolation to Integration: Building an Adaptive Expert Forest for Pre-Trained Model-based Class-Incremental Learning

Miles: Metric Learning with Expandable Subspace for Pre-Trained Model-Based Class-Incremental Learning

Dynamic LoRA-Experts and Prototype-Ensemble Matching for Class-Incremental Learning