From Isolation to Integration: Building an Adaptive Expert Forest for Pre-Trained Model-based Class-Incremental Learning

Ruiqi Liu; Boyu Diao; Hangda Liu; Zhulin An; Fei Wang; Yongjun Xu

cs.LGFeb 24, 2026

From Isolation to Integration: Building an Adaptive Expert Forest for Pre-Trained Model-based Class-Incremental Learning

Authors: Ruiqi Liu, Boyu Diao, Hangda Liu, Zhulin An, Fei Wang, Yongjun Xu

Organizations: Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Abstract

Class-Incremental Learning (CIL) requires models to learn new classes without forgetting old ones. A common method is to freeze a pre-trained model and train a new, lightweight adapter for each task. While this prevents forgetting, it treats the learned knowledge as a simple, unstructured collection and fails to use the relationships between tasks. To this end, we propose the Semantic-guided Adaptive Expert Forest (SAEF), a new method that organizes adapters into a structured hierarchy for better knowledge sharing. SAEF first groups tasks into conceptual clusters based on their semantic relationships. Then, within each cluster, it builds a balanced expert tree by creating new adapters from merging the adapters of similar tasks. At inference time, SAEF finds and activates a set of relevant experts from the forest for any given input. The final prediction is made by combining the outputs of these activated experts, weighted by how confident each expert is. Experiments on several benchmark datasets show that SAEF achieves SOTA performance.

Figures & tables

Figure 1: Conceptual comparison of adapter-based CIL methods. (a) The conventional methods treat adapters as an unstructured pool, leading to inefficient full-ensemble inference. (b) Our proposed SAEF automatically organizes adapters into a semantic expert forest, enabling an efficient, adaptive search for the most relevant experts.

Figure 2: Overview of the SAEF. Our method organizes individually trained task adapters into a structured knowledge hierarchy. (1) Stage 1: Conceptual Clustering. Based on semantic prototypes, tasks are grouped into high-level conceptual clusters (e.g., ’Animal’, ’Vehicle’). (2) Stage 2: Hierarchical Construction. Within each cluster, a balanced expert tree is built via a bottom-up process of recursively fusing the most visually similar nodes. The roots of these trees are then merged to form a single Global Expert. (3) Stage 3: Adaptive Inference. For a given input, SAEF performs a parallel, multi-path search to dynamically activate a set of relevant experts. The final prediction is derived from an entropy-guided fusion of their outputs.

Method	CIFAR-100		ImageNet-R		ImageNet-A		ObjectNet
Method	$\bar{\mathcal{A}}$ (%)	$\mathcal{A}_{T}$ (%)	$\bar{\mathcal{A}}$ (%)	$\mathcal{A}_{T}$ (%)	$\bar{\mathcal{A}}$ (%)	$\mathcal{A}_{T}$ (%)	$\bar{\mathcal{A}}$ (%)	$\mathcal{A}_{T}$ (%)
Finetune	38.90	20.17	32.32	22.78	24.28	14.51	19.14	8.73
L2P [ 44 ]	85.94	79.93	75.46	69.77	49.39	41.71	63.78	52.19
DualPrompt [ 43 ]	87.87	81.15	73.10	67.18	53.71	41.67	59.27	49.33
CODA-Prompt [ 36 ]	89.11	81.96	77.97	72.27	53.54	42.73	66.07	53.29
SLCA [ 47 ]	92.49	88.55	81.17	77.00	68.66	58.74	72.55	61.30

Table 1: Average (

\bar{\mathcal{A}}

) and final task (

\mathcal{A}_{T}

) Top-1 accuracy comparison on four challenging class-incremental learning benchmarks. All methods use ViT-B/16-IN21K as the backbone and are evaluated in the exemplar-free setting. The best performance in each column is highlighted in bold .

Panel of Figure 3 — Figure 3: Comparison against rehearsal-based methods. Average accuracy ( $\bar{\mathcal{A}}$ ) is plotted as a function of the number of learned tasks. Rehearsal-based methods use 20 exemplars per class. Despite being an exemplar-free method, SAEF consistently outperforms strong rehearsal-based competitors across multiple benchmarks, and the performance gap often widens over time, demonstrating superior knowledge retention and scalability.

Panel of Figure 4 — Figure 4: Ablation and structural analysis of SAEF on ImageNet-R. Performance is measured by average accuracy ( $\bar{\mathcal{A}}$ ). (a) Ablation study demonstrating the progressive contribution of each core component. (b) Structural analysis validating our balanced forest structure ( $K=2$ ) over less structured alternatives.

Temperature ( $\tau$ )	1.0	0.5	0.2	0.1
$\bar{\mathcal{A}}$ (%)	84.54	84.49	84.32	84.31
$\mathcal{A}_{T}$ (%)	80.02	80.08	79.78	79.87

Table 2: Robustness analysis of the fusion temperature

\tau

on ImageNet-R. The optimal results are marked in bold.

Configuration	$\bar{\mathcal{A}}$ (%) $\uparrow$	Avg. Depth $\downarrow$	Theo. Speedup $\uparrow$	Time (ms) $\downarrow$
Flat Structure (Baseline)	94.44	–	1.00 $\times$	2775.43
SAEF ( $\tau_{e}=0.0$ )	94.53	4.04	2.20 $\times$	1346.51
SAEF ( $\tau_{e}=1.0$ )	94.24	1.19	5.92 $\times$	500.39
SAEF ( $\tau_{e}=2.0$ )	93.96	1.00	6.67 $\times$	443.00

Table 3: Analysis of the trade-off between inference efficiency and average accuracy by varying the entropy threshold

\tau_{e}

on CIFAR-100. The ’Flat Structure’ baseline represents a non-hierarchical method where all task adapters are queried. The setting of

\tau_{e}=1.0

(highlighted in bold ) strikes an excellent balance.

Figure 5: efficiency and performance evaluation on CIFAR-100. We plot average accuracy (

\bar{\mathcal{A}}

) against Training Throughput (samples/s). The size of each bubble is proportional to the Inference Time (ms/sample). The ideal method occupies the top-right corner with the smallest bubble, representing high accuracy, fast training, and rapid inference.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure A1 : Incremental accuracy trends of SAEF compared to SOTA CIL methods on four standard benchmarks: CIFAR-100, ImageNet-R, ImageNet-A, and ObjectNet. Across all scenarios, SAEF demonstrates consistently superior performance. This highlights its robust ability to mitigate catastrophic forgetting and maintain a clear advantage as the number of learned classes increases.

Figure A2 : Performance stability across multiple random seeds on ImageNet-R (10-task setting). The plot displays the average accuracy (

\bar{\mathcal{A}}

) as the number of learned tasks increases. All results shown are the mean performance calculated over five different random seeds, which vary the order of incoming classes. The consistent and superior performance of SAEF highlights its robustness against variations in the incremental learning sequence.

Explore similar work

Non-Forgetting Knowledge Allocation with Bi-level Competition for Class-Incremental Learning
May 28, 2026Xiang Tan, Run He, Yawen Cui +6Class-Incremental Learning Incremental Learning
South China University of Technology · Hong Kong Polytechnic University · Agency for Science, Technology and Research +3
Dynamic LoRA-Experts and Prototype-Ensemble Matching for Class-Incremental Learning
Sep 30, 2026Hongwei Zhao, Rui Liu, Yansong LiuClass-Incremental Learning Prototype-Based Classification
School of Computer Science and Engineering, Beihang University
Miles: Metric Learning with Expandable Subspace for Pre-Trained Model-Based Class-Incremental Learning
Jul 20, 2026Kai Jiang, Zisong Lin, Hongyuan Zhang +2Class-Incremental Learning Metric Learning
National Key Laboratory of Radar Signal Processing, Xidian University, Xi’an 710071, China · The University of Hong Kong, Hong Kong SAR, China · Institute of Artificial Intelligence (TeleAI) of China Telecom, China

May 28, 2026J/K move · Enter open · S save

Non-Forgetting Knowledge Allocation with Bi-level Competition for Class-Incremental Learning

Xiang Tan, Run He, Yawen Cui, Mengchen Zhao+5

South China University of Technology · Hong Kong Polytechnic University · Agency for Science, Technology and Research+3

Class-Incremental Learning Incremental Learning Catastrophic Forgetting

Open paper AskPDF arXiv

Class-Incremental Learning (CIL) with pre-trained models (PTMs) aims to sequentially adapt PTMs to new categories without forgetting old knowledge. Built upon PTMs, existing adapter-based methods mainly train models via distinct task-specific adapters, and present a uniform knowledge allocation for each adapter during inference. However, this allocation mechanism ignores the nature of task discrepancy and leads to suboptimal utilization of adapters. Also, under CIL constraint, an allocator is prone to forgetting when tasks evolve. To address these issues, we propose a Non-Forgetting Allocation with Bi-Level Competition (NoFA-BC). NoFA-BC constructs a non-forgetting allocator (NFA) by transforming the allocator training into a recursive least-squares problem and achieves an allocator equivalent to that trained with all data. Based on the NFA, a Bi-Level Competition (BLC) including an intra-task level Winner-Takes-All (WTA) mechanism and inter-task Last-Ones-Fall (LOF) elimination is proposed to provide better allocation of adapter knowledge. WTA extracts the most significant logit within a task to represent the adapter's contribution and LOF suppresses the irrelevant adapters. With BLC, participation ratio of each adapter can be tailored for each input. Moreover, a Stability Enhancement (SE) process is incorporated to further improve the performance of old tasks.

Abstract

Explore similar work

Non-Forgetting Knowledge Allocation with Bi-level Competition for Class-Incremental Learning

Dynamic LoRA-Experts and Prototype-Ensemble Matching for Class-Incremental Learning

Miles: Metric Learning with Expandable Subspace for Pre-Trained Model-Based Class-Incremental Learning