Dynamic LoRA-Experts and Prototype-Ensemble Matching for Class-Incremental Learning

Hongwei Zhao; Rui Liu; Yansong Liu

cs.LGSep 30, 2026

Dynamic LoRA-Experts and Prototype-Ensemble Matching for Class-Incremental Learning

Authors: Hongwei Zhao, Rui Liu, Yansong Liu

Organizations: School of Computer Science and Engineering, Beihang University

Abstract

Class-Incremental Learning (CIL) aims to continuously learn new classes without forgetting previously acquired knowledge. Parameter-efficient fine-tuning with pre-trained models reduces parameter overhead but can suffer from cumulative interference and suboptimal alignment between inference samples and specialized modules. We propose Dynamic LoRA-Experts and Prototype-Ensemble Matching (DLEPEM), a two-stage rehearsal-free framework. DLEPEM allocates a task-specific LoRA-Expert for each incremental task to reduce cross-task interference, then combines frozen pre-trained-model prototypes with task-adaptive LoRA-Expert prototypes for reliable task-level discrimination. Experiments on standard CIL and Few-Shot CIL benchmarks demonstrate strong performance under the evaluated protocols.

Figures & tables

Figure 1: Illustration of DLEPEM. (a) In the $t$ -th incremental task, a new LoRA-Expert $\mathbf{E}_{t}$ (with parameters $\mathbf{U}_{t}$ and $\mathbf{V}_{t}$ ) is trained to capture task-specific features. (b) Structure of the LoRA-Expert. Depending on the insertion location, the module is categorized as either an MLP-Expert or a QV-Expert. The QV-Expert integrates LoRA into the $\mathbf{W}_{q}$ and $\mathbf{W}_{v}$ projections of the MHA layer. (c) Prototype-Ensemble Matching. General prototypes from the PTM and domain-specific prototypes from the router $\mathbf{E}^{router}_{t}$ are combined and linked to their respective experts. During inference, the nearest prototype guides expert selection for each input sample.

Task	CIFAR100	CUB200	ImageNet-R	OmniBenchmark	VTAB
Classes / Task	10	20	40	30	10
# of tasks	10	10	5	10	5

Table 1: Configuration of the class-incremental learning (CIL) benchmarks.

Task	CUB200	CIFAR100	mini ImageNet
Base Classes	100	60	60
Incremental Tasks	10-way 5-shot	5-way 5-shot	5-way 5-shot
# of Tasks	1+10	1+8	1+8

Table 2: Configuration of the few-shot class-incremental learning (FSCIL) benchmarks.

Method	CIFAR100 ( $T$ =10)		CUB200 ( $T$ =10)		ImageNet-R ( $T$ =5)		OmniBenchmark ( $T$ =10)		VTAB ( $T$ =5)
Method	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$
Full Fine-Tuning	$66.26\pm 0.18$	$76.94\pm 0.02$	$55.29\pm 0.46$	$70.30\pm 0.74$	$59.90\pm 0.05$	$72.19\pm 0.12$	$47.75\pm 0.14$	$65.86\pm 0.12$	$62.95\pm 5.94$	$80.80\pm 1.50$
SimpleCIL Zhou et al. (2024)	$81.27\pm 0.01$	$87.13\pm 0.01$	$82.28\pm 6.35$	$91.85\pm 0.00$	$65.14\pm 15.29$	$59.72\pm 0.03$	$66.98\pm 8.93$	$79.35\pm 0.01$	$80.74\pm 5.26$	$90.80\pm 0.00$
L2P Wang et al. (2022a)	$84.82\pm 0.22$	$89.78\pm 0.01$	$71.98\pm 0.07$	$81.80\pm 0.03$	$72.08\pm 0.01$	$76.76\pm 0.06$	$64.45\pm 0.07$	$74.14\pm 0.03$	$64.27\pm 0.02$	$81.84\pm 0.19$
DualPrompt Wang et al. (2022b)	$85.23\pm 0.07$	$90.32\pm 0.06$	$74.23\pm 0.10$	$84.81\pm 0.02$	$69.34\pm 0.09$	$73.58\pm 0.05$	$66.16\pm 0.06$	$74.97\pm 0.03$	$78.90\pm 0.10$	$89.82\pm 0.06$
CODA-Prompt Smith et al. (2023)	$86.69\pm 0.00$	$91.31\pm 0.01$	$75.45\pm 0.00$	$84.65\pm 0.00$	$75.16\pm 0.08$	$80.46\pm 0.04$	$68.67\pm 0.00$	$77.79\pm 0.00$	$75.08\pm 0.00$	$87.24\pm 0.00$

Table 3: Performance comparison on CIL benchmarks using the same ViT-B/16-IN21K backbone. Results are reported as mean

\pm

standard deviation over three runs; the best result in each column is highlighted in bold.

Panel of Figure 2 — Figure 2: Incremental accuracy curves on the CIL benchmarks. All methods use the same ViT-B/16-IN21K backbone, and each subplot reports accuracy after successive incremental tasks for the specified dataset protocol.

Method	CUB200 ( $T$ =11)			CIFAR100 ( $T$ =9)			mini ImageNet ( $T$ =9)
Method	$\mathcal{A}_{\text{Base}}$	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{\text{Base}}$	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{\text{Base}}$	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$
L2P Wang et al. (2022a)	$91.50\pm 0.00$	$50.04\pm 0.00$	$66.70\pm 0.00$	$93.43\pm 0.00$	$55.75\pm 0.00$	$71.81\pm 0.00$	$96.53\pm 0.00$	$60.91\pm 0.00$	$76.28\pm 0.00$
CODA-Prompt Smith et al. (2023)	$91.50\pm 0.00$	$53.65\pm 0.00$	$69.30\pm 0.00$	$94.05\pm 0.00$	$57.10\pm 0.00$	$73.11\pm 0.00$	$97.15\pm 0.00$	$65.55\pm 0.00$	$78.83\pm 0.00$
InfLoRA Liang and Li (2024)	$92.45\pm 0.20$	$45.18\pm 2.50$	$66.27\pm 1.43$	$\bf{94.92\pm 0.06}$	$57.41\pm 0.31$	$74.28\pm 0.28$	$97.42\pm 0.06$	$51.52\pm 0.10$	$71.55\pm 0.03$
SD-LoRA Wu et al. (2025)	$91.92\pm 0.00$	$56.28\pm 0.00$	$70.87\pm 0.00$	$94.60\pm 0.00$	$73.51\pm 0.00$	$78.42\pm 0.00$	$\bf{97.72\pm 0.00}$	$79.07\pm 0.00$	$84.36\pm 0.00$
CPE-CLIP D’Alessandro et al. (2023)	$80.21\pm 0.85$	$63.32\pm 0.17$	$69.37\pm 0.37$	$88.32\pm 0.04$	$79.99\pm 0.18$	$83.38\pm 0.10$	$90.14\pm 0.05$	$81.55\pm 0.04$	$85.54\pm 0.04$

Table 4: Performance comparison on FSCIL benchmarks using the same ViT-B/16-IN21K backbone. Results are reported as mean

\pm

standard deviation over three runs; the best result in each column is highlighted in bold.

Figure 3: Incremental accuracy curves on the FSCIL benchmarks. The three subplots show the CUB200, CIFAR100, and mini ImageNet protocols, respectively, using the same ViT-B/16-IN21K backbone.

Ablated Components	CIFAR100 ( $T$ =10)		ImageNet-R ( $T$ =5)		CIFAR100 ( $T$ =9)		mini ImageNet ( $T$ =9)
Ablated Components	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$
w/o Dynamic LoRA-Expert	83.13 / 86.72	88.85 / 90.82	61.17 / 72.37	74.31 / 80.14	73.03 / 79.73	80.39 / 86.59	84.26 / 91.11	89.90 / 93.31
w/o Prototype-Ensemble	85.79 / 87.59	91.21 / 92.29	69.53 / 73.13	77.75 / 80.42	81.13 / 84.09	86.97 / 88.67	85.54 / 92.25	90.03 / 94.06
DLEPEM-MLP / QV	85.99 / 88.84	92.40 / 93.39	76.25 / 78.77	82.37 / 83.43	84.67 / 87.29	89.00 / 90.5	89.96 / 93.62	93.11 / 94.80

Table 5: Ablation study on CIL and FSCIL tasks. The first two benchmarks follow CIL protocols, while the last two follow FSCIL protocols. Each metric reports MLP-Expert / QV-Expert performance.

Figure 4: Comparison of routing strategies for module-identity inference. The prototype-ensemble strategy is compared with frozen-PTM prototypes, KNN matching, and router-domain prototypes under the same evaluation protocols.

Metric	CIFAR100 ( $T=10$ )	CUB200 ( $T=10$ )	ImageNet-R ( $T=5$ )	OmniBenchmark ( $T=10$ )	VTAB ( $T=5$ )
CNN	$92.40\ /\ 93.39$	$92.31\ /\ 92.09$	$82.37\ /\ 83.43$	$81.60\ /\ 82.16$	$91.84\ /\ 91.11$
Router	$91.88\ /\ 93.68$	$93.24\ /\ 93.31$	$88.43\ /\ 88.87$	$84.90\ /\ 85.55$	$93.45\ /\ 93.56$
LoRA-Expert (Oracle)	$98.04\ /\ 97.81$	$97.15\ /\ 96.31$	$89.17\ /\ 88.81$	$92.88\ /\ 91.48$	$97.66\ /\ 95.88$

Table 6: Diagnostic accuracy of the classifier, router, and oracle expert selection on five CIL benchmarks. Each entry is reported as DLEPEM-MLP / DLEPEM-QV; the oracle expert setting uses the ground-truth task identity only for diagnostic analysis.

Figure 5: Effect of different pre-trained backbones on CIL performance. The comparison evaluates DLEPEM with several ViT-based pre-training sources and reports the corresponding performance on ImageNet-R and CIFAR100.

Figure 6: Parameter–accuracy comparison on CIL benchmarks. The plots compare trainable parameter counts and final accuracy on CIFAR100 and VTAB to illustrate the parameter-performance trade-off of DLEPEM.

Method	CIFAR100 ( $T$ =10)		ImageNet-R ( $T$ =5)
Method	Training Time(s)	Inference Time(ms)	Training Time(s)	Inference Time(ms)
L2P Wang et al. (2022a)	102.12	3.77	50.12	3.84
DualPrompt Wang et al. (2022b)	93.21	3.44	45.16	3.58
CODA-Prompt Smith et al. (2023)	99.42	2.99	47.53	3.08
LAE Gao et al. (2023)	46.87	3.35	24.26	3.52
InfLoRA Liang and Li (2024)	72.36	2.03	34.96	2.24

Table 7: Training and inference time comparison. Training time is the average time per epoch for each incremental task, and inference latency is measured in milliseconds per image. All methods use the same ViT-B/16-IN21K backbone for a fair comparison.

Panel of Figure 7 — Figure 7: Hyperparameter sensitivity of DLEPEM. Subplot (a) analyzes the effect of LoRA rank and insertion position, while subplot (b) shows the effect of the distillation coefficient $\alpha$ across datasets.

Figure 8: New-task and old-task performance of LoRA-based CIL methods. The left subplot reports accuracy on newly introduced tasks as a proxy for plasticity, and the right subplot reports retained accuracy on previous tasks as a proxy for stability.

Symbol	Definition
$\mathcal{D}_{t}$ , $\mathcal{Y}_{t}$	Training set and class set of task $t$ .
$\mathbf{Y}_{t}$	The set of all classes observed up to task $t$ , i.e., $\mathbf{Y}_{t}=\mathcal{Y}_{1}\cup\cdots\cup\mathcal{Y}_{t}$ .
$\mathbf{E}_{t}$	Task- $t$ LoRA-Expert module, i.e., a group of low-rank LoRA adapters inserted into selected Transformer layers rather than a complete independent ViT. Historical experts are frozen after training.
$\mathbf{U}_{t}$ , $\mathbf{V}_{t}$ , $r$	Low-rank matrices and rank inside each adapter. For row-vector input $\mathbf{e}\in\mathbf{R}^{1\times d_{in}}$ , $\mathbf{U}_{t}\in\mathbf{R}^{d_{in}\times r}$ , $\mathbf{V}_{t}\in\mathbf{R}^{r\times d_{out}}$ , with $r=10$ by default.
$\mathbf{V}_{attn}$	Attention value matrix in Eq. ( 8 ), distinct from the LoRA matrix $\mathbf{V}_{t}$ .
$\mathbf{E}^{router}_{t}$	Router after learning task $t$ , used to extract router-domain features.

Table 8: Notation used in DLEPEM.

Explore similar work

Sep 30, 2026cs.LG

Hyperbolic Prototype Routing for Rehearsal-Free Class-Incremental Learning

Class-Incremental Learning (CIL) aims to continually learn new classes while preserving prior knowledge. Parameter-efficient fine-tuning with pre-trained models enables CIL with minimal parameter updates, but existing approaches still suffer from catastrophic forgetting caused by cumulative interference and suboptimal module-sample matching at inference. We propose Hyperbolic Prototype Routing (HyPro), a rehearsal-free framework for continual learning. HyPro allocates a dedicated LoRA-Expert module to each incremental task for isolated representation learning, then projects routing features onto a Poincare ball and performs geodesic nearest-prototype matching for reliable task-level discrimination. Extensive experiments on standard CIL and Few-Shot CIL benchmarks show that HyPro consistently improves average and final-stage accuracy over strong baselines.

HongWei Zhao, Rui Liu, Yong Chen

Beihang University · Beijing University of Posts and Telecommunications

Class-Incremental Learning Learnable Prototypes Layer-Selective Rehearsal Strategy Catastrophic Forgetting Retraining Singular Learning Theory+1

Ask PDF

Jul 20, 2026cs.CV

Miles: Metric Learning with Expandable Subspace for Pre-Trained Model-Based Class-Incremental Learning

Class Incremental Learning (CIL) aims to learn new concepts consistently from a data stream without forgetting. Unlike typical CIL methods which need to learn a model from scratch, pre-trained model (PTM) can easily adapt to a new task with fine-tuning. However, existing PTM-based CIL methods fail to achieve a trade-off between performance and computational expenditure, i.e., they either adopt the same parameter space so that leading catastrophic forgetting, or expand a new branch for each task but adding more computational cost. To this end, we propose MetrIc Learning with Expandable Subspace (Miles) to harness the prior information within pre-trained knowledge, thereby orchestrating an efficient expansion of the parameter space through guided optimization. Specifically, it decouples the learnable modules with the pre-trained model, exploiting prior information from intermediate features of the backbone network to enable more flexible parameter expansion. Then, a central loss is adopted to guide the new category to cluster towards the corresponding prototype in the new task subspace while incorporating an auxiliary distance regularization term to maintain metric equilibrium across tasks. Extensive experiments on six benchmark datasets demonstrate that Miles achieves state-of-the-art performance in various CIL settings.

Kai Jiang, Zisong Lin, Hongyuan Zhang +2

National Key Laboratory of Radar Signal Processing, Xidian University, Xi’an 710071, China · The University of Hong Kong, Hong Kong SAR, China · Institute of Artificial Intelligence (TeleAI) of China Telecom, China

Class-Incremental Learning Meta-Learning Pretraining Subspace Regularization

Ask PDF

Sep 30, 2026cs.LG

Decoupled and Distilled: Task-Adaptive LoRA-Teachers with Ensemble Knowledge Transfer for Few-Shot Class-Incremental Learning

Few-Shot Class-Incremental Learning (FSCIL) addresses the challenge of learning new classes from very limited samples while retaining knowledge of previously learned ones. Although parameter-efficient fine-tuning methods with pre-trained models show promise for class-incremental learning, strict gradient-based constraints can be unreliable under severe data scarcity, while multi-expert approaches can impose substantial inference-time costs. We propose TALON (Task-Adaptive LoRA-Teachers with Ensemble Knowledge Transfer), an inference-efficient FSCIL framework. TALON dynamically allocates an independent LoRA-Teacher to each incremental task for task-specific representation learning, then distills multiple frozen teachers into a unified LoRA-Student through Ensemble Knowledge Transfer, eliminating runtime module selection or generation. A semantic-guided distillation strategy weights teacher contributions by feature-space similarity to mitigate catastrophic forgetting and overfitting. Across three class-order runs, TALON achieves comparable or better mean average accuracy across four FSCIL benchmarks, obtaining 86.68 +/- 1.22% on CUB200, 90.39 +/- 0.27% on CIFAR100, 78.38 +/- 0.94% on ImageNet-R, and 96.34 +/- 0.33% on miniImageNet. TALON uses up to 33x fewer deployment parameters and reduces average inference time per task to 26.7 s, a 41.70% reduction relative to ASP.

Hongwei Zhao, Rui Liu, Yansong Liu +2

School of Computer Science and Engineering, Beihang University · School of Computer Science, Beijing University of Posts and Telecommunications

Class-Incremental Learning Few-Shot Learning Imagenet Cross-Task Knowledge Transfer Representation Learning

Ask PDF