Dynamic LoRA-Experts and Prototype-Ensemble Matching for Class-Incremental Learning

Hongwei Zhao; Rui Liu; Yansong Liu

cs.LGSep 30, 2026

Dynamic LoRA-Experts and Prototype-Ensemble Matching for Class-Incremental Learning

Authors: Hongwei Zhao, Rui Liu, Yansong Liu

Organizations: School of Computer Science and Engineering, Beihang University

Abstract

Class-Incremental Learning (CIL) aims to continuously learn new classes without forgetting previously acquired knowledge. Parameter-efficient fine-tuning with pre-trained models reduces parameter overhead but can suffer from cumulative interference and suboptimal alignment between inference samples and specialized modules. We propose Dynamic LoRA-Experts and Prototype-Ensemble Matching (DLEPEM), a two-stage rehearsal-free framework. DLEPEM allocates a task-specific LoRA-Expert for each incremental task to reduce cross-task interference, then combines frozen pre-trained-model prototypes with task-adaptive LoRA-Expert prototypes for reliable task-level discrimination. Experiments on standard CIL and Few-Shot CIL benchmarks demonstrate strong performance under the evaluated protocols.

Figures & tables

Figure 1: Illustration of DLEPEM. (a) In the $t$ -th incremental task, a new LoRA-Expert $\mathbf{E}_{t}$ (with parameters $\mathbf{U}_{t}$ and $\mathbf{V}_{t}$ ) is trained to capture task-specific features. (b) Structure of the LoRA-Expert. Depending on the insertion location, the module is categorized as either an MLP-Expert or a QV-Expert. The QV-Expert integrates LoRA into the $\mathbf{W}_{q}$ and $\mathbf{W}_{v}$ projections of the MHA layer. (c) Prototype-Ensemble Matching. General prototypes from the PTM and domain-specific prototypes from the router $\mathbf{E}^{router}_{t}$ are combined and linked to their respective experts. During inference, the nearest prototype guides expert selection for each input sample.

Task	CIFAR100	CUB200	ImageNet-R	OmniBenchmark	VTAB
Classes / Task	10	20	40	30	10
# of tasks	10	10	5	10	5

Table 1: Configuration of the class-incremental learning (CIL) benchmarks.

Task	CUB200	CIFAR100	mini ImageNet
Base Classes	100	60	60
Incremental Tasks	10-way 5-shot	5-way 5-shot	5-way 5-shot
# of Tasks	1+10	1+8	1+8

Table 2: Configuration of the few-shot class-incremental learning (FSCIL) benchmarks.

Method	CIFAR100 ( $T$ =10)		CUB200 ( $T$ =10)		ImageNet-R ( $T$ =5)		OmniBenchmark ( $T$ =10)		VTAB ( $T$ =5)
Method	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$
Full Fine-Tuning	$66.26\pm 0.18$	$76.94\pm 0.02$	$55.29\pm 0.46$	$70.30\pm 0.74$	$59.90\pm 0.05$	$72.19\pm 0.12$	$47.75\pm 0.14$	$65.86\pm 0.12$	$62.95\pm 5.94$	$80.80\pm 1.50$
SimpleCIL Zhou et al. (2024)	$81.27\pm 0.01$	$87.13\pm 0.01$	$82.28\pm 6.35$	$91.85\pm 0.00$	$65.14\pm 15.29$	$59.72\pm 0.03$	$66.98\pm 8.93$	$79.35\pm 0.01$	$80.74\pm 5.26$	$90.80\pm 0.00$
L2P Wang et al. (2022a)	$84.82\pm 0.22$	$89.78\pm 0.01$	$71.98\pm 0.07$	$81.80\pm 0.03$	$72.08\pm 0.01$	$76.76\pm 0.06$	$64.45\pm 0.07$	$74.14\pm 0.03$	$64.27\pm 0.02$	$81.84\pm 0.19$
DualPrompt Wang et al. (2022b)	$85.23\pm 0.07$	$90.32\pm 0.06$	$74.23\pm 0.10$	$84.81\pm 0.02$	$69.34\pm 0.09$	$73.58\pm 0.05$	$66.16\pm 0.06$	$74.97\pm 0.03$	$78.90\pm 0.10$	$89.82\pm 0.06$
CODA-Prompt Smith et al. (2023)	$86.69\pm 0.00$	$91.31\pm 0.01$	$75.45\pm 0.00$	$84.65\pm 0.00$	$75.16\pm 0.08$	$80.46\pm 0.04$	$68.67\pm 0.00$	$77.79\pm 0.00$	$75.08\pm 0.00$	$87.24\pm 0.00$

Table 3: Performance comparison on CIL benchmarks using the same ViT-B/16-IN21K backbone. Results are reported as mean

\pm

standard deviation over three runs; the best result in each column is highlighted in bold.

Panel of Figure 2 — Figure 2: Incremental accuracy curves on the CIL benchmarks. All methods use the same ViT-B/16-IN21K backbone, and each subplot reports accuracy after successive incremental tasks for the specified dataset protocol.

Method	CUB200 ( $T$ =11)			CIFAR100 ( $T$ =9)			mini ImageNet ( $T$ =9)
Method	$\mathcal{A}_{\text{Base}}$	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{\text{Base}}$	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{\text{Base}}$	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$
L2P Wang et al. (2022a)	$91.50\pm 0.00$	$50.04\pm 0.00$	$66.70\pm 0.00$	$93.43\pm 0.00$	$55.75\pm 0.00$	$71.81\pm 0.00$	$96.53\pm 0.00$	$60.91\pm 0.00$	$76.28\pm 0.00$
CODA-Prompt Smith et al. (2023)	$91.50\pm 0.00$	$53.65\pm 0.00$	$69.30\pm 0.00$	$94.05\pm 0.00$	$57.10\pm 0.00$	$73.11\pm 0.00$	$97.15\pm 0.00$	$65.55\pm 0.00$	$78.83\pm 0.00$
InfLoRA Liang and Li (2024)	$92.45\pm 0.20$	$45.18\pm 2.50$	$66.27\pm 1.43$	$\bf{94.92\pm 0.06}$	$57.41\pm 0.31$	$74.28\pm 0.28$	$97.42\pm 0.06$	$51.52\pm 0.10$	$71.55\pm 0.03$
SD-LoRA Wu et al. (2025)	$91.92\pm 0.00$	$56.28\pm 0.00$	$70.87\pm 0.00$	$94.60\pm 0.00$	$73.51\pm 0.00$	$78.42\pm 0.00$	$\bf{97.72\pm 0.00}$	$79.07\pm 0.00$	$84.36\pm 0.00$
CPE-CLIP D’Alessandro et al. (2023)	$80.21\pm 0.85$	$63.32\pm 0.17$	$69.37\pm 0.37$	$88.32\pm 0.04$	$79.99\pm 0.18$	$83.38\pm 0.10$	$90.14\pm 0.05$	$81.55\pm 0.04$	$85.54\pm 0.04$

Table 4: Performance comparison on FSCIL benchmarks using the same ViT-B/16-IN21K backbone. Results are reported as mean

\pm

standard deviation over three runs; the best result in each column is highlighted in bold.

Figure 3: Incremental accuracy curves on the FSCIL benchmarks. The three subplots show the CUB200, CIFAR100, and mini ImageNet protocols, respectively, using the same ViT-B/16-IN21K backbone.

Ablated Components	CIFAR100 ( $T$ =10)		ImageNet-R ( $T$ =5)		CIFAR100 ( $T$ =9)		mini ImageNet ( $T$ =9)
Ablated Components	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$	$\mathcal{A}_{L}$	$\bar{\mathcal{A}}$
w/o Dynamic LoRA-Expert	83.13 / 86.72	88.85 / 90.82	61.17 / 72.37	74.31 / 80.14	73.03 / 79.73	80.39 / 86.59	84.26 / 91.11	89.90 / 93.31
w/o Prototype-Ensemble	85.79 / 87.59	91.21 / 92.29	69.53 / 73.13	77.75 / 80.42	81.13 / 84.09	86.97 / 88.67	85.54 / 92.25	90.03 / 94.06
DLEPEM-MLP / QV	85.99 / 88.84	92.40 / 93.39	76.25 / 78.77	82.37 / 83.43	84.67 / 87.29	89.00 / 90.5	89.96 / 93.62	93.11 / 94.80

Table 5: Ablation study on CIL and FSCIL tasks. The first two benchmarks follow CIL protocols, while the last two follow FSCIL protocols. Each metric reports MLP-Expert / QV-Expert performance.

Figure 4: Comparison of routing strategies for module-identity inference. The prototype-ensemble strategy is compared with frozen-PTM prototypes, KNN matching, and router-domain prototypes under the same evaluation protocols.

Metric	CIFAR100 ( $T=10$ )	CUB200 ( $T=10$ )	ImageNet-R ( $T=5$ )	OmniBenchmark ( $T=10$ )	VTAB ( $T=5$ )
CNN	$92.40\ /\ 93.39$	$92.31\ /\ 92.09$	$82.37\ /\ 83.43$	$81.60\ /\ 82.16$	$91.84\ /\ 91.11$
Router	$91.88\ /\ 93.68$	$93.24\ /\ 93.31$	$88.43\ /\ 88.87$	$84.90\ /\ 85.55$	$93.45\ /\ 93.56$
LoRA-Expert (Oracle)	$98.04\ /\ 97.81$	$97.15\ /\ 96.31$	$89.17\ /\ 88.81$	$92.88\ /\ 91.48$	$97.66\ /\ 95.88$

Table 6: Diagnostic accuracy of the classifier, router, and oracle expert selection on five CIL benchmarks. Each entry is reported as DLEPEM-MLP / DLEPEM-QV; the oracle expert setting uses the ground-truth task identity only for diagnostic analysis.

Figure 5: Effect of different pre-trained backbones on CIL performance. The comparison evaluates DLEPEM with several ViT-based pre-training sources and reports the corresponding performance on ImageNet-R and CIFAR100.

Figure 6: Parameter–accuracy comparison on CIL benchmarks. The plots compare trainable parameter counts and final accuracy on CIFAR100 and VTAB to illustrate the parameter-performance trade-off of DLEPEM.

Method	CIFAR100 ( $T$ =10)		ImageNet-R ( $T$ =5)
Method	Training Time(s)	Inference Time(ms)	Training Time(s)	Inference Time(ms)
L2P Wang et al. (2022a)	102.12	3.77	50.12	3.84
DualPrompt Wang et al. (2022b)	93.21	3.44	45.16	3.58
CODA-Prompt Smith et al. (2023)	99.42	2.99	47.53	3.08
LAE Gao et al. (2023)	46.87	3.35	24.26	3.52
InfLoRA Liang and Li (2024)	72.36	2.03	34.96	2.24

Table 7: Training and inference time comparison. Training time is the average time per epoch for each incremental task, and inference latency is measured in milliseconds per image. All methods use the same ViT-B/16-IN21K backbone for a fair comparison.

Panel of Figure 7 — Figure 7: Hyperparameter sensitivity of DLEPEM. Subplot (a) analyzes the effect of LoRA rank and insertion position, while subplot (b) shows the effect of the distillation coefficient $\alpha$ across datasets.

Figure 8: New-task and old-task performance of LoRA-based CIL methods. The left subplot reports accuracy on newly introduced tasks as a proxy for plasticity, and the right subplot reports retained accuracy on previous tasks as a proxy for stability.

Symbol	Definition
$\mathcal{D}_{t}$ , $\mathcal{Y}_{t}$	Training set and class set of task $t$ .
$\mathbf{Y}_{t}$	The set of all classes observed up to task $t$ , i.e., $\mathbf{Y}_{t}=\mathcal{Y}_{1}\cup\cdots\cup\mathcal{Y}_{t}$ .
$\mathbf{E}_{t}$	Task- $t$ LoRA-Expert module, i.e., a group of low-rank LoRA adapters inserted into selected Transformer layers rather than a complete independent ViT. Historical experts are frozen after training.
$\mathbf{U}_{t}$ , $\mathbf{V}_{t}$ , $r$	Low-rank matrices and rank inside each adapter. For row-vector input $\mathbf{e}\in\mathbf{R}^{1\times d_{in}}$ , $\mathbf{U}_{t}\in\mathbf{R}^{d_{in}\times r}$ , $\mathbf{V}_{t}\in\mathbf{R}^{r\times d_{out}}$ , with $r=10$ by default.
$\mathbf{V}_{attn}$	Attention value matrix in Eq. ( 8 ), distinct from the LoRA matrix $\mathbf{V}_{t}$ .
$\mathbf{E}^{router}_{t}$	Router after learning task $t$ , used to extract router-domain features.

Table 8: Notation used in DLEPEM.

Explore similar work

Hyperbolic Prototype Routing for Rehearsal-Free Class-Incremental Learning
Sep 30, 2026HongWei Zhao, Rui Liu, Yong ChenClass-Incremental Learning Learnable Prototypes
Beihang University · Beijing University of Posts and Telecommunications
Miles: Metric Learning with Expandable Subspace for Pre-Trained Model-Based Class-Incremental Learning
Jul 20, 2026Kai Jiang, Zisong Lin, Hongyuan Zhang +2Class-Incremental Learning Meta-Learning
National Key Laboratory of Radar Signal Processing, Xidian University, Xi’an 710071, China · The University of Hong Kong, Hong Kong SAR, China · Institute of Artificial Intelligence (TeleAI) of China Telecom, China
Decoupled and Distilled: Task-Adaptive LoRA-Teachers with Ensemble Knowledge Transfer for Few-Shot Class-Incremental Learning
Sep 30, 2026Hongwei Zhao, Rui Liu, Yansong Liu +2Class-Incremental Learning Few-Shot Learning
School of Computer Science and Engineering, Beihang University · School of Computer Science, Beijing University of Posts and Telecommunications

Sep 30, 2026, cs.LGJ/K move · Enter open · S save

Hyperbolic Prototype Routing for Rehearsal-Free Class-Incremental Learning

HongWei Zhao, Rui Liu, Yong Chen

Beihang University · Beijing University of Posts and Telecommunications

Class-Incremental Learning Learnable Prototypes Layer-Selective Rehearsal Strategy Catastrophic Forgetting Retraining Singular Learning Theory+1

Open paper AskPDF arXiv

Class-Incremental Learning (CIL) aims to continually learn new classes while preserving prior knowledge. Parameter-efficient fine-tuning with pre-trained models enables CIL with minimal parameter updates, but existing approaches still suffer from catastrophic forgetting caused by cumulative interference and suboptimal module-sample matching at inference. We propose Hyperbolic Prototype Routing (HyPro), a rehearsal-free framework for continual learning. HyPro allocates a dedicated LoRA-Expert module to each incremental task for isolated representation learning, then projects routing features onto a Poincare ball and performs geodesic nearest-prototype matching for reliable task-level discrimination. Extensive experiments on standard CIL and Few-Shot CIL benchmarks show that HyPro consistently improves average and final-stage accuracy over strong baselines.

Abstract

Explore similar work

Hyperbolic Prototype Routing for Rehearsal-Free Class-Incremental Learning

Miles: Metric Learning with Expandable Subspace for Pre-Trained Model-Based Class-Incremental Learning

Decoupled and Distilled: Task-Adaptive LoRA-Teachers with Ensemble Knowledge Transfer for Few-Shot Class-Incremental Learning