Quantum-Gated Task-interaction Knowledge Distillation for Pre-trained Model-based Class-Incremental Learning

Linjie Li; Huiyu Xiao; Jiarui Cao; Zhenyu Wu; Yang Ji

cs.LGApr 13, 2026

Quantum-Gated Task-interaction Knowledge Distillation for Pre-trained Model-based Class-Incremental Learning

Authors: Linjie Li, Huiyu Xiao, Jiarui Cao, Zhenyu Wu, Yang Ji

Organizations: School of Information and Communication Engineering, BUPT, China · Engineering Research Center for Information Network, Ministry of Education, China

Abstract

Class-incremental learning (CIL) aims to continuously accumulate knowledge from a stream of tasks and construct a unified classifier over all seen classes. Although pretrained models (PTMs) have shown promising performance in CIL, they still struggle with the entanglement of multi-task subspaces, leading to catastrophic forgetting when task routing parameters are poorly calibrated or task-level representations are rigidly fixed. To address this issue, we propose a novel Quantum-Gated Task-interaction Knowledge Distillation (QKD) framework that leverages quantum gating to guide inter-task knowledge transfer. Specifically, we introduce a quantum-gated task modulation gating mechanism to model the relational dependencies among task embedding, dynamically capturing the sample-to-task relevance for both joint training and inference across streaming tasks. Guided by the quantum gating outputs, we perform task-interaction knowledge distillation guided by these task-embedding-level correlation weights from old to new adapters, enabling the model to bridge the representation gaps between independent task subspaces. Extensive experiments demonstrate that QKD effectively mitigates forgetting and achieves state-of-the-art performance.

Figures & tables

Figure 1: Illustration of QKD. Left: The current task sample is encoded by the frozen PTM and the first adapter, while old adapters generate task representations forming a task pool. These representations are mapped into the quantum space to compute task correlations. Right: The task correlations produce task scores, which weight the feature-level knowledge distillation to guide new adapter training.

Method	CIFAR B0-Inc10		CUB B0-Inc20		IN-A B0-Inc10		IN-R B0-Inc20		VTAB B0-Inc10
Method	$\overline{\mathcal{A}}(\uparrow)$	$\mathcal{A}_{B}$ ( $\uparrow$ )	$\overline{\mathcal{A}}(\uparrow)$	$\mathcal{A}_{B}$ ( $\uparrow$ )	$\overline{\mathcal{A}}(\uparrow)$	$\mathcal{A}_{B}$ ( $\uparrow$ )	$\overline{\mathcal{A}}(\uparrow)$	$\mathcal{A}_{B}$ ( $\uparrow$ )	$\overline{\mathcal{A}}(\uparrow)$	$\mathcal{A}_{B}$ ( $\uparrow$ )
Finetune	79.17	67.90	69.34	52.42	29.29	13.17	72.78	60.65	81.31	82.40
L2P [ 37 ]	89.14	84.18	78.17	65.65	41.94	34.76	76.20	69.95	81.65	61.41
Dualprompt [ 36 ]	88.50	84.50	84.58	59.80	49.98	35.02	71.55	65.37	89.32	78.65
CODA-Prompt [ 28 ]	91.44	86.70	84.69	75.15	47.89	34.23	78.23	72.62	87.24	75.08
SimpleCIL [ 43 ]	82.37	76.27	90.56	85.20	60.03	49.24	61.20	54.33	90.79	84.46

Table 1: Average and final accuracy comparison across five datasets. Bold numbers indicate the best performance, while underlined values represent the second-best. All methods are evaluated under the exemplar-free setting.

Panel of Figure 2 — Figure 2: Performance curve of various methods under different settings. The relative improvement of QKD over the second-best method at the final incremental stage is highlighted with numerical annotations.

Method	Exemplars	CIFAR B0-Inc10		IN-R B0-Inc20
Method	Exemplars	$\overline{\mathcal{A}}(\uparrow)$	$\mathcal{A}_{B}$ ( $\uparrow$ )	$\overline{\mathcal{A}}(\uparrow)$	$\mathcal{A}_{B}$ ( $\uparrow$ )
iCaRL [ 26 ]	20/class	82.46	73.87	72.42	60.67
DER [ 40 ]	20/class	86.04	77.93	80.48	74.32
FOSTER [ 32 ]	20/class	89.87	84.91	81.34	74.48
MEMO [ 47 ]	20/class	92.17	89.26	74.80	66.62
QKD(ours)	0	94.08	90.20	81.79	77.67

Table 2: In contrast to conventional exemplar-based CIL approaches, QKD operates without storing any exemplars.

Panel of Figure 3 — Figure 3: Experimental results with large base classes.

QGTM	$\mathcal{L}_{\mathrm{QKD}}$	$\mathcal{L}_{\mathrm{s}}$	CIFAR B0-Inc20		CUB B0-Inc20		IN-A B0-Inc10
QGTM	$\mathcal{L}_{\mathrm{QKD}}$	$\mathcal{L}_{\mathrm{s}}$	$\overline{\mathcal{A}}(\uparrow)$	$\mathcal{A}_{B}$ ( $\uparrow$ )	$\overline{\mathcal{A}}(\uparrow)$	$\mathcal{A}_{B}$ ( $\uparrow$ )	$\overline{\mathcal{A}}(\uparrow)$	$\mathcal{A}_{B}$ ( $\uparrow$ )
$\times$	$\times$	$\times$	78.42	71.29	65.26	61.98	42.83	37.89
$\checkmark$	$\times$	$\times$	88.59	87.77	91.73	88.28	62.15	51.79
$\checkmark$	$\checkmark$	$\times$	92.69	89.28	92.89	89.42	64.92	54.98
$\checkmark$	$\checkmark$	$\checkmark$	93.76	90.24	93.45	90.20	65.77	55.65

Table 3: Ablation Study on the QGTM,

\mathcal{L}_{\mathrm{QKD}}

and

\mathcal{L}_{\mathrm{s}}

in QKD.

Benchmark	Method	Params	Time(s)	Mem(GB)	Lat/img	Lat/batch	Avg(%)
INR B0-Inc20	MLP	1673416	300.16	2.13	13.23	136.39	80.25
	Cosine	1343432	211.25	2.12	14.07	180.24	80.32
	Attention	10798280	306.80	2.17	14.58	178.75	80.08
	Quantum	1349620	326.85	2.12	22.84	222.40	81.79
CUB B0-Inc20	MLP	1673416	139.49	3.43	19.48	264.77	92.63
	Cosine	1343432	114.77	3.43	13.63	340.64	92.72

Table 4: Computational Analysis of different task recognizers.

Panel of Figure 4 — Figure 4: Further analysis on hyperparameter robustness.

Method	CIFAR B0-Inc10		CUB B0-Inc20		VTAB B0-Inc10
Method	$\overline{\mathcal{A}}(\uparrow)$	$\mathcal{A}_{B}$ ( $\uparrow$ )	$\overline{\mathcal{A}}(\uparrow)$	$\mathcal{A}_{B}$ ( $\uparrow$ )	$\overline{\mathcal{A}}(\uparrow)$	$\mathcal{A}_{B}$ ( $\uparrow$ )
$f_{\mathrm{ViT}}(x)$	93.19	88.83	91.94	86.26	95.10	90.20
$f_{\mathrm{ViT}}(x;\mathcal{A}_{1})$	94.08	90.20	93.54	90.12	95.81	92.20
$f_{\mathrm{ViT}}(x;{[}\mathcal{A}_{1},\mathcal{A}_{2}{]})$	93.82	89.74	93.26	90.35	95.63	91.54

Table 5: Performance Comparison of Different Inputs.

Class-Incremental Learning

Class-incremental learning studies how models can add new categories over time while retaining the ability to recognize previously learned classes.

See all 160 papers

Miles: Metric Learning with Expandable Subspace for Pre-Trained Model-Based Class-Incremental Learning
Jul 20, 2026Kai Jiang, Zisong Lin, Hongyuan Zhang +2Class-Incremental Learning Metric Learning
National Key Laboratory of Radar Signal Processing, Xidian University, Xi’an 710071, China · The University of Hong Kong, Hong Kong SAR, China · Institute of Artificial Intelligence (TeleAI) of China Telecom, China
Semantic-Guided Dynamic Sparsification for Pre-Trained Model-based Class-Incremental Learning
Jan 29, 2026Ruiqi Liu, Boyu Diao, Zijia An +3Class-Incremental Learning Activation Sparsity
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Non-Forgetting Knowledge Allocation with Bi-level Competition for Class-Incremental Learning
May 28, 2026Xiang Tan, Run He, Yawen Cui +6Class-Incremental Learning Incremental Learning
South China University of Technology · Hong Kong Polytechnic University · Agency for Science, Technology and Research +3

Jul 20, 2026J/K move · Enter open · S save

Miles: Metric Learning with Expandable Subspace for Pre-Trained Model-Based Class-Incremental Learning

Kai Jiang, Zisong Lin, Hongyuan Zhang, Xueru Bai+1

National Key Laboratory of Radar Signal Processing, Xidian University, Xi’an 710071, China · The University of Hong Kong, Hong Kong SAR, China · Institute of Artificial Intelligence (TeleAI) of China Telecom, China

Class-Incremental Learning Metric Learning Incremental Learning

Open paper AskPDF arXiv

Class Incremental Learning (CIL) aims to learn new concepts consistently from a data stream without forgetting. Unlike typical CIL methods which need to learn a model from scratch, pre-trained model (PTM) can easily adapt to a new task with fine-tuning. However, existing PTM-based CIL methods fail to achieve a trade-off between performance and computational expenditure, i.e., they either adopt the same parameter space so that leading catastrophic forgetting, or expand a new branch for each task but adding more computational cost. To this end, we propose MetrIc Learning with Expandable Subspace (Miles) to harness the prior information within pre-trained knowledge, thereby orchestrating an efficient expansion of the parameter space through guided optimization. Specifically, it decouples the learnable modules with the pre-trained model, exploiting prior information from intermediate features of the backbone network to enable more flexible parameter expansion. Then, a central loss is adopted to guide the new category to cluster towards the corresponding prototype in the new task subspace while incorporating an auxiliary distance regularization term to maintain metric equilibrium across tasks. Extensive experiments on six benchmark datasets demonstrate that Miles achieves state-of-the-art performance in various CIL settings.

Abstract

Class-Incremental Learning

Explore similar work

Miles: Metric Learning with Expandable Subspace for Pre-Trained Model-Based Class-Incremental Learning

Semantic-Guided Dynamic Sparsification for Pre-Trained Model-based Class-Incremental Learning

Non-Forgetting Knowledge Allocation with Bi-level Competition for Class-Incremental Learning