Task Relevance-Driven Prompt Expansion for Federated Continual Learning
Authors: Jiahua Dong, Wenqi Liang, Yang Cong, Yichen Li, Hongyuan Yu, Henghui Ding, Yulun Zhang, Salman Khan, Fahad Shahbaz Khan
Organizations: Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · University of Trento, Trento, Italy · South China University of Technology, Guangzhou, China · School of Computer Science and Technology, Huazhong University of Science and Technology · Xiaomi, Beijing, China · Institute of Big Data, Fudan University, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China
Federated continual learning (FCL) has attracted considerable interest by enabling local clients to continually acquire knowledge from new tasks. However, most FCL methods assume that the global server has prior knowledge of the arrival times and number of categories for new tasks, which is unrealistic in the real world. Furthermore, they neglect the importance of positive inter-task semantic relationships when learning new tasks and are significantly affected by irrelevant relationships from old tasks, thereby exacerbating catastrophic forgetting. To resolve the above challenges, we develop a novel Task Relevance-driven Prompt Expansion (TRPE) model. Specifically, to eliminate the reliance on prior knowledge of new tasks, we propose an out-of-distribution detector to identify the arrival of new tasks within local clients, along with a radius-adaptive DBSCAN to infer the quantity of categories in new tasks on the proxy server. Moreover, to explore positive inter-task relationships, we calculate the affinity between the original gradient for learning new task and its gradient orthogonal to the feature space of old tasks to evaluate semantic relevance between old and new tasks from a global perspective. Then, we use this global task relevance to determine the key-prompt pair for new tasks and identify highly related old tasks. After concatenating their prompts for training, we can leverage such positive inter-task relations to improve the continual learning of newly arriving tasks. To balance the learning of new tasks and the forgetting of old ones, a soft orthogonality constraint is applied to the local key and prompt gradients. Experiments spanning diverse datasets and backbones confirm the efficacy of our TRPE model.
Figures & tables
Fig. 1: Illustration of the Federated Continual Learning (FCL) problem.
Fig. 2: Diagram of our model. It contains an out-of-distribution detector for recognizing new-task arrivals at local clients and a radius-adaptive DBSCAN module for estimating new-task category counts at the proxy server, thereby addressing dependence on prior knowledge (Sec. IV-B1 ). Our TRPE also comprises a task relevance measure to exploit positive inter-task relations for the continual learning of new tasks (Sec. IV-B2 ).
Fig. 3: Graphical illustration of the proposed task relevance measure.
Backbone
Methods
Exemplars
T=5
T=10
T=20
FAA
Imp.
CAA
Imp.
FAA
Imp.
CAA
Imp.
FAA
Imp.
CAA
Imp.
Sup-21K [ 45 ]
GLFC [ 13 ]
500
45.94
⇑ 42.50
58.67
⇑ 16.93
45.84
⇑ 41.80
53.86
⇑ 22.18
41.10
⇑ 43.20
44.12
⇑ 39.65
HFC [ 11 ] + FL
500
54.20
⇑ 34.24
64.05
⇑ 11.55
49.49
⇑ 38.15
55.68
⇑ 20.36
46.03
⇑ 38.27
47.98
⇑ 35.79
DualPrompt [ 65 ] + FL
0
71.35
⇑ 17.09
48.16
⇑ 27.44
64.48
⇑ 23.16
50.92
⇑ 25.13
54.31
⇑ 29.99
47.63
⇑ 36.14
CODA-Prompt [ 53 ] + FL
0
72.40
⇑ 16.04
60.40
⇑ 15.20
68.42
⇑ 19.22
63.41
⇑ 12.63
58.42
⇑ 25.88
55.31
⇑ 28.46
HiDe-Prompt [ 62 ] + FL
0
80.52
⇑ 7.92
57.98
⇑ 17.62
82.77
⇑ 4.87
69.18
⇑ 6.86
80.38
⇑ 3.92
70.90
⇑ 12.87
TABLE I: Comparison results on the CIFAR-100 dataset [ 22 ] under different class-incremental settings of FCL, where Imp. refers to the improvements.
Backbone
Methods
Exemplars
T=5
T=10
T=20
FAA
Imp.
CAA
Imp.
FAA
Imp.
CAA
Imp.
FAA
Imp.
CAA
Imp.
Sup-21K [ 45 ]
GLFC [ 13 ]
200
53.71
⇑ 31.49
60.20
⇑ 22.44
45.89
⇑ 38.00
50.43
⇑ 33.41
39.84
⇑ 43.51
43.24
⇑ 38.95
HFC [ 11 ] + FL
200
55.15
⇑ 30.05
63.29
⇑ 19.35
46.58
⇑ 37.31
53.44
⇑ 30.40
40.63
⇑ 42.72
45.37
⇑ 36.82
DualPrompt [ 65 ] + FL
0
64.93
⇑ 20.27
72.09
⇑ 10.55
60.20
⇑ 23.69
69.77
⇑ 14.08
52.41
⇑ 30.94
59.61
⇑ 22.58
CODA-Prompt [ 53 ] + FL
0
63.81
⇑ 21.39
68.97
⇑ 13.67
61.25
⇑ 22.64
67.62
⇑ 16.22
55.48
⇑ 27.87
57.98
⇑ 24.21
HiDe-Prompt [ 62 ] + FL
0
82.60
⇑ 2.59
80.46
⇑ 2.18
81.96
⇑ 1.93
81.09
⇑ 1.76
82.05
⇑ 1.30
80.96
⇑ 1.23
TABLE II: Comparison results on the CUB-200 dataset [ 59 ] under different class-incremental settings of FCL, where Imp. refers to the improvements.
Backbone
Methods
Exemplars
T=10
T=20
T=40
FAA
Imp.
CAA
Imp.
FAA
Imp.
CAA
Imp.
FAA
Imp.
CAA
Imp.
Sup-21K [ 45 ]
GLFC [ 13 ]
100
50.20
⇑ 16.21
57.13
⇑ 5.22
48.82
⇑ 14.38
52.97
⇑ 7.31
45.53
⇑ 15.02
51.82
⇑ 7.03
HFC [ 11 ] + FL
100
51.74
⇑ 14.67
58.91
⇑ 3.44
50.08
⇑ 9.12
55.35
⇑ 4.93
48.47
⇑ 12.08
55.65
⇑ 3.20
DualPrompt [ 65 ] + FL
0
55.36
⇑ 11.05
51.78
⇑ 10.57
50.87
⇑ 12.33
49.32
⇑ 10.97
42.66
⇑ 17.89
44.67
⇑ 14.18
CODA-Prompt [ 53 ] + FL
0
57.40
⇑ 9.01
53.10
⇑ 9.25
52.90
⇑ 10.30
51.20
⇑ 9.08
47.30
⇑ 13.25
46.90
⇑ 11.95
HiDe-Prompt [ 62 ] + FL
0
65.21
⇑ 1.19
54.67
⇑ 7.68
61.05
⇑ 2.15
57.55
⇑ 2.74
57.81
⇑ 2.74
56.24
⇑ 2.61
TABLE III: Comparison results on the ImageNet-R dataset [ 66 ] under different class-incremental settings of FCL, where Imp. refers to the improvements.
Backbone
Methods
FAA
Imp.
CAA
Imp.
Sup-21K [ 79 ]
DualPrompt [ 65 ] + FL
39.74
⇑ 8.33
38.24
⇑ 7.58
HiDe-Prompt [ 62 ] + FL
45.11
⇑ 2.96
41.65
⇑ 4.17
PGP [ 43 ] + FL
40.51
⇑ 7.56
38.96
⇑ 6.86
MFCL [ 2 ]
40.37
⇑ 7.70
40.08
⇑ 5.74
PIP [ 37 ]
40.58
⇑ 7.49
43.79
⇑ 2.03
Powder [ 41 ]
41.45
⇑ 6.62
40.26
⇑ 5.56
TABLE IV: Comparisons on the Office dataset [ 47 ] under the domain-incremental setting of FCL, where Imp. refers to the improvements.
Backbone
Methods
FAA
Imp.
CAA
Imp.
Sup-21K [ 45 ]
DualPrompt [ 65 ] + FL
67.20
⇑ 15.81
72.57
⇑ 3.37
HiDe-Prompt [ 62 ] + FL
79.71
⇑ 2.30
72.85
⇑ 3.09
PGP [ 43 ] + FL
64.23
⇑ 18.78
71.51
⇑ 4.43
MFCL [ 2 ]
73.37
⇑ 9.64
71.67
⇑ 4.27
PIP [ 37 ]
72.57
⇑ 10.44
74.33
⇑ 1.61
Powder [ 41 ]
72.15
⇑ 10.86
73.58
⇑ 2.36
TABLE V: Comparisons on the Digit dataset [ 29 ] under the domain-incremental setting of FCL, where Imp. refers to the improvements.
Variants
Base
Base+CT
Base+CL
Base+CT
Ours
+TR
+TR+DP
Modules
CT
✗
✓
✓
✓
✓
TR
✗
✗
✓
✓
✓
DP
✗
✗
✗
✓
✓
SO
✗
✗
✗
✗
✓
Sup-21K [ 45 ]
T=5
FAA
73.51
80.52
86.17
86.49
88.44
TABLE VI: Ablation studies under different class-incremental settings of FCL on the CIFAR-100 dataset [ 22 ] .
Fig. 4: Representation visualization of 10 randomly-selected classes from CIFAR-100 [ 22 ] when the backbone is Sup-21K [ 45 ] and T=10 .
Variants
Base
Base+CT
Base+CT
Base+CT
Ours
+TR
+TR+DP
Modules
CT
✗
✓
✓
✓
✓
TR
✗
✗
✓
✓
✓
DP
✗
✗
✗
✓
✓
SO
✗
✗
✗
✗
✓
Sup-21K [ 45 ]
T=5
FAA
70.89
82.60
84.09
84.70
85.20
TABLE VII: Ablation studies on the CUB-200 dataset [ 59 ] under different class-incremental settings of FCL.
Variants
Base
Base+CT
Base+CT
Base+CT
Ours
+TR
+TR+DP
Modules
CT
✗
✓
✓
✓
✓
TR
✗
✗
✓
✓
✓
DP
✗
✗
✗
✓
✓
SO
✗
✗
✗
✗
✓
Sup-21K [ 45 ]
FAA
41.59
45.11
46.88
47.08
48.07
TABLE VIII: Ablation on Office [ 47 ] under the domain-incremental setting of FCL.
Backbone
Variants
FAA
Imp.
CAA
Imp.
Sup-21K [ 45 ]
Powder [ 41 ]
57.70
⇑ 30.74
62.40
⇑ 13.20
Ours w/ IG
84.05
⇑ 4.39
73.26
⇑ 2.34
Ours w/ FI
85.62
⇑ 2.82
74.01
⇑ 1.59
Ours w/ TC
83.92
⇑ 4.52
72.79
⇑ 2.81
Ours-RandP
80.13
⇑ 8.31
71.64
⇑ 3.96
Ours-w/o TR
83.57
⇑ 4.87
73.05
⇑ 2.55
TABLE IX: Analysis of task relevance measure on CIFAR-100 [ 22 ] when T=5 .
Backbone
Variants
FAA
Imp.
CAA
Imp.
Sup-21K [ 45 ]
Powder [ 41 ]
55.30
⇑ 32.34
58.00
⇑ 18.04
Ours w/ IG
84.46
⇑ 3.18
72.08
⇑ 3.96
Ours w/ FI
83.79
⇑ 3.85
72.30
⇑ 3.74
Ours w/ TC
84.50
⇑ 3.14
73.28
⇑ 2.76
Ours-RandP
80.35
⇑ 7.29
71.50
⇑ 4.54
Ours-w/o TR
82.56
⇑ 5.08
72.45
⇑ 3.59
TABLE X: Analysis of task relevance measure on CIFAR-100 [ 22 ] when T=10 .
Task
Methods
FAA
Imp.
CAA
Imp.
T=5
DualPrompt [ 65 ] + FL
70.54
⇑ 15.85
46.30
⇑ 28.03
CODA-Prompt [ 53 ] + FL
71.28
⇑ 15.11
56.49
⇑ 17.84
HiDe-Prompt [ 62 ] + FL
79.13
⇑ 7.26
55.18
⇑ 19.10
PGP [ 43 ] + FL 5
70.15
⇑ 16.24
46.72
⇑ 27.61
MFCL [ 2 ]
65.01
⇑ 21.38
69.13
⇑ 5.20
PIP [ 37 ]
65.45
⇑ 20.94
70.28
⇑ 4.05
TABLE XI: Comparisons on CIFAR-100 [ 22 ] under the asynchronous FCL setting, where all methods use Sup-21K [ 45 ] as the backbone.
Fig. 5: Cluster visualization ( t≥2 ) grouped by radius-adaptive DBSCAN on CUB-200 [ 59 ] when the backbone is Sup-21K [ 45 ] and T=20 .
Fig. 6: Cluster visualization ( t≥2 ) grouped by radius-adaptive DBSCAN on CIFAR-100 [ 22 ] when the backbone is Sup-21K [ 45 ] and T=5 .
Methods
#Params
Memory
#Comm
Training
Test
GLFC [ 13 ]
86.64M
351.63M
3465.63M
0.38s
0.05s
HFC [ 11 ] + FL
86.64M
351.63M
3465.63M
0.38s
0.05s
DualPrompt [ 65 ] + FL
1.04M
4.10M
41.65M
0.13s
0.05s
CODA-Prompt [ 53 ] + FL
0.93M
3.74M
37.28M
0.13s
0.05s
HiDe-Prompt [ 62 ] + FL
0.84M
3.38M
33.62M
0.21s
0.09s
PGP [ 43 ] + FL
1.04M
4.10M
41.65M
0.14s
0.05s
TABLE XII: Comparisons of computation complexity on CIFAR-100 [ 22 ] under the Sup-21K backbone [ 45 ] when T=5 .
Heterogeneity
t=1
t=2
t=3
t=4
t=5
30%
60.50
60.72
61.60
66.61
77.03
40%
75.81
66.55
75.15
72.06
88.44
50%
76.80
72.37
77.80
80.75
89.07
60%
83.80
84.85
86.70
86.04
90.33
TABLE XIII: Analysis of heterogeneous data distributions on CIFAR-100 [ 22 ] under the Sup-21K backbone [ 45 ] when T=5 .
Setting
Heterogeneity
t=2
t=3
t=4
t=5
Synchronous FCL
5%
17.00
100.00
100.00
100.00
10%
79.00
100.00
100.00
100.00
15%
100.00
100.00
100.00
100.00
20%
100.00
100.00
100.00
100.00
30%
100.00
100.00
100.00
100.00
40%
100.00
100.00
100.00
100.00
TABLE XIV: Task-boundary detection accuracy (%) of the proposed out-of-distribution detector on the CIFAR-100 dataset [ 22 ] under synchronous and asynchronous FCL settings, using Sup-21K [ 45 ] as the backbone and T=5 .
Fig. 7: Hyperparameter analysis of δe (a)(b)(c)(d) and nω (e)(f) on CIFAR-100 dataset [ 22 ] when T={5,10} .
Fig. 8: Task-wise convergence analysis on CIFAR-100 [ 22 ] (a) and CUB-200 [ 59 ] (b) when T=5 .
Federated continual learning (FCL) enables collaborative model training across distributed clients on sequentially arriving tasks without revisiting past data. However, existing approaches often suffer from catastrophic forgetting, rely on replay buffers or generative models that may violate privacy constraints, or assume knowledge of task identities during inference. We propose FedProTIP (Federated Projection-based Continual Learning with Task Identity Prediction), a replay-free FCL framework that maintains shared task-specific feature subspaces across clients. Each client extracts low-rank core bases from intermediate activations using randomized singular value decomposition, capturing dominant feature directions associated with the current task. These bases are transmitted to the server and aggregated to construct global task subspaces that capture shared feature directions across clients without requiring data sharing. During training, client updates are projected onto the orthogonal complement of previously learned subspaces to reduce cross-task interference and mitigate catastrophic forgetting. The learned subspaces are also reused during inference to estimate task identity via subspace relevance, enabling task-agnostic prediction without requiring explicit task labels. Experiments on CIFAR100, ImageNet-R, and DomainNet demonstrate that FedProTIP consistently outperforms state-of-the-art federated continual learning baselines while maintaining lower training time, memory footprint, and communication cost.
Seohyeon Cha, Huancheng Chen, Haris Vikalo
Department of Electrical and Computer Engineering The University of Texas at Austin
Federated Continual Learning (FCL) is fundamental to real-world distributed learning systems, requiring models to adapt to sequential, non-IID data across clients while mitigating catastrophic forgetting and client drift. Existing approaches formulate continual learning (CL) as a sequence of per-task optimization problems, applied locally at each client and coupled through aggregation, using heuristic mechanisms such as replay, regularization, or projection-based constraints. However, forgetting in FCL is inherently a long-term, distributed phenomenon, arising from the interaction of temporal task evolution and cross-client heterogeneity, which is not explicitly regulated. In this work, we cast FCL as a stochastic control problem and propose Federated Queue-regulated Continual Learning (FedQCL), a framework based on Lyapunov drift-plus-penalty (DPP) optimization. FedQCL introduces virtual queues to track the accumulation of forgetting across tasks and clients, enabling explicit control of the stability-plasticity trade-off. By optimizing a DPP objective, the method jointly improves current-task performance while the queue-based formulation provides an interpretable and tunable mechanism to balance adaptation and retention through a single parameter, without requiring gradient projection or additional communication overhead. Empirical evaluations on standard benchmarks, including Split-CIFAR-10, Split-CIFAR-100, and Split-TinyImageNet, demonstrate that FedQCL outperforms state-of-the-art baselines with respect to accuracy while significantly reducing forgetting under heterogeneous data distributions.
Federated Continual Learning (FCL) leverages inter-client collaboration to better balance new knowledge acquisition and old knowledge retention on non-stationary data. However, existing FCL methods struggle to adapt to streaming scenarios where sequential and ephemerally accessible data chunks lack task identifiers and exhibit arbitrary class overlap, leading to confusion between old and new knowledge and an inability to sustain local inference on all encountered classes. To address this, we propose FedKACE with three components: 1) an adaptive mechanism that determines when to switch the inference model from the local to the global one to improve client-side inference performance; 2) a responsive gradient-balanced replay scheme that utilizes the ratio of the squared L2 gradient norms to balance client-specific knowledge between new acquisition and old retention; 3) a holistic buffer maintenance strategy that preserves highly informative and boundary-significant samples to enhance knowledge retention under class overlap.Experiments across multiple scenarios and theoretical analysis demonstrate the effectiveness of FedKACE.
Sixing Tan, Xianmin Liu
Faculty of Computing Harbin Institute of Technology