cs.CVOct 25, 2026

Task Relevance-Driven Prompt Expansion for Federated Continual Learning

Authors: Jiahua Dong, Wenqi Liang, Yang Cong, Yichen Li, Hongyuan Yu, Henghui Ding, Yulun Zhang, Salman Khan, 

Organizations: Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · University of Trento, Trento, Italy · South China University of Technology, Guangzhou, China · School of Computer Science and Technology, Huazhong University of Science and Technology · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Federated continual learning (FCL) has attracted considerable interest by enabling local clients to continually acquire knowledge from new tasks. However, most FCL methods assume that the global server has prior knowledge of the arrival times and number of categories for new tasks, which is unrealistic in the real world. Furthermore, they neglect the importance of positive inter-task semantic relationships when learning new tasks and are significantly affected by irrelevant relationships from old tasks, thereby exacerbating catastrophic forgetting. To resolve the above challenges, we develop a novel Task Relevance-driven Prompt Expansion (TRPE) model. Specifically, to eliminate the reliance on prior knowledge of new tasks, we propose an out-of-distribution detector to identify the arrival of new tasks within local clients, along with a radius-adaptive DBSCAN to infer the quantity of categories in new tasks on the proxy server. Moreover, to explore positive inter-task relationships, we calculate the affinity between the original gradient for learning new task and its gradient orthogonal to the feature space of old tasks to evaluate semantic relevance between old and new tasks from a global perspective. Then, we use this global task relevance to determine the key-prompt pair for new tasks and identify highly related old tasks. After concatenating their prompts for training, we can leverage such positive inter-task relations to improve the continual learning of newly arriving tasks. To balance the learning of new tasks and the forgetting of old ones, a soft orthogonality constraint is applied to the local key and prompt gradients. Experiments spanning diverse datasets and backbones confirm the efficacy of our TRPE model.

Figures & tables

Fig. 1: Illustration of the Federated Continual Learning (FCL) problem.
Fig. 2: Diagram of our model. It contains an out-of-distribution detector for recognizing new-task arrivals at local clients and a radius-adaptive DBSCAN module for estimating new-task category counts at the proxy server, thereby addressing dependence on prior knowledge (Sec. IV-B1 ). Our TRPE also comprises a task relevance measure to exploit positive inter-task relations for the continual learning of new tasks (Sec. IV-B2 ).
Fig. 3: Graphical illustration of the proposed task relevance measure.
BackboneMethodsExemplarsT=5T=5T=10T=10T=20T=20
  FAAImp.CAAImp.FAAImp.CAAImp.FAAImp.CAAImp.
Sup-21K [ 45 ]GLFC [ 13 ]50045.94⇑\Uparrow 42.5058.67⇑\Uparrow 16.9345.84⇑\Uparrow 41.8053.86⇑\Uparrow 22.1841.10⇑\Uparrow 43.2044.12⇑\Uparrow 39.65
HFC [ 11 ] + FL50054.20⇑\Uparrow 34.2464.05⇑\Uparrow 11.5549.49⇑\Uparrow 38.1555.68⇑\Uparrow 20.3646.03⇑\Uparrow 38.2747.98⇑\Uparrow 35.79
DualPrompt [ 65 ] + FL071.35⇑\Uparrow 17.0948.16⇑\Uparrow 27.4464.48⇑\Uparrow 23.1650.92⇑\Uparrow 25.1354.31⇑\Uparrow 29.9947.63⇑\Uparrow 36.14
CODA-Prompt [ 53 ] + FL072.40⇑\Uparrow 16.0460.40⇑\Uparrow 15.2068.42⇑\Uparrow 19.2263.41⇑\Uparrow 12.6358.42⇑\Uparrow 25.8855.31⇑\Uparrow 28.46
HiDe-Prompt [ 62 ] + FL080.52⇑\Uparrow 7.9257.98⇑\Uparrow 17.6282.77⇑\Uparrow 4.8769.18⇑\Uparrow 6.8680.38⇑\Uparrow 3.9270.90⇑\Uparrow 12.87
TABLE I: Comparison results on the CIFAR-100 dataset [ 22 ] under different class-incremental settings of FCL, where Imp. refers to the improvements.
BackboneMethodsExemplarsT=5T=5T=10T=10T=20T=20
  FAAImp.CAAImp.FAAImp.CAAImp.FAAImp.CAAImp.
Sup-21K [ 45 ]GLFC [ 13 ]20053.71⇑\Uparrow 31.4960.20⇑\Uparrow 22.4445.89⇑\Uparrow 38.0050.43⇑\Uparrow 33.4139.84⇑\Uparrow 43.5143.24⇑\Uparrow 38.95
HFC [ 11 ] + FL20055.15⇑\Uparrow 30.0563.29⇑\Uparrow 19.3546.58⇑\Uparrow 37.3153.44⇑\Uparrow 30.4040.63⇑\Uparrow 42.7245.37⇑\Uparrow 36.82
DualPrompt [ 65 ] + FL064.93⇑\Uparrow 20.2772.09⇑\Uparrow 10.5560.20⇑\Uparrow 23.6969.77⇑\Uparrow 14.0852.41⇑\Uparrow 30.9459.61⇑\Uparrow 22.58
CODA-Prompt [ 53 ] + FL063.81⇑\Uparrow 21.3968.97⇑\Uparrow 13.6761.25⇑\Uparrow 22.6467.62⇑\Uparrow 16.2255.48⇑\Uparrow 27.8757.98⇑\Uparrow 24.21
HiDe-Prompt [ 62 ] + FL082.60⇑\Uparrow 2.5980.46⇑\Uparrow 2.1881.96⇑\Uparrow 1.9381.09⇑\Uparrow 1.7682.05⇑\Uparrow 1.3080.96⇑\Uparrow 1.23
TABLE II: Comparison results on the CUB-200 dataset [ 59 ] under different class-incremental settings of FCL, where Imp. refers to the improvements.
BackboneMethodsExemplarsT=10T=10T=20T=20T=40T=40
  FAAImp.CAAImp.FAAImp.CAAImp.FAAImp.CAAImp.
Sup-21K [ 45 ]GLFC [ 13 ]10050.20⇑\Uparrow 16.2157.13⇑\Uparrow 5.2248.82⇑\Uparrow 14.3852.97⇑\Uparrow 7.3145.53⇑\Uparrow 15.0251.82⇑\Uparrow 7.03
HFC [ 11 ] + FL10051.74⇑\Uparrow 14.6758.91⇑\Uparrow 3.4450.08⇑\Uparrow 9.1255.35⇑\Uparrow 4.9348.47⇑\Uparrow 12.0855.65⇑\Uparrow 3.20
DualPrompt [ 65 ] + FL055.36⇑\Uparrow 11.0551.78⇑\Uparrow 10.5750.87⇑\Uparrow 12.3349.32⇑\Uparrow 10.9742.66⇑\Uparrow 17.8944.67⇑\Uparrow 14.18
CODA-Prompt [ 53 ] + FL057.40⇑\Uparrow 9.0153.10⇑\Uparrow 9.2552.90⇑\Uparrow 10.3051.20⇑\Uparrow 9.0847.30⇑\Uparrow 13.2546.90⇑\Uparrow 11.95
HiDe-Prompt [ 62 ] + FL065.21⇑\Uparrow 1.1954.67⇑\Uparrow 7.6861.05⇑\Uparrow 2.1557.55⇑\Uparrow 2.7457.81⇑\Uparrow 2.7456.24⇑\Uparrow 2.61
TABLE III: Comparison results on the ImageNet-R dataset [ 66 ] under different class-incremental settings of FCL, where Imp. refers to the improvements.
BackboneMethodsFAAImp.CAAImp.
Sup-21K [ 79 ]DualPrompt [ 65 ] + FL39.74⇑\Uparrow 8.3338.24⇑\Uparrow 7.58
HiDe-Prompt [ 62 ] + FL45.11⇑\Uparrow 2.9641.65⇑\Uparrow 4.17
PGP [ 43 ] + FL40.51⇑\Uparrow 7.5638.96⇑\Uparrow 6.86
MFCL [ 2 ]40.37⇑\Uparrow 7.7040.08⇑\Uparrow 5.74
PIP [ 37 ]40.58⇑\Uparrow 7.4943.79⇑\Uparrow 2.03
Powder [ 41 ]41.45⇑\Uparrow 6.6240.26⇑\Uparrow 5.56
TABLE IV: Comparisons on the Office dataset [ 47 ] under the domain-incremental setting of FCL, where Imp. refers to the improvements.
BackboneMethodsFAAImp.CAAImp.
Sup-21K [ 45 ]DualPrompt [ 65 ] + FL67.20⇑\Uparrow 15.8172.57⇑\Uparrow 3.37
HiDe-Prompt [ 62 ] + FL79.71⇑\Uparrow 2.3072.85⇑\Uparrow 3.09
PGP [ 43 ] + FL64.23⇑\Uparrow 18.7871.51⇑\Uparrow 4.43
MFCL [ 2 ]73.37⇑\Uparrow 9.6471.67⇑\Uparrow 4.27
PIP [ 37 ]72.57⇑\Uparrow 10.4474.33⇑\Uparrow 1.61
Powder [ 41 ]72.15⇑\Uparrow 10.8673.58⇑\Uparrow 2.36
TABLE V: Comparisons on the Digit dataset [ 29 ] under the domain-incremental setting of FCL, where Imp. refers to the improvements.
 VariantsBaseBase+CTBase+CLBase+CTOurs
+TR+TR+DP
ModulesCT✗✓✓✓✓
TR✗✗✓✓✓
DP✗✗✗✓✓
SO✗✗✗✗✓
Sup-21K [ 45 ]T ⁣= ⁣5T\!=\!5FAA73.5180.5286.1786.4988.44
TABLE VI: Ablation studies under different class-incremental settings of FCL on the CIFAR-100 dataset [ 22 ] .
Fig. 4: Representation visualization of 10 randomly-selected classes from CIFAR-100 [ 22 ] when the backbone is Sup-21K [ 45 ] and T=10T=10 .
 VariantsBaseBase+CTBase+CTBase+CTOurs
+TR+TR+DP
ModulesCT✗✓✓✓✓
TR✗✗✓✓✓
DP✗✗✗✓✓
SO✗✗✗✗✓
Sup-21K [ 45 ]T ⁣= ⁣5T\!=\!5FAA70.8982.6084.0984.7085.20
TABLE VII: Ablation studies on the CUB-200 dataset [ 59 ] under different class-incremental settings of FCL.
 VariantsBaseBase+CTBase+CTBase+CTOurs
+TR+TR+DP
ModulesCT✗✓✓✓✓
TR✗✗✓✓✓
DP✗✗✗✓✓
SO✗✗✗✗✓
Sup-21K [ 45 ]FAA41.5945.1146.8847.0848.07
TABLE VIII: Ablation on Office [ 47 ] under the domain-incremental setting of FCL.
BackboneVariantsFAAImp.CAAImp.
Sup-21K [ 45 ]Powder [ 41 ]57.70⇑\Uparrow 30.7462.40⇑\Uparrow 13.20
Ours w/ IG84.05⇑\Uparrow 4.3973.26⇑\Uparrow 2.34
Ours w/ FI85.62⇑\Uparrow 2.8274.01⇑\Uparrow 1.59
Ours w/ TC83.92⇑\Uparrow 4.5272.79⇑\Uparrow 2.81
Ours-RandP80.13⇑\Uparrow 8.3171.64⇑\Uparrow 3.96
Ours-w/o TR83.57⇑\Uparrow 4.8773.05⇑\Uparrow 2.55
TABLE IX: Analysis of task relevance measure on CIFAR-100 [ 22 ] when T=5T=5 .
BackboneVariantsFAAImp.CAAImp.
Sup-21K [ 45 ]Powder [ 41 ]55.30⇑\Uparrow 32.3458.00⇑\Uparrow 18.04
Ours w/ IG84.46⇑\Uparrow 3.1872.08⇑\Uparrow 3.96
Ours w/ FI83.79⇑\Uparrow 3.8572.30⇑\Uparrow 3.74
Ours w/ TC84.50⇑\Uparrow 3.1473.28⇑\Uparrow 2.76
Ours-RandP80.35⇑\Uparrow 7.2971.50⇑\Uparrow 4.54
Ours-w/o TR82.56⇑\Uparrow 5.0872.45⇑\Uparrow 3.59
TABLE X: Analysis of task relevance measure on CIFAR-100 [ 22 ] when T=10T=10 .
TaskMethodsFAAImp.CAAImp.
T=5T=5DualPrompt [ 65 ] + FL70.54⇑\Uparrow 15.8546.30⇑\Uparrow 28.03
CODA-Prompt [ 53 ] + FL71.28⇑\Uparrow 15.1156.49⇑\Uparrow 17.84
HiDe-Prompt [ 62 ] + FL79.13⇑\Uparrow 7.2655.18⇑\Uparrow 19.10
PGP [ 43 ] + FL 570.15⇑\Uparrow 16.2446.72⇑\Uparrow 27.61
MFCL [ 2 ]65.01⇑\Uparrow 21.3869.13⇑\Uparrow 5.20
PIP [ 37 ]65.45⇑\Uparrow 20.9470.28⇑\Uparrow 4.05
TABLE XI: Comparisons on CIFAR-100 [ 22 ] under the asynchronous FCL setting, where all methods use Sup-21K [ 45 ] as the backbone.
Fig. 5: Cluster visualization ( t≥2t\geq 2 ) grouped by radius-adaptive DBSCAN on CUB-200 [ 59 ] when the backbone is Sup-21K [ 45 ] and T=20T=20 .
Fig. 6: Cluster visualization ( t≥2t\geq 2 ) grouped by radius-adaptive DBSCAN on CIFAR-100 [ 22 ] when the backbone is Sup-21K [ 45 ] and T=5T=5 .
Methods#ParamsMemory#CommTrainingTest
GLFC [ 13 ]86.64M351.63M3465.63M0.38s0.05s
HFC [ 11 ] + FL86.64M351.63M3465.63M0.38s0.05s
DualPrompt [ 65 ] + FL1.04M4.10M41.65M0.13s0.05s
CODA-Prompt [ 53 ] + FL0.93M3.74M37.28M0.13s0.05s
HiDe-Prompt [ 62 ] + FL0.84M3.38M33.62M0.21s0.09s
PGP [ 43 ] + FL1.04M4.10M41.65M0.14s0.05s
TABLE XII: Comparisons of computation complexity on CIFAR-100 [ 22 ] under the Sup-21K backbone [ 45 ] when T=5T=5 .
Heterogeneityt=1t=1t=2t=2t=3t=3t=4t=4t=5t=5
30%60.5060.7261.6066.6177.03
40%75.8166.5575.1572.0688.44
50%76.8072.3777.8080.7589.07
60%83.8084.8586.7086.0490.33
TABLE XIII: Analysis of heterogeneous data distributions on CIFAR-100 [ 22 ] under the Sup-21K backbone [ 45 ] when T=5T=5 .
SettingHeterogeneityt=2t=2t=3t=3t=4t=4t=5t=5
Synchronous FCL5%17.00100.00100.00100.00
10%79.00100.00100.00100.00
15%100.00100.00100.00100.00
20%100.00100.00100.00100.00
30%100.00100.00100.00100.00
40%100.00100.00100.00100.00
TABLE XIV: Task-boundary detection accuracy (%) of the proposed out-of-distribution detector on the CIFAR-100 dataset [ 22 ] under synchronous and asynchronous FCL settings, using Sup-21K [ 45 ] as the backbone and T=5T=5 .
Fig. 7: Hyperparameter analysis of δe\delta_{e} (a)(b)(c)(d) and nωn_{\omega} (e)(f) on CIFAR-100 dataset [ 22 ] when T={5,10}T=\{5,10\} .
Fig. 8: Task-wise convergence analysis on CIFAR-100 [ 22 ] (a) and CUB-200 [ 59 ] (b) when T=5T=5 .
More in this topic

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 301 papers

Explore similar work

CardsList
Sep 25, 2025

Task-Agnostic Federated Continual Learning via Replay-Free Gradient Projection

Federated continual learning (FCL) enables collaborative model training across distributed clients on sequentially arriving tasks without revisiting past data. However, existing approaches often suffer from catastrophic forgetting, rely on replay buffers or generative models that may violate privacy constraints, or assume knowledge of task identities during inference. We propose FedProTIP (Federated Projection-based Continual Learning with Task Identity Prediction), a replay-free FCL framework that maintains shared task-specific feature subspaces across clients. Each client extracts low-rank core bases from intermediate activations using randomized singular value decomposition, capturing dominant feature directions associated with the current task. These bases are transmitted to the server and aggregated to construct global task subspaces that capture shared feature directions across clients without requiring data sharing. During training, client updates are projected onto the orthogonal complement of previously learned subspaces to reduce cross-task interference and mitigate catastrophic forgetting. The learned subspaces are also reused during inference to estimate task identity via subspace relevance, enabling task-agnostic prediction without requiring explicit task labels. Experiments on CIFAR100, ImageNet-R, and DomainNet demonstrate that FedProTIP consistently outperforms state-of-the-art federated continual learning baselines while maintaining lower training time, memory footprint, and communication cost.

Seohyeon Cha, Huancheng Chen, Haris Vikalo

Department of Electrical and Computer Engineering The University of Texas at Austin

Federated Continual LearningGradient ProjectionTask-Free Continual LearningFederated Learning
AskPDF
Aug 21, 2026

Federated Continual Learning as a Distributed Drift-Plus-Penalty Control Problem

Federated Continual Learning (FCL) is fundamental to real-world distributed learning systems, requiring models to adapt to sequential, non-IID data across clients while mitigating catastrophic forgetting and client drift. Existing approaches formulate continual learning (CL) as a sequence of per-task optimization problems, applied locally at each client and coupled through aggregation, using heuristic mechanisms such as replay, regularization, or projection-based constraints. However, forgetting in FCL is inherently a long-term, distributed phenomenon, arising from the interaction of temporal task evolution and cross-client heterogeneity, which is not explicitly regulated. In this work, we cast FCL as a stochastic control problem and propose Federated Queue-regulated Continual Learning (FedQCL), a framework based on Lyapunov drift-plus-penalty (DPP) optimization. FedQCL introduces virtual queues to track the accumulation of forgetting across tasks and clients, enabling explicit control of the stability-plasticity trade-off. By optimizing a DPP objective, the method jointly improves current-task performance while the queue-based formulation provides an interpretable and tunable mechanism to balance adaptation and retention through a single parameter, without requiring gradient projection or additional communication overhead. Empirical evaluations on standard benchmarks, including Split-CIFAR-10, Split-CIFAR-100, and Split-TinyImageNet, demonstrate that FedQCL outperforms state-of-the-art baselines with respect to accuracy while significantly reducing forgetting under heterogeneous data distributions.

Nazreen Shah, Naveen Kumar Reddy Somireddy, Zubair Shaban +2

IIIT Delhi India · IIT Dharwad India

Federated Continual LearningCatastrophic ForgettingNon-IID Federated Learning
AskPDF
Jan 27, 2026

Knowledge-Aware Evolution for Task-Free Streaming Federated Continual Learning with Arbitrary Class Overlap

Federated Continual Learning (FCL) leverages inter-client collaboration to better balance new knowledge acquisition and old knowledge retention on non-stationary data. However, existing FCL methods struggle to adapt to streaming scenarios where sequential and ephemerally accessible data chunks lack task identifiers and exhibit arbitrary class overlap, leading to confusion between old and new knowledge and an inability to sustain local inference on all encountered classes. To address this, we propose FedKACE with three components: 1) an adaptive mechanism that determines when to switch the inference model from the local to the global one to improve client-side inference performance; 2) a responsive gradient-balanced replay scheme that utilizes the ratio of the squared L2 gradient norms to balance client-specific knowledge between new acquisition and old retention; 3) a holistic buffer maintenance strategy that preserves highly informative and boundary-significant samples to enhance knowledge retention under class overlap.Experiments across multiple scenarios and theoretical analysis demonstrate the effectiveness of FedKACE.

Sixing Tan, Xianmin Liu

Faculty of Computing Harbin Institute of Technology

Continual LearningFederated Continual LearningReplay-Based Continual LearningTask-Free Continual LearningFederated Learning
AskPDF