Modern big-data systems generate massive, heterogeneous, and geographically dispersed streams that are large-scale and privacy-sensitive, making centralization challenging. While federated learning (FL) provides a privacy-enhancing training mechanism, it assumes a static data flow and learns a collaborative model over multiple rounds, making learning with incremental data challenging in limited-communication scenarios. This paper presents One-Shot Incremental Federated Learning (OSI-FL), the first FL framework that addresses the dual challenges of communication overhead and catastrophic forgetting. OSI-FL communicates category-specific embeddings, devised by a frozen vision-language model (VLM) from each client in a single communication round, which a pre-trained diffusion model at the server uses to synthesize new data similar to the client's data distribution. The synthesized samples are used on the server for training. However, two challenges still persist: i) tasks arriving incrementally need to retrain the global model, and ii) as future tasks arrive, retraining the model introduces catastrophic forgetting. To this end, we augment training with Selective Sample Retention (SSR), which identifies and retains the top-p most informative samples per category and task pair based on sample loss. SSR bounds forgetting by ensuring that representative retained samples are incorporated into training in further iterations. The experimental results indicate that OSI-FL outperforms baselines, including traditional and one-shot FL approaches, in both class-incremental and domain-incremental scenarios across three benchmark datasets.
Figures & tables
Fig. 1 : Motivation behind OSI-FL. One-shot FL approaches degrade in performance similar to traditional FL approaches as tasks arrive incrementally, specially when the tasks differ significantly.
Fig. 2 : T-SNE data visualization for multiple tasks, where each task contains a subset of the total classes, in an incremental FL setup (NICO Common dataset). As depicted, the distribution of data varies significantly among tasks, and as these tasks arrive incremetnally, learning the most recent task affects the knowledge about previous tasks (catastrophic forgetting).
Symbol
Explanation
C
Set of participating clients
T
Set of all tasks
n
Number of participating clients
m
Number of tasks
Dt
Data for task t
D
Global Dataset
TABLE I : Symbols used
Criteria
OSCAR
OSI-FL
Description Generation
OSCAR primarily utilizes the BLIP OPT pretrained model for caption generation, which is large, weighing around 5 GB .
In contrast, OSI-FL employs the GPT ViT pretrained model for captioning, which is compact, totaling around 0.9 GB .
Data Division
OSCAR mainly use domain-based data division, incorporating feature-skewed non-IID division to simulate a distributed setup.
OSI-FL adds a label-skewed, non-IID setup in which data is divided by label across clients. Although we also provide experimental data for a feature-skewed setup.
Learning Setup
OSCAR assume that all encodings from all clients are available at the start of the experiments, which is impractical in real-world settings.
OSI-FL incorporates a more realistic setup in which clients join the FL setup incrementally.
TABLE II : Comparison between OSCAR and OSI-FL approaches
Fig. 3 : Overview of OSI-FL framework, as new tasks arrive.
Hyperparameter
Value
Backbone
ResNet-18
Optimizer
Adam
Learning rate
0.001
Batch size
256
Global epochs
20
Weight decay
1e-4
TABLE III : Hyperparameters
Model
FL
OS
IL
FedAvg
✔
✗
✗
FedProx
✔
✗
✗
FedDyn
✔
✗
✗
FedET
✔
✗
✔
FedIL+
✔
✗
✔
FedEWC
✔
✗
✔
TABLE IV : Baselines
Model
Client Test Set Accuracy after each task ↑
T1
T2
T3
T4
T5
T6
OpenImage
FedAvg
93.69
41.54
33.97
28.94
24.75
25.22
FedProx
93.69
41.54
33.98
29.64
24.73
25.22
FedEWC
93.69
41.64
34.22
29.02
25.02
25.19
FedET
95.22
44.52
39.94
34.14
33.61
32.27
TABLE V : Accuracy (in %) on the test set for the baselines and our approach on class incremental settings after each task. Ceiling performance in italic, best performance excluding ceiling in bold.
Model
Client Test Set Accuracy after each task ↑
T1
T2
T3
T4
T5
T6
OpenImage
FedAvg
64.38
52.64
53.08
58.44
58.68
57.14
FedProx
64.38
52.55
53.62
53.94
57.25
57.20
FedEWC
64.38
52.72
53.42
59.66
59.22
58.04
OSCAR-IL
52.74
47.40
48.72
45.66
46.67
45.78
TABLE VI : Accuracy (in %) on the test set for the baselines and our approach on domain incremental settings after each task. Ceiling performance in italic, best performance excluding ceiling in bold
Fig. 4 : Impact of p , the number of retained samples, on the overall performance in OSI-FL.
Fig. 5 : Class incremental accuracy with varying number of clients
Fig. 6 : Peak GPU and CPU memory usage (in GB), and training cost (in TFLOPS) for OSI-FL and baselines (NICO Common dataset and ResNet-18 backbone).
Fig. 7 : Client upload cost in OSI-FL and baselines compared to final performance on class-incremental OpenImage dataset and ResNet-18 backbone (in million parameters, log scaled).
Federated Learning (FL) emerged as a promising distributed machine learning paradigm. However, extending FL to the class incremental learning scenarios introduces unique challenges: 1) Capacity conflict and catastrophic forgetting from the shared model overloading, 2) Heterogeneity from Non-Independent and Identically Distributed (Non-IID) data, and 3) Synchronized class misalignment. In this paper, we propose Fisher-Routed MiXture of Experts for Federated Class-Incremental Learning (FedFMX), a novel framework to address these challenges via adaptive expert specialization across clients. The crucial insight is to route each sample to an expert subset that jointly optimizes knowledge acquisition and retention. Specifically, we introduce a Fisher-Routed Expert Scoring (FRES) module to estimate expert importance via Fisher-based stability cost and gradient-based plasticity gain. Then, we design an Adaptive Expert Selection (AES) module by quantifying marginal contributions for adaptive expert subset determination. Finally, by the routing-aware regularization (RAR), we achieve load balance and efficient FL training. We theoretically prove the O(T−1) convergence rate. Extensive experiments on multiple benchmarks compared with state-of-the-art methods demonstrate the superiority of FedFMX.
Wenhao Yuan, Chenchen Lin, Jian Chen +3
Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China · School of Artificial Intelligence, Sun Yat-sen University, China.
Federated continual learning must integrate new tasks over time without losing earlier-task knowledge. Most existing methods attach an anti-forgetting mechanism to the client-trained, server-aggregated loop of federated learning, which holds back new learning to preserve earlier knowledge and burdens resource-constrained clients. We propose ReSCENE, which structurally mitigates catastrophic forgetting by having each client upload a small condensed surrogate of its local data while the server keeps the surrogates of past tasks and trains the global model on them together with the current task surrogates. For efficient server memory, we introduce temporal herding, which selects the more recent surrogates from the pool accumulated over a task into a compressed buffer. Our study provides a theoretical analysis showing that this buffer can represent the original task data more closely than full accumulation of all surrogates. Across CIFAR-10, CIFAR-100, and TinyImageNet, ReSCENE achieves the strongest accuracy over seven baselines, by up to 31.1 points of average accuracy, while requiring as little as 0.11× of the client computation and up to 179× less upload than the model-update baselines. ReSCENE further demonstrates its effectiveness when scaled to larger client populations and larger models while remaining efficient, which makes it a practical method for federated continual learning.
Federated Learning (FL) aims at unburdening the training of deep models by distributing computation across multiple devices (clients) while safeguarding data privacy. On top of that, Federated Continual Learning (FCL) also accounts for data distribution evolving over time, mirroring the dynamic nature of real-world environments. While previous studies have identified Catastrophic Forgetting and Client Drift as major factors of performance degradation in FCL, we shed light on the importance of Incremental Bias and Federated Bias, which cause models to prioritize classes that are recently introduced or locally predominant, respectively. Our proposal constrains both biases to the last layer by efficiently fine-tuning a pre-trained backbone using learnable prompts, resulting in clients that produce less biased representations and more biased classifiers. Therefore, instead of solely relying on parameter aggregation, we leverage generative prototypes to effectively balance the predictions of the global model. Our proposed methodology significantly improves the current state of the art across six datasets, each including three different scenarios.
Riccardo Salami, Pietro Buzzega, Matteo Mosconi +2
AImageLab - University of Modena and Reggio Emilia, Modena, Italy