FedACT: Concurrent Federated Intelligence across Heterogeneous Data Sources
Authors: Md Sirajul Islam, Isabelle G Chapman, N I Md Ashafuddula, Xu Yuan, Li Chen, Nian-Feng Tzeng, Klara Nahrstedt
Organizations: School of Computing and Informatics, University of Louisiana at Lafayette, USA · Department of Computer and Information Sciences, University of Delaware, USA · Department of Computer Science, University of Illinois at Urbana-Champaign, USA
Federated Learning (FL) enables collaborative intelligence across decentralized data source devices in a privacy-preserving way. While substantial research attention has been drawn to optimizing the learning process for an individual task, real-world applications increasingly require multiple machine learning tasks simultaneously training their models across a shared pool of devices. Naively applying single-FL optimization techniques in multi-FL systems results in suboptimal system performance, particularly due to device heterogeneity and resource inefficiency. To address such a critical open challenge, we introduce {\em FedACT}, a novel resource heterogeneity-aware device scheduling approach designed to efficiently schedule heterogeneous devices across multiple concurrent FL jobs, with the goal of minimizing their average job completion time (JCT). {\em FedACT} dynamically assigns devices to FL jobs based on an alignment scoring mechanism that evaluates the compatibility between available resources of devices and resource demands of jobs. Additionally, it incorporates participation fairness to ensure balanced contributions from devices across jobs, further enhancing the accuracy levels of learned global models. An optimal scheduling plan is formulated in {\em FedACT} by prioritizing devices with higher alignment scores, while ensuring fair participation across jobs. To evaluate the effectiveness of the proposed scheduling algorithm, we carried out comprehensive experiments using diverse FL jobs and benchmark datasets. Experimental results demonstrate that {\em FedACT} reduces the average JCT by up to 8.3× and improves model accuracy by up to 44.5%, compared to the state-of-the-art baselines.
Figures & tables
Fig. 1 : An example of Multi-Job Federated Learning: smartphones simultaneously improve next-word prediction for keyboards while enhancing image quality.
Notation
Definition
K ; ∣K∣
Set of all devices; the total number of devices
M ; T ; m
The total number of jobs; total training time; job index
Dkm ; dkm ; ∣Dkm∣
The local dataset on Device k for Job m ; batch size; size of Dkm
Dm ; ∣Dm∣
Global dataset of Job m ; size of Dm
Fkm(w) ; Fm(w)
Local loss function of Device k for Job m ; global loss function of Job m
wkm,r(j)
Local model of Job m in Device k in the j -th local update of Round r
TABLE I : Notations and Descriptions
Fig. 2 : An overview of the training procedure within the FedACT framework.
Group A
Group B
Datasets
Cifar-10
Emnist (L)
Emnist (D)
Fashion-Mnist
Cifar-10
Mnist
Features
32 × 32
28 × 28
28 × 28
28 × 28
32 × 32
28 × 28
Model Architecture
VGG-16
CNN
LeNet-5
CNN
ResNet-18
AlexNet
Model Parameters
26,233K
3,785K
62K
225K
598k
3,275K
Dataset Size
50k/10k
124.8k/20.8k
240k/40k
60K/10K
50K/10K
60K/10K
Local epochs
5
5
5
5
5
5
TABLE II : A summary of the datasets and models used in our experiments. The dataset size indicates the number of training samples and test samples (number of training samples/number of test samples). Emnist (L) and Emnist (D) represent Emnist-Letters and Emnist-Digits, respectively.
Convergence Accuracy
Time (min)
Random
Greedy
Genetic
MJ-FL
FedACT
Random
Greedy
Genetic
MJFL
FedACT
IID
LeNet
0.9924
0.990
0.9931
0.9937
0.9943
LeNet(0.99)
15.13
17.25
6.42
5.18
3.92
CNN
0.942
0.926
0.928
0.935
0.942
CNN(0.93)
58.41
30.17
109.68
19.18
11.52
VGG
0.611
0.52
0.557
0.602
0.605
VGG(0.55)
134.3
/
229.7
59.10
43.25
Non-IID
TABLE III : Elapsed wall-clock time needed to reach the convergence accuracy and target accuracy (shown in parentheses) for different methods in Group A. “/” denotes that the method failed to achieve the target accuracy.
Convergence Accuracy
Time (min)
Random
Greedy
Genetic
MJ-FL
FedACT
Random
Greedy
Genetic
MJFL
FedACT
IID
AlexNet
0.9937
0.9834
0.9938
0.9939
0.9942
AlexNet(0.9933)
37.02
/
20.28
21.90
13.58
CNN
0.868
0.868
0.867
0.868
0.871
CNN(0.867)
123.10
36.88
40.08
31.35
21.08
ResNet
0.786
0.742
0.753
0.784
0.796
ResNet(0.74)
64.66
52.68
31.26
16.65
11.42
Non-IID
TABLE IV : Elapsed wall-clock time needed to reach the convergence accuracy and target accuracy (shown in parentheses) for different methods in Group B. “/” denotes that the method failed to achieve the target accuracy.
IID/Non-IID
IID/Non-IID
Job
VGG
CNN
LeNet
ResNet
CNN
AlexNet
Target Accuracy
0.55/0.55
0.93/0.80
0.99/0.984
0.74/0.50
0.867/0.73
0.9933/0.976
Time (min)
140.7/2488.3
49.6/52.4
19.1/49.5
*/896.7
32.1/36.8
67.02/116.5
TABLE V : Elapsed wall-clock time needed to reach the target accuracy in Single-Job FL using FedAvg. “*” denotes that it fails to reach the target accuracy.
Fig. 3 : Test accuracy versus elapsed wall-clock time for different jobs in Group A with the IID distribution.
Fig. 4 : Test accuracy versus elapsed wall-clock time for different jobs in Group B with the IID distribution.
Fig. 5 : Test accuracy versus elapsed wall-clock time for different jobs in Group A with the Non-IID distribution.
Fig. 6 : Test accuracy versus elapsed wall-clock time for different jobs in Group B with the Non-IID distribution.
Fig. 7 : Elapsed wall-clock training time required for each job of Group A to achieve the target convergence accuracy under IID and Non-IID data distributions.
Fig. 8 : Elapsed wall-clock training time required for each job of Group B to achieve the target convergence accuracy under IID and Non-IID data distributions.
Federated learning (FL) is a decentralized approach that enables collaborative model training without exposing raw data. Instead of transferring sensitive data, it allows devices to share only model weights, keeping personal data locally and secure. However, in real world settings, the data held by devices is often not evenly distributed and devices mostly differ in computing power and memory capacity. These differences make FL harder to maintain consistent performance across the system. To address these issues, we propose FedMTFI, a novel architecture that combines multi-teacher knowledge distillation (MTKD) with feature importance to improve the FL process in heterogeneous environments. In FedMTFI, clients are clustered based on similar hardware and model types. Each cluster trains a specific model on not independently and identically distributed (non-IID) data. Within a cluster, every client updates that model using only its own local private data. The server then aggregates the locally trained models in each cluster using FedAvg to form multiple prototype models. Then these prototypes serve as teacher models to train a global generalized student model using MTKD. What makes FedMTFI more unique is the integration of Shapley values (SHAP) to emphasize important features during distillation, which enhances both accuracy and interpretability. Experimental results show that FedMTFI achieves higher accuracy than traditional FL algorithms and performs more effectively under non-IID data conditions.
Scheduling clients for model training is critical in federated learning due to both data and system heterogeneity. Most previous works focus on the quality of the scheduled clients to achieve faster convergence, shorter wall-clock convergence time, or better average model performance. They rarely consider the diversity of clients, which is important to counter heterogeneity and improve performance for the worst-off clients. In this work, we advocate the use of determinantal point processes (DPPs) to model and enhance the diversity in client scheduling. We first design the kernel matrices of DPPs using gradient information and quality scores, which inherently enables a flexible quality-diversity trade-off. Applying fast MAP inference over DPPs, we propose Adaptive Determinantal Client Scheduling (ADCS) in FL. We further quantify the gradient approximation error of ADCS and develop convergence analysis for general biased client selection in FL with non-convex loss functions. We conduct comparative numerical experiments showing that ADCS outperforms state-of-the-art client scheduling algorithms, including both quality-based and diversity-based ones.
Federated learning (FL) is a communication-efficient distributed learning paradigm. However, client drift remains one of the most critical challenges, hindering the efficient training of a global model. In this study, we propose a novel latent information sharing scheme that directly mitigates data heterogeneity across clients. Our theoretical and empirical results show that sharing a small amount of hidden-layer activations significantly improves training efficiency while preserving convergence guarantees and data privacy. Furthermore, we compare our method with existing FL approaches designed to address client drift, including FedProx, SCAFFOLD, FedPVR, FedProto, and SplitFed, and demonstrate superior model accuracy under a fixed round budget without incurring excessive communication overhead. Overall, this work presents a promising new knowledge aggregation scheme and provides a comprehensive analysis of the impact of activation sharing on federated optimization.