Federated Learning (FL) is a paradigm for training machine learning (ML) models in collaborative settings while preserving participants' privacy by keeping raw data local. A key requirement for the use of FL in production is reliability, as insufficient reliability can compromise the validity, stability, and reproducibility of learning outcomes. FL inherently operates as a distributed system and is therefore susceptible to crash failures, network partitioning, and other fault scenarios. Despite this, the impact of such failures on FL outcomes has not yet been studied systematically. In this paper, we address this gap by investigating the impact of missing participants in FL. To this end, we conduct extensive experiments on image, tabular, and time-series data and analyze how the absence of participants affects model performance, taking into account influencing factors such as data skewness, different availability patterns, and model architectures. Furthermore, we examine scenario-specific aspects, including the utility of the global model for missing participants. Our experiments provide detailed insights into the effects of various influencing factors. In particular, we show that data skewness has a strong impact, often leading to overly optimistic model evaluations and, in some cases, even altering the effects of other influencing factors.
Figures & tables
Parameter
Value
Local Rounds, Global Rounds
5 Rounds, 30 Rounds
Aggregation Strategy, Optimizer (Learning rate)
FedAvg, Adam (0.0001)
Availability Timing, Available Rounds
Start, 2
Metrics (Classification, Forecasting)
Macro F1-score, R2-score
Table 1: Base configuration for all datasets
Modifiers
Description
Range of values
DSkew
Data Skew
Base Skew , High Skew, Manual Skew
NParti
Number of Participants
4 , 6, 8
FChar.Ti
Participant Availability
Start , Middle, End, StartEnd
DMod
Data Modality
Image , Tabular, Time-series
MArch (Image)
Model Architecture
CNN , ResNet50
MArch (GermanSolarFarm)
LSTM , 1D-CNN
Table 2: Overview of all modifiers used in the experiments with description and possible values. Default values are shown in bold
Figure 1: Image Datasets: Comparison of the evaluation with and without the failed participant across base skew (BS) and manual skew (MS)
Figure 2: Tabular Datasets: Comparison of the evaluation with and without the failed participant for the BS and MS
Figure 3: GermanSolarFarm: Comparison of the evaluation with and without the missing participant across different models
Figure 4: CIFAR10 CNN: Different availability phases for the missing participant across different skews. The baseline represents the absence of the failure
Figure 5: Influence of the ResNet architecture on the availability phases for image datasets with BS. The baseline represents the absence of the failure
Figure 6: Adult: Availability phases of the missing participant across HS and MS. The baseline represents the absence of the failure
Figure 7: Covertype: Availability phases for the missing participant across the BS and HS. The baseline represents the absence of the failure
Figure 8: Manuel Skew: Different availability phases for the missing participant across CIFAR10 and Covertype. The baseline represents the absence of the failure
Figure 9: GermanSolarFarm: Different availability phases for missing participants across different model architectures, where the baseline represents the absence of the failure
Figure 10: Image BS: Comparison of the model performance for the missing participant during failure and participation (baseline) across different numbers of participants
Figure 11: Tabular BS: Comparison of the model performance for the missing participant during failure and participation (baseline) across different numbers of participants
Figure 12: Time-series: Comparison of the model performance for the missing participant during failure and participation (baseline) across different numbers of participants
Figure 13: MS: Comparison of the global model performance for the missing participant during failure and participation (baseline) across different numbers of participants
Round
P0-SV
P1-SV
P2-SV
P3-SV
0-BS
-0.014305
0.039144
-0.036371
0.018734
1-BS
-0.004299
0.014654
-0.036120
0.031978
0-HS
0.045645
-0.013814
0.000002
-0.025670
1-HS
0.027890
-0.036587
-0.032981
0.049456
Table 3: CIFAR10 : SVs for base skew (BS) and high skew (HS) for the first rounds until failure
Figure 14: CIFAR10 : Quantity distribution per participant across different skews
Figure 15: CIFAR10 : Impact of different participants dropping out of the process
Performance evaluation is essential for assessing the quality of machine learning (ML) models and guiding deployment decisions. In federated learning (FL), assessing the performance is challenging because data are distributed across participants. Consequently, the coordinator must rely on locally computed evaluation metrics and aggregate them to assess the global model. A key challenge is that common aggregation strategies, such as weighted averaging based on the local samples per participant, do not always produce the same results as centralized evaluation. Existing definitions of performance evaluation are largely tailored to accuracy and do not generalize to other metrics, leading to inconsistencies between participant-based and centralized evaluation. However, such discrepancies are inconsistent with the FL objective and lead to a wrong calculation of the metric. To address this issue, we examine the underlying reasons for these discrepancies and propose FLAM, a performance evaluation method based on aggregatable measures that yields the same results as centralized evaluation without the need for a global test dataset.
Fabian Stricker, Jose A. Peregrina, David Bermbach +1
Institute of Data-Centric Software Systems (IDSS), Hochschule Karlsruhe, Karlsruhe, Germany · Scalable Software Systems Research Group, TU Berlin, Berlin, Germany
Federated learning (FL) is a popular distributed learning framework where multiple clients perform local training and a server aggregates the locally updated models. FL enables decentralized training while preserving the privacy of clients' datasets. However, non-independent and identically distributed (non-IID) or noisy datasets can lead to low model accuracy or high convergence latency. Precluding these clients through client selection may mitigate the problem, but heavily biased client selections may also degrade the learning performance. In this study, we first experimentally measure the impact of non-IID data (including skews in data quantity and label distribution), noisy data, and fairness in client selection on model accuracy and convergence. We then propose a privacy-preserving scoring method to assess each client's contribution in FL, with experiments conducted to demonstrate the effectiveness of the proposed assessment.
Yuan-Heng Tsai, Li-Hsing Yen, Yan-Wei Chen
Department of Computer Science, National Yang Ming Chiao Tung University, Hsinchu, Taiwan.
Federated learning offers a privacy-friendly collaborative learning framework, yet its success, like any joint venture, hinges on the contributions of its participants. Existing client evaluation methods predominantly focus on model performance, such as accuracy or loss, which represents only one dimension of a machine learning model's overall utility. In contrast, this work investigates the critical, yet overlooked, issue of client contributions towards a model's trustworthiness -- specifically, its reliability (tolerance to noisy data), resilience (resistance to adversarial examples), and fairness (measured via demographic parity). To quantify these multifaceted contributions, we employ the state-of-the-art approximation of the Shapley value, a principled method for value attribution. Our results reveal that no single client excels across all dimensions, which are largely independent from each other, highlighting a critical flaw in current evaluation scheme: no single metric is adequate for comprehensive evaluation and equitable rewarding allocation.