We study federated learning (FL) with adversarial clients, where the goal is to minimize the average loss of the honest (non-adversarial) clients without knowing their identity. Under heterogeneity, a single shared model parameter is statistically inappropriate: it cannot capture the distinct data-generating processes across clients, incurring an irreducible model-heterogeneity bias and severely limiting robustness to adversarial clients (a.k.a. Byzantine-robustness). We address this problem through representation learning, where each client learns a personalized linear head, while collaboratively estimating a shared nonlinear representation through Byzantine-robust aggregation. We demonstrate that the heterogeneity among honest representation gradients is controlled by the representation error and statistical errors that decay either with the number of data samples per client (τ) or the number of iterations (T). In particular, our non-asymptotic parameter recovery error bound reveals three terms: (i) an initialization-dependent error that goes away with T, (ii) finite-sample noise terms that decreases with τ and the number of honest clients, and (iii) a stochastic gradient variance term that also reduces with T. Importantly, with no irreducible model-heterogeneity bias in our bounds. We extend the regression analysis to multiclass classification, and empirically validate it on CIFAR-10, FEMNIST, and School Exam Score datasets.
Figures & tables
NNM+Krum
NNM+TrMean
Dataset
Loss
Method
ALIE
Mimic
ALIE
Mimic
CIFAR-10
Cross-entropy
Baseline
50.46±0.57
46.39±1.76
50.46±0.71
47.17±1.58
Cross-entropy
Rep. learning
88.79±0.42
88.50±0.59
88.76±0.44
88.54±0.61
Multiclass
Rep. learning
88.33±0.53
87.95±0.51
88.42±0.52
87.89±0.60
FEMNIST
Cross-entropy
Baseline
72.95±1.95
70.34±1.23
73.09±2.03
70.66±1.47
Cross-entropy
Rep. learning
91.89±0.31
92.43±0.15
92.02±0.28
92.51±0.07
Table 1: Average local test accuracy (%) on CIFAR-10 and FEMNIST with 50 honest and 5 Byzantine clients per round. Entries report mean ± standard deviation over three seeds.
Figure 1: FEMNIST accuracy under the Mimic attack using cross-entropy loss and a fixed number of five Byzantine clients. Each curve reports the mean accuracy and standard deviation across three random seeds as the number of honest clients per round increases.
NNM+Krum
NNM+TrMean
Method
ALIE
Mimic
ALIE
Mimic
Baseline
0.7611±0.0452
0.6532±0.0201
0.7672±0.0465
0.6531±0.0226
Rep. learning
0.6324±0.0272
0.6300±0.0276
0.6316±0.0275
0.6301±0.0276
Table 2: School Exam test MSE using the squared loss with 20 honest and 5 Byzantine clients per round. Results report mean ± standard deviation over three seeds. Lower is better.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
CIFAR-10
FEMNIST
School Exam Score
∣H∣ / f
50/5
50/5
20/5
Communication rounds
100
200
500
Mini-batch size
10
10
32
Representation dimension
64
64
64
Representation learning rate
0.01
0.01
0.05
Head learning rate
0.01
0.01
0.01
Appendix
Table 3: Hyperparameters used in the numerical experiments. Here ∣H∣ and f denote the numbers of honest and adversarial updates received at each communication round, respectively.
Federated learning (FL) is intrinsically heterogeneous: honest clients may have different data-generating models. On top of that, adversarial clients can make heterogeneity even more pronounced by sharing arbitrary updates. Existing analyses typically control the interaction between statistical heterogeneity and adversarial behavior through gradient-dissimilarity conditions. However, the underlying bound is imposed a priori and may yield conservative guarantees even for least-squares regression. We instead derive the gradient heterogeneity from the statistical model of linear and nonlinear regression with fresh data samples at every round. Our bounds separate heterogeneity among the honest clients' ground-truth model parameters, finite-sample label noise, and initialization. We then demonstrate that, for any (f,κ)-robust aggregator with coefficient κ=O(f/n), where f is the number of adversarial clients and n the total number of clients (with f/n<1/2), convergence holds after an explicit sample burn-in.
Leonardo F. Toso, James Anderson, Nirupam Gupta +1
Columbia University, USA · University of Copenhagen, Denmark · Sorbonne Université and Université Paris Cité, CNRS, LPSM, France
Federated learning distributes data among n clients, making it vulnerable to malicious attacks and data heterogeneity, which together pose challenges for robust learning. To tackle this issue, centered clipping and Huber aggregators have been exploited for Byzantine robustness. In this paper, we first demonstrate their equivalence via convex conjugate theory, and show that they can yield biased solutions in the presence of outliers, leading to failure under high data heterogeneity and a substantial fraction of outliers. Next, we propose a new robust aggregation rule that utilizes the truncated-quadratic (TQ) loss, effectively mitigating the biases of existing methods, such as centered clipping and Huber aggregators. We show that our aggregator achieves order-optimal Byzantine-robust learning under nonconvex loss functions and heterogeneous data, ultimately enhancing the reliability of federated learning systems. Additionally, we provide a robust deviation estimation strategy for TQ, demonstrating its effectiveness. Furthermore, we show that TQ maintains robustness even when only an estimate of the number of Byzantine clients is available. Finally, experimental results on MNIST, Fashion-MNIST, and CIFAR-10, indicate that our aggregator provides better robustness performance than the competing techniques.
Zhi-Yong Wang, Hao Nan Sheng, Werner Stefan +3
State Key Laboratory of Ocean Sensing, Ocean College, Zhejiang University, Hangzhou, China · Department of Electrical Engineering, City University of Hong Kong, Hong Kong, China · Department of Information and Communications Engineering, Aalto University, 02150 Espoo, Finland +1
This paper proposes a robust decentralized personalized federated learning method R-DPFL, that enables clients to reduce the impact of Byzantine attacks via robust neighborhood direction estimation and history-based update trend prediction, rather than purely aggregating client models as in the existing work. In R-DPFL, each client first computes the current-round model update by aggregating the received neighborhood update vectors. It then predicts what this update should be based on its historical values and local model changes. Finally, R-DPFL computes the difference between these two quantities, adaptively clips this difference, and adds it to the local update. We prove convergence of the learning process through rigorous analysis and show that honest clients maintain stable personalized descent dynamics under Byzantine neighbor perturbations without requiring consensus among neighboring models. Extensive experiments on CIFAR-10 demonstrate that RDPFL consistently outperforms state-of-the-art decentralized and personalized federated learning baselines under heterogeneous and adversarial settings.
Xiao Ma, Hong Shen, Hui Tian +2
Faculty of Applied Sciences, Macao Polytechnic University, Macao SAR, China