FedAdaVR: Adaptive Variance Reduction for Robust Federated Learning under Limited Client Participation
Organizations: School of Computing and Mathematical Sciences, University of Leicester, Leicester, United Kingdom · School of Informatics, University of Edinburgh, Edinburgh, United Kingdom · Department of Computer Science, University of Exeter, Exeter, United Kingdom
Abstract
Federated learning (FL) encounters substantial challenges due to heterogeneity, leading to gradient noise, client drift, and partial client participation errors, the last of which is the most pervasive but remains insufficiently addressed in current literature. In this paper, we propose FedAdaVR, a novel FL algorithm aimed at solving heterogeneity issues caused by partial client participation by incorporating an adaptive optimiser with a variance reduction technique. This method takes advantage of the most recent stored updates from clients, even when they are absent from the current training round, thereby emulating their presence. Furthermore, we propose FedAdaVR-Quant, a server-side memory compression variant that stores historical client updates in quantised form (while keeping client-server communication in full precision), reducing the server memory footprint of FedAdaVR by 50% (FP16), 75% (Int8), and 87.5% (Int4) while maintaining highly competitive, and occasionally superior model performance. We analyse the non-convex convergence behaviour of FedAdaVR with Adagrad and prove that our proposed framework eliminates the first-order partial client participation bias while improving monotonically with cohort size. Extensive experiments conducted on multiple vision and NLP datasets demonstrate that FedAdaVR consistently outperforms state-of-the-art baseline methods.
Figures & tables
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Step | Standard FedVARP | Standard FedOpt | Naive Stack (FedVARP + FedOpt) | FedAdaVR |
| Signal | ||||
| (Adam, Yogi) | (None) | |||
| (Adam) | (None) | |||
| (Yogi) | (None) | same on | same on | |
| (Adagrad) | (None) | |||
| Bias correction | — | none (all three) | none | , (Adam, Yogi); none (Adagrad) |
| Dataset | ||||
| Parameter | MNIST | F-MNIST | CIFAR-10 | Shakespeare |
| # of clients | 500 | 500 | 250 | 904 |
| # of participating clients | 5 | 5 | 5 | 5 |
| % of participating clients | 1% | 1% | 2% | |
| # of clients for evaluation | 250 | 250 | 250 | 225 |
| Local model | LeNet-5 | LeNet-5 | ResNet-18 | GRU |
| Partition | ||||||||
| Type | Dataset | IID | Non-IID | Dirichlet | LQ-1 | LQ-2 | LQ-3 | Natural ID |
| Training rounds | MNIST | 60 | 60 | 60 | 100 | 60 | 60 | x |
| F-MNIST | 100 | 100 | 150 | 350 | 150 | 150 | x | |
| CIFAR-10 | 200 | 300 | 400 | 1500 | 500 | 500 | x | |
| Shakespeare | x | x | x | x | x | x | 200 | |
| Selected rounds | MNIST | 10 | 10 | 10 | 10 | 10 | 10 | x |
| Local Model | Parameters (Millions) | # of Clients (Thousands) | Approximate Memory (GB) | |||
| FP32 | FP16 | Int8 | Int4 | |||
| SqueezeNet 1.1 [ Iandola et al., 2016 ] | 1.2 | 1 | 4.6 | 2.3 | 1.15 | 0.58 |
| 10 | 46.0 | 23.0 | 11.5 | 5.8 | ||
| 100 | 460 | 230 | 115 | 58 | ||
| ShuffleNet V2 [ Ma et al., 2018 ] | 2.3 | 1 | 8.8 | 4.4 | 2.2 | 1.1 |
| 10 | 88.0 | 44.0 | 22.0 | 11.0 | ||
| Algorithm | Communication Cost | Server Memory | Client Computation |
| FedAvg | |||
| FedAdam | |||
| FedAdagrad | |||
| FedYogi | |||
| FedProx | + Proximal Term | ||
| SCAFFOLD | + Control Variates |
| Algorithm | Time/Round (s) | Time to Reach Target Accuracy (s) | |||
| 30% | 35% | 40% | 45% | ||
| FedAvg | 166.43 | 103,020 | 136,972 | 220,353 | 232,336 |
| FedAdam | 168.32 | - | - | - | - |
| FedAdagrad | 167.89 | - | 229,673 | - | - |
| FedYogi | 168.54 | - | 249,945 | - | - |
| FedProx | 170.57 | 123,834 | 157,095 | 203,490 | - |
| Algorithm | Criteria | Accuracy Threshold | |||||
| 20% | 25% | 30% | 35% | 40% | 45% | ||
| FedAvg | Rounds | 335 | 488 | 619 | 823 | 1324 | 1396 |
| Norm. Rounds | 3.3 | 2.3 | 2.1 | 2.0 | 2.0 | 1.5 | |
| FedAdagrad | Rounds | 355 | 646 | 988 | 1368 | x | x |
| Norm. Rounds | 3.5 | 3.1 | 3.3 | 3.2 | x | x | |
| FedAdam | Rounds | x | x | x | x | x | x |
| Algorithm | Optimiser | |||||
| 0.001 | 0.005 | 0.01 | 0.1 | 1.0 | ||
| FedVARP | x | x | x | 43.038 | 49.434 | 50.288 |
| FedAdaVR | Adabelief | 45.271 | 49.453 | 49.514 | x | x |
| Adam | 35.733 | 48.541 | 49.343 | x | x | |
| Adagrad | 38.379 | 50.048 | 50.577 | x | x | |
| Yogi | 27.921 | 48.440 | 49.549 | x | x | |
| Partition | ||||||||
| Algorithm | Optimiser | IID | Non_IID | Dir. | LQ-1 | LQ-2 | LQ-3 | |
| FedAvg | - | - | 91.06 | 90.44 | 81.75 | 75.26 | 74.04 | 78.78 |
| FedProx | - | - | 90.34 | 86.74 | 85.24 | 78.81 | 69.61 | 80.32 |
| SCAFFOLD | - | - | 89.63 | 89.59 | 90.31 | 38.45 | 84.40 | 88.79 |
| FedNova | - | - | 90.85 | 89.15 | 82.44 | 70.85 | 66.69 | 69.79 |
| FedAdam | - | - | 11.24 | 10.10 | 10.39 | 11.04 | 10.21 | 10.89 |
| Partition | ||||||||
| Algorithm | Optimiser | IID | Non_IID | Dir. | LQ-1 | LQ-2 | LQ-3 | |
| FedAvg | - | - | 81.35 | 77.43 | 77.08 | 63.53 | 59.70 | 71.92 |
| FedProx | - | - | 81.33 | 76.92 | 75.8 | 48.12 | 59.12 | 69.66 |
| SCAFFOLD | - | - | 67.18 | 55.81 | 62.0 | 48.49 | 41.57 | 52.24 |
| FedNova | - | - | 80.10 | 76.46 | 76.23 | 66.23 | 63.57 | 71.15 |
| FedAdam | - | - | 10.00 | 10.00 | 10.00 | 10.00 | 10.00 | 10.00 |
| Partition | ||||||||
| Algorithm | Optimiser | IID | Non_IID | Dirichlet | LQ-1 | LQ-2 | LQ-3 | |
| FedAvg | - | - | 68.4 | 65.6 | ||||
| FedProx | - | - | 66.9 | 63.4 | ||||
| SCAFFOLD | - | - | 62.8 | 60.9 | 48.89 | 13.09 | 32.35 | 39.64 |
| FedNova | - | - | 68.2 | 65.1 | ||||
| FedAdam | - | - | 41.6 | 36.4 | ||||
Federated Learning
Federated learning trains shared models across data holders without requiring them to pool their raw data.
See all 1,025 papers