cs.LGAug 25, 2026
SaveFlower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment
Organizations: Flower Labs · University of Cambridge · University of Auckland · University of Melbourne ·
Abstract
Federated learning (FL) has emerged as a key approach for training models across decentralized data, yet benchmarking in FL remains difficult to reproduce, compare, and extend. Existing evaluations are often tied to custom infrastructure, released as incomplete research code, and conducted primarily in simulation, which limits portability and practical relevance. We present Flower Hub, a platform for publishing, discovering, and executing decentralized and federated applications. We show how it enables reproducible benchmarking by packaging benchmarks as executable, versioned applications with standardized metadata, pinned dependencies, and explicit evaluation workflows. We instantiate this approach with a multi-domain benchmark suite spanning cross-silo and cross-device settings, and including tasks in medical imaging, financial tabular learning, legal instruction tuning, phishing URL detection, and audio tagging. We further demonstrate that the same benchmarking application can run across both simulation and deployment runtimes without changing the application code, enabling unified evaluation across varying learning environments. Beyond model quality, our benchmark design supports system-aware reporting, including runtime and communication metrics. This work advances benchmarking in FL settings from ad hoc code artifacts towards portable, executable, and reusable benchmark applications.
Figures & tables
Figure 1: Overview of Flower Hub. Flower Hub is an open-source benchmarking platform for federated and decentralized learning that decouples application logic from infrastructure, enabling lightweight, portable and shareable benchmark applications. This design allows researchers to focus on algorithmic development and evaluation. Apps can be executed smoothly across both simulation and real-world deployment settings without code modifications, thereby accelerating the transition from development to deployment. Each published benchmarking app follows a standardized package structure, enhancing reproducibility and facilitating sharing, reuse, and extensibility. An integrated system monitor provides real-time tracking of key performance metrics. Together, these features position Flower Hub as a foundation for a collaborative federated benchmarking ecosystem.
| Datasets | Domain | Data modality | Total # samples | # clients | FL type |
|---|---|---|---|---|---|
| fed-brats 2 | Medical | Image | 1.6 K | 5 | Cross-silo |
| fed-fraud-paysim-banks 3 | Finance | Tabular | 6.4 M | 5 | Cross-silo |
| fed-legal 4 | Law | Text | 83.6 K | 5 | Cross-silo |
| fed-phishing-urls 5 | Security | URL | 1.1 M | 100 | Cross-device |
| fed-urbansound8k 6 | Sensing | Audio | 8.7 K | 50 | Cross-device |
Table 1: Summary of the federated learning benchmark datasets, including their application domain, data modality, number of samples, number of clients, and FL setting.
Figure 2: Distribution of sample counts across clients for the five proposed benchmark datasets.
| Tasks | Task type | Model | Model size | Trainable | Evaluation metrics |
|---|---|---|---|---|---|
| Medical imaging | Segmentation | 3D U-Net | 1.77 M | 1.77 M | Dice score |
| Financial fraud detection | Binary CLS | MLP | 10.75 K | 10.75 K | PR-AUC |
| Legal instruction tuning | Instruction tuning | LLM | 3.08 B | 7.67 M | F1 score |
| Phishing URL detection | Binary CLS | 1D CNN | 0.57 M | 0.57 M | ROC-AUC |
| On-device audio tagging | CLS | CNN | 24.40 K | 24.40 K | Accuracy |
Table 2: Summary of the training and evaluation pipelines for the benchmark tasks. “CLS” denotes classification, and “Trainable” refers to the number of trainable model parameters.
| Medical (%) | Finance (%) | Legal (%) | URL (%) | Audio (%) | |
|---|---|---|---|---|---|
| FedAvg | 78.57 (0.09) | 56.40 (0.45) | 72.15 (0.63) | 98.45 (0.05) | 48.70 (1.03) |
| FedProx | 78.58 (0.28) | 55.21 (1.39) | 72.29 (1.44) | 98.48 (0.01) | 52.56 (3.71) |
| FedAvgM | 78.31 (0.39) | 54.86 (0.56) | 71.96 (1.19) | 98.38 (0.02) | 47.44 (2.15) |
| FedAdam | 73.57 (3.76) | 14.71 (17.02) | 70.02 (1.85) | 93.64 (0.20) | 25.78 (5.68) |
| FedAdagrad | 64.02 (10.29) | 44.80 (2.37) | 72.97 (1.50) | 91.02 (0.47) | 22.54 (2.66) |
| FedYogi | 75.18 (1.85) | 2.88 (0.10) | 70.19 (1.41) | 94.80 (0.18) | 27.80 (5.70) |
Table 3: Comparison of aggregation strategies across five tasks. The evaluation metrics are Dice score, PR-AUC, token-level F1 score, ROC-AUC, and accuracy for medical image segmentation, financial fraud detection, legal instruction tuning, phishing URL detection, and audio tagging, respectively. All experiments are conducted over three runs with different random seeds, reporting the mean performance with standard deviation in parentheses. The best results are highlighted in bold , and the second-best are underlined .
Figure 3: Changes in evaluation metrics over FL rounds for the medical image segmentation, legal instruction tuning, and phishing URL detection tasks.
| Tasks | Comm. (GB) | CPU Mem. (GB) | GPU Mem. (GB) | Training (s) | Latency (s) |
|---|---|---|---|---|---|
| Medical | 3.85 | 36.23 | 2.45 | 10.05 K | 40.52 K |
| Finance | 0.01 | 2.78 | 0.02 | 363.01 | 13.25 K |
| Legal | 2.86 | 9.47 | 21.24 | 14.63 K | 42.43 K |
| URL | 0.85 | 1.91 | 0.31 | 27.34 | 244.00 |
| Audio | 0.28 | 7.33 | 0.10 | 156.67 | 2.17 K |
Table 4: System performance across benchmark tasks in simulation mode, measured using FedAvg. The legal task is executed on an H200 GPU, while all other tasks use an A100 GPU. “Comm.” denotes the total communication cost over all training rounds. “CPU Mem.” and “GPU Mem.” represent the average peak memory usage across rounds. “Training” indicates the cumulative client training time, and “Latency” refers to the total end-to-end time required to complete federated training.
Figure 4: Deployment profiles for FedMedSeg and FedFinFraud under FedAvg. Stacked bars show per-round wall-clock time split into client training and non-training overhead (e.g., preprocessing, communication, and synchronization). Right panels show cumulative communication over FL rounds.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Overview of Flower infrastructure.
Figure 6: Example of an application detail page on Flower Hub. ① indicates the navigation link to the main applications page. ② denotes the publisher account name, which links to the publisher’s profile page. ③ displays the application name, and ④ shows the full application identifier in the @account/app format. ⑤ provides a button to execute the application on SuperGrid. ⑥ reports the number of stars received by the application, while ⑦ and ⑧ indicate the total number of downloads and executions, respectively. ⑨ presents the command for fetching the application to a local environment. ⑩ corresponds to the information panel, including the README, code details, versioning, license, and verification metadata. The interface shown reflects the design at the time of publication and may evolve in future versions.
Figure 7: Number of samples and class distribution across clients in the proposed federated audio tagging dataset.
Figure 8: Distribution of fraud rates and phishing rates across clients for the financial fraud detection and phishing URL detection benchmarks.
| Tasks | num-rounds | fraction-train | local-epochs | batch-size | lr-max | lr-min | optimizer |
|---|---|---|---|---|---|---|---|
| Medical | 20 | 1.0 | 3 | 4 | 1e-4 | 1e-5 | Adam |
| Finance | 20 | 1.0 | 2 | 1024 | 1e-4 | 1e-5 | Adam |
| Legal | 10 | 1.0 | 1 | 16 | 5e-5 | 1e-5 | AdamW |
| URL | 20 | 0.1 | 1 | 128 | 1e-3 | 1e-4 | AdamW |
| Audio | 100 | 0.3 | 2 | 32 | 1e-2 | 1e-4 | Adam |
Table 5: Selected hyperparameter configurations for the five benchmark tasks.
Figure 9: Configuration example for the fed-med-seg application. (Left) The configuration block specifies metadata, including the application name, version, description, license, and pinned dependencies. (Right) The configuration block defines the hyperparameters used for federated training and evaluation.
Figure 10: Cross-region deployment topology used in our deployment-mode experiments. The federated server was hosted in Canada, while five clients were deployed on Vast.ai instances in the USA, Poland, Spain, and Japan. Edges indicate client-server communication during federated training.
| Role | GPU | CPU | VRAM |
|---|---|---|---|
| Server | 1 RTX 5090 | AMD Ryzen 9 5900XT 16-Core | 32 GB |
| Client 0 | 1 RTX 5090 | AMD EPYC 9J14 96-Core | 32 GB |
| Client 1 | 1 RTX 5090 | AMD Ryzen 7 7700X 8-Core | 32 GB |
| Client 2 | 1 RTX 5090 | AMD EPYC 9654 96-Core | 32 GB |
| Client 3 | 1 RTX 5090 | AMD EPYC 7642 48-Core | 32 GB |
| Client 4 | 1 RTX 4090 | AMD EPYC 7662 64-Core | 48 GB |
Table 6: Vast.ai instance configuration used for deployment-mode experiments.
Figure 11: Changes in evaluation metrics over federated learning rounds for the financial fraud detection and audio tagging tasks. FedAdam, FedAdagrad, and FedYogi exhibit suboptimal or unstable performance on these tasks.
| Benchmark | Mean CPU | Peak CPU | Mean GPU | Peak GPU |
|---|---|---|---|---|
| FedMedSeg | 23.27 GiB | 23.31 GiB | 2.45 GiB | 2.45 GiB |
| FedFinFraud | 1.73 GiB | 1.74 GiB | 20.15 MiB | 20.15 MiB |
Table 7: Client-side memory footprint during deployment-mode FedAvg runs.
More in this topic
Benchmark Design
Benchmark design creates tasks, datasets, metrics, and protocols that make AI capabilities measurable and comparisons credible.
See all 391 papers