cs.CROct 28, 2026
SaveTrustworthy Distributed Machine Learning in Cloud-Edge-IoT Systems: A Heterogeneity-Aware Survey of Security, Privacy, and Robustness
Organizations: Cyber Florida, University of South Florida, Tampa, FL, USA · Department of Computer and Information Science, Fordham University, New York, NY, USA · Data Science Department, University of North Texas, Denton, 76207, USA · Department of Mathematics and Computer Science at the Royal Military College of Canada (RMC) in Kingston, Ontario, Canada ·
Abstract
Distributed machine learning (DML), including conventional distributed training and federated learning (FL), enables collaborative model development across cloud-edge-IoT systems without fully centralizing raw data. However, distributing data, computation, updates, and orchestration increases the attack surface and introduces coupled challenges across privacy, robustness, reliability, fairness, and governance. This survey presents a security-centered, heterogeneity-aware roadmap for trustworthy DML, with FL treated as a prominent high-risk case. We first characterize the assets, adversaries, trust assumptions, and coordination patterns that define DML/FL security boundaries. We then analyze how statistical, system, and trust heterogeneity, including non-IID data, resource asymmetry, intermittent participation, and cross-silo/cross-device trust differences, affect convergence, fairness, attack detectability, and defense reliability. Next, we synthesize attack-defense interactions across poisoning, backdoors, Byzantine manipulation, privacy leakage, model extraction, network/orchestration attacks, Sybil behavior, and strategic participation. We further compare layered defenses, including robust aggregation, drift-aware optimization, differential privacy, secure aggregation, homomorphic encryption, secure multiparty computation, trusted execution environments, watermarking, provenance, authenticated communication, and monitoring. Finally, we discuss deployment, evaluation, governance, and open research guidelines, emphasizing reproducible, non-independent and identically distributed (non-IID) partitions, attack-aware benchmarks, privacy budgets, attack success rate, client-level disparity, latency, communication, memory, energy/carbon overhead, and auditable lifecycle artifacts.
Figures & tables
| Survey | Year | DML / FL | Threat / Trust Model | Heterogeneity | Attack-Defense | Privacy / Protected Comp. | Deploy. / Eval. / Gov. | Main Emphasis |
| Verbraaken et al. [ 1 ] | 2020 | ✓ | ✗ | ✗ | ✗ | Broad distributed deep learning systems survey focused mainly on scalability, orchestration, and infrastructure rather than security. | ||
| Li et al. [ 3 ] | 2020 | ✓ | ✓ | Foundational FL survey covering architectures, statistical heterogeneity, communication efficiency, optimization, and privacy-related challenges. | ||||
| Yang et al. [ 6 ] | 2020 | ✓ | ✓ | Early FL survey addressing concepts, communication settings, application domains, and adversarial considerations. | ||||
| Kairouz et al. [ 4 ] | 2021 | ✓ | ✓ | Landmark FL survey spanning optimization, personalization, privacy, fairness, robustness, and open problems, but not centered on a security-systems roadmap. | ||||
| Rodríguez-Barroso et al. [ 7 ] | 2023 | ✓ | ✓ | ✓ | Comprehensive FL threats survey with attack/defense taxonomies and empirical analysis across privacy and integrity threats. | |||
| Zhang et al. [ 8 ] | 2024 | ✓ | ✓ | Trustworthy FL survey covering privacy, security, fairness, explainability, and accountability under a broad trustworthiness lens. |
TABLE I : Comparison of representative surveys related to secure distributed and federated machine learning. ✓ indicates substantial coverage, indicates partial or secondary coverage, and ✗ indicates limited or no meaningful coverage. The comparison reflects relative emphasis based on survey scope rather than an exhaustive claim about all topics mentioned in each paper
Fig. 1 : Roadmap for heterogeneity-aware trustworthy DML in cloud-edge-IoT systems. Statistical non-IID heterogeneity can exacerbate poisoning, Byzantine manipulation, privacy leakage, model extraction, network tampering, and Sybil/free-rider behavior. These risks motivate a layered defense portfolio spanning heterogeneity-aware optimization, robust and secure aggregation, differential privacy, protected computation, intellectual property (IP) protection, and client vetting. Evaluation should report utility, attack success rate, client-level fairness, privacy budgets, communication/energy overhead, standardized non-IID partitions, public attack suites, and governance artifacts.
Fig. 2 : Effect of heterogeneity on Byzantine-robust aggregation. Robust rules such as median, trimmed mean, Krum, and geometric median are more reliable when benign updates are concentrated; under non-IID data, honest dispersion can reduce attack detectability. Preprocessing methods such as nearest-neighbor mixing and bucketing can partially mitigate this effect. Data points are from [ 6 ] , implemented using [ 46 ] .
| Attack class | Threat assumptions and adversary capability | Goal and affected stage | Primary defenses | Defense limitations and tensions |
| Data/model poisoning and backdoors | Malicious clients or data owners can corrupt local data, labels, loss functions, or submitted updates; attackers may be single-client, colluding, or adaptive. | Availability degradation or targeted integrity failure during local training, update submission, and aggregation. | Robust aggregation, clipping, anomaly scoring, backdoor validation, client vetting, provenance. | Robust rules often assume concentrated benign updates; severe non-IID data can make honest minority updates appear anomalous. Secure aggregation can hide update-level signals needed for anomaly detection. DP noise and clipping may mask weak malicious signals or reduce the utility of rare classes. |
| Byzantine manipulation | A bounded or unbounded fraction of clients may submit arbitrary, coordinated, or optimization-aware updates independent of any local dataset. | Aggregation-time integrity and convergence disruption. | Median/trimmed mean, Krum, geometric median/RFA, sign-based aggregation, norm bounding. | Guarantees weaken under collusion, high dimensionality, adaptive attacks, client drift, and heterogeneous benign updates. Cryptographic confidentiality does not ensure the semantic correctness of encrypted updates. |
| Gradient leakage, membership inference, and property inference An honest-but-curious server, compromised aggregator, eavesdropper, or inference API user observes gradients, updates, model snapshots, or outputs. | Confidentiality breach during training, aggregation, or inference. | DP-SGD/PATE, secure aggregation, HE/MPC, TEEs, output control, privacy accounting. | Secure aggregation hides individual updates but not necessarily leakage from the final model. DP provides formal privacy but may reduce accuracy, fairness, and robustness, especially for small or non-IID clients. HE/MPC/TEEs add latency, memory, communication, energy, or hardware-trust costs. | |
| Model extraction and IP theft | Querying adversary, malicious tenant, or insider can repeatedly access prediction APIs or deployed artifacts. | Model/IP theft during inference and deployment. | Rate limiting, confidence masking, query monitoring, watermarking, fingerprinting, and provenance. | Output restriction can reduce usability and calibration. Watermarking/fingerprinting supports ownership verification but does not prevent poisoning or privacy leakage and may be weakened by fine-tuning, pruning, or model distillation. |
| Network and orchestration attacks | On-path adversaries, compromised gateways, or orchestration faults can replay, delay, downgrade, substitute, or tamper with updates and control messages. | Confidentiality, integrity, and availability failures during communication, scheduling, and synchronization. | Authenticated transport, update signing, replay protection, attestation, secure aggregation, monitoring. | Channel security cannot stop malicious authenticated clients. Attestation and signing increase key management complexity and may not address denial-of-service attacks, stragglers, or stale-update attacks. |
| Sybil, free-rider, and game-theoretic attacks | Open or weakly vetted clients create multiple identities, submit low-effort updates, or strategically adapt to rewards, sampling, and detection rules. | Bias aggregation, manipulate incentives, or obtain benefits without useful contribution during admission, sampling, and training. | Identity management, capped sampling, reputation, contribution auditing, incentive design, anomaly monitoring. | Privacy-preserving aggregation limits direct auditing of contributions. Reputation systems may be gamed, and strong identity controls can be costly or impractical in cross-device IoT settings. |
TABLE II : Analytical comparison of major DML/FL attack classes and defense-composition limits in cloud-edge-IoT systems
| Phase | Mechanism | Reference |
| Local training | Drift-aware optimization | [ 3 , 38 , 70 , 71 , 35 ] , [ 72 , 73 , 74 ] |
| Differential privacy | [ 75 , 76 ] | |
| Cryptographic protection | [ 77 , 78 , 79 ] , [ 80 , 81 ] , [ 82 ] | |
| Before aggregation | Secure aggregation | [ 61 ] |
| During the aggregation | Robust aggregation | [ 53 , 54 , 40 , 55 ] |
| After aggregation | Watermarking | [ 83 , 84 ] |
TABLE III : Defense mechanisms in IoT-oriented DML
| Attack | Test accuracy reported in [ 41 ] : IID and non-IID ( ), 25% malicious clients | Test error reported in [ 88 ] : non-IID ( ), varying attacker counts |
| Inner Product Manipulation (IPM) | Under IPM with , CC achieves 77.77% test accuracy with IID data and 56.97% with non-IID data. Under the stronger IPM attack with , its test accuracy falls to 10.00% in both settings. | CC maintains a test error of approximately 60% as the number of IPM attackers increases from 3 to 18. |
| Sign Flipping (SF) | Under SF, CC achieves only 10.00% test accuracy with both IID and non-IID data. | CC reaches a test error of approximately 90% when the number of SF attackers exceeds 9. |
| Label Flipping (LF) | Under LF, CC achieves 69.80% test accuracy with IID data and 52.41% with non-IID data. | CC reaches a test error of approximately 90% when the number of LF attackers exceeds 12. |
| A Little Is Enough (ALIE) | Under ALIE, CC achieves 78.53% test accuracy with IID data and 59.29% with non-IID data. | CC reaches a test error of approximately 90% when the number of ALIE attackers exceeds 12. |
TABLE IV : Centered Clipping (CC) results reported by two independent studies on CIFAR-10 with FedAvg. The left column summarizes test accuracy from [ 41 ] using a fixed malicious-client fraction of 25% under IID and non-IID data (Dirichlet for the non-IID partition). The right column summarizes test error from [ 88 ] for non-IID data (Dirichlet, ) as the number of attackers varies. Because the studies use different data partitions, attacker configurations, and evaluation metrics, their numerical results are not directly comparable
| IoT domain / setting | Use cases | DML pattern / topology | IoT-specific constraints and threats | Recommended defenses / practices | Metrics to report |
| Industrial sensing / IIoT | Predictive maintenance; quality inspection; adaptive scheduling | Cross-silo FL; plant gateway-edge/cloud hybrid; periodic or event-driven synchronization | Plant/site skew, heterogeneous sensors, legacy devices, intermittent links, constrained gateways, physical tampering, gateway compromise, data locality, and operational safety | Authenticated transport, device identity management, gateway-level anomaly screening, non-IID-aware aggregation, robust aggregation, provenance, drift monitoring, and fail-safe rollback policies [ 93 , 94 , 95 , 96 , 97 ] | OEE/throughput, defect rate, downtime, false alarms, ASR, communication, latency, memory, energy, and fairness across lines/sites |
| Vehicular IoT | Cooperative perception; fleet learning; controller/model updates | Cross-device FL; hierarchical vehicle-RSU-edge/cloud learning | High mobility, spatiotemporal skew, intermittent V2X links, stale updates, strict tail latency, physical/environmental interference, and safety-critical poisoning/backdoors | Hierarchical aggregation, stale-update control, signed updates, distribution-aware robust aggregation, poisoning/backdoor stress tests, edge placement for latency-sensitive tasks, and local fallback policies [ 98 , 99 , 3 , 6 ] | mAP or task utility, tail/edge latency, stale-update rate, link dropout, ASR, backdoor persistence, client-drift metrics, and communication/energy cost |
| Smart healthcare and wearables | Wearable monitoring; collaborative diagnostics; outcome prediction; imaging | Cross-device wearable FL; cross-silo hospital FL; edge/cloud orchestration | PHI sensitivity, patient/site-level non-IID data, cohort shift, label imbalance, battery limits, small memory, intermittent participation, sensor drift, missing measurements, and device loss or compromise | Lightweight local training, dropout-tolerant secure aggregation, DP - SGD/PATE with explicit privacy accounting, selected HE/MPC or TEE support for sensitive operations, gateway buffering, attestation for managed devices, and fairness auditing across patients/sites [ 100 , 101 , 3 , 99 ] | AUROC/AUPRC, DP , per-site or per-patient disparity, dropout rate, ASR, latency, memory, communication, and battery/energy overhead |
| Smart cities / CPS | Traffic control; energy response; environmental monitoring; anomaly detection | Cross-domain FL across agencies/utilities; edge federation across cameras, meters, roadside units, and gateways | Device heterogeneity, district-level domain shift, heterogeneous ownership, PII exposure, public-service reliability, physical-layer disruption, and unreliable clients/gateways | Cross-domain governance, secure aggregation, DP when PII is present, personalization/FedBN or clustered FL for spatial skew, authenticated communication, audit logs, and resilience testing under device, gateway, and link failures [ 102 , 103 , 104 , 105 , 99 ] | Service reliability, availability, travel-time or response utility, privacy budget, ASR, fairness by district/site, communication overhead, latency, and energy |
| IoT cybersecurity | DDoS/IDS; malware detection; log and telemetry analytics | Cross-tenant or cross-organization FL; streaming edge telemetry; gateway/cloud aggregation | Highly non-IID traffic, adaptive adversaries, poisoned telemetry, compromised clients/gateways, tenant isolation, model/IP exposure, and evolving attacks | Byzantine-robust aggregation, clipping/norm bounds, secure aggregation where compatible, attack-aware benchmarks, provenance, authenticated telemetry, query/interface monitoring, and model watermarking/fingerprinting when IP risk is high [ 6 , 7 , 106 , 107 ] | TPR@FPR, false positives, ASR/backdoor persistence, detection latency, communication, memory, energy, and model-extraction risk |
| Cloud-edge IoT services | Managed IoT training/inference; federated MLOps; edge analytics | Device/gateway streams with cloud-hosted training, inference, monitoring, and rollback | Edge-cloud placement, multi-tenancy, access control, model extraction, orchestration faults, data residency, cost, and energy/carbon constraints | Access control, authenticated APIs, rate limiting, output control, provenance, deployment gates, model rollback, explicit placement of filtering/aggregation/privacy accounting, and energy/carbon-aware scheduling [ 108 , 106 , 109 , 110 ] | Cost/hour, time-to-accuracy, SLO adherence, tail latency, extraction risk, memory, energy, kgCO 2 e, and rollback frequency |
TABLE V : IoT-centered DML deployment settings, constraints, design guidelines, and reporting metrics. Abbrev.: DP = differential privacy; DP-SGD = differentially private stochastic gradient descent; HE = homomorphic encryption; MPC = secure multiparty computation; TEE = trusted execution environment; RSU = roadside unit; ASR = attack success rate; SLO = service-level objective; OEE = overall equipment effectiveness; PHI = protected health information; AUROC = area under the receiver operating characteristic curve; AUPRC = area under the precision-recall curve; DDoS = distributed denial of service; IDS = intrusion detection system; TPR = true-positive rate; FPR = false-positive rate
| Ref. | Non-IID data partitioning | Supported attacks and defenses | Reported metrics |
| [ 42 ] | Federated image datasets (FEMNIST and CelebA), text datasets (Sentiment140, Shakespeare, and Reddit), and a synthetic tabular dataset. | No support. | Statistical metrics: device-level test accuracy, training loss, and number of samples per user. System metrics: total computation across devices, measured in FLOPs, and client-server network traffic, measured in bytes written. |
| [ 135 ] | Label-distribution skew based on fixed-sample and Dirichlet partitioning; feature-distribution skew based on Gaussian and cube-based sampling; quantity skew based on a Dirichlet distribution; and FEMNIST. | No support. | Test accuracy, computation time (seconds), and communication volume (MB). |
| [ 121 ] | Twenty real-world FL datasets spanning different scales and task categories, together with a client system-behavior dataset. | Limited support. | Accuracy or loss, rounds to target accuracy, time to target accuracy, power (W), energy (J), latency (seconds), wall-clock time, communication cost, and device-availability duration. |
| [ 46 ] | Dirichlet- and gamma-similarity-based distributions. | Attacks: six Byzantine attacks. Defenses: four pre-aggregation methods and ten Byzantine-robust aggregation methods. | Test accuracy, training loss, and worst-case accuracy. |
| [ 134 ] , [ 138 ] | Dirichlet partitioning; LEAF datasets (FEMNIST and Shakespeare); and PubMedQA. | Attacks: four model-poisoning, two data-poisoning, and three data-reconstruction methods. Defenses: twelve pre-aggregation, three aggregation-stage, and two post-aggregation methods. | Test accuracy and test loss. |
| [ 133 ] | Dirichlet partitioning and FEMNIST. | Attacks: eight model-poisoning, two data-poisoning, and five hybrid-poisoning methods. Defenses: six methods against data poisoning and fifteen methods against model poisoning. | Test accuracy, attack success rate (ASR), targeted attack impact, and targeted defense robustness. |
TABLE VI : Comparison of open-source federated learning benchmark tools