Organizations: School of Software Engineering, Sun Yat-sen University, Zhuhai 519082, China · School of Computing and Information Systems, The University of Melbourne, Melbourne, VIC, Australia
Federated learning heavily relies on distributed gradient descent techniques. In the situation where gradient information is not available, the gradients need to be estimated from zeroth-order information, which typically involves computing finite-differences along isotropic random directions. This method suffers from high estimation errors, as the geometric features of the objective landscape may be overlooked during the isotropic sampling. In this work, we propose a non-isotropic sampling method to improve zeroth-order gradient estimation. The method uses historical solution trajectories to identify a promising subspace and constructs a non-isotropic Gaussian smoothing that balances exploitation in this subspace with global exploration. We implement this method in zeroth-order federated settings, and show that the convergence rate aligns with existing ones while introducing no significant overheads in communication or local computation. The effectiveness of our proposal is verified on several numerical experiments in comparison to several commonly-used zeroth-order federated optimization algorithms. The experiments cover both independent and identically distributed data and heterogeneous client data distributions.
Figures & tables
Figure 1: Training loss versus the number of function evaluations obtained in the IID setting
Figure 2: Training loss versus the number of function evaluations obtained in the non-IID setting
Figure 3: Sensitivity analysis of ZOFedHT on logistic regression over the rcv1 dataset. The horizontal axis denotes α ranging from 0.1 to 0.9, the vertical axis denotes τ∈{1,3,5,10,20,50} , and each entry reports the final test loss
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Test loss versus the number of function evaluations obtained in the IID setting
Figure 5: Test loss versus the number of function evaluations obtained in the non-IID setting
Problem
IID
Non-IID
ZOFedAvg-GD
ZOFedAvg-SGD
ZOFedHT
ZOFedProx
ZOFedAvg-GD
ZOFedAvg-SGD
ZOFedHT
ZOFedProx
Fashion-MNIST (SVM)
0.2576
0.2109
0.1552
0.1678
0.2575
0.2124
0.1826
0.2043
Fashion-MNIST (MLP)
0.2307
0.2253
0.2259
0.2233
0.2358
0.2345
0.2340
0.2324
Fashion-MNIST (LR)
0.2467
0.2371
0.2216
0.2378
0.2473
0.2381
0.2340
0.2388
MNIST (SVM)
0.2951
0.2727
0.2326
0.2333
0.2966
0.2737
0.2592
0.2650
MNIST (MLP)
0.3158
0.2895
0.2565
0.2830
0.3525
0.2970
0.2986
0.2968
Appendix
Table 1: Comparison of final test loss under IID and non-IID settings. Lower values indicate better performance
Zeroth-order optimization uses function values or comparisons, rather than gradients, to optimize objectives when differentiation is unavailable or costly.
Federated zeroth-order optimization (FedZOO) reduces the memory and communication costs of collaborative training for edge intelligence by avoiding backpropagation while communicating only random seeds and scalar coefficients. However, existing methods lack efficient mechanisms for exact unlearning to fully remove the influence of training data when users request deletion. A key insight is that total variation stability of the learning algorithm creates overlap between model distributions before and after data deletion, allowing exact unlearning to reuse computation through coupling. Based on this observation, we jointly design two stable zeroth-order learning algorithms and their corresponding exact-unlearning procedures: FedZOO-S obtains stability through client and mini-batch subsampling, enabling FedZOUL-S to perform efficient unlearning by preserving earlier updates and restarting from the first affected step. FedZOO-NS further combines subsampling with Gaussian noise to increase overlap between update distributions; FedZOUL-NS exploits this overlap through rejection--reflection coupling to reuse affected updates when the coupling is accepted. We prove exact unlearning for both algorithms: the model and saved state have the same joint distribution as retraining with the corresponding learner on retained data. We further establish stability and convergence guarantees for the learning algorithms and derive upper bounds on expected unlearning cost. Experiments demonstrate effective unlearning with competitive model utility and up to 56.1% less cumulative unlearning time than re-optimization. Compared with first-order baselines, our methods reduce communication overhead by up to approximately four orders of magnitude.
Weijun Yin, Yanqing Yao, Falko Dressler +2
School of Cyber Science and Technology, Beihang University, Beijing 100191, China · Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University, Beijing 100083, China · School of Electrical Engineering and Computer Science, TU Berlin, 10623 Berlin, Germany +1
Federated Learning (FL) enables distributed model training on edge devices while preserving data privacy. However, clients tend to have non-Independent and Identically Distributed (non-IID) data, which often leads to client-drift, and therefore diminishing convergence speed and model performance. While adaptive optimizers have been proposed to mitigate these effects, they frequently introduce computational complexity or communication overhead unsuitable for resource-constrained IoT environments. This paper introduces Federated Zero Mean Gradients (FedZMG), a novel, parameter-free, client-side optimization algorithm designed to tackle client-drift by structurally regularizing the optimization space. Advancing the idea of Gradient Centralization, FedZMG projects local gradients onto a zero-mean hyperplane, effectively neutralizing the "intensity" or "bias" shifts inherent in heterogeneous data distributions without requiring additional communication or hyperparameter tuning. A theoretical analysis is provided, proving that FedZMG reduces the effective gradient variance and guarantees tighter convergence bounds compared to standard FedAvg. Extensive empirical evaluations on EMNIST, CIFAR100, and Shakespeare datasets demonstrate that FedZMG achieves better convergence speed and final validation accuracy compared to the baseline FedAvg and the adaptive optimizer FedAdam, particularly in highly non-IID settings.
TelSiP Research Laboratory, Department of Electrical and Electronic Engineering, School of Engineering, University of West Attica, Ancient Olive Grove Campus, 250 Thivon Str., GR-12241 Athens, Greece
In this paper, we present Federated Robust Curvature Optimization (FedRCO), a novel second-order optimization framework designed to improve convergence speed and reduce communication cost in Federated Learning systems under statistical heterogeneity. Existing second-order optimization methods are often computationally expensive and numerically unstable in distributed settings. In contrast, FedRCO addresses these challenges by integrating an efficient approximate curvature optimizer with a provable stability mechanism. Specifically, FedRCO incorporates three key components: (1) a Gradient Anomaly Monitor that detects and mitigates exploding gradients in real-time, (2) a Fail-Safe Resilience protocol that resets optimization states upon numerical instability, and (3) a Curvature-Preserving Adaptive Aggregation strategy that safely integrates global knowledge without erasing the local curvature geometry. Theoretical analysis shows that FedRCO can effectively mitigate instability and prevent unbounded updates while preserving optimization efficiency. Extensive experiments show that FedRCO achieves superior robustness against diverse non-IID scenarios while achieving higher accuracy and faster convergence than both state-of-the-art first-order and second-order methods.
Yuanqiao Zhang, Tiantian He, Yuan Gao +5
Key Laboratory of Collaborative Intelligence Systems of Ministry of Education, Xidian University, Xi’an, Shaanxi, China · Centre for Frontier AI Research, Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · State Key Laboratory of ISN, School of Cyber Engineering, Xidian University, Xi’an, Shaanxi, China +2