cs.LGSep 24, 2024

A Historical Trajectory Assisted Optimization Method for Zeroth-Order Federated Learning

Authors: Yu Cai, Canxi Huang, Zike Li, Jing Gong, Xiaoyu He

Organizations: School of Software Engineering, Sun Yat-sen University, Zhuhai 519082, China · School of Computing and Information Systems, The University of Melbourne, Melbourne, VIC, Australia

Abstract

Federated learning heavily relies on distributed gradient descent techniques. In the situation where gradient information is not available, the gradients need to be estimated from zeroth-order information, which typically involves computing finite-differences along isotropic random directions. This method suffers from high estimation errors, as the geometric features of the objective landscape may be overlooked during the isotropic sampling. In this work, we propose a non-isotropic sampling method to improve zeroth-order gradient estimation. The method uses historical solution trajectories to identify a promising subspace and constructs a non-isotropic Gaussian smoothing that balances exploitation in this subspace with global exploration. We implement this method in zeroth-order federated settings, and show that the convergence rate aligns with existing ones while introducing no significant overheads in communication or local computation. The effectiveness of our proposal is verified on several numerical experiments in comparison to several commonly-used zeroth-order federated optimization algorithms. The experiments cover both independent and identically distributed data and heterogeneous client data distributions.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Efficient Exact Unlearning for Federated Zeroth-Order Optimization

    Oct 26, 2026Weijun Yin, Yanqing Yao, Falko Dressler +2Federated UnlearningFederated Learning

  2. FedZMG: Efficient Client-Side Optimization in Federated Learning

    Feb 20, 2026Fotios Zantalis, Evangelos Zervas, Grigorios KoulourasFederated LearningNon-IID Federated Learning

  3. Taming the Instability: A Robust Second-Order Optimizer for Federated Learning over Non-IID Data

    Mar 30, 2026Yuanqiao Zhang, Tiantian He, Yuan Gao +5Federated LearningSecond-Order Optimization