A Momentum-Based Variance-Reduced Algorithm for Federated Multiobjective Optimization
Authors: Yong Zhao, Chunlin You, Minh N. Dao, Zai-Yun Peng
Organizations: College of Mathematics and Statistics, Chongqing Jiaotong University, Chongqing 400074, China. · Key Laboratory of Complex Systems Optimization and Intelligent Control of Chongqing Municipal Education Commission, Chongqing 400074, China. · School of Science, RMIT University, Melbourne, VIC 3000, Australia. · School of Mathematics, Yunnan Normal University, 650092 Kunming, Yunnan, China. · Yunnan Key Laboratory of Modern Analytical Mathematics and Applications, Kunming, 650500, People’s Republic of China.
Federated learning has traditionally been formulated as a single-objective optimization problem, primarily focused on maximizing model utility. In real-world applications, however, machine learning models often need to optimize multiple and potentially conflicting objectives simultaneously. This motivates federated multiobjective optimization (FMOO), which provides a natural framework for jointly handling multiple task-specific objectives in federated learning. In this paper, we propose a momentum-based variance-reduced algorithm for federated multiobjective optimization. The method incorporates a momentum-driven gradient estimator into the local updates to reduce the variance of stochastic updates, leading to an improved convergence rate. We establish theoretical guarantees showing that the expected Pareto stationarity measure of a randomly selected output iterate decays at a rate of O(T−2/3), improving upon the O(T−1/2) rates established for existing methods such as FSMGDA and FedCMOO. Numerical experiments on federated multiobjective optimization benchmarks demonstrate the effectiveness and competitive performance of the proposed algorithm.
Figures & tables
minλst≥0∑s∈[S]λstgst2,s.t.∑s∈[S]λst=1
Algorithm 1 FSMGDA-M-VR
Figure 1: Mean test accuracy with MultiMNIST, MNIST+FMNIST and CIFAR10+MNIST datasets.
Figure 2: Mean test loss in MultiMNIST, MNIST+FMNIST and CIFAR10+MNIST datasets.
Experiment
MNIST+FMNIST
MultiMNIST
CIFAR10+MNIST
FSMGDA
93.2 ± 1.1 / 73.4 ± 0.9
91.1 ± 0.6 / 87.1 ± 0.9
54.2 ± 0.7 / 93.3 ± 1.1
FedCMOO
94.9 ± 1.0 / 77.5 ± 0.8
94.3 ± 0.7 / 91.1 ± 0.8
55.2 ± 0.6 / 93.7 ± 1.0
FSMGDA-M-VR
96.4 ± 0.5 / 76.0 ± 0.4
94.5 ± 0.3 / 91.1 ± 0.3
57.2 ± 0.3 / 96.0 ± 0.5
Table 1: The final accuracies (%) of the first/second objectives in 2-objective settings. The bold values indicate the best accuracy for each objective.
Figure 3: Test accuracy versus cumulative samples accessed per client on the MultiMNIST, MNIST+FMNIST, and CIFAR10+MNIST datasets.
Figure 4: Test loss versus cumulative samples accessed per client on the MultiMNIST, MNIST+FMNIST, and CIFAR10+MNIST datasets.
Figure 5: Comparison of average local training progress (loss ↓ and accuracy ↑) across clients among FSMGDA, FedCMOO, and FSMGDA-M-VR.
Figure 6: Mean test accuracy of different batch sizes on MultiMNIST, MNIST+FMNIST and CIFAR10+MNIST datasets.
Figure 7: Mean test loss of different batch sizes on MultiMNIST, MNIST+FMNIST and CIFAR10+MNIST datasets.
Figure 8: Mean test accuracy of different client numbers on MultiMNIST, MNIST+FMNIST and CIFAR10+MNIST datasets.
Figure 9: Mean test loss of different client numbers on MultiMNIST, MNIST+FMNIST and CIFAR10+MNIST datasets.
Figure 10: Mean test accuracy for different numbers of local steps K .
Figure 11: Mean test loss for different numbers of local steps K .
1Cheriton School of Computer Science, University of Waterloo, Waterloo, ON, Canada · 3Vector Institute · 2Computer Science, The Chinese University of Hong Kong