A Momentum-Based Variance-Reduced Algorithm for Federated Multiobjective Optimization
Authors: Yong Zhao, Chunlin You, Minh N. Dao, Zai-Yun Peng
Organizations: College of Mathematics and Statistics, Chongqing Jiaotong University, Chongqing 400074, China. · Key Laboratory of Complex Systems Optimization and Intelligent Control of Chongqing Municipal Education Commission, Chongqing 400074, China. · School of Science, RMIT University, Melbourne, VIC 3000, Australia. · School of Mathematics, Yunnan Normal University, 650092 Kunming, Yunnan, China. · Yunnan Key Laboratory of Modern Analytical Mathematics and Applications, Kunming, 650500, People’s Republic of China.
Federated learning has traditionally been formulated as a single-objective optimization problem, primarily focused on maximizing model utility. In real-world applications, however, machine learning models often need to optimize multiple and potentially conflicting objectives simultaneously. This motivates federated multiobjective optimization (FMOO), which provides a natural framework for jointly handling multiple task-specific objectives in federated learning. In this paper, we propose a momentum-based variance-reduced algorithm for federated multiobjective optimization. The method incorporates a momentum-driven gradient estimator into the local updates to reduce the variance of stochastic updates, leading to an improved convergence rate. We establish theoretical guarantees showing that the expected Pareto stationarity measure of a randomly selected output iterate decays at a rate of O(T−2/3), improving upon the O(T−1/2) rates established for existing methods such as FSMGDA and FedCMOO. Numerical experiments on federated multiobjective optimization benchmarks demonstrate the effectiveness and competitive performance of the proposed algorithm.
Figures & tables
minλst≥0∑s∈[S]λstgst2,s.t.∑s∈[S]λst=1
Algorithm 1 FSMGDA-M-VR
Figure 1: Mean test accuracy with MultiMNIST, MNIST+FMNIST and CIFAR10+MNIST datasets.
Figure 2: Mean test loss in MultiMNIST, MNIST+FMNIST and CIFAR10+MNIST datasets.
Experiment
MNIST+FMNIST
MultiMNIST
CIFAR10+MNIST
FSMGDA
93.2 ± 1.1 / 73.4 ± 0.9
91.1 ± 0.6 / 87.1 ± 0.9
54.2 ± 0.7 / 93.3 ± 1.1
FedCMOO
94.9 ± 1.0 / 77.5 ± 0.8
94.3 ± 0.7 / 91.1 ± 0.8
55.2 ± 0.6 / 93.7 ± 1.0
FSMGDA-M-VR
96.4 ± 0.5 / 76.0 ± 0.4
94.5 ± 0.3 / 91.1 ± 0.3
57.2 ± 0.3 / 96.0 ± 0.5
Table 1: The final accuracies (%) of the first/second objectives in 2-objective settings. The bold values indicate the best accuracy for each objective.
Figure 3: Test accuracy versus cumulative samples accessed per client on the MultiMNIST, MNIST+FMNIST, and CIFAR10+MNIST datasets.
Figure 4: Test loss versus cumulative samples accessed per client on the MultiMNIST, MNIST+FMNIST, and CIFAR10+MNIST datasets.
Figure 5: Comparison of average local training progress (loss ↓ and accuracy ↑) across clients among FSMGDA, FedCMOO, and FSMGDA-M-VR.
Figure 6: Mean test accuracy of different batch sizes on MultiMNIST, MNIST+FMNIST and CIFAR10+MNIST datasets.
Figure 7: Mean test loss of different batch sizes on MultiMNIST, MNIST+FMNIST and CIFAR10+MNIST datasets.
Figure 8: Mean test accuracy of different client numbers on MultiMNIST, MNIST+FMNIST and CIFAR10+MNIST datasets.
Figure 9: Mean test loss of different client numbers on MultiMNIST, MNIST+FMNIST and CIFAR10+MNIST datasets.
Figure 10: Mean test accuracy for different numbers of local steps K .
Figure 11: Mean test loss for different numbers of local steps K .
Task-wise federated multi-objective optimization (FedMOO) trains a shared model for competing prediction objectives under heterogeneous data, partial participation, and communication constraints. Existing methods commonly derive task weights from gradient or update geometry. This requires task-specific information or iterative server-side optimization. We introduce FedHV, which maps reference-relative objective slacks to closed-form inverse-slack weights. Each client optimizes one weighted loss and returns objective estimates with its model update. The protocol adds exactly 2m auxiliary scalars per participating client, yielding Theta(d + m) total per-client communication, compared with the Theta(md) task-specific communication of FSMGDA, and requires no additional synchronization stage. We analyze the resulting one-round-delayed weights under client heterogeneity, multi-step local updates, partial participation, and finite-sample objective reports. Under a fixed-horizon positive-slack reference condition, with the prescribed horizon-dependent step size and vanishing report error, FedHV achieves an O(T^(-1/2)) rate for the average squared log-hypervolume gradient norm; persistent report error determines the resulting stationarity neighborhood. The same bound controls the squared Pareto-stationarity residual. Across six Dirichlet-partitioned non-IID settings from four vision benchmark families and three training seeds, FedHV exceeds FSMGDA and FedCMOO in mean accuracy in five settings. Among these methods and uniform scalarization, it achieves the highest worst-task accuracy in four settings and improves the difficult CIFAR-10 objective in both CIFAR10-MNIST settings.
Amirardalan Dehghanpour, Seyed Mohammad Azimi-Abarghouyi, Christopher G. Brinton
Independent Researcher · Chalmers University of Technology · Purdue University
Many machine learning problems involve multiple inherent trade-offs that are best addressed by gradient-based multi-objective optimization (MOO) algorithms. Existing methods are often proposed with various motivations, analyzed case by case, and differ algorithmically in how the component gradients are aggregated at each step. In this work, we develop a unifying framework for gradient aggregation in MOO, establishing (optimal) rates of convergence to Pareto stationarity, the standard measure of performance in MOO. Central to our analysis is a sufficient alignment condition, from which we derive a theorem showing that non-conflicting directions, when chosen within the convex hull of gradients, form a fundamental sufficient condition for convergence. We further show that feasibility can be ensured through projection onto the dual cone, broadening the scope of methods that admit convergence guarantees. In parallel, we present a primal optimization perspective of gradient aggregation that encompasses established algorithms, clarifies their theoretical relationships, and enables the design of new variants. As an illustration, we introduce capped MGDA, derived from a CVaR-based formulation, and demonstrate its robustness in adversarial federated learning. Finally, we validate our theory through experiments on synthetic problems and practical benchmarks.
Zeou Hu, Kelvin Ho, Yaoliang Yu
1Cheriton School of Computer Science, University of Waterloo, Waterloo, ON, Canada · 3Vector Institute · 2Computer Science, The Chinese University of Hong Kong
Federated stochastic bilevel optimization has been actively studied in recent years due to its widespread applications in machine learning. However, most existing federated stochastic bilevel optimization algorithms require the computation of second-order Hessian and Jacobian matrices, which leads to longer running times in practice. To address these challenges, we propose a novel federated stochastic variance-reduced bilevel gradient descent algorithm that relies solely on first-order oracles. Specifically, our approach does not require the computation of second-order Hessian and Jacobian matrices, significantly reducing running time. Furthermore, we introduce a novel learning rate mechanism, i.e., a constant single-timescale learning rate, to coordinate the update of different variables. We also present a new strategy to establish the convergence rate of our algorithm. Finally, the extensive experimental results confirm the efficacy of our proposed algorithm.