stat.MLNov 3, 2026

One-shot Distributed Learning via Weighted Averaging with Server-side Information

Authors: Pavle Tepavčević, Dušan Jakovetić, Miloš Savić

Organizations: Department of Mathematics and Informatics, Faculty of Sciences, University of Novi Sad, Serbia

Abstract

Distributed learning has drawn considerable attention due to its capability of learning from distributed data without aggregating raw observations. However, frequent exchange of model updates can become the main bottleneck of communication. Therefore, one-shot approaches where each worker fits a local model and sends it to the server to aggregate emerge. It is known that averaging can achieve the so-called pooled-data mean squared error (MSE) order of a hypothetical node that has access to data from the server and all the workers, and performs empirical risk minimization (ERM) on it. However, when the number of workers increases too fast compared to the local sample size, nonzero local estimation bias prevents averaging from achieving this order. In this paper, we study whether independent server data can overcome this limitation. We consider mm workers with nn observations each and a server with pp observations, all i.i.d., giving N=mn+pN=mn+p observations in total. Under certain regularity conditions, a pooled-data ERM achieves MSE of order O(N−1)\mathcal O(N^{-1}). We analyze deterministic nonnegative averaging of convex-loss M-estimators, with equal worker weights and the server's own estimator included. Under our assumptions, including nonzero local bias of order n−αn^{-α}, α≥1α\geq1, sample-size weighted averaging achieves the pooled-data MSE order whenever m=O(n2α−1)m=\mathcal O(n^{2α-1}). Above this worker scale, it does so if and only if p=Ω(nm⋅m/n2α−1)p=Ω(nm\cdot m/n^{2α-1}). Hence, server-only estimation, whose MSE is of order O(p−1)\mathcal{O}(p^{-1}), can achieve pooled-data order while sample-size weighting fails. Analytical examples and synthetic experiments illustrate when averaging matches the pooled-data MSE order, complemented by CIFAR-10 experiments comparing aggregation strategies with linear and nonlinear classifiers.

Figures & tables

Explore similar work

Dec 15, 2023

Optimal Transport Aggregation for Distributed Mixture-of-Experts

This paper considers the setting in which local Mixture-of-experts (MoE) models are fitted independently on distributed datasets and must be combined into a single global MoE model. Weighted averaging the local mixtures, whose experts are not aligned across machines, often yields a large mixture with many experts instead of KK and does not provide an estimator of the true model parameters. We propose to reduce this large mixture to a KK-expert MoE by minimizing an expected transportation divergence, in which the transport plan depends on the covariates. A relaxation of this problem separates the experts from the gating network: the experts are computed by a majorization--minimization algorithm, which decreases the objective monotonically, and the gating network is then fitted to the gates implied by the optimal plan, with an explicit bound on the resulting loss. We show that the aggregated estimator is consistent whenever the local estimators are, without requiring the local models to have the same labeling across machines. The procedure only needs a single one-way communication round. Experiments on simulated and real data show accuracy close to centralized estimation at a fraction of its computation time. Our source code is available at https://github.com/nhat-thien/Distributed-Mixture-Of-Experts.
Sep 17, 2026

FedeRage: Provably Convergent Agnostic Federated Learning under General Client Drift

Federated learning (FL) enables collaborative model training without sharing raw data, but its performance degrades under non-IID data and stochastic client participation. Remedies built on classical Federated Averaging (FedAvg) typically presuppose that client participation probabilities are known to the server, which is rarely the case in deployed systems. We first discuss and then characterize the optimization problem that distributionally agnostic FedAvg actually solves when participation is entirely unknown, possibly highly skewed, and of variable size across rounds: uniform aggregation is shown to minimize a well-defined stochastic objective, weighted by the participation-induced marginal, at a standard O(1/T)\mathcal{O}(1/\sqrt{T}) rate for convex and possibly nonsmooth losses. Building on this characterization, we propose Federated Risk-Averse Averaging (FedeRage), a risk-averse extension of FedAvg that embeds the Conditional Value-at-Risk (CVaR) into the local objective within a natural distributionally robust optimization (DRO) framework. FedeRage implicitly upweights high-loss and infrequently participating clients while adding only a single scalar per-client, and admits an O(κ/T)\mathcal{O}(κ/\sqrt{T}) rate in which the factor κκ is the upper bound on the ``price" of risk aversion. In contrast with aggregation-alignment schemes based on optimal transport, which require the availability distribution as an input, FedeRage remains agnostic to it. Several experiments on three heterogeneous benchmarks indicate consistent improvements over state-of-the-art methods in accuracy, fairness, and convergence speed.
Apr 3, 2026

Enhancing Robustness of Federated Learning via Server Learning

This paper explores the use of server learning for enhancing the robustness of federated learning against malicious attacks even when clients' training data are not independent and identically distributed. We propose a heuristic algorithm that uses server learning and client update filtering in combination with geometric median aggregation. We demonstrate via experiments that this approach can achieve significant improvement in model accuracy even when the fraction of malicious clients is high, even more than 50%50\% in some cases, and the dataset utilized by the server is small and could be synthetic with its distribution not necessarily close to that of the clients' aggregated data.