Group Distributionally Robust Optimization over Decentralized Networks

Haomin Bai; Junyan Sun; Quan Cheng; Yuanyu Wan; Lijun Zhang

math.OCOct 21, 2026

Group Distributionally Robust Optimization over Decentralized Networks

Authors: Haomin Bai, Junyan Sun, Quan Cheng, Yuanyu Wan, Lijun Zhang

Organizations: State Key Laboratory of Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · School of Software Technology, Zhejiang University

Abstract

We study group distributionally robust optimization (GDRO) over decentralized networks, where workers collaborate to learn robust models by minimizing the maximum local risk. Existing methods optimize only dual-regularized surrogates and establish guarantees for a globally averaged model that is not available to workers. In this paper, we directly optimize the original GDRO objective and establish guarantees for local models in semi-decentralized (SemiDec) and fully decentralized (Dec) settings. A key difficulty is that applying the primal-dual decomposition from centralized GDRO yields worker-dependent losses whose evaluation requires unavailable cross-worker information. To address this challenge, we develop a novel error decomposition that recasts decentralized GDRO as (i) a decentralized online convex optimization (D-OCO) problem and a prediction-with-expert-advice problem in SemiDec, and as (ii) two coupled D-OCO problems in Dec, with discrepancy terms capturing inconsistencies among local decisions. By isolating consensus errors in analysis, we develop modular frameworks that support different consensus routines. We then establish the first per-worker convergence guarantees for optimizing the original GDRO objective over decentralized networks. Instantiations with accelerated gossip and Choco-gossip achieve an $O(\sqrt{(\log m)/T})$ convergence rate over $T$ update rounds, while experiments demonstrate improved worst-worker performance over baselines.

Figures & tables

Figure 1: Worst-worker accuracy over update rounds on Adult, MNIST, and CIFAR-10. The top and bottom rows correspond to SemiDec and Dec, respectively. Curves report the mean over five random seeds, with shaded regions showing one standard deviation.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Worst-worker loss over update rounds on Adult, MNIST, and CIFAR-10. The top and bottom rows correspond to SemiDec and Dec, respectively. Parenthesized numbers denote the consensus rounds per update.

Group Distributionally Robust Optimization

See all 11 papers

Jun 3, 2026

Near-Optimal Decentralized Stochastic Convex Optimization over Networks

We study decentralized stochastic smooth convex optimization, where

M

workers minimize an average objective using local stochastic gradients and neighbor-only communication over a fixed gossip network. A central question in this setting is to determine the largest number of workers that can be used under a total budget of

N

gradient samples while still preserving the centralized

O(1/\sqrt N)

statistical rate. We introduce an accelerated decentralized method that preserves this rate for up to

\smash{M\lesssim \sqrtρ\,N^{3/4}}

workers, where

ρ

is the spectral gap of the gossip network, improving the best prior maximal scaling of

\smash{M\lesssim ρ\sqrt N}

. The method is based on a one-step-delayed stochastic acceleration scheme that enables workers to interleave minibatching with accelerated gossip while controlling residual disagreement, and its guarantee depends only logarithmically on the optimum-local heterogeneity. We also establish a matching lower bound for linear-span decentralized first-order methods, showing that the method is optimal up to logarithmic factors.

Nitai Kluger, Amit Attia, Tomer Koren

Blavatnik School of Computer Science, Tel Aviv University · Google Research Tel Aviv

Stochastic Optimization Convergence Accelerated Gradient Methods Decentralized Optimization

Ask PDF

Jun 5, 2026

Accelerated Decentralized Stochastic Gradient Descent for Strongly Convex Optimization

Decentralized stochastic optimization is a fundamental paradigm for large-scale learning over networks, where agents communicate only with their neighbors and no central coordinator is required. For strongly convex problems, communication efficiency is mainly determined by the condition number

κ=L/μ

and the network spectral gap

1-β

. Although deterministic decentralized methods can simultaneously achieve accelerated

\sqrtκ

and

1/\sqrt{1-β}

dependences, no existing stochastic method attains both improvements at once. In this paper, we propose Multi-Gossip Accelerated DSGD (MG-ADSGD), a decentralized stochastic algorithm that combines Nesterov-type primal--dual extrapolation with multi-round fast gossip averaging. The key idea is to couple the gossip depth with the mini-batch size so that additional communication rounds simultaneously improve consensus accuracy and reduce gradient variance. We show that MG-ADSGD achieves the communication complexity

\widetilde{\mathcal O}\!\left( \frac{σ^2}{μnε}\log\frac{1}ε + \sqrt{\fracκ{1-β}}\log\frac{1}ε \right),

where

ε

denotes the target accuracy,

n

is the number of nodes, and

σ^2

is the gradient variance. To the best of our knowledge, this bound yields the best currently available communication complexity for decentralized stochastic strongly convex optimization, up to logarithmic factors that are independent of

ε

Ming Sun, Kun Yuan

Center for Machine Learning Research Peking University

Gradient Descent Accelerated Gradient Methods Decentralized Optimization Stochastic Gradient Descent

Ask PDF

Jan 28, 2026

Fast and Efficient Asynchronous Gossip Algorithm for Robust and Non-Smooth Convex Decentralized Learning

Asynchronous primal-dual methods for decentralized non-smooth convex optimization often require each node to maintain

\mathcal{O}(d)

auxiliary variables, where

d

is its degree. This dependence on degree increases memory requirements and can amplify the effects of stale information, especially in dense networks. Motivated by the challenge of frugal memory management in decentralized learning, we introduce Goal-PD, an asynchronous gossip-based primal-dual algorithm that maintains only two variables per node, regardless of the node's degree. We establish almost-sure convergence of Goal-PD to a minimizer of the underlying optimization problem, and prove linear convergence when the objective functions are piecewise linear-quadratic. For decentralized mean estimation, we show that pairwise averaging is a special case of Goal-PD, which establishes a direct link between the proposed primal-dual framework and classical gossip. Experiments on synthetic and real datasets over various network topologies, with non-smooth objectives including median estimation, show that Goal-PD converges faster than existing asynchronous baselines while requiring significantly less memory by design.

Anna van Elst, Olivier Fercoq, Igor Colin +1

LTCI, Télécom Paris, Institut Polytechnique de Paris

Decentralized Learning Nonsmooth Convex Optimization Decentralized Optimization

Ask PDF

Abstract

Group Distributionally Robust Optimization

Explore similar work

Near-Optimal Decentralized Stochastic Convex Optimization over Networks

Accelerated Decentralized Stochastic Gradient Descent for Strongly Convex Optimization

Fast and Efficient Asynchronous Gossip Algorithm for Robust and Non-Smooth Convex Decentralized Learning