Communication-Efficient Distributed Learning with Differential Privacy

Xiaoxing Ren; Yuwen Ma; Nicola Bastianello; Karl H. Johansson; Thomas Parisini; Andreas A. Malikopoulos

cs.LGApr 2, 2026

Communication-Efficient Distributed Learning with Differential Privacy

Authors: Xiaoxing Ren, Yuwen Ma, Nicola Bastianello, Karl H. Johansson, Thomas Parisini, Andreas A. Malikopoulos

Organizations: School of Civil and Environmental Engineering, Cornell University, Ithaca, New York, United States. · Department of Electronic and Electrical Engineering, University College London, London WC1E 7JE, United Kingdom. · School of Electrical Engineering and Computer Science, and Digital Futures, KTH Royal Institute of Technology, Stockholm, Sweden. · Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom ·

Abstract

We address nonconvex learning problems over undirected networks. In particular, we focus on the challenge of designing an algorithm that is both communication-efficient and that guarantees the privacy of the agents' data. The first goal is achieved through a local training approach, which reduces communication frequency. The second goal is achieved by perturbing gradients during local training, specifically through gradient clipping and additive noise. We prove that the resulting algorithm converges to a stationary point of the problem within a bounded distance. Additionally, we provide theoretical privacy guarantees within a differential privacy framework that ensure agents' training data cannot be inferred from the trained model shared over the network. We show the algorithm's superior performance on a classification task under the same privacy budget, compared with state-of-the-art methods.

Figures & tables

Algorithm [Ref.]	Time
PORTER [ 21 ]	$\tau(t_{g}+2t_{c})$
PriSMA [ 22 ]	$\tau(2t_{g}+t_{c})$
LT-ADMM-DP	$\tau t_{g}+t_{c}$

TABLE I : Computation time over

\tau

iterations.

Panel of Figure 2 — Fig. 1 : Numerical results for three algorithms with differential privacy.

Communication-Efficient Distributed Training

Communication-efficient distributed training reduces the data exchanged among devices so models can be trained faster and over limited-bandwidth networks.

See all 227 papers

DPDL: Towards Differential Privacy Preservation in Decentralized Stochastic Learning on Non-IID Data
Jun 3, 2026Yunsheng Yuan, Xue Xiao, Lina Wang +1Decentralized Learning Privacy-Preserving ML
School of Computer Science and Technology, Shandong University, Qingdao 266237, China · Inspur Cloud Information Technology Co, Ltd.
Privacy Preserving Gossip Learning
Sep 13, 2026Erkan Bayram, Mohamed-Ali Belabbas, Tamer BaşarContinual Learning Decentralized Learning
University of Illinois Urbana Champaign, IL, USA
A Unified Framework for Fair and Personalized Decentralized Learning under Communication Constraints
Aug 27, 2026Krishnendu S. Tharakan, Carlo FischioneDecentralized Learning Communication-Efficient Distributed Training
School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, Stockholm, Sweden

Jun 3, 2026J/K move · Enter open · S save

DPDL: Towards Differential Privacy Preservation in Decentralized Stochastic Learning on Non-IID Data

Yunsheng Yuan, Xue Xiao, Lina Wang, Feng Li

School of Computer Science and Technology, Shandong University, Qingdao 266237, China · Inspur Cloud Information Technology Co, Ltd.

Decentralized Learning Privacy-Preserving ML Differentially Private Federated Learning Differential Privacy

Open paper AskPDF arXiv

In the paradigm of decentralized learning, a group of agents collaborate to train a global model using distributed datasets without a central server. Although the power of collaboration has been verified by many state-of-the-art studies, it entails extensive gradient information exchanging among the agents and thus induces high risk of privacy leakage for the individual agents. Moreover, in real-world applications, the training data are usually non-identically and independently distributed across the agents, inducing more challenges to enable privacy-preserved decentralized learning. To address these issues, we propose a privacy-preserved decentralized learning algorithm with non-IID data, DPDL, which leverages the notion of Differential Privacy (DP) in cross-gradient aggregation through a similarity-based calibration technique. Specifically, in each round, each agent perturbs the cross-gradients (i.e., the derivatives of its neighbors' local model in its private local data) by Gaussian noise mechanism before sharing them with its neighbors; it then adopt cosine similarity to calibrate the received perturbed cross-gradients such that the aggregation of the calibrated cross-gradients can be utilized to effectively update local model in a momentum-like manner. Our rigorous theoretical analysis not only reveals the minimum noise level required to achieve a specific level of privacy preservation, but also illustrates that our algorithm still achieves a linear speedup in training with non-IID data. We finally conduct extensive experiments on real-world dataset to validate the effectiveness of our algorithm in defending privacy attacks and in training accurate models.

Abstract

Communication-Efficient Distributed Training

Explore similar work

DPDL: Towards Differential Privacy Preservation in Decentralized Stochastic Learning on Non-IID Data

Privacy Preserving Gossip Learning

A Unified Framework for Fair and Personalized Decentralized Learning under Communication Constraints