LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communication

Wei Liu; Anweshit Panda; Ujwal Pandey; Haven Cook; George M. Slota; Naigang Wang; Jie Chen; Yangyang Xu

cs.LGApr 11, 2026

LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communication

Authors: Wei Liu, Anweshit Panda, Ujwal Pandey, Haven Cook, George M. Slota, Naigang Wang, Jie Chen, Yangyang Xu

Organizations: Department of Mathematical Sciences, Rensselaer Polytechnic Institute · Department of Computer Science, Rensselaer Polytechnic Institute · IBM T. J. Watson Research Center · MIT-IBM Watson AI Lab, IBM Research

Abstract

In the decentralized distributed learning, achieving fast convergence and low communication cost is essential for scalability and high efficiency. Adaptive gradient methods, such as Adam, have demonstrated strong practical performance in deep learning and centralized distributed settings. However, their convergence properties remain largely unexplored in decentralized settings involving multiple local training steps, such as federated learning. To address this limitation, we propose LoDAdaC, a unified multiple Local Training (MLT) Decentralized framework with Adam-type updates and Compressed communication (CC). LoDAdaC accommodates a broad class of optimizers for its local adaptive updates, including AMSGrad, Adam, and AdaGrad; it is compatible with standard (possibly biased) compressors such as low-bit quantization and sparsification. MLT and CC enable LoDAdaC to achieve multiplied reduction of communication cost, while the technique of adaptive updates enables fast convergence. We rigorously prove the combined advantage through complexity analysis. In addition, experiments on image classification and GPT-style language model training validate our theoretical findings and show that LoDAdaC significantly outperforms existing decentralized algorithms in terms of convergence speed and communication efficiency.

Figures & tables

MLT Methods	CC	AG	#Iter	CommCost	MLTsave	CCsave
PD-SGD ( Ge and Chang, 2023 )	✗	✗	$\frac{1}{n\epsilon^{4}}$	$\frac{1}{nK\epsilon^{4}}$	✓	$-$
LSGT ( Li et al., 2019 )	✗	✗	$\frac{1}{n\epsilon^{4}}$	$\frac{1}{nK\epsilon^{4}}$	✓	$-$
DFedAvgM ( Sun et al., 2022 )	✓	✗	$\max\{\frac{K}{\epsilon^{4}},\frac{K\epsilon^{4}}{s^{2}}\}^{*}$	$\max\{\frac{1-s}{\epsilon^{4}},\frac{(1-s)\epsilon^{4}}{s^{2}}\}^{*}$	✗	✓
SQuARM-SGD ( Singh et al., 2021 )	✓	✗	$\frac{1}{n\epsilon^{4}}$	$\frac{1-\eta}{nK\epsilon^{4}}$	✓	✓
LM-DFL ( Chen et al., 2024 )	✓	✗	$\max\{\frac{1}{K^{3}\epsilon^{4}},\frac{K^{3}s^{4}}{\epsilon^{4}}\}^{*}$	$\max\{\frac{1-s}{K^{4}\epsilon^{4}},\frac{(1-s)K^{2}s^{4}}{\epsilon^{4}}\}^{*}$	✗	✗
LoDAdaC (this paper)	✓	✓	$\frac{1}{n\epsilon^{4}}$	$\frac{1-\eta}{nK\epsilon^{4}}$	✓	✓

Table 1: Comparison between the proposed method and selected approaches that use MLT for nonconvex decentralized distributed learning. “CC” indicates whether compressed communication is employed; “AG” denotes the use of adaptive gradient updates; “#Iter” specifies the number of total iterations (per agent) to obtain an

\epsilon

-stationary point of problem ( 2 ), see Definition 1.2 ; “CommCost” refers to the total communication cost, where each communication round incurs a unit cost in the absence of compression; “MLTsave” indicates whether the number of communication rounds can be theoretically reduced by employing MLT; and “CCsave” reflects whether the total communication cost can be effectively reduced by compression. Here, the

O(\cdot)

notation is omitted in the table,

\epsilon

is assumed to be sufficiently small, and the number of local steps

K

is at most

O(\epsilon^{-1})

{}^{*}s

refers to the compression error given in ( Sun et al., 2022 ) , satisfying

\mathbb{E}_{\mathcal{Q}}\left[\|{\bm{x}}-\mathcal{Q}[{\bm{x}}]\|^{2}\right]\leq s^{2}d

Optimizer	Description
Vanilla SGD	$\beta_{1}=0$ and $r_{t}\equiv{\bm{0}}$ , i.e., ${\bm{u}}_{i}^{t}={\bm{0}}$ for all $i$ and $t$
Momentum (Heavy-ball) SGD	$\beta_{1}\in(0,1)$ and $r_{t}\equiv{\bm{0}}$ , i.e., ${\bm{u}}_{i}^{t}={\bm{0}}$ for all $i$ and $t$
AMSGrad	$\widehat{{\bm{u}}}_{i}^{t}=\beta_{2}\widehat{{\bm{u}}}_{i}^{t-1}+(1-\beta_{2}){\bm{g}}_{i}^{t}\circ{\bm{g}}_{i}^{t}$ , ${\bm{u}}_{i}^{t}=\max\{{\bm{u}}_{i}^{t-1},\widehat{{\bm{u}}}_{i}^{t}\}$ , $\widehat{{\bm{u}}}_{i}^{-1}={\bm{0}}$ , and $\beta_{2}\in(0,1)$
Adam	${{\bm{u}}}_{i}^{t}=\beta_{2}{{\bm{u}}}_{i}^{t-1}+(1-\beta_{2}){\bm{g}}_{i}^{t}\circ{\bm{g}}_{i}^{t}$ , with $\beta_{2}\in\left[\frac{\sqrt{TK}}{\sqrt{TK}+1},1\right)$
Adam-mini	${{\bm{u}}}_{i}^{t}=\beta_{2}{{\bm{u}}}_{i}^{t-1}+(1-\beta_{2})\mathrm{mean}({\bm{g}}_{i}^{t}\circ{\bm{g}}_{i}^{t})$ , with $\beta_{2}\in\left[\frac{\sqrt{TK}}{\sqrt{TK}+1},1\right)$
Averaged AdaGrad	${{\bm{u}}}_{i}^{t}=\frac{1}{t+1}\sum_{s=0}^{t}{\bm{g}}_{i}^{s}\circ{\bm{g}}_{i}^{s}$

Table 2: Representative optimizers of Algorithm 1 with specific selections of

\beta_{1}

and

r_{t}

Panel of Figure 1 — Figure 1: Optimizer Comparison: Plotted above are the training loss, test accuracy, and consensus error of CIFAR-10 (top) and the training loss, validation loss, and consensus error of tiny-shakespeare (bottom) with training done on all of the various optimizers.

Panel of Figure 2 — Figure 2: Number of Local Updates: Plotted above are the training loss, test accuracy, and consensus error of CIFAR-10 (top) and the training loss, validation loss, and consensus error of tiny-shakespeare (bottom) with training done using the Adam optimizer across a number of possible $K$ values from 1 to 50.

Panel of Figure 3 — Figure 3: Communication Volume: Plotted above are the training loss, test accuracy, and consensus error of CIFAR-10 (top) and the training loss, validation loss, and consensus error of tiny-shakespeare (bottom) with training done using the Adam optimizer across a number of possible $K$ and top- $k$ values.

Panel of Figure 4 — Figure 4: Larger Agent Counts and Differing Topology: Plotted above are the training loss, test accuracy, and consensus error of CIFAR-10 with training done using the Adam optimizer when scaling to 4, 9, and 16 agents on ring topology (top) and 2D grid topology (bottom). All experiments were run with $K=20$ local updates per communication round.

Panel of Figure 5 — Figure 5: Non-IID Data: Plotted above are the training loss, test accuracy, and consensus error when using the Adam optimizer on CIFAR-10 with non-IID data across $K$ values from 1 to 50. We distribute data following a standard Dirichlet distribution process using $\alpha=1.0$ (top) and $\alpha=0.5$ (bottom).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Convergence performance for FashionMNIST: Plotted above are the training loss and test accuracy of FashionMNIST. The top row compares optimizer performance with Top- $k$ 30% compression and a local update count of $K=20$ . The middle row demonstrates the reduction in communication rounds based on the number of local updates with Top- $k$ compression of 30%. The bottom row compares the total communication volume scaled relative to the uncompressed baseline with no local updates.

Panel of Figure 7 — Figure 7: Additional optimizer comparisons: Plotted above are accuracy (for FashionMNIST and CIFAR-10) and validation loss (for tiny-shakespeare) for the tested optimizers across a range of local update counts $K$ . For each subplot, FashionMNIST is on the right, CIFAR-10 is in the middle, and tiny-shakespeare is on the right.

Panel of Figure 8 — Figure 8: Additional number of local updates comparisons: Plotted above are accuracy (for FashionMNIST and CIFAR-10) and validation loss (for tiny-shakespeare) using AdaGrad (left) and AMSGrad (right), comparing across a range of local update values $K$ .

Panel of Figure 9 — Figure 9: Additional Top- $k$ compression comparisons: Plotted above are accuracy (for FashionMNIST and CIFAR-10) and validation loss (for tiny-shakespeare) using Adam (left), AdaGrad (middle), and AMSGrad (right), comparing across a range of Top- $k$ compression values.

Panel of Figure 10 — Figure 10: Additional Scaling Results: Plotted above are the training loss, test accuracy, and consensus error of CIFAR-10 when scaling to 4, 9, and 16 agents using ring topology and 2D grid topology with the AdaGrad and AMSGrad optimizers. All experiments were run with $K=20$ local updates per communication round.

Decentralized Optimization

Decentralized optimization develops methods for agents connected by networks to solve shared problems using local data and peer-to-peer communication.

See all 60 papers

Unifying Local Communications and Local Updates for LLM Pretraining
Jun 9, 2026Pietro Cagnasso, Eugene Belilovsky, Edouard OyallonLanguage Model Pretraining Communication-Efficient Distributed Training
Concordia University · Mila · CNRS, Sorbonne University
SPADE-DFL: Communication-Efficient Decentralized Federated Learning via Derivative-Free Linearized ADMM
Sep 24, 2026Mengli Wei, Mengkai Zhu, Jiawen Chen +2Communication-Efficient Distributed Training Black-Box Optimization
School of Mathematics, Southeast University, Nanjing 210096, China
FedA2L: Adaptive layer-wise learning rate adjustment in decentralized federated learning
Aug 10, 2026Van Truong Vo, Khoa Nguyen, Taehong KimNon-IID Federated Learning Federated Learning
School of Information and Communication Engineering, Chungbuk National University, Cheongju, South Korea

Jun 9, 2026J/K move · Enter open · S save

Unifying Local Communications and Local Updates for LLM Pretraining

Pietro Cagnasso, Eugene Belilovsky, Edouard Oyallon

Concordia University · Mila · CNRS, Sorbonne University

Language Model Pretraining Communication-Efficient Distributed Training Distributed Optimization Decentralized Learning Decentralized Optimization

Open paper AskPDF arXiv

Communication-efficient pre-training of LLMs is increasingly important as training draws on compute distributed across clusters, data centers, and lower-bandwidth links. Many practical methods reduce communication frequency but still rely on synchronous All-Reduce operations that maintain identical model states and tie progress to global collectives. This can become a bottleneck when bandwidth or worker speed is heterogeneous. We introduce GASLoC, a novel decentralized pre-training algorithm that generalizes the notion of communication acceleration to the recently popular "outer optimizer" to allow a practical gossip-based training framework that is compatible with adaptive optimizers, allows for local optimizer steps, and can utilize sparse randomized peer communication. Empirically, on a number of standard LLM training tasks, we demonstrate that GASLoC outperforms state-of-the-art decentralized algorithms in single step per communication setting for a number of topologies and, unlike existing decentralized methods in the LLM setting, it allows to obtain performance competitive with DiLoCo when utilizing multiple local steps. In the heterogeneous bandwidth setting we demonstrate the advantage of GASLoC showing that it can significantly outperform DiLoCo.

Abstract

Decentralized Optimization

Explore similar work

Unifying Local Communications and Local Updates for LLM Pretraining

SPADE-DFL: Communication-Efficient Decentralized Federated Learning via Derivative-Free Linearized ADMM

FedA2L: Adaptive layer-wise learning rate adjustment in decentralized federated learning