DPA: Decentralized Primal Averaging with Quasi-Global Momentum for Highly Heterogeneous Data
Organizations: Department of Information and Communications Engineering, Aalto University, 02150 Espoo, Finland. · Department of Electronic Systems, Faculty of Information Technology and Electrical Engineering, Norwegian University of Science and Technology, 7034 Trondheim, Norway; also with the Department of Information and Communications Engineering, Aalto University, 02150 Espoo, Finland.
Abstract
Decentralized training of deep learning models keeps training data local and supports on-device learning over networks. In realistic scenarios, heterogeneity across agents' local data poses an optimization challenge and can severely degrade test accuracy, especially on sparse communication graphs. Building on recent advances in centralized primal averaging, we propose Decentralized Primal Averaging (DPA), a decentralized learning algorithm that combines primal averaging with a local quasi-global momentum buffer. DPA maintains three coupled sequences for optimization, gradient evaluation, and output averaging, while using consecutive mixed iterates to construct a network-aware momentum direction without transmitting an auxiliary tracking variable. For smooth nonconvex objectives, we establish a convergence bound for DPA and identify conditions for linear speedup in the number of agents for sufficiently many iterations. We also introduce DPA-1G, a variant that uses one communication round per iteration and matches the per-iteration communication cost of DSGD-style baselines. Experiments following the GUT decentralized image-classification benchmark show that DPA compares favorably with DSGD, gradient tracking, QG-DSGDm, momentum tracking, and QG-GUTm across datasets, architectures, topologies, and heterogeneity levels, with the largest gains under high heterogeneity on sparse rings. At the same total communication cost, both DPA variants achieve higher mean test accuracy than the best baseline on CIFAR-10 and CIFAR-100 with ResNet-20. Our code is publicly available.
Figures & tables
| Method | Fashion-MNIST / LeNet-5 | CIFAR-10 / VGG-11 | CIFAR-10 / ResNet-20 | CIFAR-100 / ResNet-20 |
|---|---|---|---|---|
| LE std / L5E std | LE std / L5E std | LE std / L5E std | LE std / L5E std | |
| DSGD | ||||
| GT | ||||
| QG-DSGDm | ||||
| MT | ||||
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset / Model | DSGD | GT | QG-DSGDm | MT | QG-GUTm | DPA / DPA-1G | |
|---|---|---|---|---|---|---|---|
| Fashion-MNIST / LeNet-5 | |||||||
| Fashion-MNIST / LeNet-5 | |||||||
| CIFAR-10 / VGG-11 | |||||||
| CIFAR-10 / VGG-11 | |||||||
| CIFAR-10 / ResNet-20 | |||||||
| CIFAR-10 / ResNet-20 |
| Model / Graph | DSGD | GT | QG-DSGDm | MT | QG-GUTm | DPA / DPA-1G | |
|---|---|---|---|---|---|---|---|
| ResNet-20 / Ring | |||||||
| ResNet-20 / Ring |
| Model / Graph | QG-DSGDm | QG-GUTm | DPA / DPA-1G | |
|---|---|---|---|---|
| VGG-11 / Ring | ||||
| VGG-11 / Ring | ||||
| ResNet-20 / Torus | ||||
| ResNet-20 / Dyck |
| Dataset / Model | Graph | Scaling coefficient | ||
|---|---|---|---|---|
| Fashion-MNIST / LeNet-5 | Ring | |||
| Fashion-MNIST / LeNet-5 | Ring | |||
| CIFAR-10 / VGG-11 | Ring | |||
| CIFAR-10 / VGG-11 | Ring | |||
| CIFAR-10 / ResNet-20 | Ring | |||
| CIFAR-100 / ResNet-20 | Ring |
| Dataset / Model | Graph | Epochs | ||||
|---|---|---|---|---|---|---|
| Fashion-MNIST / LeNet-5 | Ring | 16 | 100 | |||
| CIFAR-10 / VGG-11 | Ring | 16, 32 | 200 | |||
| CIFAR-10 / ResNet-20 | Ring | 16, 32 | 200 | |||
| CIFAR-100 / ResNet-20 | Ring | 16 | 200 | |||
| CIFAR-10 / ResNet-20 | Torus | 32 | 200 | |||
| CIFAR-10 / ResNet-20 | Dyck | 32 | 200 |
| Method | LE std | L5E std | |
|---|---|---|---|
| DSGD | |||
| GT | |||
| QG-DSGDm | |||
| MT | |||
| QG-GUTm | |||
| DPA-1G |
| Graph | Method | LE std | L5E std |
|---|---|---|---|
| Torus | QG-DSGDm | ||
| QG-GUTm | |||
| DPA-1G | |||
| DPA | |||
| Dyck | QG-DSGDm | ||
| QG-GUTm |
| Method | LE std | L5E std | |
|---|---|---|---|
| QG-DSGDm | |||
| QG-GUTm | |||
| DPA-1G | |||
| DPA | |||
| QG-DSGDm | |||
| QG-GUTm |
| Method | |||||
|---|---|---|---|---|---|
| Best baseline (QG-GUTm) | |||||
| DPA-1G | |||||
| DPA |
| Method | |||||
|---|---|---|---|---|---|
| Best baseline method | QG-DSGDm | QG-DSGDm | QG-GUTm | QG-GUTm | QG-DSGDm |
| Best baseline accuracy | |||||
| DPA-1G | |||||
| DPA |
| Training progress | QG-DSGDm | QG-GUTm | DPA-1G | DPA |
|---|---|---|---|---|
| 10% (epoch 20) | ||||
| 20% (epoch 40) | ||||
| 40% (epoch 80) | ||||
| 50% (epoch 100) | ||||
| 60% (epoch 120) | ||||
| 80% (epoch 160) |
| Method | Total time | Training time | GPU-hours | GPU util. | Peak mem. | GB/agent | Grad. evals/agent |
|---|---|---|---|---|---|---|---|
| DSGD (direct) | 45.84 min | 34.52 min | 3.056 | 61.83% | 4982 MiB | 42.72 | 19,600 |
| QG-DSGDm (direct) | 46.81 min | 35.51 min | 3.121 | 62.64% | 4990 MiB | 42.72 | 19,600 |
| QG-GUTm | 115.80 min | 104.78 min | 7.720 | 84.40% | 5006 MiB | 42.72 | 19,600 |
| GT | 51.62 min | 40.33 min | 3.441 | 60.98% | 4990 MiB | 85.45 | 19,600 |
| MT | 52.59 min | 41.23 min | 3.506 | 67.42% | 4990 MiB | 85.45 | 19,600 |
| DPA-1G | 47.17 min | 35.79 min | 3.145 | 62.79% | 4998 MiB | 42.72 | 15,680 |
| Fashion-MNIST / LeNet-5 | CIFAR-10 / ResNet-20 | CIFAR-100 / ResNet-20 | ||||
| Method | LE | After averaging | LE | After averaging | LE | After averaging |
| DSGD | ||||||
| GT | ||||||
| QG-DSGDm | ||||||
| MT | ||||||
Decentralized Learning
Decentralized learning trains models through local computation and peer-to-peer information exchange rather than relying on a central coordinator.
See all 106 papers