Distributed Training

Distributed training aims to accelerate machine learning model training by distributing the workload across multiple computing nodes. Current research focuses on improving efficiency by addressing challenges like straggler nodes (slow devices), communication bottlenecks (especially in architectures like Parameter Servers and AllReduce), and the unique needs of specific model types such as Graph Neural Networks. These advancements are crucial for training increasingly large and complex models, enabling faster development and deployment of AI applications across various domains, from recommendation systems to power grid optimization.

Papers

July 29, 2024

Rina: Enhancing Ring-AllReduce with In-network Aggregation in Distributed Model Training
Zixuan Chen, Xuandong Liu, Minglin Li, Yinfan Hu, Hao Mei, Huifeng Xing, Hao Wang, Wanxin Shi, Sen Liu, Yang Xu
Distributed Learning Multi Hop All Reduce ToR Network Distributed Training Network Aggregation

April 15, 2024

AntDT: A Self-Adaptive Distributed Training Framework for Leader and Straggler Nodes
Youshao Xiao, Lin Ju, Zhenglei Zhou, Siyuan Li, Zhaoxin Huan, Dalong Zhang, Rujie Jiang, Lin Wang, Xiaolu Zhang, Lei Liang, Jun Zhou
Straggler Mitigation Distributed Training

November 12, 2023

GraNNDis: Efficient Unified Distributed Training Framework for Deep GNNs on Large Clusters
Jaeyong Song, Hongsun Jang, Jaewon Jung, Youngsok Kim, Jinho Lee
Graph Neural Network Cluster Structure Mini Batch Automatic Batching GNN Training Framework Distributed Training

July 13, 2023

Proof of Training (PoT): Harnessing Crypto Mining Power for Distributed AI Training
Peihao Li
Training Data Partial Proof Decentralized Training Protocol Design Byzantine Fault Tolerance Verifiable Learning Distributed Training

December 15, 2022

Distributed-Training-and-Execution Multi-Agent Reinforcement Learning for Power Control in HetNet
Kaidi Xu, Nguyen Van Huynh, Geoffrey Ye Li
Multi Agent Reinforcement Learning Multi Agent Deep Reinforcement Learning Power Allocation Heterogeneous Network Power Control Distributed Training

February 7, 2022

Parallel Successive Learning for Dynamic Distributed Model Training over Heterogeneous Wireless Networks
Seyyedali Hosseinalipour, Su Wang, Nicolo Michelusi, Vaneet Aggarwal, Christopher G. Brinton, David J. Love, Mung Chiang
Stochastic Gradient Descent Heterogeneous Wireless Network Distributed Training Decentralized Cooperation Successive Affine Learning

Distributed Training

Papers

Rina: Enhancing Ring-AllReduce with In-network Aggregation in Distributed Model Training

AntDT: A Self-Adaptive Distributed Training Framework for Leader and Straggler Nodes

GraNNDis: Efficient Unified Distributed Training Framework for Deep GNNs on Large Clusters

Proof of Training (PoT): Harnessing Crypto Mining Power for Distributed AI Training

Distributed-Training-and-Execution Multi-Agent Reinforcement Learning for Power Control in HetNet

Parallel Successive Learning for Dynamic Distributed Model Training over Heterogeneous Wireless Networks