Distributed Training

Distributed training studies how to divide model computation and data across GPUs, computers, and sometimes geographically separated or unreliable workers. Common approaches include data, tensor, pipeline, and expert parallelism, alongside sharding, gradient compression, asynchronous updates, and communication-aware scheduling. Systems work measures throughput, memory use, and scaling on language models, graph neural networks, and scientific workloads, while fault-tolerant collectives, checkpointing, and robust aggregation address failures and malicious workers. A central challenge is balancing faster, larger-scale training against communication and synchronization costs without harming convergence, model quality, or privacy.