Communication-Efficient Distributed Training

Overview

This field studies how multiple devices can train a shared or coordinated model without spending too much time and bandwidth exchanging updates, activations, or parameters. Common approaches include doing more local optimization between synchronizations, compressing messages with quantization, sparsification, or low-rank projections, and exchanging compact alternatives such as logits or prototypes. Systems work also reduces communication costs through topology-aware collectives, pipeline scheduling, and overlapping data transfers with computation, especially for large language models and graph workloads. The central tension is to save communication while preserving convergence, accuracy, privacy, and robustness to heterogeneous data, unreliable links, and uneven hardware.

Latest papers

CardsList

245 papers20 papers in the last four weeks, down 37% on the four weeks before. 0.2% of all new papers.

  1. A cautionary tale on the cost-effectiveness of collaborative AI in real-world medical applications

    Dec 9, 2024Francesco Cremonesi, Lucia Innocenti, Sebastien Ourselin +3Communication-Efficient Distributed TrainingFederated Learning

  2. Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learning

    Oct 31, 2024Minghui Chen, Meirui Jiang, Xin Zhang +3Communication-Efficient Distributed TrainingCross-Silo Federated Learning

  3. The Limits and Potentials of Local SGD for Distributed Heterogeneous Learning with Intermittent Communication

    May 19, 2024Kumar Kshitij Patel, Margalit Glasgow, Ali Zindari +5Communication-Efficient Distributed TrainingDistributed Optimization