Overview

Distributed training studies how to divide model computation and data across GPUs, computers, and sometimes geographically separated or unreliable workers. Common approaches include data, tensor, pipeline, and expert parallelism, alongside sharding, gradient compression, asynchronous updates, and communication-aware scheduling. Systems work measures throughput, memory use, and scaling on language models, graph neural networks, and scientific workloads, while fault-tolerant collectives, checkpointing, and robust aggregation address failures and malicious workers. A central challenge is balancing faster, larger-scale training against communication and synchronization costs without harming convergence, model quality, or privacy.

Latest papers

CardsList

142 papers12 papers in the last four weeks, down 29% on the four weeks before. 0.1% of all new papers.

  1. LACCO: Local Accumulation with Computation--Communication Overlap

    Nov 13, 2026Daniele Kopyshevskiy, Benjamin Th{é}rien, Nicolas Pinto +1Communication-Efficient Distributed TrainingDistributed Training

  2. RPEG: Training Reasoning Language Models Scalably and Cost-Effectively

    Nov 6, 2026Maciej Besta, Leonard Schmidt, Pierre Pang +3LLM TrainingDistributed Training

  3. KOSMOS: Self-scheduling Persistent Kernels for Communication-Compute Overlap

    Nov 5, 2026Alexander Magro, Aristotle Martin, Ye Wang +1GPU Kernel OptimizationDistributed Training

  4. One-shot Distributed Learning via Weighted Averaging with Server-side Information

    Nov 3, 2026Pavle Tepavčević, Dušan Jakovetić, Miloš SavićDistributed TrainingFederated Learning Aggregation

  5. FP4Quant: Efficient 4-Bit Communication Compression for Model-Parallel LLM Training

    Nov 3, 2026Man Liu, Xingchen Liu, Hairui Zhao +2Communication-Efficient Distributed Training4-Bit Quantization

  6. DynSL: Resource-Adaptive Model Training across Heterogeneous Edge Devices

    Nov 2, 2026Boris Radovič, Istemi Ekin Akkus, Marco Canini +4Distributed TrainingEdge Computing

  7. Dobb: Shard-Local Preconditioning in Block-Diagonal Eigenbases

    Oct 29, 2026Shi-Yuk Fong, Shuhua Yu, Jiawei ZhangDistributed OptimizationCommunication-Efficient Distributed Training

  8. MCCL: Training Pytorch models across a fleet of Macs at Skywalker Sound

    Oct 28, 2026Kevin Sasso, Chris Fiegel, Francis Aitken +2Distributed TrainingHigh-Performance Computing

  9. The Endgame of FSDP Communication: Layout-Aware Collectives on Copy Engines, RDMA, and NVSwitch

    Oct 28, 2026Zejian Wang, Yifan Zhao, Kaihui Ren +11Communication-Efficient Distributed TrainingDistributed Training

  10. Satellite Intelligence: Distributed Training, Collaborative Inference,and LLM Adaptation in LEO Constellation

    Oct 26, 2026Zheng Lin, Zhanwei Wang, Yuxin Zhang +9LEO Satellite NetworksLLM Inference

  11. Democratizing LLM Training with the Edge: Challenges, Systems, and the Road Ahead

    Oct 23, 2026Ziyue Luo, Jiaxuan Cai, Cedric LeDenmat +27LLM TrainingDistributed Training

  12. Train Locally, Learn Together: Distributed Multi-Agent Training via ADMM

    Oct 17, 2026Yiwei Chen, Changsheng Wang, Yicheng Lang +3Distributed TrainingMulti-Agent Consensus

  13. ETH: Data-Balanced Heterogeneous Pipeline Parallelism for Training Large Models

    Oct 15, 2026Mianjie Yu, Zhongmin Zhao, Zizhao Mo +9Distributed TrainingPipeline Parallelism

  14. Coordination-Free Workload Regularization for Variable-Cost Distributed Training

    Oct 15, 2026Tao Huang, Chen Zhu, Antonio Alonso +4Distributed Training

  15. Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training

    Oct 14, 2026Anshu Raina, Peyman Razaghi, Yuankai Chen +7Distributed TrainingLLM Training

  16. How Inner Optimizers Shape Replica Scaling

    Oct 13, 2026Eugene Belilovsky, Stefan Horoi, Edouard Oyallon +1LLM TrainingDistributed Training

  17. A Dependency-Aware Multi-GPU Training Framework for Temporal Graph Neural Networks

    Oct 11, 2026Yang Zhao, Yue Dai, Liang Qiao +8Temporal GNNsDistributed Training

  18. Cordial Learning: Distributed Training with Correlated Data

    Oct 2, 2026Sarah Shitrit, Ilai BistritzDistributed TrainingCommunication-Efficient Distributed Training

  19. HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Training

    Sep 30, 2026Mengyuan Fan, Peizhuang Cong, Zixiao Huang +7Expert ParallelismHeterogeneous Computing

  20. Scaling Zero-Order Pretraining through Model Sharding

    Sep 29, 2026Francois Chaubard, Mykel J. Kochenderfer, Chris RéZeroth-Order OptimizationLanguage Model Pretraining

  21. AutoLoCo: Communication Efficient Distributed LLM Training via Adaptive Synchronization

    Sep 29, 2026Pengyu He, Yan Zhang, Ruien Li +1Communication-Efficient Distributed TrainingDistributed Training

  22. Mixture-of-Kittens: MoE Megakernel for NVL72s

    Sep 28, 2026Stuart H. Sul, Nash Brown, Henry Wildermuth +3Distributed TrainingMixture-of-Experts Models

  23. TopoEP: Topology-Aware Load Balancing for Expert-Parallel MoE Training

    Sep 28, 2026Jiacheng Zhu, Xie Zhao, Gongming Zhao +3Expert ParallelismExpert Load Balancing

  24. FairMean: Promoting Fairness in Distributed Learning under Label Poisoning Attacks

    Sep 22, 2026Huigan Zheng, Jiaojiao Zhang, Yongxiang LiuAlgorithmic FairnessAdaptive Loss Weighting