Distributed Training

Ask about this topic

Overview

Distributed training studies how to divide model computation and data across GPUs, computers, and sometimes geographically separated or unreliable workers. Common approaches include data, tensor, pipeline, and expert parallelism, alongside sharding, gradient compression, asynchronous updates, and communication-aware scheduling. Systems work measures throughput, memory use, and scaling on language models, graph neural networks, and scientific workloads, while fault-tolerant collectives, checkpointing, and robust aggregation address failures and malicious workers. A central challenge is balancing faster, larger-scale training against communication and synchronization costs without harming convergence, model quality, or privacy.

Latest papers

CardsList

133 papers17 papers in the last four weeks, up 31% on the four weeks before. 0.1% of all new papers.

  1. Dobb: Shard-Local Preconditioning in Block-Diagonal Eigenbases

    Oct 29, 2026Shi-Yuk Fong, Shuhua Yu, Jiawei ZhangDistributed OptimizationCommunication-Efficient Distributed Training

    Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI, USA · Meta FAIR, Menlo Park, CA, USA

  2. MCCL: Training Pytorch models across a fleet of Macs at Skywalker Sound

    Oct 28, 2026Kevin Sasso, Chris Fiegel, Francis Aitken +2Distributed TrainingHigh-Performance Computing

    Skywalker Sound, Nicasio, California, USA.

  3. The Endgame of FSDP Communication: Layout-Aware Collectives on Copy Engines, RDMA, and NVSwitch

    Oct 28, 2026Zejian Wang, Yifan Zhao, Kaihui Ren +11Communication-Efficient Distributed TrainingDistributed Training

    KlingAI Research, Shanghai, China · KlingAI Research, Beijing, China

  4. Satellite Intelligence: Distributed Training, Collaborative Inference,and LLM Adaptation in LEO Constellation

    Oct 26, 2026Zheng Lin, Zhanwei Wang, Yuxin Zhang +9LEO Satellite NetworksLLM Inference

    Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, L-1855 Luxembourg, Luxembourg · Department of Computer and Electronic Engineering, University of Hong Kong, Pok Fu Lam, Hong Kong, China · Institute of Space Internet, Fudan University, Shanghai, China +5

  5. Democratizing LLM Training with the Edge: Challenges, Systems, and the Road Ahead

    Oct 23, 2026Ziyue Luo, Jiaxuan Cai, Cedric LeDenmat +27LLM TrainingDistributed Training

    The Ohio State University · University of Wisconsin–Madison · Rochester Institute of Technology +5

  6. Train Locally, Learn Together: Distributed Multi-Agent Training via ADMM

    Oct 17, 2026Yiwei Chen, Changsheng Wang, Yicheng Lang +3Distributed TrainingMulti-Agent Consensus

    Michigan State University · University of Michigan, Ann Arbor

  7. ETH: Data-Balanced Heterogeneous Pipeline Parallelism for Training Large Models

    Oct 15, 2026Mianjie Yu, Zhongmin Zhao, Zizhao Mo +9Distributed TrainingPipeline Parallelism

    University of Macau · Great Bay University · Sun Yat-sen University +1

  8. Coordination-Free Workload Regularization for Variable-Cost Distributed Training

    Oct 15, 2026Tao Huang, Chen Zhu, Antonio Alonso +4Distributed Training

    LinkedIn Corporation, Sunnyvale, CA, USA · The University of Texas at Dallas, Richardson, TX, USA

  9. Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training

    Oct 14, 2026Anshu Raina, Peyman Razaghi, Yuankai Chen +7Distributed TrainingLLM Training

    Advanced Micro Devices, Inc.

  10. How Inner Optimizers Shape Replica Scaling

    Oct 13, 2026Eugene Belilovsky, Stefan Horoi, Edouard Oyallon +1LLM TrainingDistributed Training

    Mila · Concordia University · Université de Montréal +1

  11. DisaggKD: Efficient Knowledge Distillation through Teacher-Student Disaggregation and a Fused Loss

    Oct 13, 2026Ruixuan Tu, Yuyang DongLanguage Model DistillationKnowledge Distillation

    The University of Tokyo, RIKEN AIP · SB Intuitions Corp.

  12. Scaling Repeated-Data Pretraining: Batch Size, Regularization, and GPU-Time Tradeoffs

    Oct 11, 2026Sewon MinLLM TrainingData Repetition in Language Model Training

    UC Berkeley & Allen Institute for AI

  13. A Dependency-Aware Multi-GPU Training Framework for Temporal Graph Neural Networks

    Oct 11, 2026Yang Zhao, Yue Dai, Liang Qiao +8Temporal GNNsDistributed Training

    University of Science and Technology of China, Hefei, Anhui, China · Illinois Institute of Technology, Chicago, Illinois, USA

  14. What Gradients Add to Text Leakage in Split Language Models, Counted per Token and per Document

    Oct 2, 2026Georgios Politis, Evangelos PappasPrivacy Leakage in Language ModelsGradient Inversion Attacks

  15. Cordial Learning: Distributed Training with Correlated Data

    Oct 2, 2026Sarah Shitrit, Ilai BistritzDistributed TrainingCommunication-Efficient Distributed Training

    School of Electrical and Computer Engineering Tel Aviv University Tel Aviv, Israel · School of Industrial and Intelligent Systems Engineering and School of Electrical and Computer Engineering Tel Aviv University Tel Aviv, Israel

  16. HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Training

    Sep 30, 2026Mengyuan Fan, Peizhuang Cong, Zixiao Huang +7Expert ParallelismHeterogeneous Computing

    State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Infinigence AI · Tsinghua University

  17. Argus: A Real-EKS Study of When Predicting Spot Interruptions Beats Simple Checkpointing

    Sep 30, 2026Angshuman Chakravertty, MD RayyanFault-Tolerant Distributed TrainingCloud Computing

    SVKM’s NMIMS (Deemed to be University)

  18. Scaling Zero-Order Pretraining through Model Sharding

    Sep 29, 2026Francois Chaubard, Mykel J. Kochenderfer, Chris RéZeroth-Order OptimizationLanguage Model Pretraining

    Stanford University

  19. AutoLoCo: Communication Efficient Distributed LLM Training via Adaptive Synchronization

    Sep 29, 2026Pengyu He, Yan Zhang, Ruien Li +1Communication-Efficient Distributed TrainingDistributed Training

    Department of Electrical Engineering, California Institute of Technology · Department of Computer Science and Technology, Tsinghua University · Department of Computer Sciences, University of Wisconsin - Madison

  20. Mixture-of-Kittens: MoE Megakernel for NVL72s

    Sep 28, 2026Stuart H. Sul, Nash Brown, Henry Wildermuth +3Distributed TrainingMixture-of-Experts Models

    Department of Computer Science, Stanford University · Cursor Research

  21. TopoEP: Topology-Aware Load Balancing for Expert-Parallel MoE Training

    Sep 28, 2026Jiacheng Zhu, Xie Zhao, Gongming Zhao +3Expert ParallelismExpert Load Balancing

    University of Science and Technology of China

  22. FairMean: Promoting Fairness in Distributed Learning under Label Poisoning Attacks

    Sep 22, 2026Huigan Zheng, Jiaojiao Zhang, Yongxiang LiuAlgorithmic FairnessAdaptive Loss Weighting

    Sun Yat-Sen University · Pengcheng Laboratory · Great Bay University

  23. Accelerating Sharded Data Parallelism at Scale with Federated Learning

    Sep 17, 2026Gianluca Mittone, Marco AldinucciCommunication-Efficient Distributed TrainingHigh-Performance Computing

    University of Turin, Turin, Italy

  24. Efficiently Distributed Federated Learning

    Sep 17, 2026Gianluca Mittone, Robert Birke, Marco AldinucciDistributed TrainingFederated Learning

    Computer Science Department, University of Turin, Turin, Italy

  25. OPEN-1B: A Fully Auditable Training Run

    Sep 15, 2026John Donaghy, Brian Wilcox, Oğuzhan Ersoy +6Neural Network VerificationLLM Auditing

    Gensyn

  26. LLM-Based Schema-Aware Split Learning for Privacy-Preserving Mental Distress Prediction Across Heterogeneous Surveys

    Sep 14, 2026Md Khalid Syfullah, Alvi Ataur KhalilPrivacy-Preserving MLClinical Prediction

    Transformative Innovation for Trustworthy AI and Network Security (TITANS) Lab, Computer Science, Southern Illinois University Carbondale, USA

  27. Affinity-Aware Sharding for Delayed Tensor Parallelism

    Sep 12, 2026Eloi de ReynalTensor ParallelismDistributed Training

    Kog Team

  28. 4D Parallelism Unlocks Exascale Bayesian Neural Networks for High-Fidelity Atmospheric Modeling

    Sep 11, 2026Deifilia Kieckhefen, Juan Pedro Gutiérrez Hermosillo Muriedas, Lars Helge Heyen +12Bayesian Neural NetworksUncertainty Quantification

    Karlsruhe Institute of Technology, Germany · Forschungszentrum Jülich, Germany · CSC – IT Center for Science Ltd., Finnland +1

  29. Clustering-Based Balanced Sampling and Allocation with Data Parallelism for High-Performance Fine-Tuning

    Sep 11, 2026Hyunjin Kim, Youngeun Nam, Jaemin Han +2LLM Fine-TuningInstruction-Tuning Data Selection

    KAIST

  30. Generalization Analysis of Distributed Kernel-based Robust Gradient Descent Algorithms

    Sep 10, 2026Jun-Yi Meng, Zheng-Chu Guo, Yuan MaoKernel RegressionCommunication-Efficient Distributed Training

    School of Mathematical Sciences, Zhejiang University, Hangzhou 310058, P. R. China · College of Informatics, Huazhong Agricultural University, Wuhan 430070, P. R. China