Distributed Training

Ask about this topic

Overview

Distributed training studies how to divide model computation and data across GPUs, computers, and sometimes geographically separated or unreliable workers. Common approaches include data, tensor, pipeline, and expert parallelism, alongside sharding, gradient compression, asynchronous updates, and communication-aware scheduling. Systems work measures throughput, memory use, and scaling on language models, graph neural networks, and scientific workloads, while fault-tolerant collectives, checkpointing, and robust aggregation address failures and malicious workers. A central challenge is balancing faster, larger-scale training against communication and synchronization costs without harming convergence, model quality, or privacy.

Latest papers

CardsList

133 papers17 papers in the last four weeks, up 31% on the four weeks before. 0.1% of all new papers.

  1. Resilient AI Supercomputer Networking using MRC and SRv6

    May 5, 2026Joao Araujo, Alex Chow, Mark Handley +47Distributed Training

  2. DBLP: Phase-Aware Bounded-Loss Transport for Burst-Resilient Distributed ML Training

    May 3, 2026Zechen Ma, Zixi Qu, Jinyan Yi +2Communication-Efficient Distributed TrainingDistributed Training

    Department of Computer Science, University of Toronto, Toronto, Canada

  3. Accelerating Long-Tail Generation in Synchronous RLHF Training via Adaptive Tensor Parallelism

    May 3, 2026Long Zhao, Qinghe Wang, Jiaan Zhu +5Tensor ParallelismLLM Inference Acceleration

    Anhui University · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · University of Science and Technology of China +2

  4. Adaptation of AI-accelerated CFD Simulations to the IPU platform

    May 1, 2026P. Rosciszewski, A. Krzywaniak, S. Iserte +2AI for SciencePDE Surrogate Modeling

    Graphcore Poland · Dpt. of Construction and Mechanical Engineering, Universitat Jaume I, Spain · Barcelona Supercomputing Center, Spain +2

  5. FloatSOM: GPU-Accelerated, Distributed, Topology-Flexible Self-Organizing Maps

    Apr 29, 2026Tony Xu, Sarah Klamt, Katherine Turner +3GPU AccelerationHigh-Performance Computing

    John Curtin School of Medical Research, Australian National University · Department of Information Technology and Electrical Engineering, ETH Zürich · Mathematical Data Science Centre, Australian National University +2

  6. A Survey on Split Learning for LLM Fine-Tuning: Models, Systems, and Privacy Optimizations

    Apr 27, 2026Zihan Liu, Yizhen Wang, Rui Wang +2Privacy-Preserving Language ModelsLLM Fine-Tuning

    Zhejiang University, Ningbo, China

  7. TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Training

    Apr 27, 2026Man Liu, Xingchen Liu, Xingjian Tian +8Communication-Efficient Distributed TrainingTensor Parallelism

    Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences · University of Science and Technology of China

  8. FreeScale: Distributed Training for Sequence Recommendation Models with Minimal Scaling Cost

    Apr 27, 2026Chenhao Feng, Haoli Zhang, Shakhzod Ali-Zade +17Sequential RecommendationCommunication-Efficient Distributed Training

    1Meta Inc.

  9. CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Training

    Apr 27, 2026Rezaul Karim, Austin Wen, Wang Zongzuo +3Communication-Efficient Distributed TrainingDistributed Optimization

    Toronto Ascend Team

  10. Decoupled DiLoCo for Resilient Distributed Pre-training

    Apr 23, 2026Arthur Douillard, Keith Rush, Yani Donchev +14Language Model PretrainingDistributed Optimization

    1Google DeepMind · 2Google Research

  11. Towards Energy Impact on AI-Powered 6G IoT Networks: Centralized vs. Decentralized

    Apr 21, 2026Anjie Qiu, Donglin Wang, Sanket Partani +2Energy-Efficient MLInternet of Things

    *Institute for Wireless Communication and Navigation (WiCoN), University of Kaiserslautern-Landau (RPTU) · +Intelligent Networks Research Group (IN), German Research Center for Artificial Intelligence (DFKI)

  12. Scalable and Adaptive Parallel Training of Graph Transformer on Large Graphs

    Apr 17, 2026Jun-Liang Lin, Kamesh Madduri, Mahmut Taylan KandemirDistributed TrainingGraph Transformer

    The Pennsylvania State University University Park, PA, USA

  13. Training Time Prediction for Mixed Precision-based Distributed Training

    Apr 17, 2026Minchul Kang, Changyong Shin, Jinwoo Jeong +5Deep Learning OptimizationDistributed Training

    Korea University · KT Corporation

  14. PINNACLE: An Open-Source Computational Framework for Classical and Quantum PINNs

    Apr 17, 2026Shimon Pisnoy, Hemanth Chandravamsi, Ziv Chen +4Neural PDE SolversPDE Surrogate Modeling

    †Faculty of Mechanical Engineering, Technion Israel Institute of Technology, Haifa, Israel. · ‡Corresponding author. · §The Andrew and Erna Viterbi Faculty of Electrical & Computer Engineering, Technion Israel Institute of Technology, Haifa, Israel. +2

  15. SparseBalance: Load-Balanced Long Context Training with Dynamic Sparse Attention

    Apr 15, 2026Hongtao Xu, Jianchao Tan, Yuxuan Hu +8Sparse AttentionDistributed Training

    School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China · Meituan, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

  16. SL-FAC: A Communication-Efficient Split Learning Framework with Frequency-Aware Compression

    Apr 8, 2026Zehang Lin, Miao Yang, Haihan Zhu +9Activation CompressionCommunication-Efficient Distributed Training

    School of Computer and Information Engineering, Xiamen University of Technology, Xiamen, 361000, China · School of Artificial Intelligence, Hebei University of Technology · Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong +5

  17. NestPipe: Large-Scale Recommendation Training on 1,500+ Accelerators via Nested Pipelining

    Apr 8, 2026Zhida Jiang, Zhaolong Xing, Huichao Chai +12Distributed TrainingPipeline Parallelism

    Huawei

  18. Communication-free Sampling and 4D Hybrid Parallelism for Scalable Mini-batch GNN Training

    Apr 3, 2026Cunyang Wei, Siddharth Singh, Aishwarya Sarkar +7Graph Neural NetworksDistributed Training

    University of Maryland, College Park, USA · NVIDIA, Inc. · Iowa State University, USA +3

  19. ModTrans: Translating Real-world Models for Distributed Training Simulator

    Apr 2, 2026Yi LyuDistributed Training

    University of Wisconsin-Madison Madison, WI, USA

  20. EmbedPart: Embedding-Driven Graph Partitioning for Scalable Graph Neural Network Training

    Apr 1, 2026Nikolai Merkel, Ruben Mayer, Volker Markl +1Graph PartitioningGraph Neural Networks

    Technical University of Munich (TUM) · University of Bayreuth · TU Berlin, BIFOLD, DFKI +1

  21. OptINC: Optical In-Network-Computing for Scalable Distributed Learning

    Mar 30, 2026Sijie Fei, Grace Li Zhang, Bing Li +1Gradient QuantizationCommunication-Efficient Distributed Training

    Technical University of Munich, Munich, Germany · Technical University of Darmstadt, Darmstadt, Germany · Technical University of Ilmenau, Ilmenau, Germany

  22. Gradient Manipulation in Distributed Stochastic Gradient Descent with Strategic Agents: Truthful Incentives with Convergence Guarantees

    Mar 30, 2026Ziqin Chen, Yongqiang WangDistributed TrainingDecentralized Learning

    Department of Electrical and Computer Engineering, Clemson University, Clemson, SC 29634 USA

  23. Tula: Optimizing Time, Cost, and Generalization in Distributed Large-Batch Training

    Mar 18, 2026Sahil Tyagi, Feiyi WangDistributed TrainingHyperparameter Optimization

    Oak Ridge National Laboratory, Oak Ridge, Tennessee, USA

  24. Byzantine-Robust and Communication-Efficient Distributed Training: Compressive and Cyclic Gradient Coding

    Mar 17, 2026Chengxi Li, Youssef Allouah, Rachid Guerraoui +2Distributed TrainingCommunication-Efficient Distributed Training

    Division of Information Science and Engineering, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, 10044 Stockholm, Sweden · Stanford University, CA 94305, United States · School of Computer and Communication Sciences, the Swiss Federal Institute of Technology in Lausanne (EPFL), 1015 Lausanne, Switzerland

  25. Thousand-GPU Large-Scale Training and Optimization Recipe for AI-Native Cloud Embodied Intelligence Infrastructure

    Mar 11, 2026Yongjian Guo, Yunxuan Ma, Haoran Sun +22Distributed TrainingEmbodied AI

  26. Exploiting Adaptive Channel Pruning for Communication-Efficient Split Learning

    Mar 10, 2026Jialei Tan, Zheng Lin, Xiangming Cai +4Distributed TrainingNeural Network Pruning

    School of Physics and Information Engineering, Fuzhou University, Fuzhou 350108, China · Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong · School of Electrical Engineering and Telecommunications, University of New South Wales Sydney, Sydney, NSW 2052, Australia +3

  27. Split Federated Learning Architectures for High-Accuracy and Low-Delay Model Training

    Mar 9, 2026Yiannis Papageorgiou, Yannis Thomas, Ramin Khalili +1Split Federated LearningFederated Learning

    Department of Informatics, Athens University of Economics and Business, Athens, Greece · Huawei Technologies Duesseldorf Gmbh, Munich, Germany

  28. Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet

    Mar 9, 2026Joel Lidin, Amir Sarfi, Erfan Miahi +6Language Model PretrainingDistributed Training

    Covenant AI · Mila

  29. A Distributed Gaussian Process Model for Multi-Robot Mapping

    Mar 7, 2026Seth Nabarro, Mark van der Wilk, Andrew J. DavisonSparse Gaussian ProcessesGaussian Belief Propagation

    Dyson Robotics Lab, Imperial College London · Department of Computer Science, University of Oxford

  30. NEST: Network- and Memory-Aware Device Placement For Distributed Deep Learning

    Mar 6, 2026Irene Wang, Vishnu Varma Venkata, Arvind Krishnamurthy +1Distributed TrainingCommunication-Efficient Distributed Training

    Georgia Institute of Technology, Atlanta, GA, USA · University of Washington, Seattle, WA, USA