Communication-Efficient Distributed Training

Ask about this topic

Overview

This field studies how multiple devices can train a shared or coordinated model without spending too much time and bandwidth exchanging updates, activations, or parameters. Common approaches include doing more local optimization between synchronizations, compressing messages with quantization, sparsification, or low-rank projections, and exchanging compact alternatives such as logits or prototypes. Systems work also reduces communication costs through topology-aware collectives, pipeline scheduling, and overlapping data transfers with computation, especially for large language models and graph workloads. The central tension is to save communication while preserving convergence, accuracy, privacy, and robustness to heterogeneous data, unreliable links, and uneven hardware.

Latest papers

CardsList

230 papers30 papers in the last four weeks, up 43% on the four weeks before. 0.2% of all new papers.

  1. Dobb: Shard-Local Preconditioning in Block-Diagonal Eigenbases

    Oct 29, 2026Shi-Yuk Fong, Shuhua Yu, Jiawei ZhangDistributed OptimizationCommunication-Efficient Distributed Training

    Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI, USA · Meta FAIR, Menlo Park, CA, USA

  2. The Endgame of FSDP Communication: Layout-Aware Collectives on Copy Engines, RDMA, and NVSwitch

    Oct 28, 2026Zejian Wang, Yifan Zhao, Kaihui Ren +11Communication-Efficient Distributed TrainingDistributed Training

    KlingAI Research, Shanghai, China · KlingAI Research, Beijing, China

  3. Layerwise Repacking: Variable-Length Hybrid-Attention Training without an Extra Redistribution Pass

    Oct 26, 2026Jiahao Zhang, Junwei Feng, Dongliang Huang +3Communication-Efficient Distributed TrainingHybrid Attention

    KlingAI Research Beijing, China

  4. Democratizing LLM Training with the Edge: Challenges, Systems, and the Road Ahead

    Oct 23, 2026Ziyue Luo, Jiaxuan Cai, Cedric LeDenmat +27LLM TrainingDistributed Training

    The Ohio State University · University of Wisconsin–Madison · Rochester Institute of Technology +5

  5. On Johnson-Lindenstrauss Transforms for Byzantine-Robust Optimization

    Oct 20, 2026Aleksandr Shestakov, Vladimir Aletov, Aleksandr BeznosikovRobust OptimizationDistributed Optimization

    Basic Research of Artificial Intelligence Laboratory · MBZUAI · Federated Learning Problems Laboratory

  6. Bridging Compressed Communication and Local Steps via Spectral Method in Federated Learning

    Oct 16, 2026Dmitriy Bystrov, George Chirkov, Daniil Medyakov +2Federated LearningCommunication-Efficient Distributed Training

    Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · Federated Learning Problems Laboratory · Innopolis University

  7. TALLOP: Joint Optimization of Local Training and Communication Compression for Time-Constrained Federated Learning

    Oct 16, 2026Honoka Anada, Yafei Ou, Shinya Takamaeda-YamazakiFederated LearningCommunication-Efficient Distributed Training

    The University of Tokyo / RIKEN · RIKEN

  8. Understanding Stability of Outer Nesterov Momentum on Stochastic Pseudo-Gradients

    Oct 15, 2026Nishant Sharma, Laura Gomezjurado Gonzalez, Ganesh Talluri +2Neural Network StabilityNeural Network Training Dynamics

    Chandar Research Lab · Mila – Quebec AI Institute · Université de Montréal +4

  9. ETH: Data-Balanced Heterogeneous Pipeline Parallelism for Training Large Models

    Oct 15, 2026Mianjie Yu, Zhongmin Zhao, Zizhao Mo +9Distributed TrainingPipeline Parallelism

    University of Macau · Great Bay University · Sun Yat-sen University +1

  10. How Inner Optimizers Shape Replica Scaling

    Oct 13, 2026Eugene Belilovsky, Stefan Horoi, Edouard Oyallon +1LLM TrainingDistributed Training

    Mila · Concordia University · Université de Montréal +1

  11. NBCLT: Parameter-Agnostic Norm-Ball Clipped Local Training

    Oct 13, 2026Linxuan Pan, Shucheng Yang, Junchi YangCommunication-Efficient Distributed TrainingDecentralized Learning

  12. FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention

    Oct 13, 2026Guoxia Wang, Qianyue He, Siming Wu +6Context ParallelismAttention Mechanisms

    Baidu Inc.

  13. Chain Dimension Scaling: Communication-Efficient Loop-Style Scaling for Industrial Short-Video Ranking

    Oct 12, 2026Zhimeng Jiang, Yuening Li, Ruixiao Sun +4Communication-Efficient Distributed TrainingLearning to Rank

    Google

  14. A Dependency-Aware Multi-GPU Training Framework for Temporal Graph Neural Networks

    Oct 11, 2026Yang Zhao, Yue Dai, Liang Qiao +8Temporal GNNsDistributed Training

    University of Science and Technology of China, Hefei, Anhui, China · Illinois Institute of Technology, Chicago, Illinois, USA

  15. BSIPipe: Bidirectional Sequence-Level Interleaved Pipeline Parallelism for Long-Sequence Large Language Model Training

    Oct 10, 2026Yuzhou Ni, Ling ChenPipeline ParallelismLLM Training

    State Key Laboratory of Blockchain and Data Security, Zhejiang University · College of Computer Science and Technology, Zhejiang University

  16. Echo: Error-Compensated 4-Bit Tensor-Parallel Communication for LLM Training and Inference

    Oct 9, 2026Xingyu Xie, Xiaoyong Xie, Zhouchen Lin +1Error FeedbackCommunication-Efficient Distributed Training

    Department of Mathematics, National University of Singapore · State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University

  17. Learning to Sync at the Edge: Client-Side Adaptive Synchronization for Federated Learning

    Oct 9, 2026Ebasa Temesgen, Ammar Ahmed, Nathan Lechamo +2Federated LearningEdge ML

    Department of Computer Science & Engineering, University of Minnesota, Minneapolis, MN, USA · Department of Computer Science, University of Texas at San Antonio, San Antonio, TX, USA

  18. Deploying RDMA for Distributed Inference and Training At Scale

    Oct 8, 2026Yibo Huang, Yi Cui, Yukai Huang +12Communication-Efficient Distributed TrainingDistributed Inference

    Fudan University · ByteDance Inc. · University of Michigan

  19. Fed-GRPO: Reward-Signal-Driven Federated Group Relative Policy Optimization

    Oct 8, 2026Pengxin Guo, Shuang Zeng, Zonggen Li +3Federated RLRL for Language Model Reasoning

    The University of Hong Kong · Sun Yat-sen University

  20. Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling

    Oct 7, 2026Radha Gulhane, Quentin Anthony, Beren MillidgeCommunication-Efficient Mixture-of-ExpertsMixture-of-Experts Language Models

    Zyphra

  21. NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale

    Oct 6, 2026Songlin Jiang, Zhiyu Li, Terry Kong +5Communication-Efficient Distributed TrainingDistributed RL

    Aalto University · NVIDIA

  22. Tram-FL: Reducing Communication and Computation Costs through Sequential Model Circulation in Decentralized Federated Learning

    Oct 6, 2026Kota Maejima, Takayuki Nishio, Asato Yamazaki +1Communication-Efficient Distributed TrainingFederated Learning

    School of Engineering, Institute of Science Tokyo, Tokyo 152-8550, Japan

  23. Memory-Efficient Expert Routing for Distributed MoE Training

    Oct 5, 2026Arnab Kanti Tarafder, Jaume Guasch-Martí, Gokcen Kestor +1Expert RoutingMixture-of-Experts Models

    William & Mary · Barcelona Supercomputing Center

  24. Distributed Subliminal Learning: Replacing Model Updates with Random-Carrier Outputs

    Oct 4, 2026Dario Fenoglio, Gabriele Dominici, Martin Gjoreski +1Communication-Efficient Distributed TrainingFederated Knowledge Distillation

    Università della Svizzera italiana (USI), Lugano, Switzerland · MIT Media Lab, Massachusetts Institute of Technology, Cambridge, MA, USA

  25. Cordial Learning: Distributed Training with Correlated Data

    Oct 2, 2026Sarah Shitrit, Ilai BistritzDistributed TrainingCommunication-Efficient Distributed Training

    School of Electrical and Computer Engineering Tel Aviv University Tel Aviv, Israel · School of Industrial and Intelligent Systems Engineering and School of Electrical and Computer Engineering Tel Aviv University Tel Aviv, Israel

  26. FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projection

    Oct 1, 2026Junkang LiuFederated Learning AggregationMemory-Efficient Fine-Tuning

    Tianjin University

  27. Beyond Uniform Compression: Budgeted Transmission Allocation for Extreme Federated Learning

    Sep 30, 2026Pengfei Li, Mohammad KhalilCommunication-Efficient Distributed TrainingLow-Rank Compression

    Centre for the Science of Learning & Technology (SLATE), University of Bergen, Bergen, Norway

  28. Importance-Aware Feature Sparsification for Wireless Split Learning

    Sep 30, 2026Bumjun Kim, Yoon Huh, Wan ChoiCommunication-Efficient Distributed TrainingSplit Federated Learning

    Department of Electrical and Computer Engineering, Seoul National University (SNU), and the Institute of New Media and Communications, SNU, Seoul 08826, Korea

  29. AutoLoCo: Communication Efficient Distributed LLM Training via Adaptive Synchronization

    Sep 29, 2026Pengyu He, Yan Zhang, Ruien Li +1Communication-Efficient Distributed TrainingDistributed Training

    Department of Electrical Engineering, California Institute of Technology · Department of Computer Science and Technology, Tsinghua University · Department of Computer Sciences, University of Wisconsin - Madison

  30. Communication-Efficient Agnostic Federated Learning via Faster Convergence and Compression

    Sep 29, 2026Haomin Bai, Junyan Sun, Sifan Yang +2Error FeedbackOnline Federated Learning

    State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · Department of Computer Science, City University of Hong Kong, Hong Kong, China