Communication-Efficient Distributed Training

Overview

This field studies how multiple devices can train a shared or coordinated model without spending too much time and bandwidth exchanging updates, activations, or parameters. Common approaches include doing more local optimization between synchronizations, compressing messages with quantization, sparsification, or low-rank projections, and exchanging compact alternatives such as logits or prototypes. Systems work also reduces communication costs through topology-aware collectives, pipeline scheduling, and overlapping data transfers with computation, especially for large language models and graph workloads. The central tension is to save communication while preserving convergence, accuracy, privacy, and robustness to heterogeneous data, unreliable links, and uneven hardware.

Latest papers

CardsList

245 papers20 papers in the last four weeks, down 37% on the four weeks before. 0.2% of all new papers.

  1. Communication-Efficient Distributionally Robust Federated Averaging with Multi-Source Data

    Nov 15, 2026Yingzhu Liu, Zhongkui Li, Pengcheng You +1Federated LearningFederated Learning Aggregation

  2. LACCO: Local Accumulation with Computation--Communication Overlap

    Nov 13, 2026Daniele Kopyshevskiy, Benjamin Th{é}rien, Nicolas Pinto +1Communication-Efficient Distributed TrainingDistributed Training

  3. Train High, Ship Low: Efficient LoRA Synchronization for Distributed On-Policy Post-Training

    Nov 9, 2026Shenghui Li, Li Ju, Thiemo VoigtCommunication-Efficient Distributed TrainingGradient Compression

  4. Wireless AI Networks: Co-Adapting Communication Resources and Distributed Intelligence

    Nov 9, 2026Ziqiang Ye, Yue Xiao, Qiyan Zhan +3Wireless Resource AllocationWireless Communications

  5. Network Contention State Aware Dynamic Optimizer for Accelerator Collective Communication Libraries

    Nov 7, 2026Rajat Bisht, Tevfik Kosar, Anthony Skjellum +2Communication-Efficient Distributed Training

  6. DisMUSE: An Efficient Distributed Singular-Spectrum-Based Optimization Approach with Theoretical Convergence Guarantee

    Nov 6, 2026Zhen Qin, Zhiyao Zhang, Yingbin Liang +1Distributed OptimizationSpectral Optimization

  7. Compressed Per-Player Local Methods for Multiplayer Federated Learning: Exact Convergence via Increment Sharing

    Nov 5, 2026Bhavika Balani, TaeHo Yoon, Nicolas LoizouFederated LearningCommunication-Efficient Distributed Training

  8. Split-KV: Efficient Context Parallelism for Training Diffusion Drafters

    Nov 4, 2026Geng Zhang, Fangzhao Zhang, Cong Xie +2Context ParallelismCommunication-Efficient Distributed Training

  9. Scheduling Data to Compute: Hierarchical Data Pipelining for LLM Training on AMD GPUs

    Nov 3, 2026Gengyu Rao, Ghadeer Ahmed H Alabandi, Su Ann Chong +28LLM TrainingGPU Kernel Optimization

  10. FP4Quant: Efficient 4-Bit Communication Compression for Model-Parallel LLM Training

    Nov 3, 2026Man Liu, Xingchen Liu, Hairui Zhao +2Communication-Efficient Distributed Training4-Bit Quantization

  11. ParaBlock++: Communication-Computation Parallel Blockwise Hierarchical Federated Learning

    Oct 31, 2026Kai Wang, Chee Wei TanBlock Coordinate DescentFederated Learning

  12. Representative Learning for Distributed Support Vector Machines

    Oct 31, 2026Joseph A. Casey, Boxiang Wang, Keren LiSupport Vector MachinesCommunication-Efficient Distributed Training

  13. Connectivity-Aware Client Selection in Decentralized Federated Learning

    Oct 30, 2026Usevalad Milasheuski, Monica Nicoli, Stefano SavazziClient Selection in Federated LearningDecentralized Learning

  14. Dobb: Shard-Local Preconditioning in Block-Diagonal Eigenbases

    Oct 29, 2026Shi-Yuk Fong, Shuhua Yu, Jiawei ZhangDistributed OptimizationCommunication-Efficient Distributed Training

  15. The Endgame of FSDP Communication: Layout-Aware Collectives on Copy Engines, RDMA, and NVSwitch

    Oct 28, 2026Zejian Wang, Yifan Zhao, Kaihui Ren +11Communication-Efficient Distributed TrainingDistributed Training

  16. Democratizing LLM Training with the Edge: Challenges, Systems, and the Road Ahead

    Oct 23, 2026Ziyue Luo, Jiaxuan Cai, Cedric LeDenmat +27LLM TrainingDistributed Training

  17. On Johnson-Lindenstrauss Transforms for Byzantine-Robust Optimization

    Oct 20, 2026Aleksandr Shestakov, Vladimir Aletov, Aleksandr BeznosikovRobust OptimizationDistributed Optimization

  18. Bridging Compressed Communication and Local Steps via Spectral Method in Federated Learning

    Oct 16, 2026Dmitriy Bystrov, George Chirkov, Daniil Medyakov +2Federated LearningCommunication-Efficient Distributed Training

  19. Understanding Stability of Outer Nesterov Momentum on Stochastic Pseudo-Gradients

    Oct 15, 2026Nishant Sharma, Laura Gomezjurado Gonzalez, Ganesh Talluri +2Neural Network StabilityNeural Network Training Dynamics

  20. ETH: Data-Balanced Heterogeneous Pipeline Parallelism for Training Large Models

    Oct 15, 2026Mianjie Yu, Zhongmin Zhao, Zizhao Mo +9Distributed TrainingPipeline Parallelism

  21. How Inner Optimizers Shape Replica Scaling

    Oct 13, 2026Eugene Belilovsky, Stefan Horoi, Edouard Oyallon +1LLM TrainingDistributed Training

  22. A Dependency-Aware Multi-GPU Training Framework for Temporal Graph Neural Networks

    Oct 11, 2026Yang Zhao, Yue Dai, Liang Qiao +8Temporal GNNsDistributed Training

  23. BSIPipe: Bidirectional Sequence-Level Interleaved Pipeline Parallelism for Long-Sequence Large Language Model Training

    Oct 10, 2026Yuzhou Ni, Ling ChenPipeline ParallelismLLM Training