Communication-Efficient Distributed Training

Ask about this topic

Overview

This field studies how multiple devices can train a shared or coordinated model without spending too much time and bandwidth exchanging updates, activations, or parameters. Common approaches include doing more local optimization between synchronizations, compressing messages with quantization, sparsification, or low-rank projections, and exchanging compact alternatives such as logits or prototypes. Systems work also reduces communication costs through topology-aware collectives, pipeline scheduling, and overlapping data transfers with computation, especially for large language models and graph workloads. The central tension is to save communication while preserving convergence, accuracy, privacy, and robustness to heterogeneous data, unreliable links, and uneven hardware.

Latest papers

CardsList

230 papers30 papers in the last four weeks, up 43% on the four weeks before. 0.2% of all new papers.

  1. Catastrophic Forgetting Resilient One-Shot Incremental Federated Learning

    Feb 19, 2026Obaidullah Zaland, Zulfiqar Ahmad Khan, Monowar BhuyanFederated LearningFederated Continual Learning

    Department of Computing Science, Umeå University, Umeå, SE-90187, Sweden

  2. ZorBA: Zeroth-order Federated Fine-tuning of LLMs with Heterogeneous Block Activation

    Feb 19, 2026Chuiyang Meng, Ming Tang, Vincent W. S. WongFederated Fine-TuningMemory-Efficient Fine-Tuning

    Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, Canada · Department of Computer Science and Engineering, Southern University of Science and Technology, China

  3. Fractional-Order Federated Learning

    Feb 17, 2026Mohammad Partohaghighi, Roummel Marcia, YangQuan ChenFederated LearningFederated Learning Aggregation

    Electrical Engineering and Computer Science, University of California at Merced, USA · Department of Applied Mathematics, University of California Merced, Merced, CA, USA · Mechatronics, Embedded Systems and Automation (MESA) Lab, Department of Mechanical Engineering, School of Engineering, University of California, Merced, CA 95343, USA

  4. DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reduce

    Feb 9, 2026Wenchen Han, Shay Vargaftik, Michael Mitzenmacher +1Communication-Efficient Distributed TrainingHigh-Performance Computing

    University College London · VMware Research by Broadcom · Harvard University +1

  5. ERIS: Enhancing the Privacy-Utility-Scalability Trade-Off in Federated Learning via Federated Shard Aggregation

    Feb 9, 2026Dario Fenoglio, Pasquale Polverino, Akash Dhasade +3Federated LearningFederated Learning Aggregation

    Universit`a della Svizzera italiana Lugano, Switzerland · Carnegie Mellon University Pittsburgh, PA, United States

  6. Digital Twin-Driven Communication-Efficient Federated Anomaly Detection for Industrial IoT

    Jan 5, 2026Mohammed Ayalew Belay, Adil Rasheed, Pierluigi Salvo RossiCommunication-Efficient Distributed TrainingIndustrial Anomaly Detection

    Department of Electronic Systems, Norwegian University of Science and Technology, 7034 Trondheim, Norway

  7. Over-the-Air Federated Learning: Rethinking Edge AI Through Signal Processing

    Dec 3, 2025Seyed Mohammad Azimi-Abarghouyi, Carlo Fischione, Kaibin HuangFederated Learning AggregationCommunication-Efficient Distributed Training

    School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, Stockholm, Sweden · Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong SAR, China

  8. EDGC: Entropy-driven Dynamic Gradient Compression for Efficient LLM Training

    Nov 13, 2025Qingao Yi, Jiaang Duan, Jun Zhang +5Communication-Efficient Distributed TrainingGradient Compression

    University of Shanghai for Science and Technology, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China · Zhejiang University, Hangzhou, China

  9. A Byzantine-Robust Federated Learning with Nesterov Momentum

    Nov 4, 2025Lihan Xu, Xiaoyi Fan, Gang Wang +3Federated Learning AggregationByzantine-Robust Federated Learning

    School of Cyber Science and Engineering, Xi’an Jiaotong University, Xi’an 710049, China. · the Artificial Intelligence Research Institute and Guangdong-Hong Kong-Macao Joint Laboratory for Emotional Intelligence and Pervasive Computing, Shenzhen MSU-BIT University, Shenzhen 518172, China. · the State Key Laboratory of Autonomous Intelligent Unmanned Systems, Beijing Institute of Technology, Beijing 100081, China.

  10. OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Scheduling

    Sep 28, 2025Jihu Guo, Tenghui Ma, Wei Gao +6Communication-Efficient Distributed TrainingHeterogeneous Computing

    FDU & Shanghai AI Laboratory · HKUST · Independent Researcher +2

  11. An Efficient Subspace Algorithm for Federated Learning on Heterogeneous Data

    Sep 5, 2025Jiaojiao Zhang, Yizhao Fan, Yuqi Xu +1Communication-Efficient Distributed TrainingNon-IID Federated Learning

    Intelligent Computing Research Center, Great Bay University, Dongguan, China · Center for Machine Learning Research, Peking University, Beijing, China

  12. FedUHD: Unsupervised Federated Learning using In-Memory Hyperdimensional Computing

    Aug 16, 2025You Hak Lee, Keming Fan, Xiaofan Yu +4Compute-in-MemoryFederated Learning Aggregation

    University of California San Diego, USA

  13. FlexP-SFT: A Flexible Aggregation-Free Framework for On-Device Personalized Split Federated Fine-Tuning of LLMs

    Aug 14, 2025Jiaxiang Geng, Tianjun Yuan, Pengchao Han +3Communication-Efficient Distributed TrainingLLM Fine-Tuning

    Duke Kunshan University, Suzhou, China · Guangdong University of Technology, Guangzhou, China · Jade Bird Fire (JBF), China +1

  14. FedCoT: Communication-Efficient Federated Reasoning Enhancement for Large Language Models

    Aug 7, 2025Chuan Li, Qianyi Zhao, Fengran Mo +1Communication-Efficient Distributed TrainingMedical LLMs

    East China Normal University, China · University of Montreal, Canada

  15. Better Convergence Guarantees for Sign-Based Momentum Methods

    Jul 16, 2025Wei Jiang, Dingzhi Yu, Sifan Yang +3Sign-Based OptimizationMomentum Methods

    National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China

  16. NoLoCo: No-all-reduce Low Communication Training Method for Large Models

    Jun 12, 2025Jari Kolehmainen, Nikolay Blagoev, Semih Kara +3Communication-Efficient Distributed TrainingDecentralized Learning

    1Gensyn

  17. Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism

    Jun 2, 2025Sameera Ramasinghe, Thalaiyasingam Ajanthan, Gil Avraham +2Communication-Efficient Distributed TrainingDecentralized Learning

    Pluralis Research

  18. A cautionary tale on the cost-effectiveness of collaborative AI in real-world medical applications

    Dec 9, 2024Francesco Cremonesi, Lucia Innocenti, Sebastien Ourselin +3Communication-Efficient Distributed TrainingFederated Learning

    Epione Research Group, Inria Center of Universit´e Cˆote d’Azur, France · King’s College London, London, UK

  19. Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learning

    Oct 31, 2024Minghui Chen, Meirui Jiang, Xin Zhang +3Communication-Efficient Distributed TrainingModel Merging

    University of British Columbia · Vector Institute · Chinese University of Hong Kong +1

  20. The Limits and Potentials of Local SGD for Distributed Heterogeneous Learning with Intermittent Communication

    May 19, 2024Kumar Kshitij Patel, Margalit Glasgow, Ali Zindari +5Communication-Efficient Distributed TrainingDistributed Optimization

    Toyota Technological Institute, Chicago · Stanford University · CISPA Helmholtz Center +1