Communication-Efficient Distributed Training

Ask about this topic

Overview

This field studies how multiple devices can train a shared or coordinated model without spending too much time and bandwidth exchanging updates, activations, or parameters. Common approaches include doing more local optimization between synchronizations, compressing messages with quantization, sparsification, or low-rank projections, and exchanging compact alternatives such as logits or prototypes. Systems work also reduces communication costs through topology-aware collectives, pipeline scheduling, and overlapping data transfers with computation, especially for large language models and graph workloads. The central tension is to save communication while preserving convergence, accuracy, privacy, and robustness to heterogeneous data, unreliable links, and uneven hardware.

Latest papers

CardsList

230 papers30 papers in the last four weeks, up 43% on the four weeks before. 0.2% of all new papers.

  1. CARGO: Carbon-Aware Gossip Orchestration in Smart Shipping

    Mar 29, 2026Alexandros S. Kalafatelis, Nikolaos Nomikos, Vasileios Nikolakakis +2Carbon-Aware ComputingFederated Learning

    Department of Ports Management and Shipping, National and Kapodistrian University of Athens, 34400 Euboea, Greece · Laskaridis Shipping Co. Ltd, 14562, Kifisia Greece

  2. A Theoretical Framework for Energy-Aware Gradient Pruning in Federated Learning

    Mar 23, 2026Emmanouil M. AthanasakosFederated LearningEnergy-Efficient ML

    National and Kapodistrian University of Athens, Department of Informatics and Telecommunications, Athens, Greece

  3. Privacy-Preserving Federated Action Recognition via Differentially Private Selective Tuning and Efficient Communication

    Mar 22, 2026Idris Zakariyya, Pai Chet Ng, Kaushik Bhargav Sivangi +3Differentially Private Federated LearningVideo Action Recognition

    School of Computing Science, University of Glasgow, G12 8QQ, Glasgow, UK · Infocomm Technology Cluster, Singapore Institute of Technology, Singapore · The Edward S. Rogers Sr. Dept. of Electrical and Computer Engineering, University of Toronto, Canada

  4. Federated Hyperdimensional Computing for Resource-Constrained Industrial IoT

    Mar 20, 2026Nikita Zeulin, Olga Galinina, Nageen Himayat +1Industrial Internet of ThingsHyperdimensional Computing

    Tampere University, Tampere, Finland · Tampere Institute for Advanced Study, Tampere University, Finland · Intel Corporation, Santa Clara, CA, United States

  5. Byzantine-Robust and Communication-Efficient Distributed Training: Compressive and Cyclic Gradient Coding

    Mar 17, 2026Chengxi Li, Youssef Allouah, Rachid Guerraoui +2Distributed TrainingCommunication-Efficient Distributed Training

    Division of Information Science and Engineering, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, 10044 Stockholm, Sweden · Stanford University, CA 94305, United States · School of Computer and Communication Sciences, the Swiss Federal Institute of Technology in Lausanne (EPFL), 1015 Lausanne, Switzerland

  6. Joint Routing and Model Pruning for Decentralized Federated Learning in Bandwidth-Constrained Multi-Hop Wireless Networks

    Mar 16, 2026Xiaoyu He, Weicai Li, Tiejun Lv +1Federated LearningCommunication-Efficient Distributed Training

    School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing 100876, China · Center for Target Cognition Information Processing Science and Technology, and the Key Laboratory of Modern Measurement and Control Technology, Ministry of Education, both at Beijing Information Science and Technology University, Beijing, China

  7. Accelerating Byzantine-Robust Distributed Learning with Compressed Communication via Double Momentum and Variance Reduction

    Mar 16, 2026Yanghao Li, Changxin Liu, Yuhao YiDistributed OptimizationCommunication-Efficient Distributed Training

    Sichuan University · East China University of Science and Technology

  8. HO-SFL: Hybrid-Order Split Federated Learning with Backprop-Free Clients and Dimension-Free Aggregation

    Mar 16, 2026Qiyuan Chen, Xian Wu, Yi Wang +1Communication-Efficient Distributed TrainingZeroth-Order Optimization

    Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China.

  9. SCOPE: Scalable Training-Free Semantic Coreset Selection for Long-Tailed Federated Learning

    Mar 13, 2026Md Anwar Hossen, Nathan R. Tallent, Luanzheng Guo +1Federated LearningCommunication-Efficient Distributed Training

    Iowa State University, Ames, Iowa, USA · Pacific Northwest National Laboratory, Richland, WA, USA

  10. CA-HFP: Curvature-Aware Heterogeneous Federated Pruning with Model Reconstruction

    Mar 13, 2026Gang Hu, Yinglei Teng, Pengfei Wu +1Federated LearningHeterogeneous Federated Learning

  11. Prioritizing Gradient Sign Over Modulus: An Importance-Aware Framework for Wireless Federated Learning

    Mar 11, 2026Yiyang Yue, Jiacheng Yao, Wei Xu +3Federated LearningCommunication-Efficient Distributed Training

    National Mobile Communications Research Laboratory (NCRL), Southeast University, Nanjing 210096, China · Purple Mountain Laboratories, Nanjing 211111, China · Zhejiang Lab, Hangzhou 311121, China +3

  12. FedLECC: Cluster- and Loss-Guided Client Selection for Federated Learning under Non-IID Data

    Mar 9, 2026Daniel M. Jimenez-Gutierrez, Giovanni Giunta, Mehrdad Hassanzadeh +3Client Selection in Federated LearningNon-IID Federated Learning

    Sapienza University of Rome Via Ariosto 25 Rome, 00185, Italy

  13. Split Federated Learning Architectures for High-Accuracy and Low-Delay Model Training

    Mar 9, 2026Yiannis Papageorgiou, Yannis Thomas, Ramin Khalili +1Split Federated LearningFederated Learning

    Department of Informatics, Athens University of Economics and Business, Athens, Greece · Huawei Technologies Duesseldorf Gmbh, Munich, Germany

  14. Scalable Training of Mixture-of-Experts Models with Megatron Core

    Mar 8, 2026Zijie Yan, Hongxiao Bai, Xin Yao +42Low-Precision TrainingMixture-of-Experts Models

    NVIDIA

  15. Compressed Proximal Federated Learning for Non-Convex Composite Optimization on Heterogeneous Data

    Mar 8, 2026Pu Qiu, Chen Ouyang, Yongyang Xiong +3Error FeedbackFederated Learning

    School of Intelligent Systems Engineering, Sun Yat-Sen University, Shenzhen 518107, P.R China. · Department of Automation, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China. · Department of Mechanical Engineering, University of Victoria, Victoria, BC V8W 2Y2, Canada.

  16. NEST: Network- and Memory-Aware Device Placement For Distributed Deep Learning

    Mar 6, 2026Irene Wang, Vishnu Varma Venkata, Arvind Krishnamurthy +1Distributed TrainingCommunication-Efficient Distributed Training

    Georgia Institute of Technology, Atlanta, GA, USA · University of Washington, Seattle, WA, USA

  17. Heterogeneous Decentralized Diffusion Models

    Mar 6, 2026Zhiying Jiang, Raihan Seraj, Marcos Villagra +1Diffusion ModelsCommunication-Efficient Distributed Training

    Bagel Labs

  18. Semantic Communication-Enhanced Split Federated Learning for Vehicular Networks: Architecture, Challenges, and Case Study

    Mar 5, 2026Lu Yu, Zheng Chang, Ying-Chang LiangSplit Federated LearningFederated Learning

    School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China · Faculty of Information Technology, University of Jyväskylä, FIN-40014 Jyväskylä, Finland · Center for Intelligent Networking and Communications (CINC), University of Electronic Science and Technology of China, 611731 Chengdu, China

  19. EdgeFLow: Serverless Federated Learning via Sequential Model Migration in Edge Networks

    Mar 3, 2026Yuchen Shi, Qijun Hou, Pingyi Fan +1Federated LearningCommunication-Efficient Distributed Training

    Dept. of Electronic Engineering, BNRist, Tsinghua University. · Dept. of Electronic and Computer Engineering, HKUST.

  20. CUCo: An Agentic Framework for Compute and Communication Co-design

    Mar 2, 2026Yoga Sri Varshan Varadharajan, Bodun Hu, Saurabh Agarwal +1GPU Kernel OptimizationLLM Inference Acceleration

    UT Austin

  21. Decentralized Federated Learning by Partial Message Exchange

    Mar 2, 2026Shan Sha, Shenglong Zhou, Xin Wang +2Communication-Efficient Distributed TrainingFederated Learning

    School of Mathematics and Statistics, Beijing Jiaotong University, Beijing, China. · Department of Electrical and Electronic Engineering, Faculty of Engineering, Imperial College London, London, U.K.

  22. Rudder: Steering Prefetching in Distributed GNN Training using LLM Agents

    Feb 26, 2026Aishwarya Sarkar, Sayan Ghosh, Nathan Tallent +3Communication-Efficient Distributed TrainingGraph Neural Networks

    Iowa State University, Ames, IA, USA · Pacific Northwest National Laboratory, Richland, WA, USA · Amazon GenAI, Cupertino, CA, USA +1

  23. veScale-FSDP: Flexible and High-Performance FSDP at Scale

    Feb 25, 2026Zezhou Wang, Youjie Li, Zhiqi Lin +9Distributed TrainingCommunication-Efficient Distributed Training

    ByteDance Seed

  24. Learning to Collaborate via Structures: Cluster-Guided Item Alignment for Federated Recommendation

    Feb 25, 2026Yuchun Tu, Zhiwei Li, Bingli Sun +2Recommender SystemsFederated Learning

    Beihang University, Beijing, China · University of Technology Sydney, Sydney, Australia · University of Edinburgh, Edinburgh, UK

  25. GFPL: Generative Federated Prototype Learning for Resource-Constrained and Data-Imbalanced Vision Task

    Feb 25, 2026Shiwei Lu, Yuhang He, Jiashuo Li +2Federated LearningFederated Learning Aggregation

    Xi’an Jiaotong University · Air Force Engineering University

  26. Efficient Scaling of LLM Training with Flexible Context Parallelism

    Feb 25, 2026Yifan Niu, Han Xiao, Dongyi Liu +2Context ParallelismEfficient Language Model Training

    The Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Co., Ltd. · The Hong Kong University of Science and Technology

  27. Benchmarking Federated Learning in Edge Computing Environments: A Systematic Review and Performance Evaluation

    Feb 24, 2026Sales Aribe, Gil Nicholas CagandeFederated LearningCommunication-Efficient Distributed Training

    Information Technology Department, Bukidnon State University, Malaybalay City, Philippines

  28. Event-Triggered Gossip for Distributed Learning

    Feb 22, 2026Zhiyuan Zhai, Xiaojun Yuan, Wei Ni +3Communication-Efficient Distributed TrainingDecentralized Learning

  29. Communication-Efficient Personalized Adaptation via Federated-Local Model Merging

    Feb 20, 2026Yinan Zou, Md Kamran Chowdhury Shisher, Christopher G. Brinton +1Federated LearningFederated Fine-Tuning

    Elmore Family School of Electrical and Computer Engineering, Purdue University.

  30. SeedFlood: A Step Toward Scalable Decentralized Fine-Tuning of LLMs

    Feb 20, 2026Jihun Kim, Dongyeop Lee, Namhoon LeeCommunication-Efficient Distributed TrainingZeroth-Order Optimization

    POSTECH, pohang, Republic of Korea.