Communication Cost

Communication cost, the overhead associated with transmitting data in distributed computing and machine learning, is a critical bottleneck in many applications. Current research focuses on reducing this cost through techniques like model compression (e.g., using diffusion models or quantization), efficient aggregation strategies (e.g., hierarchical communication or aggregation before communication), and optimized communication protocols (e.g., event-triggered synchronization or all-to-all based gradient averaging). Addressing communication cost is crucial for enabling scalable and efficient distributed systems, particularly in resource-constrained environments like federated learning and large language model training, impacting both the feasibility and performance of these technologies.

Papers

October 26, 2024

Anatomical 3D Style Transfer Enabling Efficient Federated Learning with Extremely Low Communication Costs
Yuto Shibata, Yasunori Kudo, Yohei Sugawara
Multi Organ Segmentation Communication Cost 3D Style Transfer Multi Organ Class Diversity

September 29, 2024

DiffCP: Ultra-Low Bit Collaborative Perception via Diffusion Model
Ruiqing Mao, Haotian Wu, Yukuan Jia, Zhaojun Nan, Yuxuan Sun, Sheng Zhou, Deniz Gündüz, Zhisheng Niu
Diffusion Model Collaborative Filtering Collaborative Perception Communication Cost Feature Swapping

August 1, 2024

CDFGNN: a Systematic Design of Cache-based Distributed Full-Batch Graph Neural Network Training with Communication Reduction
Shuai Zhang, Zite Jiang, Haihang You
Design Principle Communication Cost Full Batch Gradient Descent

April 22, 2024

Distributed Learning for Wi-Fi AP Load Prediction
Dariush Salami, Francesc Wilhelmi, Lorenzo Galati-Giordano, Mika Kasslin
Machine Learning LeArning Abstract Knowledge Distillation Communication Cost

December 22, 2023

Federated Q-Learning: Linear Regret Speedup with Low Communication Cost
Zhong Zheng, Fengyu Gao, Lingzhou Xue, Jing Yang
Reinforcement Learning Optimal Policy Single Agent Episodic Markov Decision Process Communication Cost

December 19, 2023

Convergence Visualizer of Decentralized Federated Distillation with Reduced Communication Costs
Akihito Taya, Yuuki Nishiyama, Kaoru Sezaki
Low Temperature Distillation Model Convergence Federated Distillation Decentralized Algorithm Consensus Algorithm Communication Cost Mass Convergence Event

December 12, 2023

Communication Cost Reduction for Subgraph Counting under Local Differential Privacy via Hash Functions
Quentin Hillebrand, Vorapong Suppakitpaisarn, Tetsuo Shibuya
Differential Privacy Subgraph Structure Local Differential Privacy Large Graph Communication Cost Hash Function Graph Structure Information Subgraph Counting

December 11, 2023

Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes
Zhen Qin, Daoyuan Chen, Bingchen Qian, Bolin Ding, Yaliang Li, Shuiguang Deng
Large Language Model Communication Cost Sub Byte Federated Tuning

December 6, 2023

Revisiting the Optimality of Word Lengths
Tiago Pimentel, Clara Meister, Ethan Gotlieb Wilcox, Kyle Mahowald, Ryan Cotterell
Near Optimality Communication Cost Word Length Channel Capacity

October 15, 2023

FLrce: Resource-Efficient Federated Learning with Early-Stopping Strategy
Ziru Niu, Hai Dong, A. Kai Qin, Tao Gu
Deep Learning Federated Learning Early Stopping Communication Cost Global Deep

October 9, 2023

Rethinking Memory and Communication Cost for Efficient Large Language Model Training
Chan Wu, Hanxiao Zhang, Lin Ju, Jinjing Huang, Youshao Xiao, Zhaoxin Huan, Siyuan Li, Fanzhuang Meng, Lei Liang, Xiaolu Zhang, Jun Zhou
Large Language Model Communication Cost Large Language Model Training Whale Optimization Algorithm Memory and Communication

August 8, 2023

Cooperative Multi-agent Bandits: Distributed Algorithms with Optimal Individual Regret and Constant Communication Costs
Lin Yang, Xuchuang Wang, Mohammad Hajiesmaili, Lijun Zhang, John C. S. Lui, Don Towsley
Practical Algorithm Simple Regret Multi Agent Multi Armed Bandit Communication Cost Cooperative Bandit

June 21, 2023

Mitigating Communication Costs in Neural Networks: The Role of Dendritic Nonlinearity
Xundong Wu, Pengfei Zhao, Zilin Yu, Lei Ma, Ka-Wa Yip, Huajin Tang, Gang Pan, Tiejun Huang
Neural Network Integral Role Communication Cost Dendritic Tree Dendritic Like Structure Dendritic Neuron

June 16, 2023

ZeRO++: Extremely Efficient Collective Communication for Giant Model Training
Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Connor Holmes, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, Yuxiong He
Model Training Better Zero GPU Cluster Communication Cost Collective Communication Reduce SCATTER Block Quantization

May 29, 2023

Reducing Communication for Split Learning by Randomized Top-k Sparsification
Fei Zheng, Chaochao Chen, Lingjuan Lyu, Binhui Yao
Timely Communication Vertical Federated Learning Split Learning Communication Cost Top K Sparsification

April 13, 2023

Decentralized federated learning methods for reducing communication cost and energy consumption in UAV networks
Deng Pan, Mohammad Ali Khoshkholghi, Toktam Mahmoodi
Single Drone Energy Consumption Decentralized Federated Learning Decentralized FL UAV Network Learning Method Communication Cost

March 27, 2023

Asynchronous Online Federated Learning with Reduced Communication Requirements
Francois Gauthier, Vinay Chakravarthi Gogineni, Stefan Werner, Yih-Fang Huang, Anthony Kuh
Asynchronous Federated Learning Random Participation Computational Power Communication Cost

January 21, 2023

A Communication-Efficient Adaptive Algorithm for Federated Learning under Cumulative Regret
Sudeep Salgia, Qing Zhao, Tamir Gabay, Kobi Cohen
Simple Regret Online Learning Algorithm Communication Cost Order Optimal Regret Online Stochastic

December 11, 2022

ABC: Aggregation before Communication, a Communication Reduction Framework for Distributed Graph Neural Network Training and Effective Partition
Junwei Su
Timely Communication Graph Neural Data Aggregation Large Graph GNN Training Communication Cost GNN Layer GNN Training Framework

December 4, 2022

Quantum Federated Learning with Entanglement Controlled Circuits and Superposition Coding
Won Joon Yun, Jae Pyoung Kim, Hankyul Baek, Soyi Jung, Jihong Park, Mehdi Bennis, Joongheon Kim
Quantum Circuit Quantum Computer Intermediate Scale Quantum Communication Cost Quantum Federated Learning

Communication Cost

Papers

Anatomical 3D Style Transfer Enabling Efficient Federated Learning with Extremely Low Communication Costs

DiffCP: Ultra-Low Bit Collaborative Perception via Diffusion Model

CDFGNN: a Systematic Design of Cache-based Distributed Full-Batch Graph Neural Network Training with Communication Reduction

Distributed Learning for Wi-Fi AP Load Prediction

Federated Q-Learning: Linear Regret Speedup with Low Communication Cost

Convergence Visualizer of Decentralized Federated Distillation with Reduced Communication Costs

Communication Cost Reduction for Subgraph Counting under Local Differential Privacy via Hash Functions

Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes

Revisiting the Optimality of Word Lengths

FLrce: Resource-Efficient Federated Learning with Early-Stopping Strategy

Rethinking Memory and Communication Cost for Efficient Large Language Model Training

Cooperative Multi-agent Bandits: Distributed Algorithms with Optimal Individual Regret and Constant Communication Costs

Mitigating Communication Costs in Neural Networks: The Role of Dendritic Nonlinearity

ZeRO++: Extremely Efficient Collective Communication for Giant Model Training

Reducing Communication for Split Learning by Randomized Top-k Sparsification

Decentralized federated learning methods for reducing communication cost and energy consumption in UAV networks

Asynchronous Online Federated Learning with Reduced Communication Requirements

A Communication-Efficient Adaptive Algorithm for Federated Learning under Cumulative Regret

ABC: Aggregation before Communication, a Communication Reduction Framework for Distributed Graph Neural Network Training and Effective Partition

Quantum Federated Learning with Entanglement Controlled Circuits and Superposition Coding