Communication-Efficient Distributed Training

Follow this topic Ask about this topic

Momentum

20 papers in the last four weeks, up 300% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 151

All topics

TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Training
Apr 27, 2026Man Liu, Xingchen Liu, Xingjian Tian +8Communication-Efficient Distributed Training Tensor Parallelism
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences · University of Science and Technology of China
FreeScale: Distributed Training for Sequence Recommendation Models with Minimal Scaling Cost
Apr 27, 2026Chenhao Feng, Haoli Zhang, Shakhzod Ali-Zade +17Sequential Recommendation Communication-Efficient Distributed Training
1Meta Inc.
CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Training
Apr 27, 2026Rezaul Karim, Austin Wen, Wang Zongzuo +3Communication-Efficient Distributed Training Distributed Optimization
Toronto Ascend Team
FedSLoP: Memory-Efficient Federated Learning with Low-Rank Gradient Projection
Apr 27, 2026Yutong He, Zhengyang Huang, Jiahe Geng +1Communication-Efficient Distributed Training Gradient Compression
Peking University · Beihang University
Scalable and Verifiable Federated Learning for Cross-Institution Financial Fraud Detection
Apr 25, 2026Prajwal Panth, Nishant NigamFederated Learning Aggregation Communication-Efficient Distributed Training
School of Computer Engineering, KIIT Deemed to be University, Bhubaneswar, India · School of Electronics Engineering, KIIT Deemed to be University, Bhubaneswar, India
Efficient Federated RLHF via Zeroth-Order Policy Optimization
Apr 20, 2026Deyi Wang, Qining Zhang, Lei YingFederated RL Communication-Efficient Distributed Training
University of Michigan, Ann Arbor
UCCL-Zip: Lossless Compression Supercharged GPU Communication
Apr 19, 2026Shuang Ma, Chon Lam Lao, Zhiying Xu +8GPU Kernel Optimization GPU Acceleration
UC Davis · Harvard University · Amazon Web Services +1
Robust Synchronisation for Federated Learning in The Face of Correlated Device Failure
Apr 17, 2026Stefan Behfar, Richard MortierCommunication-Efficient Distributed Training Group Fairness
Computer Lab, University of Cambridge, Cambridge, United Kingdom
Breaking the Training Barrier of Billion-Parameter Universal Machine Learning Interatomic Potentials
Apr 17, 2026Yuanchang Zhou, Hongyu Wang, Yiming Du +12Machine Learning Interatomic Potentials Communication-Efficient Distributed Training
State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences · Independent Researcher · Sun Yat-Sen University
Optimizing Stochastic Gradient Push under Broadcast Communications
Apr 16, 2026Tuan Nguyen, Ting HeCommunication-Efficient Distributed Training Federated Learning
Pennsylvania State University State College, PA, USA
Exploiting Correlations in Federated Learning: Opportunities and Practical Limitations
Apr 16, 2026Adrian Edin, Michel Kieffer, Mikael Johansson +1Communication-Efficient Distributed Training Gradient Compression
Department of Electrical Engineering (ISY), Linköping University, SE-581 83 Linköping, Sweden · CentraleSupélec, CNRS, L2S, Université Paris-Saclay, Gif-sur-Yvette, FR-91192, France · Division of Decision and Control Systems, School of EECS, KTH Royal Institute of Technology, SE-100 44 Stockholm, Sweden
TensorHub: Scalable and Elastic Weight Transfer for LLM RL Training
Apr 10, 2026Chenhao Ye, Huaizheng Zhang, Mingcong Han +11RL for Language Models Communication-Efficient Distributed Training
ByteDance Seed · University of Wisconsin–Madison
HO-SFL: Hybrid-Order Split Federated Learning with Backprop-Free Clients and Dimension-Free Aggregation
Mar 16, 2026Qiyuan Chen, Xian Wu, Yi Wang +1Communication-Efficient Distributed Training Zeroth-Order Optimization
Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China.
Decentralized Federated Learning by Partial Message Exchange
Mar 2, 2026Shan Sha, Shenglong Zhou, Xin Wang +2Communication-Efficient Distributed Training Federated Learning
School of Mathematics and Statistics, Beijing Jiaotong University, Beijing, China · Department of Electrical and Electronic Engineering, Faculty of Engineering, Imperial College London, London, U.K.
SeedFlood: A Step Toward Scalable Decentralized Fine-Tuning of LLMs
Feb 20, 2026Jihun Kim, Dongyeop Lee, Namhoon LeeCommunication-Efficient Distributed Training Zeroth-Order Optimization
POSTECH, pohang, Republic of Korea.
DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reduce
Feb 9, 2026Wenchen Han, Shay Vargaftik, Michael Mitzenmacher +1Communication-Efficient Distributed Training High-Performance Computing
University College London · VMware Research by Broadcom · Harvard University +1
Digital Twin-Driven Communication-Efficient Federated Anomaly Detection for Industrial IoT
Jan 5, 2026Mohammed Ayalew Belay, Adil Rasheed, Pierluigi Salvo RossiCommunication-Efficient Distributed Training Industrial Anomaly Detection
Department of Electronic Systems, Norwegian University of Science and Technology, 7034 Trondheim, Norway
Over-the-Air Federated Learning: Rethinking Edge AI Through Signal Processing
Dec 3, 2025Seyed Mohammad Azimi-Abarghouyi, Carlo Fischione, Kaibin HuangFederated Learning Aggregation Communication-Efficient Distributed Training
School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, Stockholm, Sweden · Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong SAR, China
EDGC: Entropy-driven Dynamic Gradient Compression for Efficient LLM Training
Nov 13, 2025Qingao Yi, Jiaang Duan, Jun Zhang +5Communication-Efficient Distributed Training Gradient Compression
University of Shanghai for Science and Technology, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China · Zhejiang University, Hangzhou, China
A Nesterov-Accelerated Byzantine-Robust Federated Learning
Nov 4, 2025Lihan Xu, Xiaoyi Fan, Gang Wang +3Federated Learning Aggregation Byzantine-Robust Federated Learning
School of Cyber Science and Engineering, Xi’an Jiaotong University, Xi’an 710049, China · Artificial Intelligence Research Institute and Guangdong-Hong Kong-Macao Joint Laboratory for Emotional Intelligence and Pervasive Computing, Shenzhen MSU-BIT University, Shenzhen 518172, China · State Key Laboratory of Autonomous Intelligent Unmanned Systems, Beijing Institute of Technology, Beijing 100081, China
OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Scheduling
Sep 28, 2025Jihu Guo, Tenghui Ma, Wei Gao +6Communication-Efficient Distributed Training Heterogeneous Computing
FDU & Shanghai AI Laboratory · HKUST · Independent Researcher +2
An Efficient Subspace Algorithm for Federated Learning on Heterogeneous Data
Sep 5, 2025Jiaojiao Zhang, Yizhao Fan, Yuqi Xu +1Communication-Efficient Distributed Training Non-IID Federated Learning
Intelligent Computing Research Center, Great Bay University, Dongguan, China · Center for Machine Learning Research, Peking University, Beijing, China
FedUHD: Unsupervised Federated Learning using In-Memory Hyperdimensional Computing
Aug 16, 2025You Hak Lee, Keming Fan, Xiaofan Yu +4Compute-in-Memory Federated Learning Aggregation
University of California San Diego, USA
FlexP-SFT: A Flexible Aggregation-Free Framework for On-Device Personalized Split Federated Fine-Tuning of LLMs
Aug 14, 2025Jiaxiang Geng, Tianjun Yuan, Pengchao Han +3Communication-Efficient Distributed Training LLM Fine-Tuning
Duke Kunshan University, Suzhou, China · Guangdong University of Technology, Guangzhou, China · Jade Bird Fire (JBF), China +1
FedCoT: Communication-Efficient Federated Reasoning Enhancement for Large Language Models
Aug 7, 2025Chuan Li, Qianyi Zhao, Fengran Mo +1Communication-Efficient Distributed Training Medical LLMs
East China Normal University, China · University of Montreal, Canada
Better Convergence Guarantees for Sign-Based Momentum Methods
Jul 16, 2025Wei Jiang, Dingzhi Yu, Sifan Yang +3Sign-Based Optimization Momentum Methods
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China
NoLoCo: No-all-reduce Low Communication Training Method for Large Models
Jun 12, 2025Jari Kolehmainen, Nikolay Blagoev, Semih Kara +3Communication-Efficient Distributed Training Decentralized Learning
1Gensyn
Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism
Jun 2, 2025Sameera Ramasinghe, Thalaiyasingam Ajanthan, Gil Avraham +2Communication-Efficient Distributed Training Decentralized Learning
Pluralis Research
A cautionary tale on the cost-effectiveness of collaborative AI in real-world medical applications
Dec 9, 2024Francesco Cremonesi, Lucia Innocenti, Sebastien Ourselin +3Communication-Efficient Distributed Training Federated Learning
Epione Research Group, Inria Center of Universit´e Cˆote d’Azur, France · King’s College London, London, UK
Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learning
Oct 31, 2024Minghui Chen, Meirui Jiang, Xin Zhang +3Communication-Efficient Distributed Training Model Merging
University of British Columbia · Vector Institute · Chinese University of Hong Kong +1
The Limits and Potentials of Local SGD for Distributed Heterogeneous Learning with Intermittent Communication
May 19, 2024Kumar Kshitij Patel, Margalit Glasgow, Ali Zindari +5Communication-Efficient Distributed Training Distributed Optimization
Toyota Technological Institute, Chicago · Stanford University · CISPA Helmholtz Center +1

Apr 27, 2026, cs.DCJ/K move · Enter open · S save

TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Training

Man Liu, Xingchen Liu, Xingjian Tian, Bing Lu+7

Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences · University of Science and Technology of China

Communication-Efficient Distributed Training Tensor Parallelism Distributed Training

Open paper AskPDF arXiv

Handling communication overhead in large-scale tensor-parallel training remains a critical challenge due to the dense, near-zero distributions of intermediate tensors, which exacerbate errors under frequent communication and introduce significant computational overhead during compression. To this end, we propose TACO (Tensor-parallel Adaptive COmmunication compression), a robust FP8-based framework for compressing TP intermediate tensors. First, we employ a data-driven reshaping strategy combined with an Adaptive Scale-Hadamard Transform to enable high-fidelity FP8 quantization, while its Dual-Scale Quantization mechanism ensures numerical stability throughout training. Second, we design a highly fused compression operator to reduce memory traffic and kernel launch overhead, allowing efficient overlap with communication. Finally, we integrate TACO with existing state-of-the-art methods for Data and Pipeline Parallelism to develop a compression-enabled 3D-parallel training framework. Detailed experiments on GPT models and Qwen model demonstrate up to 1.87X end-to-end throughput improvement while maintaining near-lossless accuracy, validating the effectiveness and efficiency of TACO in large-scale training.

Momentum

Related topics

Latest papers 151