Communication-Efficient Distributed Training

Follow this topic Ask about this topic

Momentum

20 papers in the last four weeks, up 300% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 151

All topics

F-TIS: Harnessing Diverse Models in Collaborative GRPO
May 21, 2026Nikolay Blagoev, Oğuzhan Ersoy, Wendelin Boehmer +1RL for Language Models Communication-Efficient Distributed Training
Gensyn · University of Neuchatel · TU Delft
Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search
May 20, 2026Chaimaa Medjadji, Sylvain Kubler, Yves Le Traon +3Communication-Efficient Distributed Training Federated Knowledge Distillation
Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, Luxembourg · Blekinge Institute of Technology, Sweden · ADSLabs, Umea University, Sweden
LOSCAR-SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging
May 20, 2026Yassine Maziane, Ammar Mahran, Artavazd Maranjyan +1Communication-Efficient Distributed Training Distributed Optimization
KAUST
A Tabular Schedule Abstraction for Communication-Aware Evaluation of Pipeline-Parallel LLM Training
May 19, 2026Daniel Barley, Jonathan Leis, Benjamin Klenk +1Communication-Efficient Distributed Training Pipeline Parallelism
Hardware and Artificial Intelligence (HAWAII) Lab, Heidelberg University, Heidelberg, Germany · NVIDIA Corporation, Santa Clara, CA, USA
Statistical Limits and Efficient Algorithms for Differentially Private Federated Learning
May 18, 2026Arnab Auddy, Xiangni Peng, Subhadeep PaulFederated Learning Aggregation Communication-Efficient Distributed Training
Department of Statistics · The Ohio State University
TSFLora: Token-Compressed Split Fine-Tuning for Wireless Edge Networks
May 17, 2026Xianke Qiang, Zheng Chang, Li Wang +1Communication-Efficient Distributed Training Edge Computing
School of Computer Science and Engineering, UESTC, 611731 Chengdu, China · School of Computer Science, Beijing University of Posts and Telecommunications, Beijing, 100876, China. · Center for Intelligent Networking and Communications, UESTC, 611731 Chengdu, China.
Towards Family-Grouped Hierarchical Federated Learning on Sub-5KB Models: A Feasibility Study of Privacy-Preserving ECG Monitoring for Ultra-Resource-Constrained Wearables
May 15, 2026Hangyu WuHierarchical Federated Learning Communication-Efficient Distributed Training
Shenzhen Coddie Technology co.,ltd, Shenzhen, 518000, China
BiFedKD: Bidirectional Federated Knowledge Distillation Framework for Non-IID and Long-Tailed ECG Monitoring
May 14, 2026Zixuan Shu, Tiancheng Cao, Hen-Wei HuangLong-Tail Learning Communication-Efficient Distributed Training
School of Electrical and Electronic Engineering, Nanyang Technological University, Republic of Singapore · Lee Kong Chian School of Medicine, Nanyang Technological University, Republic of Singapore
DisAgg: Distributed Aggregators for Efficient Secure Aggregation in Federated Learning
May 13, 2026Haaris Mehmood, Giorgos Tatsis, Dimitrios Alexopoulos +4Federated Learning Aggregation Communication-Efficient Distributed Training
Samsung R&D Institute UK (SRUK) · Information Technologies Institute (CERTH-ITI)
TurboGR: An Accelerated Training System for Large-Scale Generative Recommendation
May 13, 2026Huichao Chai, Zhixin Wu, Xuemiao Li +8Communication-Efficient Distributed Training Generative Recommendation
Huawei
D-VLA: A High-Concurrency Distributed Asynchronous Reinforcement Learning Framework for Vision-Language-Action Models
May 13, 2026Yucheng Guo, Yongjian Guo, Zhong Guan +9Reinforcement Learning Communication-Efficient Distributed Training
JDT AI Infra · Tsinghua University · Tianjin University +2
LightSplit: Practical Privacy-Preserving Split Learning via Orthogonal Projections
May 13, 2026Mert Cihangiroglu, Alessandro Pegoraro, Phillip Rieger +2Communication-Efficient Distributed Training Privacy-Preserving ML
University of Pavia · 2Technical University of Darmstadt
Federated Language Models Under Bandwidth Budgets: Distillation Rates and Conformal Coverage
May 11, 2026Prasanjit Dubey, Xiaoming HuoRetrieval-Augmented Generation Communication-Efficient Distributed Training
H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA 30332, U.S.A.
DisagMoE: Computation-Communication overlapped MoE Training via Disaggregated AF-Pipe Parallelism
May 10, 2026Zhichen Zeng, Chi-Chih Chang, Jiayi Wang +10Expert Parallelism Communication-Efficient Distributed Training
ByteDance Seed · University of Washington · Cornell University
LBI: Parallel Scan Backpropagation via Latent Bounded Interfaces
May 9, 2026Shaun Christopher Lee, Sangeetha Abdu JyothiDeep Learning Optimization Backpropagation
University of California, Irvine
Cosine-Gated Adam-Decay: Drop-In Staleness-Aware Outer Optimization for Decoupled DiLoCo
May 9, 2026Vatsal Shah, Jiahao SunCommunication-Efficient Distributed Training Distributed Optimization
On the Tradeoffs of On-Device Generative Models in Federated Predictive Maintenance Systems
May 8, 2026Usevalad Milasheuski, Piero Baraldi, Enrico Zio +1Communication-Efficient Distributed Training Predictive Maintenance
Consiglio Nazionale delle Ricerche, IEIIT institute, 20133 Milan, Italy · Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB), Politecnico di Milano, 20133 Milan, Italy · Department of Energy, Politecnico di Milano, 20156 Milan, Italy +1
UMEDA: Unified Multi-modal Efficient Data Fusion for Privacy-Preserving Graph Federated Learning via Spectral-Gated Attention and Diffusion-Based Operator Alignment
May 8, 2026Shih-Yu Lai, Hirozumi Yamaguchi, Shang-Tse Chen +2Multimodal Federated Learning Federated Learning Aggregation
National Taiwan University · RIKEN-CCS · The University of Osaka +1
SparseRL-Sync: Lossless Weight Synchronization with ~100x Less Communication
May 8, 2026Lucas Hu, Ranchi Zhao, Isaac Zhu +4Reinforcement Learning Communication-Efficient Distributed Training
Scitix
Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization
May 7, 2026Tatsuhiro Nakamori, Laura Gomezjurado Gonzalez, Ganesh Talluri +5Communication-Efficient Distributed Training Distributed Optimization
1Keio University · 2Stanford University · 3Midwestern University +3
Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism
May 6, 2026Sajal Dash, Feiyi WangCommunication-Efficient Distributed Training High-Performance Computing
Oak Ridge National Laboratory Oak Ridge, USA
SignMuon: Communication-Efficient Distributed Muon Optimization
May 4, 2026Neel Mishra, Kushagara Trivedi, Pawan KumarSign-Based Optimization Communication-Efficient Distributed Training
1Microsoft · 2IIIT Hyderabad
FedPLT: Scalable, Resource-Efficient, and Heterogeneity-Aware Federated Learning via Partial Layer Training
May 4, 2026Ahmad Dabaja, Rachid El-AzouziCommunication-Efficient Distributed Training Heterogeneous Federated Learning
Laboratoire Informatique d’Avignon (LIA), Avignon University, France
DBLP: Phase-Aware Bounded-Loss Transport for Burst-Resilient Distributed ML Training
May 3, 2026Zechen Ma, Zixi Qu, Jinyan Yi +2Communication-Efficient Distributed Training Distributed Training
Department of Computer Science, University of Toronto, Toronto, Canada
AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMs
May 1, 2026Wenxiang Lin, Juntao Huang, Luhan Zhang +5LLM Quantization Communication-Efficient Distributed Training
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen · Huawei Technologies Ltd.
Scaling Federated Linear Contextual Bandits via Sketching
May 1, 2026Hantao Yang, Hong Xie, Xutong Liu +1Communication-Efficient Distributed Training Contextual Bandits
University of Science and Technology of China · University of Washington Tacoma
Hierarchical Federated Learning for Networked AI: From Communication Saving to Architecture-Aware Design
May 1, 2026Seyed Mohammad Azimi-Abarghouyi, Mehdi Bennis, Leandros TassiulasHierarchical Federated Learning Communication-Efficient Distributed Training
Department of Electrical Engineering, Chalmers University of Technology, Gothenburg, Sweden · Centre for Wireless Communications, University of Oulu, Oulu, Finland · Department of Electrical Engineering, Yale University, New Haven, USA
ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training
Apr 30, 2026Wenxiang Lin, Xinglin Pan, Ruibo Fan +2Communication-Efficient Distributed Training LLM Training
Harbin Institute of Technology, Shenzhen, China · The Hong Kong University of Science and Technology (Guangzhou), China · The Hong Kong University of Science and Technology, Hong Kong SAR
SplitFT: An Adaptive Federated Split Learning System For LLMs Fine-Tuning
Apr 29, 2026Yimeng Shan, Zhaorui Zhang, Sheng Di +3Communication-Efficient Distributed Training LLM Fine-Tuning
Hong Kong Polytechnic University Hong Kong · Argonne National Labratory USA · University of California, Merced USA +1
Fed-DLoRA: Efficient Wireless Federated Learning with Dynamic Low-Rank Adaptation
Apr 27, 2026Huaicheng Li, Junhui Zhao, Haoyu Quan +1Communication-Efficient Distributed Training Low-Rank Adaptation
School of Electronic and Information Engineering, Beijing Jiaotong University, Beijing 100044, China · School of Communication and Information Engineering, Nanjing University of Posts and Telecommunications, Nanjing 210003, China

May 21, 2026, cs.LGJ/K move · Enter open · S save

F-TIS: Harnessing Diverse Models in Collaborative GRPO

Nikolay Blagoev, Oğuzhan Ersoy, Wendelin Boehmer, Lydia Yiyu Chen

Gensyn · University of Neuchatel · TU Delft

RL for Language Models Communication-Efficient Distributed Training Group Relative Policy Optimization Decentralized Learning RL Post-Training Off-Policy RL

Open paper AskPDF arXiv

Reinforcement learning methods such as GRPO have seen great popularity in LLM post-training. In GRPO, models produce completions to a set of prompts, which are rewarded, and the policy is updated towards the relatively high reward completions. Due to the auto-regressive nature of models, the generation phase of such style of training can be extremely time consuming. As a solution, prior work has sought to distribute the inference step across many nodes, working parallel. These works assume primarily homogeneous models in the training in order to keep samples as close to on-policy as possible. This assumption may be impractical in decentralized systems, where parties with various computes and preferences may wish to collaborate on the same task. Thus, decentralized training requires an approach that can handle heterogeneous models - different models collaborating on the same tasks. However, this leads to highly off-policy samples presented during training, which prior work has identified that off-policy samples can hurt GRPO convergence. To enable heterogeneity, we propose Filtered Truncated Importance Sampling (F-TIS) - a GRPO-style training paradigm that can use off-policy samples to improve local model's learning. Our framework allows various models to collaborate in the same RL training run while being communication efficient. We extensively evaluate F-TIS in various heterogeneous setups and we show that it exhibits identical final model convergence to purely on-sample training. Furthermore, we observe in some setups better generalization on out-of-distribution tasks than on-policy training, increasing model's performance by up to 12%.

Momentum

Related topics

Latest papers 151