Communication-Efficient Distributed Training

Follow this topic Ask about this topic

Momentum

20 papers in the last four weeks, up 300% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 151

All topics

Theoretical Foundations of Communication-Efficient, Robust, and Practical Distributed and Federated Optimization
Aug 6, 2026Grigory MalinovskyByzantine-Robust Federated Learning Communication-Efficient Distributed Training
King Abdullah University of Science and Technology Thuwal, Kingdom of Saudi Arabia
ML-for-ML
Aug 6, 2026Yutong Zhao, Noga H. Rotman, Gianni Antichi +1Communication-Efficient Distributed Training Distributed Training
UCL and BUPT · United Kingdom · UCL +3
SNI-GNN: SmartNIC-Assisted Full-Graph GNN Training with In-Network Embedding Prediction
Aug 6, 2026Guofan Yu, Sitian Chen, Zhenheng Tang +2Graph Neural Networks Communication-Efficient Distributed Training
Hong Kong Baptist University · Hong Kong University of Science and Technology · Hong Kong University of Science and Technology (Guangzhou)
DG-FedReuse: Proxy-Gradient-Gated Cached-Update Reuse with Matched Sparse Uplink Accounting
Aug 5, 2026Rahil Aftab, Vineet Kumar Rakesh, Soumya Mazumdar +1Federated Learning Aggregation Communication-Efficient Distributed Training
Department of Computer Science and Engineering Jamia Hamdard New Delhi 110062, India · Engineering Science Homi Bhabha National Institute Anushaktinagar, Mumbai 400094, Maharashtra, India · Computer and Informatics Group Variable Energy Cyclotron Centre 1/AF, Bidhannagar, Kolkata 700064, West Bengal, India +1
FedCritic-MIMO: Communication-Efficient Serverless Federated Critic Learning for Massive-MIMO Resource Control in Open and Disaggregated 6G RANs
Aug 4, 2026Amin Farajzadeh, Melike Erol-KantarciFederated RL 6G Networks
School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, ON K1N 6N5, Canada · with the NETCORE Lab
FraQ: Efficient Coordinate-Space Recompression for Federated Low-Rank Adaptation
Aug 4, 2026Shenghui Li, Thiemo VoigtFederated Learning Aggregation Communication-Efficient Distributed Training
Uppsala University, Uppsala, Sweden · Research Institutes of Sweden, Stockholm, Sweden
FedRings: A Scalable and Topology-Aware Federated Learning Framework for LEO Satellite Constellations
Aug 4, 2026Ziwu Liu, Inês Pinto Gouveia, Rehana Yasmin +2Communication-Efficient Distributed Training Decentralized Learning
CEMSE Division, King Abdullah University of Science and Technology (KAUST) Thuwal 23955-6900, Kingdom of Saudi Arabia
Sign compression for Muon: SignMuon, MuonSign, and the Limits of Error Feedback
Jul 31, 2026Maria Smirnova, Alexey KravatskiySign-Based Optimization Communication-Efficient Distributed Training
MIRIAI Moscow, Russia
Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformers
Jul 23, 2026Hariharan Ramesh, Jyotikrishna DassFederated Learning Aggregation Communication-Efficient Distributed Training
Department of Electrical and Computer Engineering University of Arizona, Tucson, AZ 85719
Information-Theoretically Secure Aggregation for Lightweight Federated Learning: Resilient to Dropouts and Adversaries
Jul 23, 2026Hyeong-Gun Joo, Songnam Hong, Dong-Joon ShinSecure Multi-Party Computation Federated Learning Aggregation
Department of Electronic Engineering, Hanyang University, Seoul, South Korea
SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD
Jul 22, 2026Dongfang Li, Xiaodong Luo, Ruoyu Sun +62Communication-Efficient Distributed Training LLM Post-Training
Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts
Jul 21, 2026Minyu Cui, Anna Wingkvist, Morgan EricssonGPU Kernel Optimization Communication-Efficient Distributed Training
Linnaeus University · Växjö, Sweden
AutoEncoder-Compressed Parallel Split Learning for Pre-trained Model Fine-Tuning
Jul 20, 2026Bas Meuwissen, Vasileios Tsouvalas, Nirvana MeratniaFine-Tuning Communication-Efficient Distributed Training
Eindhoven University of Technology, Eindhoven, The Netherlands
Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models
Jul 14, 2026Gil Avraham, Violetta Shevchenko, Hadi Mohaghegh Dolatabadi +9Communication-Efficient Distributed Training Fault-Tolerant Distributed Training
Pluralis Research
Mixed-Timescale Differential Coding for Downlink Model Broadcast in Wireless Federated Learning
Jul 14, 2026Chung-Hsuan Hu, Zheng Chen, Erik G. LarssonCommunication-Efficient Distributed Training Federated Learning
Department of Electrical Engineering (ISY), Linköping University, Linköping, SE-58183 Sweden
PFAdapter: Hierarchical LoRA Decomposition for Personalized Federated MLLMs
Jul 13, 2026Jing Liu, Kun Yang, Yan Wang +5Multimodal Federated Learning Federated Learning Aggregation
College of Future Information Technology, Fudan University, Shanghai 200433, China · Division of Natural and Applied Sciences, Duke Kunshan University, Suzhou 215316, China · Department of Electrical and Computer Engineering, The University of British Columbia, BC V6T 1Z4, Canada +8
GeoFP8: Geometry-Aware FP8 Gradient Compression for Distributed LLM Training
Jul 8, 2026Jieying Wang, Zizhong Wang, Fangru Linghu +3Language Model Pretraining Communication-Efficient Distributed Training
Department of Computer Science, Rutgers University, Piscataway, NJ, USA · Department of Electrical and Computer Engineering, Rutgers University, Piscataway, NJ, USA
Robust Federated Learning Under Real-World Client Churn
Jul 8, 2026Dhruv Garg, Neha Lakhani, Debopam Sanyal +3Federated Learning Aggregation Communication-Efficient Distributed Training
Georgia Tech USA · Cisco Research USA
Entropy-Guided Tensor Compression for Multimodal Federated Learning on Edge Devices
Jul 7, 2026Quoc Bao Phan, Tuy Tan NguyenTensor Networks Multimodal Federated Learning
Department of Electrical and Computer Engineering, Center for Advanced Power Systems, FAMU-FSU College of Engineering, Florida State University, Tallahassee, FL 32310, USA
Can Model Merging Improve Aggregation in DiLoCo?
Jul 3, 2026Stefan Horoi, Benjamin Thérien, Guy Wolf +1Communication-Efficient Distributed Training Distributed Optimization
Université de Montréal · Mila – Quebec AI Institute · Concordia University
Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
Jul 2, 2026Xuan-Phi Nguyen, Shrey Pandit, Yiran Zhao +3Communication-Efficient Distributed Training High-Performance Computing
Salesforce AI Research
SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication
Jul 2, 2026Mingkai Zheng, Junlin Chen, Haotian Xie +1Language Model Pretraining Communication-Efficient Distributed Training
Rutgers University
TallyTrain: Communication-Efficient Federated Distillation
Jun 30, 2026Radhakrishna Achanta, Will ReedFederated Learning Aggregation Communication-Efficient Distributed Training
Cisco Systems Inc.
HSAP: A Hierarchical Sequence-aware Parallelism for Hybrid-Context Generative Models
Jun 29, 2026Songxin Zhang, Zejian Xie, Zhuoyang Song +4Communication-Efficient Distributed Training LLM Training
Department of Statistics and Data Science, Southern University of Science and Technology · International Digital Economy Academy
Federated Hash Projected Latent Factor Learning
Jun 24, 2026Jialan HeCommunication-Efficient Distributed Training Privacy-Preserving ML
College of Computer and Information Science Southwest University Chongqing 400715, China
TL++: Accuracy and Privacy Preserving Traversal Learning for Distributed Intelligent Systems
Jun 24, 2026Erdenebileg Batbaatar, Young YoonCommunication-Efficient Distributed Training Privacy-Preserving ML
Neouly, Republic of Korea · Department of Computer Engineering, Hongik University, Republic of Korea
Not Every Sync Is Safe: Calibrated DiLoCo Scheduling for Shared AI Infrastructure
Jun 24, 2026Maxwell Twelftree, David Lemphers, An-chi He +1Communication-Efficient Distributed Training Distributed Optimization
Maincode, Melbourne, Australia
Priority-Aware Learning-Unlearning Correction for Dynamic Decentralized LoRA Fine-Tuning
Jun 22, 2026Nuocheng Yang, Yechen He, Sihua Wang +3Federated Unlearning Communication-Efficient Distributed Training
Beijing Laboratory of Advanced Information Network, and the Beijing Key Laboratory of Network System Architecture and Convergence, Beijing University of Posts and Telecommunications, Beijing 100876, China · Information Systems Technology and Design Pillar, Singapore University of Technology and Design, 487372, Singapore
Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo
Jun 22, 2026Chamin Hewa Koneputugodage, Thalaiyasingam Ajanthan, Sameera Ramasinghe +7Communication-Efficient Distributed Training Distributed Optimization
1Pluralis Research
FeLoG: Scalable and Efficient Distributed Graph Embedding with Feedback Loop Mechanism
Jun 20, 2026Peng Fang, Arijit Khan, Ziqiang Wu +4Communication-Efficient Distributed Training Graph Embedding
Huazhong University of Science and Technology · Bowling Green State University

Aug 6, 2026, cs.LGJ/K move · Enter open · S save

Theoretical Foundations of Communication-Efficient, Robust, and Practical Distributed and Federated Optimization

Grigory Malinovsky

King Abdullah University of Science and Technology Thuwal, Kingdom of Saudi Arabia

Byzantine-Robust Federated Learning Communication-Efficient Distributed Training Low-Rank Adaptation Federated Learning Heterogeneous Federated Learning

Open paper AskPDF arXiv

Machine learning and optimization have advanced together, with practical demands motivating new theory and theoretical breakthroughs enabling new applications. Modern large-scale training relies on classical optimization principles, but the constraints of distributed systems require these foundations to be reconsidered. This thesis addresses seven challenges at the intersection of theory and practice, focusing on key bottlenecks in federated learning and distributed optimization. First, we introduce ProxSkip and prove that local gradient steps can accelerate communication, providing a theoretical foundation for this widely used heuristic. Second, we develop Variance Reduced ProxSkip, which eliminates the neighborhood error of stochastic local updates while balancing communication and local computation. Third, we show that local steps retain their communication acceleration under partial client participation. Fourth, we prove that server-side stepsizes and sampling without replacement improve convergence in heterogeneous settings. Fifth, for Random Reshuffling, we demonstrate that compressing gradient differences rather than gradients yields better theoretical and practical performance. Sixth, we establish that Byzantine robustness and partial participation can be achieved simultaneously using gradient-difference clipping. Finally, we develop the first theoretical framework for low-rank adaptation based on randomized asymmetric chains, providing new insights into fine-tuning large models. Across these contributions, we introduce novel algorithmic frameworks, establish sharp guarantees under realistic assumptions, and support the theory with numerical experiments.

Momentum

Related topics

Latest papers 151