Distributed Training

Follow this topic Ask about this topic

Momentum

13 papers in the last four weeks, up 225% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 82

All topics

DMuon: Efficient Distributed Muon Training with Near-Adam Overhead
Jun 25, 2026Vincent Chen, Starrick Liu, Regis Cheng +8Deep Learning Optimization Distributed Optimization
X SQUARE ROBOT TEAM
TL++: Accuracy and Privacy Preserving Traversal Learning for Distributed Intelligent Systems
Jun 24, 2026Erdenebileg Batbaatar, Young YoonCommunication-Efficient Distributed Training Privacy-Preserving ML
Neouly, Republic of Korea · Department of Computer Engineering, Hongik University, Republic of Korea
Not Every Sync Is Safe: Calibrated DiLoCo Scheduling for Shared AI Infrastructure
Jun 24, 2026Maxwell Twelftree, David Lemphers, An-chi He +1Communication-Efficient Distributed Training Distributed Optimization
Maincode, Melbourne, Australia
Decentralised AI Training and Inference with BlockTrain
Jun 23, 2026Peter Toth, Dan OprisaEfficient Transformer Inference Decentralized Learning
Spheroid Labs∗
Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo
Jun 22, 2026Chamin Hewa Koneputugodage, Thalaiyasingam Ajanthan, Sameera Ramasinghe +7Communication-Efficient Distributed Training Distributed Optimization
1Pluralis Research
Online Dynamic Batching with Formal Guarantees for LLM Training
Jun 18, 2026Dian Li, Zekun Wang, Yaoru Wang +1LLM Fine-Tuning Distributed Training
Tencent
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
Jun 17, 2026Lorenzo Sani, Zeyu Cao, Meghdad Kurmanji +5Expert Parallelism Communication-Efficient Distributed Training
Cambridge ML Systems Lab (CaMLSys), University of Cambridge · Flower Labs
Mixtures of Subspaces for Bandwidth Efficient Context Parallel Training
Jun 15, 2026Sameera Ramasinghe, Ajanthan Thalaiyasingam, Hadi Mohaghegh Dolatabadi +5Communication-Efficient Distributed Training Long-Context Language Modeling
Pluralis Research
Clipping Makes Distributed and Federated Asynchronous SGD Robust to Stragglers
Jun 11, 2026Samuel Erickson, Mikael JohanssonGradient Clipping Stochastic Optimization Convergence
School of EECS, KTH Royal Institute of Technology, Stockholm, Sweden.
Piper: A Programmable Distributed Training System
Jun 9, 2026Megan Frisella, Shubham Tiwari, Andy Ruan +5Distributed Optimization Distributed Training
University of Washington Seattle, WA, USA · University of Washington and Shanghai Jiao Tong University Seattle, WA, USA
ASTRA-sim 3.0: Next-Level Distributed Machine Learning Simulations via High-Fidelity GPU and Infrastructure Modeling
Jun 9, 2026William Won, Jinsun Yoo, Tuan Ta +16Distributed Training
AMD Research and Advanced Development · Georgia Institute of Technology · Keysight +1
Resource-aware Computation-Communication Overlap for multi-GPU ML Workloads
Jun 8, 2026Minyu Cui, Miquel PericasGPU Kernel Optimization Communication-Efficient Distributed Training
Chalmers University of Technology and University of Gothenburg, Gothenburg, Sweden
UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing
Jun 2, 2026Xinming Wei, Chao Jin, Tuo Dai +10Expert Parallelism Expert Load Balancing
School of Computer Science, Peking University · Xiaohongshu Inc. · Shanghai AI Laboratory +1
Demystifying Pipeline Parallelism: First Theory for PipeDream
Jun 2, 2026Ivan Ilin, Peter RichtárikNonconvex Stochastic Optimization Distributed Training
KAUST
Don't Let a Few Network Failures Slow the Entire AllReduce
Jun 1, 2026Peiqing Chen, Jiedong Jiang, Nengneng Yu +4Communication-Efficient Distributed Training High-Performance Computing
University of Maryland, College Park · Utrecht University · Kyoto University
Does Distributed Training Undermine Compute Governance?
May 28, 2026Robi RahmanDistributed Training AI Governance
Machine Intelligence Research Institute
ScaleAcross Explorer: Exploring Communication Optimization for Scale-Across AI Model Training
May 23, 2026Minghao Li, Alicia Golden, Samuel Hsia +14Communication-Efficient Distributed Training Distributed Training
1Harvard University · 2Meta Platforms, Inc.
High-speed Networking for Giga-Scale AI Factories
May 20, 2026Sajy Khashab, Albert Gran Alcoz, Alon Gal +11Distributed Training
NVIDIA · Technion - Israel Institute of Technology and NVIDIA
A Readiness-Driven Runtime for Pipeline-Parallel Training under Runtime Variability
May 18, 2026Ruitao Liu, Xinyang Tian, Shuo Chen +4High-Performance Computing Distributed Training
Tsinghua University · Scitix AI
Guard: Scalable Straggler Detection and Node Health Management for Large-Scale Training
May 18, 2026Guanliang Liu, Abhinandan Patni, Congzhu Lin +14Distributed Training Anomaly Detection
Store Foundational AI, Amazon, Seattle, WA, USA.
TierCheck: Tiered Checkpointing for Fault Tolerance in Large Language Model Training
May 18, 2026Shujie Han, Feng Jiang, Patrick P. C. Lee +5Fault-Tolerant Distributed Training Distributed Training
Northwestern Polytechnical University · The Chinese University of Hong Kong · National University of Defense Technology
A Few GPUs, A Whole Lotta Scale: Faithful LLM Training Emulation with PrismLLM
May 15, 2026Shaoke Xi, ChonLam Lao, Boyi Jia +11High-Performance Computing Distributed Training
Alibaba Group · Harvard University · Shanghai Jiao Tong University +1
Byzantine-Robust Distributed Sparse Learning Revisited
May 13, 2026Yuxuan Wang, Lixin Zhang, Kangqiang LiRobust Regression Distributed Optimization
School of Mathematical Sciences, Zhejiang University, Hangzhou, 310027, Zhejiang, China · School of Statistics and Mathematics, Zhejiang Gongshang University, Hangzhou, 310018, Zhejiang, China · Information Center, Hubei Provincial Tobacco Monopoly Administration, Wuhan, 430030, Hubei, China
DynaTrain: Fast Online Parallelism Switching for Elastic LLM Training
May 12, 2026Yuanqing Wang, Yuchen Zhang, Hao Lin +9Efficient Language Model Training Distributed Training
Peking University · Infinigence AI · Institute of Computing Technology, CAS +2
MLCommons Chakra: Advancing Performance Benchmarking and Co-design using Standardized Execution Traces
May 11, 2026Srinivas Sridharan, Theodor-Adrian Badea, Andy Balogh +26Benchmark Design Hardware-Software Co-Design
1NVIDIA · 2Keysight · 3AMD +7
ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload
May 11, 2026Ziyue Liu, Zhengyang Wang, Ruijie Zhang +7Language Model Pretraining Fault-Tolerant Distributed Training
University of California at Santa Barbara · Argonne National Laboratory
ShardTensor: Domain Parallelism for Scientific Machine Learning
May 11, 2026Corey Adams, Peter Harrington, Akshay Subramaniam +4Scientific ML Distributed Training
NVIDIA · Santa Clara, CA, USA
BatchWeave: A Consistent Object-Store-Native Data Plane for Large Foundation Model Training
May 11, 2026Ting Sun, Junjie Zhang, Xiao Yan +7Fault-Tolerant Distributed Training Distributed Training
Lionrock AI Lab, China Merchants Group, Hong Kong, China · Wuhan University, Wuhan, China · The Chinese University of Hong Kong, Shenzhen, China
From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUs
May 10, 2026Daemyung Kang, Eunjin Hwang, Hanjeong Lee +11Fault-Tolerant Distributed Training Distributed Training
Lablup Inc.∗
DisagMoE: Computation-Communication overlapped MoE Training via Disaggregated AF-Pipe Parallelism
May 10, 2026Zhichen Zeng, Chi-Chih Chang, Jiayi Wang +10Expert Parallelism Communication-Efficient Distributed Training
ByteDance Seed · University of Washington · Cornell University

Jun 25, 2026, cs.DCJ/K move · Enter open · S save

DMuon: Efficient Distributed Muon Training with Near-Adam Overhead

Vincent Chen, Starrick Liu, Regis Cheng, Dance Yang+7

X SQUARE ROBOT TEAM

Deep Learning Optimization Distributed Optimization Muon Optimizer Distributed Training

Open paper AskPDF arXiv

Matrix-orthogonalization-based optimizers, exemplified by Muon, have demonstrated strong convergence behavior across a wide range of modern deep learning workloads. The matrix-aware updates offer a compelling alternative to conventional element-wise optimization, particularly as model architectures continue to grow in scale and heterogeneity. Yet contemporary distributed training infrastructure built around the assumption of element-wise optimizers is poorly matched to matrix-level optimizers such as Muon, whose updates couple entire weight matrices and require costly Newton-Schulz iterations. Vanilla Muon implementations incur more than 2x the cost of forward and backward passes. To close this gap, we present DMuon, an open-source distributed Muon implementation that integrates into existing training pipelines as a drop-in module, with no framework-level modifications. Across both embodied foundation model and large language model (LLM) training workloads, DMuon achieves a 1.48x-3.01x speedup in end-to-end step time and a 6.85x-163.00x speedup in optimizer-step time, bringing per-step latency to near-AdamW levels and enabling efficient scaling in our model training.

Momentum

Related topics

Latest papers 82