Distributed Training

Follow this topic Ask about this topic

Momentum

13 papers in the last four weeks, up 225% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 82

All topics

HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Training
Sep 30, 2026Mengyuan Fan, Peizhuang Cong, Zixiao Huang +7Expert Parallelism Heterogeneous Computing
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Infinigence AI · Tsinghua University
Argus: A Real-EKS Study of When Predicting Spot Interruptions Beats Simple Checkpointing
Sep 30, 2026Angshuman Chakravertty, MD RayyanFault-Tolerant Distributed Training Cloud Computing
SVKM’s NMIMS (Deemed to be University)
AutoLoCo: Communication Efficient Distributed LLM Training via Adaptive Synchronization
Sep 29, 2026Pengyu He, Yan Zhang, Ruien Li +1Communication-Efficient Distributed Training Distributed Training
Department of Electrical Engineering, California Institute of Technology · Department of Computer Science and Technology, Tsinghua University · Department of Computer Sciences, University of Wisconsin - Madison
TopoEP: Topology-Aware Load Balancing for Expert-Parallel MoE Training
Sep 28, 2026Jiacheng Zhu, Xie Zhao, Gongming Zhao +3Expert Parallelism Expert Load Balancing
University of Science and Technology of China
FairMean: Promoting Fairness in Distributed Learning under Label Poisoning Attacks
Sep 22, 2026Huigan Zheng, Jiaojiao Zhang, Yongxiang LiuAlgorithmic Fairness Adaptive Loss Weighting
Sun Yat-Sen University · Pengcheng Laboratory · Great Bay University
Accelerating Sharded Data Parallelism at Scale with Federated Learning
Sep 17, 2026Gianluca Mittone, Marco AldinucciCommunication-Efficient Distributed Training High-Performance Computing
University of Turin, Turin, Italy
Efficiently Distributed Federated Learning
Sep 17, 2026Gianluca Mittone, Robert Birke, Marco AldinucciDistributed Training Federated Learning
Computer Science Department, University of Turin, Turin, Italy
OPEN-1B: A Fully Auditable Training Run
Sep 15, 2026John Donaghy, Brian Wilcox, Oğuzhan Ersoy +6Neural Network Verification LLM Auditing
Gensyn
LLM-Based Schema-Aware Split Learning for Privacy-Preserving Mental Distress Prediction Across Heterogeneous Surveys
Sep 14, 2026Md Khalid Syfullah, Alvi Ataur KhalilPrivacy-Preserving ML Clinical Prediction
Transformative Innovation for Trustworthy AI and Network Security (TITANS) Lab, Computer Science, Southern Illinois University Carbondale, USA
Clustering-Based Balanced Sampling and Allocation with Data Parallelism for High-Performance Fine-Tuning
Sep 14, 2026Hyunjin Kim, Youngeun Nam, Jaemin Han +2LLM Fine-Tuning Instruction-Tuning Data Selection
KAIST
4D Parallelism Unlocks Exascale Bayesian Neural Networks for High-Fidelity Atmospheric Modeling
Sep 14, 2026Deifilia Kieckhefen, Juan Pedro Gutiérrez Hermosillo Muriedas, Lars Helge Heyen +12Bayesian Neural Networks Uncertainty Quantification
Karlsruhe Institute of Technology, Germany · Forschungszentrum Jülich, Germany · CSC – IT Center for Science Ltd., Finnland +1
Generalization Analysis of Distributed Kernel-based Robust Gradient Descent Algorithms
Sep 10, 2026Jun-Yi Meng, Zheng-Chu Guo, Yuan MaoKernel Regression Communication-Efficient Distributed Training
School of Mathematical Sciences, Zhejiang University, Hangzhou 310058, P. R. China · College of Informatics, Huazhong Agricultural University, Wuhan 430070, P. R. China
Scalability Analysis of Distributed Kolmogorov-Arnold Network Training on High-Performance Computing Systems
Sep 7, 2026Guangneng Chen, David Garcia Selfa, Pablo Quesada BarriusoHigh-Performance Computing Kolmogorov-Arnold Networks
University of Santiago de Compostela, 15782 Santiago de Compostela, Spain · Galicia Supercomputing Center (CESGA), 15705 Santiago de Compostela, Spain
Contribution-Aware Bandwidth Allocation for Multimodal Split Learning
Sep 1, 2026Iason Ofeidis, Leandros TassiulasDistributed Training Multimodal Learning
Department of Electrical and Computer Engineering, Yale University, New Haven, CT, USA
SatDL: Jointly Optimizing Data Redistribution and Training for Satellite-Based Distributed Learning
Aug 25, 2026Hao Wu, Kin Whye Chew, Yizhan Han +2Distributed Training
School of Computing National University of Singapore Singapore
SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training
Aug 11, 2026Zhuang WangFault-Tolerant Distributed Training LLM Reliability
Independent Researcher
SwiftQK: Fast and Communication-Efficient Tensor Parallelism for Query-Key Normalization
Aug 10, 2026Gyudong Kim, Wonjun Han, Young Geun KimGPU Kernel Optimization Tensor Parallelism
Department of Computer Science and Engineering, Korea University, Seoul 02855, Republic of Korea
LGNNIC: Acceleration of Large-Scale GNN Training using SmartNICs
Aug 7, 2026Liad Gerstman, Aditya Dhakal, Dejan Milojicic +1Graph Neural Networks Communication-Efficient Distributed Training
Technion Israel Institute of Technology Haifa, Israel · Hewlett Packard Enterprise Labs Fort Collins, CO, USA · Hewlett Packard Enterprise Labs Milpitas, CA, USA
Dual-Node NVIDIA DGX Spark over Tailscale: A Remote-Access Testbed for Distributed LLM Training and Cyber-Threat-Intelligence Fine-Tuning
Aug 7, 2026Vasanth IyerLLM Fine-Tuning Distributed Training
Department of Computer Science and Digital Technologies Grambling State University
Stream Learning: Partition-Fair Gossip Learning Without Tokens
Aug 7, 2026Fabien Mathieu, Alexandre Pham, Maria Gradinariu Potop-Butucaru +1Communication-Efficient Distributed Training Decentralized Learning
NPA · Swapcard, Paris, France · Sorbonne Université, CNRS, LIP6, F-75005 Paris, France +2
ML-for-ML
Aug 6, 2026Yutong Zhao, Noga H. Rotman, Gianni Antichi +1Communication-Efficient Distributed Training Distributed Training
UCL and BUPT · United Kingdom · UCL +3
Operating Multi-Node Full Fine-Tuning on NVIDIA B300: A Field Report on Telemetry-Based Triage, Negative Results, and Operational Hardening
Aug 6, 2026Seon Ho Kim, Ui Jeong Jeon, Su Hyeon Kim +1Fine-Tuning LLM Fine-Tuning
Samsung SDS
Empowering Credit Risk Detection in Weixin Pay with Billion-Scale Deep Graph Learning
Aug 3, 2026Xin Liu, Xiyuan Chen, Chenglong Wu +3AI Risk Management Financial Fraud Detection
Tongji University Shanghai, China · Tencent Inc. Shenzhen, China
Using Non-Lipschitz Signum-based Functions for Distributed Optimization and Machine Learning: Trade-off Between Con-vergence Rate and Optimality Gap
Aug 2, 2026Mohammadreza Doostmohammadian, Amir Ahmad Ghods, Alireza Aghasi +2Sign-Based Optimization Distributed Optimization
Faculty of Mechanical Engineering, Semnan University, Semnan, Iran · Department of Electrical Engineering and Computer Science, Oregon State University, Oregon, USA · Institute of Computational Mathematics and Information Technologies, Kazan Federal University, Russia +1
Pipelined Gradient Coding
Jul 22, 2026Xian Su, Jun LiDistributed Training
Department of Computer Science · Florida International University · Miami, FL, USA +3
The Power of Backdoor Absorption in Community Training
Jul 7, 2026Issam Seddik, Sami Souihi, Mohamed Tamaazousti +1Algorithmic Auditing Backdoor Attacks
Université Paris-Saclay, CEA LIST, Palaiseau, France
PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint
Jul 2, 2026Haotian Xie, Junlin Chen, Mingkai Zheng +2Fault-Tolerant Distributed Training Distributed Training
Rutgers University · George Mason University
Unveiling the Non-Monotonic Effect of Privacy on Generalization under Byzantine Robustness
Jul 1, 2026Thomas Boudou, Batiste Le Bars, Nirupam Gupta +1Differential Privacy Distributed Training
Inria, Université de Montpellier, INSERM · Montpellier, France · Inria, Université de Lille +3
MALOQ: Massively Accelerated Learning of Operators for Quantum Transport
Jun 27, 2026Manasa Kaniselvan, Alexander Maeder, Denghui Lu +2Quantum Chemistry High-Performance Computing
D-ITET, ETH Zurich, Zurich, Switzerland
Optimizing Teacher-Student Partitioning for Scalable Knowledge Distillation on HPC Systems
Jun 26, 2026Adrian P. Dieguez, Victor Conchello Vendrell, Alex Batlle +3High-Performance Computing Distributed Training
Qualcomm AI Research

Sep 30, 2026, cs.DCJ/K move · Enter open · S save

HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Training

Mengyuan Fan, Peizhuang Cong, Zixiao Huang, Si Xu+6

State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Infinigence AI · Tsinghua University

Expert Parallelism Heterogeneous Computing Distributed Training Mixture-of-Experts Models Pipeline Parallelism

Open paper AskPDF arXiv

As model sizes continue to scale, distributed training has become inevitable. Automatic parallelization techniques can derive efficient training parallelism strategies at low cost while achieving superior performance. The difficulty of this problem is jointly determined by the complexity of the model and the underlying compute cluster. Meanwhile, mixture-of-experts (MoE) models are increasingly emerging as the dominant architecture and the rapid evolution of accelerator hardware has made cluster heterogeneity commonplace, posing substantial challenges to automatic parallelization. However, existing approaches typically target either MoE architectures or heterogeneous clusters, failing to generalize to scenarios where both challenges coexist. To this end, we present HAPMoE, a heterogeneity-aware automatic parallelism planner for MoE training. HAPMoE builds a lightweight MoE-aware cost model and efficiently searches a six-dimensional parallel space, producing parallel plans directly deployable on Megatron-LM. Experiments show that HAPMoE improves end-to-end training throughput by up to 3.2

\times

over baselines across heterogeneous clusters. Its non-uniform pipeline partitioning yields an additional up to 78% gains, and its pruning-enhanced dynamic programming algorithm completes the search within 1 minute, demonstrating high efficiency and practical value in complex hardware environments.

Momentum

Related topics

Latest papers 82