Fault-Tolerant Distributed Training

Follow this topic Ask about this topic

Fault-tolerant distributed training studies how to keep learning reliable when workers, communication links, or data pipelines fail, from large-cluster pretraining to federated and decentralized learning. System approaches include checkpointing, hot-swapping, in-place recovery, fault-tolerant collectives, and quorum-based synchronization; algorithmic approaches include Byzantine-robust aggregation, gradient tracking, and asynchronous updates. Researchers evaluate these methods on language-model training and standard image, tabular, and federated-learning tasks, while operational studies use cluster telemetry and injected faults to expose failures that small tests can miss. A central tension is how to detect and recover quickly without adding enough communication, computation, or synchronization overhead to slow training or change its learning behavior.

Latest papers

23 papers4 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Democratizing LLM Training with the Edge: Challenges, Systems, and the Road Ahead
Oct 23, 2026Ziyue Luo, Jiaxuan Cai, Cedric LeDenmat +27LLM Training Distributed Training
The Ohio State University · University of Wisconsin–Madison · Rochester Institute of Technology +5
Leto: Fast In-Place Recovery for LLM Training on Surviving Hardware
Sep 30, 2026Geon-Woo Kim, Joon Ha Kim, Daehyeok KimFault-Tolerant Distributed Training LLM Training
The University of Texas at Austin Austin, Texas, USA
Robustifying Asynchronous SGD via Soft Throttling
Sep 30, 2026Kaoru Otsuka, Maxime Meyer, Yuki Takezawa +2Distributed Optimization Fault-Tolerant Distributed Training
Okinawa Institute of Science and Technology · National University of Singapore · Toyota Motor Corporation +1
Argus: A Real-EKS Study of When Predicting Spot Interruptions Beats Simple Checkpointing
Sep 30, 2026Angshuman Chakravertty, MD RayyanFault-Tolerant Distributed Training Cloud Computing
SVKM’s NMIMS (Deemed to be University)
TrainSDC: Characterizing and Mitigating Silent Data Corruption in Large Language Model Training
Aug 31, 2026Zhipeng Xia, Haotian Xu, Siyu Yun +4Fault-Tolerant Distributed Training Language Model Robustness
Zhejiang University · Huawei
Resilient Decentralized Wireless Federated Learning via Gradient Tracking with AdamW
Aug 26, 2026Nguyen Van Thieu, Ti Ti Nguyen, Ons Aouedi +2Decentralized Learning Communication-Efficient Distributed Training
Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, Luxembourg
SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training
Aug 11, 2026Zhuang WangFault-Tolerant Distributed Training LLM Reliability
Independent Researcher
Operating Multi-Node Full Fine-Tuning on NVIDIA B300: A Field Report on Telemetry-Based Triage, Negative Results, and Operational Hardening
Aug 6, 2026Seon Ho Kim, Ui Jeong Jeon, Su Hyeon Kim +1Fine-Tuning LLM Fine-Tuning
Samsung SDS
Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models
Jul 14, 2026Gil Avraham, Violetta Shevchenko, Hadi Mohaghegh Dolatabadi +9Communication-Efficient Distributed Training Fault-Tolerant Distributed Training
Pluralis Research
PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint
Jul 2, 2026Haotian Xie, Junlin Chen, Mingkai Zheng +2Fault-Tolerant Distributed Training Distributed Training
Rutgers University · George Mason University
Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo
Jun 22, 2026Chamin Hewa Koneputugodage, Thalaiyasingam Ajanthan, Sameera Ramasinghe +7Communication-Efficient Distributed Training Distributed Optimization
1Pluralis Research
Don't Let a Few Network Failures Slow the Entire AllReduce
Jun 1, 2026Peiqing Chen, Jiedong Jiang, Nengneng Yu +4Communication-Efficient Distributed Training High-Performance Computing
University of Maryland, College Park · Utrecht University · Kyoto University
TierCheck: Tiered Checkpointing for Fault Tolerance in Large Language Model Training
May 18, 2026Shujie Han, Feng Jiang, Patrick P. C. Lee +5Fault-Tolerant Distributed Training Distributed Training
Northwestern Polytechnical University · The Chinese University of Hong Kong · National University of Defense Technology
ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload
May 11, 2026Ziyue Liu, Zhengyang Wang, Ruijie Zhang +7Language Model Pretraining Fault-Tolerant Distributed Training
University of California at Santa Barbara · Argonne National Laboratory
BatchWeave: A Consistent Object-Store-Native Data Plane for Large Foundation Model Training
May 11, 2026Ting Sun, Junjie Zhang, Xiao Yan +7Fault-Tolerant Distributed Training Distributed Training
Lionrock AI Lab, China Merchants Group, Hong Kong, China · Wuhan University, Wuhan, China · The Chinese University of Hong Kong, Shenzhen, China
From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUs
May 10, 2026Daemyung Kang, Eunjin Hwang, Hanjeong Lee +11Fault-Tolerant Distributed Training Distributed Training
Lablup Inc.∗
Decoupled DiLoCo for Resilient Distributed Pre-training
Apr 23, 2026Arthur Douillard, Keith Rush, Yani Donchev +14Language Model Pretraining Distributed Optimization
1Google DeepMind · 2Google Research
Byzantine-Robust Distributed SGD: A Unified Analysis and Tight Error Bounds
Apr 11, 2026Boyuan Ruan, Xiaoyu Wang, Ya-Feng LiuDistributed Optimization Fault-Tolerant Distributed Training
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · Ministry of Education Key Laboratory of Mathematics and Information Networks, School of Mathematical Sciences, Beijing University of Posts and Telecommunications, Beijing 102206, China
Fully Byzantine-Resilient Distributed Multi-Agent Q-Learning
Apr 3, 2026Haejoon Lee, Dimitra PanagouMulti-Agent Reinforcement Learning Distributed RL
Department of Robotics, University of Michigan, Ann Arbor, MI, USA
Revealing the influence of participant failures on model quality in cross-silo Federated Learning
Mar 26, 2026Fabian Stricker, David Bermbach, Christian ZirpinsFederated Learning Fault-Tolerant Distributed Training
SENTINEL: Stagewise Integrity Verification for Pipeline Parallel Decentralized Training
Mar 3, 2026Hadi Mohaghegh Dolatabadi, Thalaiyasingam Ajanthan, Sameera Ramasinghe +5Distributed Training Fault-Tolerant Distributed Training
Pluralis Research
R2CC: Reliable and Resilient Collective Communication for Distributed Learning
Dec 31, 2025Wei Wang, Nengneng Yu, Sixian Xiong +1Fault-Tolerant Distributed Training Distributed Inference
University of Maryland, College Park
Tight Stability Bounds for Robust Distributed Learning: Byzantine Failures Hurt Generalization More than Data Poisoning
Jun 22, 2025Thomas Boudou, Batiste Le Bars, Nirupam Gupta +1Byzantine-Robust Federated Learning Algorithmic Stability
PreMeDICaL team, Inria, Idesp, Inserm, Université de Montpellier, Montpellier, France · Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189, CRIStAL, F-59000 Lille, France · Department of Computer Science, University of Copenhagen, Copenhagen, Danemark

Oct 23, 2026J/K move · Enter open · S save

Democratizing LLM Training with the Edge: Challenges, Systems, and the Road Ahead

Ziyue Luo, Jiaxuan Cai, Cedric LeDenmat, Srijith Nair+26

The Ohio State University · University of Wisconsin–Madison · Rochester Institute of Technology+5

LLM Training Distributed Training Federated Learning Communication-Efficient Distributed Training Fault-Tolerant Distributed Training

Open paper AskPDF arXiv

The growing computational power of modern GPU data centers has let a few industrial giants train and deploy large language models (LLMs) on massive amounts of data. Yet the sheer scale of these models and their training data makes centralized development increasingly hard to sustain, and at the same time shuts out smaller institutions that hold relevant private data but lack comparable infrastructure. In this paper, we organize our positions on distributed LLM training around the pretraining and fine-tuning paradigms, covering federated learning, split learning, and decentralized approaches, and we assess their readiness for geo-distributed settings with heterogeneous hardware and unreliable connectivity. We then present a {\em unified} system architecture built around a cloud-hosted parameter server that coordinates heterogeneous clients over outbound gRPC connections, reusing one communication, compression, and fault-tolerance stack for both pretraining and a family of fine-tuning methods. We validate the design with real geo-distributed experiments on GPT-2 Medium and Llama3-1B pretraining across multiple university sites, showing stable convergence under realistic conditions. Finally, we chart the road ahead, identifying open problems that span frontier model support, elastic fault tolerance, adaptive compression, federated alignment, privacy and trust, incentive design, and system--algorithm co-design, all of which must be addressed to make democratized LLM training a reality.