Fault-Tolerant Distributed Training

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 16

All topics

Leto: Fast In-Place Recovery for LLM Training on Surviving Hardware
Sep 30, 2026Geon-Woo Kim, Joon Ha Kim, Daehyeok KimFault-Tolerant Distributed Training LLM Training
The University of Texas at Austin Austin, Texas, USA
Robustifying Asynchronous SGD via Soft Throttling
Sep 30, 2026Kaoru Otsuka, Maxime Meyer, Yuki Takezawa +2Distributed Optimization Fault-Tolerant Distributed Training
Okinawa Institute of Science and Technology · National University of Singapore · Toyota Motor Corporation +1
Argus: A Real-EKS Study of When Predicting Spot Interruptions Beats Simple Checkpointing
Sep 30, 2026Angshuman Chakravertty, MD RayyanFault-Tolerant Distributed Training Cloud Computing
SVKM’s NMIMS (Deemed to be University)
TrainSDC: Characterizing and Mitigating Silent Data Corruption in Large Language Model Training
Aug 31, 2026Zhipeng Xia, Haotian Xu, Siyu Yun +4Fault-Tolerant Distributed Training Language Model Robustness
Zhejiang University · Huawei
SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training
Aug 11, 2026Zhuang WangFault-Tolerant Distributed Training LLM Reliability
Independent Researcher
Operating Multi-Node Full Fine-Tuning on NVIDIA B300: A Field Report on Telemetry-Based Triage, Negative Results, and Operational Hardening
Aug 6, 2026Seon Ho Kim, Ui Jeong Jeon, Su Hyeon Kim +1Fine-Tuning LLM Fine-Tuning
Samsung SDS
Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models
Jul 14, 2026Gil Avraham, Violetta Shevchenko, Hadi Mohaghegh Dolatabadi +9Communication-Efficient Distributed Training Fault-Tolerant Distributed Training
Pluralis Research
PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint
Jul 2, 2026Haotian Xie, Junlin Chen, Mingkai Zheng +2Fault-Tolerant Distributed Training Distributed Training
Rutgers University · George Mason University
Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo
Jun 22, 2026Chamin Hewa Koneputugodage, Thalaiyasingam Ajanthan, Sameera Ramasinghe +7Communication-Efficient Distributed Training Distributed Optimization
1Pluralis Research
Don't Let a Few Network Failures Slow the Entire AllReduce
Jun 1, 2026Peiqing Chen, Jiedong Jiang, Nengneng Yu +4Communication-Efficient Distributed Training High-Performance Computing
University of Maryland, College Park · Utrecht University · Kyoto University
TierCheck: Tiered Checkpointing for Fault Tolerance in Large Language Model Training
May 18, 2026Shujie Han, Feng Jiang, Patrick P. C. Lee +5Fault-Tolerant Distributed Training Distributed Training
Northwestern Polytechnical University · The Chinese University of Hong Kong · National University of Defense Technology
ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload
May 11, 2026Ziyue Liu, Zhengyang Wang, Ruijie Zhang +7Language Model Pretraining Fault-Tolerant Distributed Training
University of California at Santa Barbara · Argonne National Laboratory
BatchWeave: A Consistent Object-Store-Native Data Plane for Large Foundation Model Training
May 11, 2026Ting Sun, Junjie Zhang, Xiao Yan +7Fault-Tolerant Distributed Training Distributed Training
Lionrock AI Lab, China Merchants Group, Hong Kong, China · Wuhan University, Wuhan, China · The Chinese University of Hong Kong, Shenzhen, China
From Detection to Recovery: Operational Analysis on LLM Pre-training with 504 GPUs
May 10, 2026Daemyung Kang, Eunjin Hwang, Hanjeong Lee +11Fault-Tolerant Distributed Training Distributed Training
Lablup Inc.∗
Decoupled DiLoCo for Resilient Distributed Pre-training
Apr 23, 2026Arthur Douillard, Keith Rush, Yani Donchev +14Language Model Pretraining Distributed Optimization
1Google DeepMind · 2Google Research
Tight Stability Bounds for Robust Distributed Learning: Byzantine Failures Hurt Generalization More than Data Poisoning
Jun 22, 2025Thomas Boudou, Batiste Le Bars, Nirupam Gupta +1Byzantine-Robust Federated Learning Algorithmic Stability
PreMeDICaL team, Inria, Idesp, Inserm, Université de Montpellier, Montpellier, France · Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189, CRIStAL, F-59000 Lille, France · Department of Computer Science, University of Copenhagen, Copenhagen, Danemark

Sep 30, 2026, cs.DCJ/K move · Enter open · S save

Leto: Fast In-Place Recovery for LLM Training on Surviving Hardware

Geon-Woo Kim, Joon Ha Kim, Daehyeok Kim

The University of Texas at Austin Austin, Texas, USA

Fault-Tolerant Distributed Training LLM Training

Open paper AskPDF arXiv

Hardware-operable failures (HOFs) interrupt large language model (LLM) training but permit recovery on the same hardware without reset, repair, or replacement. Existing recovery systems nevertheless reload checkpoints, recompute lost progress, and rebuild process state, idling GPUs that could otherwise continue training. We present Leto, a fault-tolerant training system that leverages surviving hardware to enable efficient in-place recovery. Our key insight is that the state needed to resume training can be retained or prepared outside the active training process while remaining on the same hardware. Leto retains the working model state and the reusable process state, and preinitializes the remaining state in a shadow trainer. We devise two-tier erasure protection and chunk-level transactional updates to keep the retained model state recoverable and consistent, and reclaim the shadow state when active training needs its GPU memory. Evaluation on 6- and 72-GPU NVIDIA A100 clusters shows that Leto recovers 3.6--6.5

\times

faster than the best-performing checkpointing baselines and improves productive training time by up to 13.7 percentage points. Large-scale simulation shows over 95% productive training time on a 131,072-GPU cluster.

Momentum

Related topics

Latest papers 16