Modern ML training and inference now span hundreds to tens of thousands of GPUs, where failures can waste 10-15% of GPU hours due to slow recovery. Common network errors and link fluctuations can cause up to 50% of job interruptions, triggering timeouts that often terminate entire jobs, forcing expensive checkpoint rollback during training and request reprocessing during inference. We present R2CC, a fault-tolerant communication library that provides lossless, low-overhead failover by exploiting multi-NIC hardware. R2CC performs rapid connection migration, bandwidth-aware load redistribution, and resilient collective algorithms to maintain progress under failures. We evaluate R2CC on four 8-GPU H100 InfiniBand servers and via large-scale ML simulators modeling up to 1024 GPUs with diverse failure patterns. Experiments show that R2CC is highly robust to inter-node network failures, incurring less than 1.1% training and 3% inference overhead under active failure scenarios. Compared to existing fault-tolerant approaches, R2CC reduces failure-induced overhead by up to 92% for training and 98% for inference. R2CC is open-sourced at https://github.com/r2cc-project/R2CC.
Figures & tables
Figure 1: R2CC (with failure) vs. Ideal (failure-free)
Figure 2: Software layering of collective communication.
Figure 3: R2CC Overview.
Figure 4: R2CC-HotRepair. Affected traffic is switched to a pre-established healthy backup path after a failure.
Figure 5: R2CC-Balance. After HotRepair, R2CC redistributes the repaired traffic across the remaining healthy paths to avoid overloading a single path.
Figure 6: R2CC-AllReduce reduces the degraded node’s load from 2D to 7/4D . In this 4-node example with lost-bandwidth fraction 1/4 , Stage 1 runs a global AllReduce with all 4 nodes on 3/4D and a partial AllReduce with 3 healthy nodes on 1/4D , and Stage 2 broadcasts the 1/4D partial result back. Thus node A carries 2⋅3/4D+1/4D=7/4D . Since the data and bandwidth in Stage 1 are proportional, the total completion times are nearly the same for each rank.
Figure 7: Overhead Analysis. Completion-time overhead vs. no-failure as a function of lost bandwidth fraction X .
Figure 8: Megatron Training Performance Evaluation
Figure 9: Simulated 7B model training on 4–64 8 × A100 servers.
Figure 10
Figure 12: p50/p99 TTFT vs QPS under NIC failure for vLLM
Figure 13: p50/p95 TPOT under NIC failure for serving Llama-3.1-405B with TP = 8, PP = 2.
Figure 14: Inference performance under failure compared to DéjàVu and non-fault-tolerant baseline.
Figure 14
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
n
X=81
X=41
X=21
4
0.470%
0.833%
1.190%
16
0.02419%
0.04419%
0.06783%
64
0.00145%
0.00265%
0.00411%
Appendix
Table 1: Relative approximation gap g(n,X)=(T(X)−T(Y∗))/T(Y∗) at representative (n,X) values.
Figure 17: Relative approximation gap g(n,X)=(T(X)−T(Y∗))/T(Y∗) of the Y=X rule vs. lost bandwidth fraction X , for n∈{4,16,64} . y -axis on log scale.
Failure Type
Support
When Supported (Boundary)
In-flight inter-node communication failures (R2CC’s primary scope)
NIC hardware / port failure (incl. NIC–ToR)
Yes
Node/process stays alive, and the node still has at least one remaining healthy inter-node NIC/path.
Inter-node link / cable / ToR port down (single rail/link)
Yes
An alternate inter-node path exists (e.g., another rail/NIC); not a full network partition.
Failure is confined to a subset of connections while endpoints remain alive and an alternate NIC/path exists.
Link flapping (up → down → up)
Partial
Only when flapping causes an in-flight transport failure (e.g. timeout) visible to the collective; pure throughput jitter is not handled.
Multi-hop fabric fault (Inter-switch link / spine switch failure)
Partial
Usually masked by fabric rerouting; if it surfaces as a transport error, handled via backup connections.
Appendix
Table 2: R2CC failure model and scope. Support means the system can keep an ongoing collective running without communicator re-initialization or job restart, under the stated boundary conditions.
Figure 18: Topology-aware logical re-ranking.
Figure 19: p50/p95 TTFT/TPOT overhead vs number of failures under multiple NIC failures for serving with pipeline parallelism
Figure 20: H100 NCCL-Test Benchmark.
Figure 21: Checkpoint and restore in a training workload
Figure 22: Training loss under a persistent NIC-port failure (GPT-2 124M, WikiText-103, 3 seeds).
Condition
Test PPL
Max. paired Δ
NCCL 2.23.4, no failure
76.085
–
R2CC, no failure
76.085
0.000%
R2CC-Balance, failure
76.085
+0.008%
R2CC-AllReduce, failure
76.086
+0.023%
Appendix
Table 3: Final test perplexity (mean of three seeds). Δ is the worst-case increase over the NCCL baseline run with the same seed.
Time
Re-capture
Replayed graph (4 GiB AllReduce)
/ iter
overhead ∗
Pre-failure graph, healthy
2.03 s
–
Pre-failure graph, HotRepair
4.07 s
–
Re-captured as R2CC-Balance
3.04 s
1.7 ms
Re-captured as R2CC-AllReduce
2.79 s
0.33 s
Appendix
Table 4: CUDA Graphs on three CloudLab servers, each with two V100S GPUs and three NIC ports limited to 10 Gbps. ∗ Re-capture overhead is the time of the iteration that performs the re-capture, including the eager collective that R2CC-AllReduce needs to create its sub-communicators, minus the time of an iteration replayed from the new graph.
Fault-tolerant distributed training keeps machine-learning jobs accurate and productive despite failed, delayed, corrupted, or unreliable workers and networks.
Network failures are among the most frequent hardware faults in large-scale GPU clusters and a leading cause of training-job interruptions. Modern collective communication libraries such as NCCL mitigate network failures by rerouting traffic through surviving NICs on the same server, trading reduced inter-node bandwidth for uninterrupted training. However, the degraded server remains on the critical path of the standard ring algorithm, slowing the entire collective. We present the first information-theoretic lower bound on AllReduce completion time under asymmetric network bandwidth and show that when the straggler retains at least half of its original bandwidth, the unavoidable overhead relative to the fault-free optimum is only O(1/p) for p GPUs. We then design OptCC, a four-stage pipelined AllReduce algorithm that approaches this lower bound. Experiments on SimAI confirm that OptCC closes the gap left by existing fault-tolerant schemes: under practical network failures with up to 50% bandwidth loss, OptCC completes AllReduce within 2-6% of NCCL's fault-free ring performance, whereas the state-of-the-art incurs up to 57% overhead.
Peiqing Chen, Jiedong Jiang, Nengneng Yu +4
University of Maryland, College Park · Utrecht University · Kyoto University
Pre-training large language models on massive GPU clusters has made hardware faults routine rather than rare, driving the need for resilient training systems. Yet existing frameworks either focus on specific parallelism schemes or risk drifting away from a failure-free training trajectory. We propose ReCoVer, a resilient LLM pre-training system that upholds a single invariant: each iteration keeps the number of microbatches constant, ensuring per-iteration gradients remain stochastically equivalent to a failure-free run. The framework is organized as three decoupled protocol layers: (1) Fault-tolerant collectives that isolate faults from propagating across replicas; (2) in-step fine-grained recovery that preserves intra-iteration progress and prevents gradient corruption; (3) versatile-workload policy that dynamically redistributes microbatch quotas across the survivors. The design is parallelism-agnostic, integrating directly with both 3D parallelism and Hybrid Sharded Data Parallel (HSDP) as a drop-in substrate. We evaluate our implementation on end-to-end pre-training tasks for up to 512 GPUs, ReCoVer successfully preserves the training trajectory from a failure-free reference despite of 256 GPUs lost spread across the run. For comparison with checkpoint-and-restart baselines, ReCoVer demonstrates 2.23× higher effective throughput after successive failures. This advantage results in ReCoVer processing 74.9% more tokens at 234 GPU-hours, with the gap widening as the training prolongs.
Ziyue Liu, Zhengyang Wang, Ruijie Zhang +7
University of California at Santa Barbara · Argonne National Laboratory
Tail latency dominates the performance of synchronous pretraining jobs when running at very large scales. We describe a three-pronged approach: (1) a new RDMA-based transport protocol, MRC, sprays across many paths and actively load-balances between them, eliminating the issue of flow collisions (2) the use of multi-plane Clos topologies to get the benefits of high switch radix and redundancy, allowing training clusters well over 100K GPUs to be built as two-tier topologies while increasing physical redundancy, and (3) the use of static source-routing using SRv6 to allow MRC the freedom to bypass failures by itself. We describe our experiences running MRC and static SRv6 routing in production in OpenAI and Microsoft's largest training clusters, where it has been used to train the latest frontier models. We demonstrate how MRC allows AI training jobs to ride out many network failures that previously would have interrupted training.