R2CC: Reliable and Resilient Collective Communication for Distributed Learning

Wei Wang; Nengneng Yu; Sixian Xiong; Zaoxing Liu

cs.DCDec 31, 2025

R2CC: Reliable and Resilient Collective Communication for Distributed Learning

Authors: Wei Wang, Nengneng Yu, Sixian Xiong, Zaoxing Liu

Organizations: University of Maryland, College Park

Abstract

Modern ML training and inference now span hundreds to tens of thousands of GPUs, where failures can waste 10-15% of GPU hours due to slow recovery. Common network errors and link fluctuations can cause up to 50% of job interruptions, triggering timeouts that often terminate entire jobs, forcing expensive checkpoint rollback during training and request reprocessing during inference. We present R2CC, a fault-tolerant communication library that provides lossless, low-overhead failover by exploiting multi-NIC hardware. R2CC performs rapid connection migration, bandwidth-aware load redistribution, and resilient collective algorithms to maintain progress under failures. We evaluate R2CC on four 8-GPU H100 InfiniBand servers and via large-scale ML simulators modeling up to 1024 GPUs with diverse failure patterns. Experiments show that R2CC is highly robust to inter-node network failures, incurring less than 1.1% training and 3% inference overhead under active failure scenarios. Compared to existing fault-tolerant approaches, R2CC reduces failure-induced overhead by up to 92% for training and 98% for inference. R2CC is open-sourced at https://github.com/r2cc-project/R2CC.

Figures & tables

Figure 1: R2CC (with failure) vs. Ideal (failure-free)

Figure 2: Software layering of collective communication.

Figure 4: R2CC-HotRepair. Affected traffic is switched to a pre-established healthy backup path after a failure.

Figure 5: R2CC-Balance. After HotRepair, R2CC redistributes the repaired traffic across the remaining healthy paths to avoid overloading a single path.

Figure 6: R2CC-AllReduce reduces the degraded node’s load from $2D$ to $7/4D$ . In this 4-node example with lost-bandwidth fraction $1/4$ , Stage 1 runs a global AllReduce with all 4 nodes on $3/4D$ and a partial AllReduce with 3 healthy nodes on $1/4D$ , and Stage 2 broadcasts the $1/4D$ partial result back. Thus node $A$ carries $2\!\cdot\!3/4D+1/4D=7/4D$ . Since the data and bandwidth in Stage 1 are proportional, the total completion times are nearly the same for each rank.

Figure 7: Overhead Analysis. Completion-time overhead vs. no-failure as a function of lost bandwidth fraction $X$ .

Figure 8: Megatron Training Performance Evaluation

Panel of Figure 9 — Figure 9: Simulated 7B model training on 4–64 8 $\times$ A100 servers.

Figure 12: p50/p99 TTFT vs QPS under NIC failure for vLLM

Figure 13: p50/p95 TPOT under NIC failure for serving Llama-3.1-405B with TP = 8, PP = 2.

Figure 14: Inference performance under failure compared to DéjàVu and non-fault-tolerant baseline.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

$n$	$X=\tfrac{1}{8}$	$X=\tfrac{1}{4}$	$X=\tfrac{1}{2}$
4	$0.470\%$	$0.833\%$	$1.190\%$
16	$0.02419\%$	$0.04419\%$	$0.06783\%$
64	$0.00145\%$	$0.00265\%$	$0.00411\%$

Appendix

Table 1: Relative approximation gap

g(n,X)=(T(X)-T(Y^{*}))/T(Y^{*})

at representative

(n,X)

values.

Figure 17: Relative approximation gap $g(n,X)=(T(X)-T(Y^{*}))/T(Y^{*})$ of the $Y=X$ rule vs. lost bandwidth fraction $X$ , for $n\in\{4,16,64\}$ . $y$ -axis on log scale.

Failure Type	Support	When Supported (Boundary)
In-flight inter-node communication failures (R2CC’s primary scope)
NIC hardware / port failure (incl. NIC–ToR)	Yes	Node/process stays alive, and the node still has at least one remaining healthy inter-node NIC/path.
Inter-node link / cable / ToR port down (single rail/link)	Yes	An alternate inter-node path exists (e.g., another rail/NIC); not a full network partition.
RDMA transport / QP-level failure (error CQE, QP error, WQE flush)	Yes	Failure is confined to a subset of connections while endpoints remain alive and an alternate NIC/path exists.
Link flapping (up $\rightarrow$ down $\rightarrow$ up)	Partial	Only when flapping causes an in-flight transport failure (e.g. timeout) visible to the collective; pure throughput jitter is not handled.
Multi-hop fabric fault (Inter-switch link / spine switch failure)	Partial	Usually masked by fabric rerouting; if it surfaces as a transport error, handled via backup connections.

Appendix

Table 2: R2CC failure model and scope. Support means the system can keep an ongoing collective running without communicator re-initialization or job restart, under the stated boundary conditions.

Figure 18: Topology-aware logical re-ranking.

Figure 19: p50/p95 TTFT/TPOT overhead vs number of failures under multiple NIC failures for serving with pipeline parallelism

Panel of Figure 20 — Figure 20: H100 NCCL-Test Benchmark.

Figure 21: Checkpoint and restore in a training workload

Panel of Figure 22 — Figure 22: Training loss under a persistent NIC-port failure (GPT-2 124M, WikiText-103, 3 seeds).

Condition	Test PPL	Max. paired $\Delta$
NCCL 2.23.4, no failure	76.085	–
R2CC, no failure	76.085	$0.000\%$
R2CC-Balance, failure	76.085	$+0.008\%$
R2CC-AllReduce, failure	76.086	$+0.023\%$

Appendix

Table 3: Final test perplexity (mean of three seeds).

\Delta

is the worst-case increase over the NCCL baseline run with the same seed.

Replayed graph (4 GiB AllReduce)	/ iter	overhead ∗
	Time	Re-capture
Pre-failure graph, healthy	2.03 s	–
Pre-failure graph, HotRepair	4.07 s	–
Re-captured as R2CC-Balance	3.04 s	1.7 ms
Re-captured as R2CC-AllReduce	2.79 s	0.33 s

Appendix

Table 4: CUDA Graphs on three CloudLab servers, each with two V100S GPUs and three NIC ports limited to 10 Gbps. ∗ Re-capture overhead is the time of the iteration that performs the re-capture, including the eager collective that R2CC-AllReduce needs to create its sub-communicators, minus the time of an iteration replayed from the new graph.

Fault-Tolerant Distributed Training

Fault-tolerant distributed training keeps machine-learning jobs accurate and productive despite failed, delayed, corrupted, or unreliable workers and networks.

See all 23 papers

Don't Let a Few Network Failures Slow the Entire AllReduce
Jun 1, 2026Peiqing Chen, Jiedong Jiang, Nengneng Yu +4Communication-Efficient Distributed Training High-Performance Computing
University of Maryland, College Park · Utrecht University · Kyoto University
ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload
May 11, 2026Ziyue Liu, Zhengyang Wang, Ruijie Zhang +7Language Model Pretraining Fault-Tolerant Distributed Training
University of California at Santa Barbara · Argonne National Laboratory
Resilient AI Supercomputer Networking using MRC and SRv6
May 5, 2026Joao Araujo, Alex Chow, Mark Handley +47Distributed Training

Jun 1, 2026J/K move · Enter open · S save

Don't Let a Few Network Failures Slow the Entire AllReduce

Peiqing Chen, Jiedong Jiang, Nengneng Yu, Yuefeng Wang+3

University of Maryland, College Park · Utrecht University · Kyoto University

Communication-Efficient Distributed Training High-Performance Computing Fault-Tolerant Distributed Training Distributed Training

Open paper AskPDF arXiv

Network failures are among the most frequent hardware faults in large-scale GPU clusters and a leading cause of training-job interruptions. Modern collective communication libraries such as NCCL mitigate network failures by rerouting traffic through surviving NICs on the same server, trading reduced inter-node bandwidth for uninterrupted training. However, the degraded server remains on the critical path of the standard ring algorithm, slowing the entire collective. We present the first information-theoretic lower bound on AllReduce completion time under asymmetric network bandwidth and show that when the straggler retains at least half of its original bandwidth, the unavoidable overhead relative to the fault-free optimum is only O(1/p) for p GPUs. We then design OptCC, a four-stage pipelined AllReduce algorithm that approaches this lower bound. Experiments on SimAI confirm that OptCC closes the gap left by existing fault-tolerant schemes: under practical network failures with up to 50% bandwidth loss, OptCC completes AllReduce within 2-6% of NCCL's fault-free ring performance, whereas the state-of-the-art incurs up to 57% overhead.

Abstract

Fault-Tolerant Distributed Training

Explore similar work

Don't Let a Few Network Failures Slow the Entire AllReduce

ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload

Resilient AI Supercomputer Networking using MRC and SRv6