Distributed AI training involves recurring rounds of data exchange between multiple pairs of GPU nodes. Slowdown in even one flow due to congestion can cause the entire communication round to slowdown. Current approaches for evading congestion in AI clusters assume global control over the entire workload (e.g. coordinating the schedule of all jobs) or assume infrastructural support (e.g. adaptive routing in switches). They are thus ill-suited in a shared cloud setting where AI jobs belonging to one user can face external congestion from other users' jobs or background traffic beyond its own control. In this paper, we build a system, REACT, that tunes the recurring pattern of data exchange between GPU nodes (known as communication collectives) in response to congestion. REACT works at the application (communication library) layer, where it detects congestion at runtime using readily available flow stats, and tunes the collective pattern to alleviate congestion - changing the set of incident flows while retaining the semantics of information exchange (e.g. selecting which node aggregates data in an AllReduce tree). REACT requires no explicit support from the underlying network infrastructure and can be unilaterally deployed by individual users in a shared cloud setting. We prototype REACT as a shim layer over NCCL, and evaluate it on a shared academic GPU cluster - enabling REACT improves communication performance (algorithm bandwidth) by 13%-38% under network congestion. Our simulations across a range of congestion scenarios further reveal up to 75% performance improvement, highlighting the effectiveness of our approach.
Figures & tables
Figure 1: Example showing benefit of swapping the congested node 3 with 2 in NCCL Tree-Allreduce algorithm. (a) and (b) show the logical flows between nodes in Tree-Allreduce. (c) and (d) show how the congested physical link is avoided by interchanging positions in the logical tree algorithm (the red and blue arrows are the chunks sent from 1 and 5 to 3 before and to 2 after replacement).
Figure 2: TEN of NCCL dual-tree AllReduce
Transforms
Ring
Recursive D.
Tree
①
Spine, Failure
All
Spine, Failure
②
✘
✘
All
③
✘
Spine, Failure
All
Table 1: Comparing the types of congestion (§ 5.2 ) resolved by the transforms for popular CAs (Ring-based algorithms, Recursive Doubling AllGather, Tree based algorithms). Spine is Spine-ToR congestion, Failure is network failures, All is all congestion types in § 5.2 . (✘) marks that the transform is not able to help with the CP, topology and congestion scenarios we consider.
Figure 3: State diagram of REACT . The orange box marks the distributed training workload running on the GPUs that uses the input collective pattern for communication. The gray boxes run on a separate thread on the CPU– collecting flow level data from all nodes, running the control loop, finding a new collective pattern, and sharing it with the training job processes on all nodes.
Figure 4: REACT architecture.
#Nodes
p
Speedup ×
# changes
Mean
p95
4
0
0.964
0.999
34
1
1.138
1.101
35.6
2
1.348
1.145
33.2
8
0
0.963
0.980
41.25
1
1.180
1.165
40
Table 2: Ratio of mean (or p95) collective completion time over epochs for baseline to REACT , across node counts and congestion degree, p , for Tree AllReduce on GPU cluster deployment.
Figure 5: Mean algorithm bandwidth improvement for a sweep of 10 randomly generated scenarios under various degrees of network congestion on a Fattree topology and 20MB message size. x -axis marks the number of source and destination pairs that initiate p external flows each.
Figure 6: Percentage performance improvement with REACT enabled for both jobs compared to without REACT, under different degrees of overlap. Job 1 runs 8 node NCCL Tree ALLReduce and Job 2 runs Ring AllReduce. Both run 20MB message sizes on a Star topology. We run 5 random seeds for each overlap scenario.
Figure 7: Running 2 jobs simultaneously from Alibaba trace with only network sharing (Spine-ToR congestion). (a) runs two Ring AllReduce jobs simultaneously from Alibaba trace. (b) runs two jobs- one Recursive Doubling algorithm and other with Ring AllReduce- simultaneously. We compare performance when REACT is only enabled for Job 1 and when it is enabled for both jobs compared to without REACT (gray). The numbers on the bar are the percent change from respective baselines.
Figure 8: 8 node nccl Tree Allreduce, 20MB message size, Fattree topology, partial link failures.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Ring (Fig 9(a) ) and Recursive Doubling Collective (Fig 9(b) ) patterns. Fig 9(c) shows a Global permutation Swap for Ring AllReduce. If 0→1 is congested at the link between two red switches, swapping 1 and 2 can alleviate congestion.
Parameters
Definition
B
Batch size of epochs
δ1
Steady congestion parameter (§ 5.2 )
δ2
irregular congestion parameter (§ 5.2 )
Kt
Number of trials each turn
ϵ
Threshold of min. expected perf. improvement
H
How much older data we maintain
Appendix
Table 3: Parameters used by REACT
Figure 10: The intra-node and network topology in the shared academic GPU cluster. Within a server, four A100 GPUs are interconnected by NVLink (not shown for clarity). The ToRs in Slingshot network topology are all fully connected with each other. The hsn -high speed interface on each server is connected to exactly one TOR. Blue nodes show an example 4 node job, red arrows marks an external congested flow injected to model congestion.
Figure 11: CDF of epoch times for TreeAllReduce measured on the shared GPU cluster.
Figure 12: 8 node nccl Tree Allreduce, 20MB message size, Star topology
Figure 13: Algorithm bandwidth improvement in a fattree topology for the p95 case. Comparing the effects of ECMP and collective tuning in alleviating latency deterioration due to network congestion. Increasing ECMP ports/QPs comes at additional cost while collective tuning works even with less QP configurations.
Network failures are among the most frequent hardware faults in large-scale GPU clusters and a leading cause of training-job interruptions. Modern collective communication libraries such as NCCL mitigate network failures by rerouting traffic through surviving NICs on the same server, trading reduced inter-node bandwidth for uninterrupted training. However, the degraded server remains on the critical path of the standard ring algorithm, slowing the entire collective. We present the first information-theoretic lower bound on AllReduce completion time under asymmetric network bandwidth and show that when the straggler retains at least half of its original bandwidth, the unavoidable overhead relative to the fault-free optimum is only O(1/p) for p GPUs. We then design OptCC, a four-stage pipelined AllReduce algorithm that approaches this lower bound. Experiments on SimAI confirm that OptCC closes the gap left by existing fault-tolerant schemes: under practical network failures with up to 50% bandwidth loss, OptCC completes AllReduce within 2-6% of NCCL's fault-free ring performance, whereas the state-of-the-art incurs up to 57% overhead.
Peiqing Chen, Jiedong Jiang, Nengneng Yu +4
University of Maryland, College Park · Utrecht University · Kyoto University
Deep learning training commonly runs on shared multi-tenant GPU servers, where exclusive allocation provides isolation but can leave resources underutilized and increase queueing time. Collocation can improve efficiency, but interference-agnostic placement may cause severe slowdowns, while inaccurate memory information can lead to out-of-memory (OOM) failures. We present AEGIS, a server-scale runtime scheduling system for controlled collocation of deep learning training workloads on shared multi-GPU servers. AEGIS integrates memory feasibility, post-placement observation, runtime-pressure filtering, placement, and OOM-aware recovery in a single scheduling loop. After placement, AEGIS observes workload activity before permitting further collocation, then uses low-overhead telemetry to determine whether a GPU can safely accept additional work. OOM failures trigger retries under progressively safer memory conditions, eventually falling back to exclusive execution. This online approach avoids costly offline pairwise compatibility profiling. We evaluate AEGIS using vision, Transformer, recommendation, and LLM-style workloads across three production-derived traces. AEGIS reduces geometric-mean makespan by 16% relative to Lucid, 21% relative to Horus, and 27% relative to exclusive allocation. Sensitivity studies show that activity-anchored observation and runtime-pressure filtering balance conservative isolation against interference-agnostic collocation, improving makespan while limiting sharing-induced per-task slowdown.
Ehsan Yousefzadeh-Asl-Miandoab, Büşra Karatay Demiray, Florina M. Ciorba +2
IT University of Copenhagen Denmark · University of Basel Switzerland
Modern ML training and inference now span hundreds to tens of thousands of GPUs, where failures can waste 10-15% of GPU hours due to slow recovery. Common network errors and link fluctuations can cause up to 50% of job interruptions, triggering timeouts that often terminate entire jobs, forcing expensive checkpoint rollback during training and request reprocessing during inference. We present R2CC, a fault-tolerant communication library that provides lossless, low-overhead failover by exploiting multi-NIC hardware. R2CC performs rapid connection migration, bandwidth-aware load redistribution, and resilient collective algorithms to maintain progress under failures. We evaluate R2CC on four 8-GPU H100 InfiniBand servers and via large-scale ML simulators modeling up to 1024 GPUs with diverse failure patterns. Experiments show that R2CC is highly robust to inter-node network failures, incurring less than 1.1% training and 3% inference overhead under active failure scenarios. Compared to existing fault-tolerant approaches, R2CC reduces failure-induced overhead by up to 92% for training and 98% for inference. R2CC is open-sourced at https://github.com/r2cc-project/R2CC.