HetCCL: Efficient LLM Training on Heterogeneous Vendor GPUs
Authors: Heehoon Kim, Chris Jaehwan Lee, Taejeoung Kim, Jongwon Park, Jinpyo Kim, Pyongwon Suh, Ryan H. Choi, Sangwoo Lee, Jaejin Lee
Organizations: Moreh Inc., Seoul, South Korea · Dept. of Computer Science and Engineering, Seoul National University, Seoul, South Korea · Samsung Research, Seoul, South Korea · Graduate School of Data Science, Seoul National University, Seoul, South Korea
The rapid growth of large language models is driving organizations to expand their GPU clusters, often with GPUs from multiple vendors. However, current deep learning frameworks lack support for collective communication across heterogeneous GPUs, leading to inefficiency and higher costs. We present HetCCL, a collective communication library that unifies vendor-specific backends and enables RDMA-based communication across GPUs without requiring driver modifications. HetCCL introduces two novel mechanisms that enable cross-vendor communication while leveraging optimized vendor libraries, NVIDIA NCCL and AMD RCCL. Evaluations on a multi-vendor GPU cluster show that HetCCL matches NCCL and RCCL performance in homogeneous setups while uniquely scaling in heterogeneous environments, enabling practical, high-performance training with both NVIDIA and AMD GPUs without changes to existing deep learning applications.
Figures & tables
Figure 1: Communication between GPUs across nodes.
Figure 2: Comparison between the previous approach and HetCCL.
Figure 3: RDMA-based communication between NVIDIA and AMD GPUs in different nodes.
Figure 4
Figure 6: Multi-node execution flow on heterogeneous vendor GPU clusters.
Figure 7: Performance of point-to-point communication with RDMA mechanism.
Figure 8: Performance of representative collective communication operations.
Figure 9: LLM training speedup. 4N/8N or 4A/8A denotes homogeneous NVIDIA or AMD GPU training, while 4A+4N/8A+8N denotes heterogeneous NVIDIA+AMD GPU training.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
System
Vendor Setting
Max #GPUs
GPU Configuration
HeteroG [ 70 ]
Single-vendor
12
NVIDIA V100, P100, GTX 1080Ti
HetSeq [ 12 ]
Single-vendor
16
NVIDIA Titan Xp, GTX 1080Ti, P100
HetPipe [ 49 ]
Single-vendor
16
NVIDIA TITAN V, TITAN RTX, RTX 2060, Quadro P4000
AMP [ 31 ]
Single-vendor
16
NVIDIA V100, T4
Metis [ 66 ]
Single-vendor
32
NVIDIA P100, T4, V100
HetCCL
Cross-vendor
16
NVIDIA V100, AMD Radeon Pro W7800
Appendix
Table 1: Heterogeneity type and evaluation scale of representative related systems.
NVIDIA (2 nodes)
AMD (2 nodes)
Mainboard
GIGABYTE MZ52-G40
ASRock ROMED8-2T
CPU
1 × AMD EPYC 7452 32-Core
2 × AMD EPYC 7313 16-Core
Memory
8 × DDR4-2666 64GB
4 × DDR4-3200 16GB
GPU
4 × NVIDIA Tesla V100-PCIe
4 × AMD Radeon Pro W7800
NIC
Mellanox ConnectX-6 InfiniBand HDR
Mellanox ConnectX-6 InfiniBand HDR
GPU PCIe
Gen3
Gen4
Appendix
Table 2: Hardware configuration of the four-node multi-vendor GPU cluster.
Model
Parameter size
Sequence length
Vocab size
GPT
125M
1024
50257
355M
LLaMA
1B
8192
32000
3B
Appendix
Table 3: Specifications of models used in the experiments.
ZeRO Stage
Stage 1
Stage 2
Stage 3 (FSDP)
Optimizer State (OS)
Sharded
Sharded
Sharded
Gradient (G)
Replicated
Sharded
Sharded
Parameter (P)
Replicated
Replicated
Sharded
Communication Ops
All-Gather (OS), All-Reduce (G)
All-Gather (P), Reduce-Scatter (G)
All-Gather (P), Reduce-Scatter (G)
Appendix
Table 4: Communication patterns across DeepSpeed ZeRO stages.
Figure 11: Performance of other representative collective communication operations.
Figure 12: Training loss convergence of LLaMA-1B on WikiText-103.
CCL (GPU Vendor)
Validation PPL
Test PPL
NCCL (NVIDIA GPUs)
15.70
16.30
RCCL (AMD GPUs)
15.54
16.15
HetCCL (NVIDIA + AMD GPUs)
15.77
16.47
Appendix
Table 5: Validation and test perplexity of LLaMA-1B on WikiText-103.
Figure 13: Collective communication performance across different NCCL versions.
Figure 14: Point-to-point communication performance across different NCCL versions.
CCL (# of GPUs)
LLaMA 7B
LLaMA 13B
Throughput (tokens/s)
Speedup
Throughput (tokens/s)
Speedup
RCCL (8xAMD)
691
1.00
386
1.00
NCCL (8xNVIDIA)
1289
1.87
716
1.85
HetCCL (4xAMD + 4xNVIDIA)
932
1.35
537
1.39
HetCCL (8xAMD + 8xNVIDIA)
1806
2.61
1021
2.64
Appendix
Table 6: LLaMA 7B and 13B model training with Megatron-DeepSpeed.
CCL (# of GPUs)
Throughput (tokens/s)
Speedup
RCCL (8xAMD)
2792
1.00
NCCL (8xNVIDIA)
4287
1.54
HetCCL (4xAMD + 4xNVIDIA)
3435
1.23
HetCCL (8xAMD + 8xNVIDIA)
6441
2.31
Appendix
Table 7: Sparse MoE model training with expert parallelism.
Figure 15: Point-to-point communication performance: GPU-aware MPI vs. HetCCL.
Figure 16: Collective communication performance on 16 GPUs: GPU-aware MPI vs. HetCCL.
Figure 17: Single node All-Reduce performance on NVIDIA and AMD high-end GPU systems.
Operation
NCCL
RCCL
HetCCL (NVIDIA)
HetCCL (AMD)
HetCCL (HET)
All-Reduce
5.80±0.02
7.17±0.02
5.80±0.02
7.85±0.03
5.81±0.02
All-Gather
5.83±0.01
7.37±0.02
5.82±0.03
7.55±0.04
5.84±0.02
Reduce-Scatter
5.85±0.02
7.35±0.03
5.84±0.02
7.91±0.03
5.84±0.02
Appendix
Table 8: Run-to-run variability in collective communication bandwidth across two nodes. Values are mean ± standard deviation over three independent runs, in GB/s.
Configuration
Throughput (tokens/s)
RCCL ( 8× AMD)
10,718±141
NCCL ( 8× NVIDIA)
20,889±20
HetCCL ( 8× AMD + 8× NVIDIA)
30,640±229
Appendix
Table 9: Run-to-run variability in end-to-end training throughput for LLaMA-1B with ZeRO-3 and FP16. Values are mean ± standard deviation over three independent runs.
Figure 18: Performance impact of RDMA on point-to-point GPU communication.
Model
End-to-end speedup
Profiling overhead (s)
GPT 125M
1.22 ×
1.4
GPT 355M
1.19 ×
2.9
LLaMA 1B
1.09 ×
26.6
LLaMA 3B
1.08 ×
80.2
Appendix
Table 10: Performance impact of load balancing on LLM training.
Model
Communication (%)
Computation (%)
Synchronization (%)
GPT 125M
13.26
86.72
0.02
GPT 355M
16.96
83.03
0.01
LLaMA 1B
22.85
77.14
0.01
LLaMA 3B
29.09
70.91
0.00
Appendix
Table 11: End-to-end training latency breakdown across models.
Training Large Language Models (LLMs) on heterogeneous clusters presents significant challenges for collective communication, as hardware from multiple vendors introduces diverse network and computational characteristics. Existing collective communication frameworks (e.g., NCCL, RCCL) designed for homogeneous environments fail to address mixed-hardware setups, while communication libraries with heterogeneous support (e.g., Gloo, OpenMPI) incur heavy overhead in the data path. This paper presents HetCCL, a framework that enables heterogeneous collective communication by efficient P2P transport across heterogeneous devices (e.g., GPUs), eliminating the host-device memory copy overhead while offloading the control to the CPUs. For combining collectives (e.g., AllReduce, ReduceScatter), HetCCL introduces a border-communicator mechanism that achieves vendor independence by using the intrinsic reduction in the combining collectives in vendor collective communication libraries. With efficient heterogeneous P2P transport and portable reduction mechanism, HetCCL proposes a hierarchical topology abstraction for heterogeneous clusters, dissecting collective communication into cluster-level primitives that guarantee optimal cross-cluster data transfer volume and optimal bandwidth utilization. We implement HetCCL with 4 different vendor support and evaluate it in 4 heterogeneous settings with benchmarks and end-to-end LLM tasks. Our evaluation shows that HetCCL achieves 17-19x higher bandwidth than Gloo in heterogeneous communications, and speeds up end-to-end training by up to 16.9% in the per-step-time.
Yuejie Wang, Tao Chang, Yuanyuan Zhao +10
Peking University · Beijing Academy of Artificial Intelligence · Infrawaves +1
Large transformer-based models increasingly depend on multi-GPU execution, which requires frequent collective communication among GPUs. Existing communication libraries often rely on many GPU threads to achieve high bandwidth or low latency, resulting in a large streaming multiprocessor (SM)-side resource footprint. This footprint can limit the resources available to other GPU work, particularly when communication and computation execute concurrently. Thus, efficient collective communication should not only achieve high collective performance but also reduce its SM-side resource usage. This paper presents T-CCL, a resource-efficient collective communication library based on the Tensor Memory Accelerator (TMA) for intra-node communication. T-CCL offloads both data movement and reduction operations to TMA and executes each collective as a pipelined series of asynchronous TMA operations, reducing the SM resources required for collective communication while maintaining high bandwidth. Evaluated across AllReduce, AllGather, and ReduceScatter collectives, T-CCL outperforms NCCL by up to 2.4x with unrestricted communication resources and up to 3.42x under restricted resource budgets, remains competitive with NCCL's recent symmetric-memory kernels, and occupies the same or fewer SMs in profiled cases. In a GEMM-collective overlap case study, switching the communication backend from NCCL to T-CCL raises the average operator-level speedup over a sequential baseline from 1.12x to 1.25x on two GPUs and from 1.04x to 1.14x on four GPUs, as T-CCL uses fewer SMs for communication, leaving more SMs available to the overlapped GEMM. Integrated into vLLM as a communication backend, T-CCL improves end-to-end inference throughput over vLLM's automatic backend dispatch by up to 1.31x, outperforming it at every evaluated batch size on both the conversation and decode-heavy workloads.
Keyvan Dadashzadeh, Yuehong Zhou, Minyu Cui +1
Chalmers University of Technology and University of Gothenburg Gothenburg, Sweden · Linnaeus University Växjö, Sweden
The rapid growth of large language models (LLMs) has made GPU communication a critical bottleneck. While prior work reduces communication volume via quantization or lossy compression, these approaches introduce numerical errors that can degrade convergence, accuracy, and stability. We present UCCL-Zip, a unified design that integrates lossless compression directly into GPU communication primitives. UCCL-Zip supports both point-to-point (P2P) and collective communication without modifying user-facing APIs or compromising numerical correctness. For P2P communication, Uzip-P2P employs a split-send pipeline that exposes transmissible data early and overlaps compression with communication, while preserving high GPU efficiency by operating on large data blocks. For collective communication, Uzip-NCCL integrates compression into NCCL's persistent kernel model via fused execution, eliminating redundant memory traffic and kernel launches. In real workloads, UCCL-Zip accelerates RL weight synchronization by up to 47.5% and reduces vLLM end-to-end inference latency by up to 10%, all without application changes.
Shuang Ma, Chon Lam Lao, Zhiying Xu +8
UC Davis · Harvard University · Amazon Web Services +1