HetCCL: Efficient LLM Training on Heterogeneous Vendor GPUs
Authors: Heehoon Kim, Chris Jaehwan Lee, Taejeoung Kim, Jongwon Park, Jinpyo Kim, Pyongwon Suh, Ryan H. Choi, Sangwoo Lee, Jaejin Lee
Organizations: Moreh Inc., Seoul, South Korea · Dept. of Computer Science and Engineering, Seoul National University, Seoul, South Korea · Samsung Research, Seoul, South Korea · Graduate School of Data Science, Seoul National University, Seoul, South Korea
The rapid growth of large language models is driving organizations to expand their GPU clusters, often with GPUs from multiple vendors. However, current deep learning frameworks lack support for collective communication across heterogeneous GPUs, leading to inefficiency and higher costs. We present HetCCL, a collective communication library that unifies vendor-specific backends and enables RDMA-based communication across GPUs without requiring driver modifications. HetCCL introduces two novel mechanisms that enable cross-vendor communication while leveraging optimized vendor libraries, NVIDIA NCCL and AMD RCCL. Evaluations on a multi-vendor GPU cluster show that HetCCL matches NCCL and RCCL performance in homogeneous setups while uniquely scaling in heterogeneous environments, enabling practical, high-performance training with both NVIDIA and AMD GPUs without changes to existing deep learning applications.
Figures & tables
Figure 1: Communication between GPUs across nodes.
Figure 2: Comparison between the previous approach and HetCCL.
Figure 3: RDMA-based communication between NVIDIA and AMD GPUs in different nodes.
Figure 4
Figure 6: Multi-node execution flow on heterogeneous vendor GPU clusters.
Figure 7: Performance of point-to-point communication with RDMA mechanism.
Figure 8: Performance of representative collective communication operations.
Figure 9: LLM training speedup. 4N/8N or 4A/8A denotes homogeneous NVIDIA or AMD GPU training, while 4A+4N/8A+8N denotes heterogeneous NVIDIA+AMD GPU training.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
System
Vendor Setting
Max #GPUs
GPU Configuration
HeteroG [ 70 ]
Single-vendor
12
NVIDIA V100, P100, GTX 1080Ti
HetSeq [ 12 ]
Single-vendor
16
NVIDIA Titan Xp, GTX 1080Ti, P100
HetPipe [ 49 ]
Single-vendor
16
NVIDIA TITAN V, TITAN RTX, RTX 2060, Quadro P4000
AMP [ 31 ]
Single-vendor
16
NVIDIA V100, T4
Metis [ 66 ]
Single-vendor
32
NVIDIA P100, T4, V100
HetCCL
Cross-vendor
16
NVIDIA V100, AMD Radeon Pro W7800
Appendix
Table 1: Heterogeneity type and evaluation scale of representative related systems.
NVIDIA (2 nodes)
AMD (2 nodes)
Mainboard
GIGABYTE MZ52-G40
ASRock ROMED8-2T
CPU
1 × AMD EPYC 7452 32-Core
2 × AMD EPYC 7313 16-Core
Memory
8 × DDR4-2666 64GB
4 × DDR4-3200 16GB
GPU
4 × NVIDIA Tesla V100-PCIe
4 × AMD Radeon Pro W7800
NIC
Mellanox ConnectX-6 InfiniBand HDR
Mellanox ConnectX-6 InfiniBand HDR
GPU PCIe
Gen3
Gen4
Appendix
Table 2: Hardware configuration of the four-node multi-vendor GPU cluster.
Model
Parameter size
Sequence length
Vocab size
GPT
125M
1024
50257
355M
LLaMA
1B
8192
32000
3B
Appendix
Table 3: Specifications of models used in the experiments.
ZeRO Stage
Stage 1
Stage 2
Stage 3 (FSDP)
Optimizer State (OS)
Sharded
Sharded
Sharded
Gradient (G)
Replicated
Sharded
Sharded
Parameter (P)
Replicated
Replicated
Sharded
Communication Ops
All-Gather (OS), All-Reduce (G)
All-Gather (P), Reduce-Scatter (G)
All-Gather (P), Reduce-Scatter (G)
Appendix
Table 4: Communication patterns across DeepSpeed ZeRO stages.
Figure 11: Performance of other representative collective communication operations.
Figure 12: Training loss convergence of LLaMA-1B on WikiText-103.
CCL (GPU Vendor)
Validation PPL
Test PPL
NCCL (NVIDIA GPUs)
15.70
16.30
RCCL (AMD GPUs)
15.54
16.15
HetCCL (NVIDIA + AMD GPUs)
15.77
16.47
Appendix
Table 5: Validation and test perplexity of LLaMA-1B on WikiText-103.
Figure 13: Collective communication performance across different NCCL versions.
Figure 14: Point-to-point communication performance across different NCCL versions.
CCL (# of GPUs)
LLaMA 7B
LLaMA 13B
Throughput (tokens/s)
Speedup
Throughput (tokens/s)
Speedup
RCCL (8xAMD)
691
1.00
386
1.00
NCCL (8xNVIDIA)
1289
1.87
716
1.85
HetCCL (4xAMD + 4xNVIDIA)
932
1.35
537
1.39
HetCCL (8xAMD + 8xNVIDIA)
1806
2.61
1021
2.64
Appendix
Table 6: LLaMA 7B and 13B model training with Megatron-DeepSpeed.
CCL (# of GPUs)
Throughput (tokens/s)
Speedup
RCCL (8xAMD)
2792
1.00
NCCL (8xNVIDIA)
4287
1.54
HetCCL (4xAMD + 4xNVIDIA)
3435
1.23
HetCCL (8xAMD + 8xNVIDIA)
6441
2.31
Appendix
Table 7: Sparse MoE model training with expert parallelism.
Figure 15: Point-to-point communication performance: GPU-aware MPI vs. HetCCL.
Figure 16: Collective communication performance on 16 GPUs: GPU-aware MPI vs. HetCCL.
Figure 17: Single node All-Reduce performance on NVIDIA and AMD high-end GPU systems.
Operation
NCCL
RCCL
HetCCL (NVIDIA)
HetCCL (AMD)
HetCCL (HET)
All-Reduce
5.80±0.02
7.17±0.02
5.80±0.02
7.85±0.03
5.81±0.02
All-Gather
5.83±0.01
7.37±0.02
5.82±0.03
7.55±0.04
5.84±0.02
Reduce-Scatter
5.85±0.02
7.35±0.03
5.84±0.02
7.91±0.03
5.84±0.02
Appendix
Table 8: Run-to-run variability in collective communication bandwidth across two nodes. Values are mean ± standard deviation over three independent runs, in GB/s.
Configuration
Throughput (tokens/s)
RCCL ( 8× AMD)
10,718±141
NCCL ( 8× NVIDIA)
20,889±20
HetCCL ( 8× AMD + 8× NVIDIA)
30,640±229
Appendix
Table 9: Run-to-run variability in end-to-end training throughput for LLaMA-1B with ZeRO-3 and FP16. Values are mean ± standard deviation over three independent runs.
Figure 18: Performance impact of RDMA on point-to-point GPU communication.
Model
End-to-end speedup
Profiling overhead (s)
GPT 125M
1.22 ×
1.4
GPT 355M
1.19 ×
2.9
LLaMA 1B
1.09 ×
26.6
LLaMA 3B
1.08 ×
80.2
Appendix
Table 10: Performance impact of load balancing on LLM training.
Model
Communication (%)
Computation (%)
Synchronization (%)
GPT 125M
13.26
86.72
0.02
GPT 355M
16.96
83.03
0.01
LLaMA 1B
22.85
77.14
0.01
LLaMA 3B
29.09
70.91
0.00
Appendix
Table 11: End-to-end training latency breakdown across models.