cs.DCJan 30, 2026

HetCCL: Efficient LLM Training on Heterogeneous Vendor GPUs

Authors: Heehoon Kim, Chris Jaehwan Lee, Taejeoung Kim, Jongwon Park, Jinpyo Kim, Pyongwon Suh, Ryan H. Choi, Sangwoo Lee, 

Organizations: Moreh Inc., Seoul, South Korea · Dept. of Computer Science and Engineering, Seoul National University, Seoul, South Korea · Samsung Research, Seoul, South Korea · Graduate School of Data Science, Seoul National University, Seoul, South Korea

Abstract

The rapid growth of large language models is driving organizations to expand their GPU clusters, often with GPUs from multiple vendors. However, current deep learning frameworks lack support for collective communication across heterogeneous GPUs, leading to inefficiency and higher costs. We present HetCCL, a collective communication library that unifies vendor-specific backends and enables RDMA-based communication across GPUs without requiring driver modifications. HetCCL introduces two novel mechanisms that enable cross-vendor communication while leveraging optimized vendor libraries, NVIDIA NCCL and AMD RCCL. Evaluations on a multi-vendor GPU cluster show that HetCCL matches NCCL and RCCL performance in homogeneous setups while uniquely scaling in heterogeneous environments, enabling practical, high-performance training with both NVIDIA and AMD GPUs without changes to existing deep learning applications.

Figures & tables

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. HetCCL: Enabling Collective Communication For Mixed-Vendor Heterogeneous Clusters

    May 29, 2026Yuejie Wang, Tao Chang, Yuanyuan Zhao +10Communication-Efficient Distributed TrainingHigh-Performance Computing

  2. T-CCL: Resource Efficient and Performant Collective Communication using Tensor Memory Accelerator

    Oct 5, 2026Keyvan Dadashzadeh, Yuehong Zhou, Minyu Cui +1GPU Kernel OptimizationLLM Inference Acceleration

  3. UCCL-Zip: Lossless Compression Supercharged GPU Communication

    Apr 19, 2026Shuang Ma, Chon Lam Lao, Zhiying Xu +8GPU Kernel OptimizationGPU Acceleration