CHARM+: Cross-Hardware Attention with Re-Merge Multistream Mechanism
Organizations: Lenovo Research
Abstract
As sequence lengths continue to grow, distributed attention based on sequence parallel (e.g., Ring-Attention) has become a mainstream solution for large language models (LLMs). Nevertheless, inherent communication dependencies limit computation-communication overlap, reducing GPU utilization. Moreover, in cross-hardware (NUMA or node) architectures, due to imbalanced communication bandwidth, frequent communication synchronization will limit overall performance. Based on these insights, we propose CHARM+, Cross-Hardware Attention with Re-Merge Multistream Mechanism. We reorganize sequential communications into several levels and achieve intra-level parallelism and computation overlap. Furthermore, we decouple cross-hardware communication from intra-hardware communication, eliminating synchronization overhead caused by bandwidth imbalance. Forward and backward algorithms were evaluated on multiple hardware architectures and consistently outperformed state-of-the-art methods. Compared to Ring-Attention, our forward and backward algorithms achieve average speedups of 1.3x and 1.7x on a dual-NUMA PCIe architecture, and 2.8x and 2.9x on a dual-node NVLink architecture, respectively.
Figures & tables
| Notation | Description | Notation | Description |
|---|---|---|---|
| Query matrix | Key matrix | ||
| Value matrix | Degree of sequence parallelism | ||
| Unhidden communication time | Function that clips value to zero | ||
| Communication time of a tensor | Computation time of Attention |
| GPU | NVLink | Num | TFLOPS | Bandwidth (GB/s) | |
|---|---|---|---|---|---|
| Cross | Intra | ||||
| L20 | ✗ | 8 | 119.5 | 14.7 | 24.0 |
| H20 | ✓ | 16 | 148.0 | 42.6 | 297.2 |
| H100 | ✓ | 32 | 989.0 | 42.5 | 365.1 |
| Batch | Seq | Heads | Ring | CHARM+ | Batch | Seq | Heads | Ring | CHARM+ |
|---|---|---|---|---|---|---|---|---|---|
| Size | Len | (MB) | (MB) | Size | Len | (MB) | (MB) | ||
| 16 | 1024 | 32 | 2,179 | 2,215 | 2 | 53,760 | 12 | 13,123 | 13,467 |
| 16 | 1024 | 64 | 3,701 | 3,881 | 4 | 53,760 | 12 | 19,301 | 20,565 |
| 64 | 8192 | 32 | 45,079 | 45,423 | 2 | 26,880 | 12 | 10,745 | 10,803 |
| 128 | 8192 | 16 | 44,720 | 45,432 | 4 | 26,880 | 12 | 14,029 | 14,425 |
| GPUs | Cross | Seq | Forward Time (ms) | ||
|---|---|---|---|---|---|
| Ring | RM+ | Ours | |||
| 4 L20 | ✗ | 1792 | 9.35 | 10.29 | – |
| 7168 | 36.86 | 39.66 | – | ||
| 8 L20 | ✓ | 1792 | 17.61 | 25.91 | 14.81 |
| 7168 | 69.85 | 98.49 | 52.40 | ||
| 4 H20 | ✗ | 1792 | 5.38 | 4.76 | – |
| B | Seq | Forward | Backward | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Ring | TASP | CHARM+ | Speedup | Ring | TASP | CHARM+ | Speedup | ||
| GPUs = | |||||||||
| 16 | 26,880 | 149.97 | 98.01 | 63.93 | 2.35 | 449.58 | – | 141.52 | 3.18 |
| 16 | 13,440 | 72.65 | 49.27 | 31.84 | 2.28 | 226.46 | – | 65.35 | 3.47 |
| 64 | 6,720 | 141.75 | 97.65 | 61.80 | 2.29 | 448.84 | – | 123.32 | 3.64 |
| 128 | 3,360 | 140.55 | 97.15 | 61.54 | 2.28 | 450.01 | – | 121.99 | 3.69 |
| Batch Size | Seq Len | Model | Ring | CHARM+ | Speedup | ||
|---|---|---|---|---|---|---|---|
| Latency | Memory | Latency | Memory | ||||
| 16 | 2,048 | GPT-350M | 10082.6 | 11,372 | 4847.0 | 11,518 | 2.08 |
| 8 | 4,096 | GPT-350M | 10032.6 | 11,348 | 4871.5 | 11,544 | 2.06 |
| 4 | 8,192 | GPT-350M | 9914.5 | 11,436 | 5173.6 | 11,552 | 1.92 |
| 2 | 16,384 | GPT-350M | 10089.5 | 12,112 | 5637.3 | 12,170 | 1.79 |
| 16 | 2,048 | GPT-1.3B | 21505.5 | 30,816 | 11738.3 | 31,602 | 1.83 |
| Batch Size | Seq Len | Model | Ring | CHARM+ | Speedup | |||
|---|---|---|---|---|---|---|---|---|
| H20 | H100 | H20 | H100 | H20 | H100 | |||
| 16 | 16,384 | GPT3-350M | 7304.6 | 3385.1 | 5601.3 | 1925.7 | 1.30 | 1.76 |
| 32 | 8,192 | GPT3-350M | 7193.7 | 3291.5 | 4669.7 | 1679.2 | 1.54 | 1.96 |
| 64 | 4,096 | GPT3-350M | 7169.6 | 3234.6 | 4452.5 | 1538.6 | 1.61 | 2.10 |
| 128 | 2,048 | GPT3-350M | 7143.6 | 3210.8 | 4242.5 | 1482.0 | 1.68 | 2.17 |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Diff | CHARM+ (Max) | CHARM+ (Mean) | Ring (Max) | Ring (Mean) |
|---|---|---|---|---|
| out | 8.34147e-04 | 4.51229e-05 | 8.07648e-04 | 4.24147e-05 |
| lse | 1.65642e-06 | 2.94155e-07 | 1.75893e-06 | 3.04284e-07 |
| dq | 1.26290e-03 | 3.29077e-05 | 1.25977e-03 | 3.27659e-05 |
| dk | 2.87713e-03 | 3.53419e-05 | 2.89413e-03 | 3.53193e-05 |
| dv | 4.47177e-03 | 3.42940e-05 | 4.48120e-03 | 3.42894e-05 |
| B | Seq | CCR | Ring | TASP | CHARM+ | Speedup | ||
|---|---|---|---|---|---|---|---|---|
| Seq = 215,040 | ||||||||
| 1 | 215,040 | 81.49 | 86.01 | 1.06 | 116.76 | 86.93 | 66.74 | 1.75 |
| 2 | 215,040 | 164.31 | 171.51 | 1.04 | 241.03 | 171.91 | 125.43 | 1.92 |
| Seq = 107,520 | ||||||||
| 1 | 107,520 | 38.34 | 23.16 | 0.60 | 44.08 | 26.36 | 21.43 | 2.06 |
| 2 | 107,520 | 74.25 | 45.20 | 0.61 | 89.20 | 51.46 | 42.28 | 2.11 |
| B | Seq | CCR | Ring | TASP | CHARM+ | Speedup | ||
|---|---|---|---|---|---|---|---|---|
| Seq = 215,040 | ||||||||
| 1 | 215,040 | 218.48 | 257.76 | 1.18 | 312.45 | – | 207.07 | 1.51 |
| 2 | 215,040 | 437.82 | 503.06 | 1.15 | 628.21 | – | 405.90 | 1.55 |
| Seq = 107,520 | ||||||||
| 1 | 107,520 | 109.45 | 68.03 | 0.62 | 114.64 | – | 64.09 | 1.79 |
| 2 | 107,520 | 218.37 | 135.44 | 0.62 | 228.70 | – | 125.01 | 1.83 |
| B | Seq | CCR | Ring | TASP | CHARM+ | Speedup | ||
|---|---|---|---|---|---|---|---|---|
| Seq = 126,976 | ||||||||
| 2 | 126,976 | 95.28 | 36.30 | 0.38 | 102.34 | 94.17 | 39.95 | 2.56 |
| 4 | 126,976 | 186.17 | 70.03 | 0.38 | 208.12 | 184.59 | 77.02 | 2.70 |
| Seq = 63,488 | ||||||||
| 2 | 63,488 | 49.95 | 12.50 | 0.25 | 46.32 | 48.54 | 19.80 | 2.34 |
| 4 | 63,488 | 96.13 | 22.28 | 0.23 | 95.79 | 93.60 | 36.62 | 2.62 |
| B | Seq | CCR | Ring | TASP | CHARM+ | Speedup | ||
|---|---|---|---|---|---|---|---|---|
| Seq = 126,976 | ||||||||
| 2 | 126,976 | 259.62 | 102.57 | 0.40 | 280.48 | – | 111.60 | 2.51 |
| 4 | 126,976 | 521.26 | 196.56 | 0.38 | 559.74 | – | 213.85 | 2.62 |
| Seq = 63,488 | ||||||||
| 2 | 63,488 | 138.65 | 29.95 | 0.22 | 141.18 | – | 46.30 | 3.05 |
| 4 | 63,488 | 269.38 | 58.46 | 0.22 | 280.96 | – | 86.70 | 3.24 |
Communication-Efficient Distributed Training
Communication-efficient distributed training reduces the data exchanged among devices so models can be trained faster and over limited-bandwidth networks.
See all 245 papers