As sequence lengths continue to grow, distributed attention based on sequence parallel (e.g., Ring-Attention) has become a mainstream solution for large language models (LLMs). Nevertheless, inherent communication dependencies limit computation-communication overlap, reducing GPU utilization. Moreover, in cross-hardware (NUMA or node) architectures, due to imbalanced communication bandwidth, frequent communication synchronization will limit overall performance. Based on these insights, we propose CHARM+, Cross-Hardware Attention with Re-Merge Multistream Mechanism. We reorganize sequential communications into several levels and achieve intra-level parallelism and computation overlap. Furthermore, we decouple cross-hardware communication from intra-hardware communication, eliminating synchronization overhead caused by bandwidth imbalance. Forward and backward algorithms were evaluated on multiple hardware architectures and consistently outperformed state-of-the-art methods. Compared to Ring-Attention, our forward and backward algorithms achieve average speedups of 1.3x and 1.7x on a dual-NUMA PCIe architecture, and 2.8x and 2.9x on a dual-node NVLink architecture, respectively.
Figures & tables
Figure 1: Communication under PCIe and NVLink. The left of (a) shows single communication at full bandwidth without contention, while the right shows multiple communications at limited bandwidth with contention. (b) shows multiple communications at full bandwidth without contention.
Figure 2: GPU Idle Time. By setting various sequence lengths, we compare GPU idle time with Ring-Attention and TASP. Our method achieves higher GPU utilization.
Figure 3: Overall Framework of CHARM+. (a) - (d) illustrate the complete process of the algorithm using an 8-GPU system with a dual-NUMA configuration. (a) displays parallel communication of q is first performed across NUMA. As a result, GPU 3 receives the q from GPU 7, eliminating the need for further cross-NUMA communication. (b) shows the parallel communication and computation of q and kv . In this step, GPU 3 communicates q with GPU 0 and kv with GPU 1. After completing the computation of Q0,4 , GPU3 communicates back to GPU 0 and receives the output. (c) performs the last step of RM+, where GPU 3 only communicates kv with GPU 2. (d) shows that once all kv results are computed, the output is communicated back to GPU 7. (e) illustrates the overall cross-NUMA communication and the full overlap of computation and communication of GPU3 happened in (a)-(d).
Notation
Description
Notation
Description
Q,q
Query matrix
K,k
Key matrix
V,v
Value matrix
n
Degree of sequence parallelism
O
Unhidden communication time
ℓ(⋅)
Function that clips value to zero
Tcomm(⋅)
Communication time of a tensor
Tcomp(⋅)
Computation time of Attention
Table 1: Notation Description.
Figure 4: The Computation and Communication Overlap on GPU 7. For 8 GPUs, (a) contains three levels. Each GPU communicates at groups of 1, 2, and 4 at different levels, aggregating the communicated kv while overlapping computation. (b) maintains consistency with (a) in the first levels, but further enables multistream communication and computation of q and kv at other each merge level. Each GPU concurrently communicates q and kv with different GPUs, computes the q of other GPU and communicates the output back. The red region indicates GPU idle time.
Figure 5: Cross-Hardware Query Parallelism. The complete process is illustrated with pipeline 0 as an example. Each intra-pipeline processes queries (Q) from other pipelines and uses the same workflow with its local KV, returning the final out back. Effective overlap allows inter-pipeline communication to be hidden by intra-pipeline, further improving performance.
GPU
NVLink
Num
TFLOPS
Bandwidth (GB/s)
Cross
Intra
L20
✗
8
119.5
14.7
24.0
H20
✓
16
148.0
42.6
297.2
H100
✓
32
989.0
42.5
365.1
Table 2: The GPU Configuration and Bandwidth.
Figure 6: Single-node Experiments on 8× L20 (PCIe) and 8× H20 (NVLink). Speedup is normalized to Ring-Attention, comparing forward, backward, and end-to-end time. In different architectures, our method consistently outperforms the baseline and TASP.
Figure 7: Multi-node Experiments on 8×H20 and 16×H20 (NVLink). Speedup is normalized to Ring-Attention and Ring-Zigzag. In the forward (top) and the backward (bottom) comparison, our method achieves consistently superior performance. Particularly, the 2×4 and 2×8 experiments highlight that our method’s performance grows as the number of GPUs increases.
Batch
Seq
Heads
Ring
CHARM+
Batch
Seq
Heads
Ring
CHARM+
Size
Len
(MB)
(MB)
Size
Len
(MB)
(MB)
16
1024
32
2,179
2,215
2
53,760
12
13,123
13,467
16
1024
64
3,701
3,881
4
53,760
12
19,301
20,565
64
8192
32
45,079
45,423
2
26,880
12
10,745
10,803
128
8192
16
44,720
45,432
4
26,880
12
14,029
14,425
Table 3: Memory Overhead. By setting various parameter scales, we compare the memory overhead with Ring-Attention. Our method shows comparable overhead to Ring-Attention.
GPUs
Cross
Seq
Forward Time (ms)
Ring
RM+
Ours
4 × L20
✗
1792
9.35
10.29
–
7168
36.86
39.66
–
8 × L20
✓
1792
17.61
25.91
14.81
7168
69.85
98.49
52.40
4 × H20
✗
1792
5.38
4.76
–
Table 4: Ablation Study. We compare the forward performance of Ring-Attention, RM+, and CHARM+ across various GPU configurations and sequence lengths. Notably, CHARM+ will be evaluated only in cross-hardware configurations.
B
Seq
Forward
Backward
Ring
TASP
CHARM+
Speedup
Ring
TASP
CHARM+
Speedup
GPUs = 2×8
16
26,880
149.97
98.01
63.93
2.35 ×
449.58
–
141.52
3.18 ×
16
13,440
72.65
49.27
31.84
2.28 ×
226.46
–
65.35
3.47 ×
64
6,720
141.75
97.65
61.80
2.29 ×
448.84
–
123.32
3.64 ×
128
3,360
140.55
97.15
61.54
2.28 ×
450.01
–
121.99
3.69 ×
Table 5: Performance on H100. Note that TASP requires the sequence length to be divisible by n(n−1) . Therefore, the sequence lengths for the four-node and two-node settings differ slightly.
Figure 8: Performance of RM+ under PCIe and NVLink. We evaluate the forward time of RM+ on 4× H20 GPUs (dashed) and 4× L20 GPUs (solid) across different batch sizes (bs). Compared to Ring-Attention, performance on L20 is lower than on H20 due to the link contention.
Batch Size
Seq Len
Model
Ring
CHARM+
Speedup
Latency
Memory
Latency
Memory
16
2,048
GPT-350M
10082.6
11,372
4847.0
11,518
2.08 ×
8
4,096
GPT-350M
10032.6
11,348
4871.5
11,544
2.06 ×
4
8,192
GPT-350M
9914.5
11,436
5173.6
11,552
1.92 ×
2
16,384
GPT-350M
10089.5
12,112
5637.3
12,170
1.79 ×
16
2,048
GPT-1.3B
21505.5
30,816
11738.3
31,602
1.83 ×
Table 6: End-to-end Performance Comparison of Ring-Attention and CHARM+ with Model Scaling
Batch Size
Seq Len
Model
Ring
CHARM+
Speedup
H20
H100
H20
H100
H20
H100
16
16,384
GPT3-350M
7304.6
3385.1
5601.3
1925.7
1.30 ×
1.76 ×
32
8,192
GPT3-350M
7193.7
3291.5
4669.7
1679.2
1.54 ×
1.96 ×
64
4,096
GPT3-350M
7169.6
3234.6
4452.5
1538.6
1.61 ×
2.10 ×
128
2,048
GPT3-350M
7143.6
3210.8
4242.5
1482.0
1.68 ×
2.17 ×
Table 7: End-to-end Performance Comparison of Ring-Attention and CHARM+ with High-end GPU
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Nsight Analysis of Ring-Attention and CHARM+
Diff
CHARM+ (Max)
CHARM+ (Mean)
Ring (Max)
Ring (Mean)
out
8.34147e-04
4.51229e-05
8.07648e-04
4.24147e-05
lse
1.65642e-06
2.94155e-07
1.75893e-06
3.04284e-07
dq
1.26290e-03
3.29077e-05
1.25977e-03
3.27659e-05
dk
2.87713e-03
3.53419e-05
2.89413e-03
3.53193e-05
dv
4.47177e-03
3.42940e-05
4.48120e-03
3.42894e-05
Appendix
Table 8: Comparison of numerical differences between CHARM+ and Ring-Attention.
B
Seq
Tcomm
Tcomp
CCR
Ring
TASP
CHARM+
Speedup
Seq = 215,040
1
215,040
81.49
86.01
1.06
116.76
86.93
66.74
1.75 ×
2
215,040
164.31
171.51
1.04
241.03
171.91
125.43
1.92 ×
Seq = 107,520
1
107,520
38.34
23.16
0.60
44.08
26.36
21.43
2.06 ×
2
107,520
74.25
45.20
0.61
89.20
51.46
42.28
2.11 ×
Appendix
Table 9: Forward performance comparison on 16-H100.
B
Seq
Tcomm
Tcomp
CCR
Ring
TASP
CHARM+
Speedup
Seq = 215,040
1
215,040
218.48
257.76
1.18
312.45
–
207.07
1.51 ×
2
215,040
437.82
503.06
1.15
628.21
–
405.90
1.55 ×
Seq = 107,520
1
107,520
109.45
68.03
0.62
114.64
–
64.09
1.79 ×
2
107,520
218.37
135.44
0.62
228.70
–
125.01
1.83 ×
Appendix
Table 10: Backward performance comparison on 16-H100.
B
Seq
Tcomm
Tcomp
CCR
Ring
TASP
CHARM+
Speedup
Seq = 126,976
2
126,976
95.28
36.30
0.38
102.34
94.17
39.95
2.56 ×
4
126,976
186.17
70.03
0.38
208.12
184.59
77.02
2.70 ×
Seq = 63,488
2
63,488
49.95
12.50
0.25
46.32
48.54
19.80
2.34 ×
4
63,488
96.13
22.28
0.23
95.79
93.60
36.62
2.62 ×
Appendix
Table 11: Forward performance comparison on 32-H100.
B
Seq
Tcomm
Tcomp
CCR
Ring
TASP
CHARM+
Speedup
Seq = 126,976
2
126,976
259.62
102.57
0.40
280.48
–
111.60
2.51 ×
4
126,976
521.26
196.56
0.38
559.74
–
213.85
2.62 ×
Seq = 63,488
2
63,488
138.65
29.95
0.22
141.18
–
46.30
3.05 ×
4
63,488
269.38
58.46
0.22
280.96
–
86.70
3.24 ×
Appendix
Table 12: Backward performance comparison on 32-H100.
Communication-efficient distributed training reduces the data exchanged among devices so models can be trained faster and over limited-bandwidth networks.