Large language models (LLM) have been the key focus of recent machine learning breakthroughs. As the model size grows exponentially, the computation needed for training is astronomical. A GPU can do a huge amount of arithmetic. However, the data its compute units need is not always available when they are ready. This data may still be in slow memory or on another GPU. While the chip waits, expensive compute sits idle, and at scale even a small drop in utilization translates into substantial cost. In this paper, we treat this waiting time as the primary bottleneck. We focus on hardware features designed for explicit data movement. These features load data from off-chip memory onto the chip, rearrange it into the layout the computation requires as it is loaded, and transfer it between GPUs. By orchestrating them, we keep a steady pipeline of data flowing to the compute units so that they remain busy. We present kernel and framework implementations that address this bottleneck using three key data-movement instructions and hardware features. The Tensor Data Mover (TDM) moves tensor data from high-bandwidth memory (HBM) into on-chip memory. Transpose-load instructions transpose tensor data into the layout that LLM computations require as it is loaded on-chip. Finally, System DMA (SDMA) and Remote DMA (RDMA) move data between GPUs over the scale-up and scale-out networks, respectively. Our results show that TDM and transpose-load provide on average 1.88x and 1.19x speedup at the kernel level, respectively. The SDMA and RDMA lead to on average 1.16x and 1.33x speedup for computation overlapped with communication. This is the first AMD LLM training system to utilize all the GPU data pipelining features. Our implementation has been integrated into open source framework and library for AMD GPUs. https://github.com/AMD-AGI/Primus-Turbohttps://github.com/AMD-AGI/Primus
A collection of 32 work-items/threads that execute in parallel. MI455 (CDNA5) supports 32 threads; the earlier MI355 (CDNA4) supports 64 threads
Workgroup
Thread block
A collection of waves/warp that can synchronize with each other using barriers;
Local Data Share (LDS)
Shared memory
A multi-bank on-chip scratch memory allocated to waves or work-groups.
Vector General Purpose Registers (VGPR)
Register
Per-work-item register storage, allocated from hardware register resources.
Tensor Data Mover (TDM)
Tensor Memory Accelerator (TMA)
DMA transfers of matrix/tensor data.
Table 1: ROCm and CUDA GPU programming terminology [ 4 , 21 ] .
ISA/intrinsic
Purpose
TENSOR_LOAD_TO_LDS
Load data from HBM to on-chip memory
TENSOR_STORE_FROM_LDS
Store data from on-chip memory to HBM
TENSORCNT
Counts the number of matrix (tensor) transfers outstanding but not yet completed
S_WAIT_TENSORCNT
Waits until number of unfinished transfers is less-than or equal-to specified value
Table 2: TDM ISA/intrinsic [ 4 ] .
Pass
Operation
Tile shapes
Transposed operand
Forward
Scores
Qi[Bq,Dq]×KjT[Dq,Bk]
KjT
Backward
Query gradient
dSij[Bq,Bk]×Kj[Bk,Dq]
none
Backward
Key gradient
dSijT[Bk,Bq]×Qi[Bq,Dq]
dSijT
Table 3: Attention kernel computes and layout in training
GPU
ISA / intrinsic
Transpose load size
Description
MI355X
ds_read_b64_tr_b16
64-bits per thread
Used for either column major matrix A or row major matrix B data load to 2 VGPRs. Element size is 16b. Two instruction load a complete matrix
MI455X
ds_load_tr16_b128
128-bits per thread
Load a 16×16 matrix of 16-bit data into VGPRs and transpose between row-major and column-major order. This instruction loads data into 4 consecutive VGPRs. A wave/warp of 32 threads moves 32×128 bits.
Table 7: Compressed Sparse Attention (CSA) config in DeepSeek V4 architecture. Every query attends to a 128-token sliding window plus the listed compressed-pool rows; T represents sequence length.
Family
Ratio
T
Forward
Backward
Step
Flash
cr0
4K
2.34 ×
2.90 ×
2.74 ×
8K
2.24 ×
2.75 ×
2.60 ×
16K
2.17 ×
2.75 ×
2.58 ×
cr4
4K
1.82 ×
1.56 ×
1.61 ×
8K
1.77 ×
1.58 ×
1.62 ×
16K
1.76 ×
1.60 ×
1.63 ×
Table 8: TDM-enabled Compressed Sparse Attention (CSA) kernel speedup over the best non-TDM implementation (higher is better).
Forward
Backward
Design (waves/SIMD)
Kernel
Flash
Pro
Flash
Pro
Non-TDM (2)
512 †
511
431
462
TDM (1)
cr4
910
954
730
1004
cr0 / cr128
1024
1024
1024 ‡
1024 ‡
Table 9: VGPRs per wave of the non-TDM and TDM CSA kernels. † spills VGPRs; ‡ spills on cr128 only.
Load
Normalized bandwidth
ds_load_b128 (no transpose)
1.00
ds_load_tr16_b128
0.99
8 × strided ds_load_u16
0.63
ds_load_b128 + XOR shuffle
0.41
Table 10: LDS-to-register bandwidth comparison, normalized to a plain ds_load_b128 (no transpose); higher is better.
Family
Ratio
T
Forward
Backward
Step
Flash
cr0
4K
1.36 ×
1.26 × †
1.29 ×
8K
1.26 ×
1.22 × †
1.23 ×
16K
1.28 ×
1.19 × †
1.22 ×
cr4
4K
1.31 ×
1.16 ×
1.19 ×
8K
1.40 ×
1.17 ×
1.21 ×
16K
1.41 ×
1.16 ×
1.21 ×
Table 11: Transpose-load-ISA-enabled kernel speedup vs strided-load version (higher is better). † The strided-load casued spills VGPRs.
Workload
Model
#GPU
Scale-up/-out
LLama 3.1 8B
LLM
8
Scale-up
GPTOSS 20B
LLM
8
Scale-up
FLUX.1-Schnell
Diffusion
8
Scale-up
Table 12: Selected MLperf dataset.
AllGather
ReduceScatter
GEMM( MNK )
CU
SDMA
CU
SDMA
8k × 6k × 4k
46%
16%
64%
15%
8k × 4k × 4k
29%
3.3%
44%
13%
8k × 28k × 4k
15%
3.6%
15%
7.3%
8k × 4k × 14k
23%
9.6%
23%
20%
Geometric mean
28%
8.0%
35%
14%
Table 13: GEMM runtime increase while overlapped with collectives, relative to the same GEMM runs alone.
Model
AG(%)
RS(%)
Sp (%)
Regional Sp
Llama3.1 8B
6.2
9.1
+2.1
1.16 ×
GPT-OSS 20B
13.5
14.6
+4.5
1.18 ×
FLUX.1
22.0 †
+3.1
1.16 ×
Table 14: Training speedup with SDMA vs. non-SDMA. AG stands for AllGather, RS for ReduceScatter, Sp for Speedup. † Flux workload only enabled AllGather SDMA.
8 GPUs
16 GPUs
32 GPUs
GEMM( MNK )
CU
DMA
CU
DMA
CU
DMA
32k × 16k × 16k
11%
4.8%
31%
4.0%
34%
3.7%
32k × 18k × 16k
9.3%
3.5%
30%
3.1%
32%
3.4%
32k × 16k × 52k
12%
5.0%
54%
4.0%
58%
3.9%
32k × 104k × 16k
12%
4.0%
36%
4.0%
36%
4.2%
Geometric mean
11%
4.3%
37%
3.8%
39%
3.9%
Table 15: GEMM runtime increase while overlapped with AllGather on 8, 16, and 32 GPUs, relative to the same GEMM run alone.