Large language models (LLM) have been the key focus of recent machine learning breakthroughs. As the model size grows exponentially, the computation needed for training is astronomical. A GPU can do a huge amount of arithmetic. However, the data its compute units need is not always available when they are ready. This data may still be in slow memory or on another GPU. While the chip waits, expensive compute sits idle, and at scale even a small drop in utilization translates into substantial cost. In this paper, we treat this waiting time as the primary bottleneck. We focus on hardware features designed for explicit data movement. These features load data from off-chip memory onto the chip, rearrange it into the layout the computation requires as it is loaded, and transfer it between GPUs. By orchestrating them, we keep a steady pipeline of data flowing to the compute units so that they remain busy. We present kernel and framework implementations that address this bottleneck using three key data-movement instructions and hardware features. The Tensor Data Mover (TDM) moves tensor data from high-bandwidth memory (HBM) into on-chip memory. Transpose-load instructions transpose tensor data into the layout that LLM computations require as it is loaded on-chip. Finally, System DMA (SDMA) and Remote DMA (RDMA) move data between GPUs over the scale-up and scale-out networks, respectively. Our results show that TDM and transpose-load provide on average 1.88x and 1.19x speedup at the kernel level, respectively. The SDMA and RDMA lead to on average 1.16x and 1.33x speedup for computation overlapped with communication. This is the first AMD LLM training system to utilize all the GPU data pipelining features. Our implementation has been integrated into open source framework and library for AMD GPUs. https://github.com/AMD-AGI/Primus-Turbohttps://github.com/AMD-AGI/Primus
A collection of 32 work-items/threads that execute in parallel. MI455 (CDNA5) supports 32 threads; the earlier MI355 (CDNA4) supports 64 threads
Workgroup
Thread block
A collection of waves/warp that can synchronize with each other using barriers;
Local Data Share (LDS)
Shared memory
A multi-bank on-chip scratch memory allocated to waves or work-groups.
Vector General Purpose Registers (VGPR)
Register
Per-work-item register storage, allocated from hardware register resources.
Tensor Data Mover (TDM)
Tensor Memory Accelerator (TMA)
DMA transfers of matrix/tensor data.
Table 1: ROCm and CUDA GPU programming terminology [ 4 , 21 ] .
ISA/intrinsic
Purpose
TENSOR_LOAD_TO_LDS
Load data from HBM to on-chip memory
TENSOR_STORE_FROM_LDS
Store data from on-chip memory to HBM
TENSORCNT
Counts the number of matrix (tensor) transfers outstanding but not yet completed
S_WAIT_TENSORCNT
Waits until number of unfinished transfers is less-than or equal-to specified value
Table 2: TDM ISA/intrinsic [ 4 ] .
Pass
Operation
Tile shapes
Transposed operand
Forward
Scores
Qi[Bq,Dq]×KjT[Dq,Bk]
KjT
Backward
Query gradient
dSij[Bq,Bk]×Kj[Bk,Dq]
none
Backward
Key gradient
dSijT[Bk,Bq]×Qi[Bq,Dq]
dSijT
Table 3: Attention kernel computes and layout in training
GPU
ISA / intrinsic
Transpose load size
Description
MI355X
ds_read_b64_tr_b16
64-bits per thread
Used for either column major matrix A or row major matrix B data load to 2 VGPRs. Element size is 16b. Two instruction load a complete matrix
MI455X
ds_load_tr16_b128
128-bits per thread
Load a 16×16 matrix of 16-bit data into VGPRs and transpose between row-major and column-major order. This instruction loads data into 4 consecutive VGPRs. A wave/warp of 32 threads moves 32×128 bits.
Table 7: Compressed Sparse Attention (CSA) config in DeepSeek V4 architecture. Every query attends to a 128-token sliding window plus the listed compressed-pool rows; T represents sequence length.
Family
Ratio
T
Forward
Backward
Step
Flash
cr0
4K
2.34 ×
2.90 ×
2.74 ×
8K
2.24 ×
2.75 ×
2.60 ×
16K
2.17 ×
2.75 ×
2.58 ×
cr4
4K
1.82 ×
1.56 ×
1.61 ×
8K
1.77 ×
1.58 ×
1.62 ×
16K
1.76 ×
1.60 ×
1.63 ×
Table 8: TDM-enabled Compressed Sparse Attention (CSA) kernel speedup over the best non-TDM implementation (higher is better).
Forward
Backward
Design (waves/SIMD)
Kernel
Flash
Pro
Flash
Pro
Non-TDM (2)
512 †
511
431
462
TDM (1)
cr4
910
954
730
1004
cr0 / cr128
1024
1024
1024 ‡
1024 ‡
Table 9: VGPRs per wave of the non-TDM and TDM CSA kernels. † spills VGPRs; ‡ spills on cr128 only.
Load
Normalized bandwidth
ds_load_b128 (no transpose)
1.00
ds_load_tr16_b128
0.99
8 × strided ds_load_u16
0.63
ds_load_b128 + XOR shuffle
0.41
Table 10: LDS-to-register bandwidth comparison, normalized to a plain ds_load_b128 (no transpose); higher is better.
Family
Ratio
T
Forward
Backward
Step
Flash
cr0
4K
1.36 ×
1.26 × †
1.29 ×
8K
1.26 ×
1.22 × †
1.23 ×
16K
1.28 ×
1.19 × †
1.22 ×
cr4
4K
1.31 ×
1.16 ×
1.19 ×
8K
1.40 ×
1.17 ×
1.21 ×
16K
1.41 ×
1.16 ×
1.21 ×
Table 11: Transpose-load-ISA-enabled kernel speedup vs strided-load version (higher is better). † The strided-load casued spills VGPRs.
Workload
Model
#GPU
Scale-up/-out
LLama 3.1 8B
LLM
8
Scale-up
GPTOSS 20B
LLM
8
Scale-up
FLUX.1-Schnell
Diffusion
8
Scale-up
Table 12: Selected MLperf dataset.
AllGather
ReduceScatter
GEMM( MNK )
CU
SDMA
CU
SDMA
8k × 6k × 4k
46%
16%
64%
15%
8k × 4k × 4k
29%
3.3%
44%
13%
8k × 28k × 4k
15%
3.6%
15%
7.3%
8k × 4k × 14k
23%
9.6%
23%
20%
Geometric mean
28%
8.0%
35%
14%
Table 13: GEMM runtime increase while overlapped with collectives, relative to the same GEMM runs alone.
Model
AG(%)
RS(%)
Sp (%)
Regional Sp
Llama3.1 8B
6.2
9.1
+2.1
1.16 ×
GPT-OSS 20B
13.5
14.6
+4.5
1.18 ×
FLUX.1
22.0 †
+3.1
1.16 ×
Table 14: Training speedup with SDMA vs. non-SDMA. AG stands for AllGather, RS for ReduceScatter, Sp for Speedup. † Flux workload only enabled AllGather SDMA.
8 GPUs
16 GPUs
32 GPUs
GEMM( MNK )
CU
DMA
CU
DMA
CU
DMA
32k × 16k × 16k
11%
4.8%
31%
4.0%
34%
3.7%
32k × 18k × 16k
9.3%
3.5%
30%
3.1%
32%
3.4%
32k × 16k × 52k
12%
5.0%
54%
4.0%
58%
3.9%
32k × 104k × 16k
12%
4.0%
36%
4.0%
36%
4.2%
Geometric mean
11%
4.3%
37%
3.8%
39%
3.9%
Table 15: GEMM runtime increase while overlapped with AllGather on 8, 16, and 32 GPUs, relative to the same GEMM run alone.
High-performance Host processors can integrate Processing-In-Memory (PIM) devices, which can accelerate memory-intensive kernels of Machine Learning (ML) models, including Large Language Models (LLMs), by leveraging the large memory bandwidth available at PIM cores. However, Host processor needs consecutive elements distributed across DRAM banks, while PIM cores need consecutive elements within their local banks. This necessitates data rearrangements in ML kernel execution that pose significant performance and programmability challenges, further exacerbated by the need to support diverse PIM devices. Current compilation approaches lack systematic optimization for diverse ML kernels and multiple PIM devices, and may largely ignore data rearrangement costs during the compute code optimization step. We show that data rearrangements and compute code optimization are interdependent, and need to be jointly optimized during the tuning process. Therefore, we design DCC, the first data-centric ML compiler for PIM systems that jointly co-optimizes data rearrangements and compute code in a unified tuning process. DCC integrates a multi-layer PIM abstraction to support multiple PIM backends. DCC enables effective co-optimization of data partitioning strategies with compute loop partitioning schemes. DCC applies PIM-specific code optimizations, and leverages a fast and accurate performance prediction model to select the bestperforming code schedule for a given kernel on a target PIM architecture. Our evaluations in various individual ML kernels show that DCC achieves up to 7.68x speedup (2.21x average) on HBM-PIM, and up to 13.17x speedup (3.92x average) on AttAcc PIM, over GPU-only execution. In end-to-end LLM inference, DCC on AttAcc accelerates GPT-3 and LLaMA-2 by 4.52x average (up to 7.71x in LLaMA-2) over GPU. DCC is open-sourced at https://github.com/SPIN-Research-Group/DCC.
Peiming Yang, Sankeerth Durvasula, Ivan Fernandez +4
University of Toronto & Vector Institute · Barcelona Supercomputing Center · ETH Zürich +2
Fine-tuning large language models (LLMs) often exceeds GPU memory limits, prompting systems to offload model states to CPU memory. However, existing offloaded training frameworks like ZeRO-Offload treat all parameters equally and update the full model on the CPU, causing severe GPU stalls, where fast, expensive GPUs sit idle, waiting for slow CPU updates and limited-bandwidth PCIe transfers. We present ZenFlow, a new offloaded training framework that decouples updates between GPU and CPU by prioritizing important gradients. ZenFlow performs in-place updates of important gradients on GPU while offloading and accumulating less important ones on CPU without blocking training iterations, fully overlapping CPU work with GPU computation. To make this importance-aware design scalable, ZenFlow leverages a novel spatio-temporal locality of important gradients, concentrated in specific positions that remain stable across iterations, to make important gradient identification efficient and lightweight while preserving fidelity. Evaluation shows that ZenFlow achieves up to 5x end-to-end speedup, 2x lower PCIe traffic, 85% less GPU stalls, and up to 72% lower cloud GPU cost, all while preserving accuracy.
Tingfeng Lan, Yusen Wu, Bin Ma +7
University of Virginia · University of California, Merced · Argonne National Laboratory +2
Fine-tuning Large Language Models (LLMs) on consumer-grade GPUs is highly cost-effective, yet constrained by limited GPU memory and slow PCIe interconnects. Pipeline parallelism combined with CPU offloading mitigates these hardware bottlenecks by reducing communication overhead. However, existing PP schedules suffer from an inherent limitation termed the weight binding issue. Binding uneven model stages (e.g., the LM head is large) to GPUs limits the pipeline's throughput to that of the GPU with the heaviest load, leading to severe pipeline bubbles. In this paper, we propose RoundPipe, a novel pipeline schedule that breaks the weight binding constraint on consumer GPU servers. RoundPipe treats GPUs as a pool of stateless execution workers and dynamically dispatches computation stages across devices in a round-robin manner, achieving a near-zero-bubble pipeline. To ensure training correctness and system efficiency, RoundPipe integrates a priority-aware transfer scheduling engine, a Layerwise Model Consistency Protocol, and an automated layer partitioning algorithm. Evaluations on an 8× RTX 4090 server demonstrate that RoundPipe achieves 1.48--2.16× speedups over state-of-the-art baselines when fine-tuning 1.7B to 32B models. Remarkably, RoundPipe enables LoRA fine-tuning of the Qwen3-235B model with 31K sequence length on a single server. RoundPipe has been open-sourced as a Python library.