Modern GPU workloads increasingly rely on device-initiated communication to let GPU kernels communicate directly without synchronizing with the host at kernel boundaries. Existing libraries commonly favor placing transport control on GPU threads, an approach known as GPUDirect Async (GDA), and treat reverse offload to host proxy threads as a slower fallback. We argue that reverse offload can outperform GDA and should be a first-class communication path. We further propose exposing reverse offload as a programming interface that lets users define workload-specific communication operations. We present KIWI, a programmable reverse-offload framework that achieves sub-microsecond one-way device-to-host latency and hundreds of millions of operations per second. We address practical challenges of scaling reverse offload, including contention on shared queues, polling overhead, and load imbalance across proxies. KIWI-P2P, a SHMEM-like library built on KIWI, outperforms state-of-the-art GDA implementations in both latency and bandwidth. More importantly, KIWI-EP redesigns low-latency mixture-of-experts dispatch and combine kernels around workload-defined operations, lowering expert-parallel communication latency by up to 2.3x over the best existing backend on AMD GPUs and time per output token in vLLM by up to 29%. The source code of KIWI is available at https://github.com/AMDResearch/kiwi.
Figures & tables
Fig. 1: GDA versus reverse offload. (a) With GDA, GPU threads drive the NIC themselves. (b) With reverse offload, the GPU hands each request to a CPU proxy, which drives the NIC. In both, the NIC moves payload data directly between GPU memories.
Fig. 2: Traditional reverse offload versus KIWI. (a) Traditional reverse offload hides the proxy and the NIC behind a fixed device API; all GPU threads share one queue, typically served by a single proxy thread. (b) KIWI exposes a single call, invoke , that runs user-defined callbacks on the CPU. Each wave owns its queue, several proxy threads poll the queues, and callbacks drive the NIC through any host library, such as LCI.
Fig. 3: KIWI’s SPSC device-to-host queue. P is the producer index, C the consumer index, and C′ the wave’s cached copy of C . Each 8-byte slot pairs 4 bytes of a descriptor with a 4-byte signal written in the same store, so those bytes are valid once their signal is. Blue slots are published but not yet popped.
Path
Latency ( μs )
Throughput (Mops/s)
KIWI-P2P (KIWI + LCI)
5.23
2.88
Flag + LCI
5.25
2.97
Flag + ibverbs
5.24
3.40
TABLE I: Eight-byte single-PUT latency and throughput between two nodes. All paths use GPU buffers and receiver-GPU polling. In the baselines, the GPU notifies a host thread through a pinned host flag, and the host thread posts the puts.
Fig. 4: Endpoint-to-proxy assignment with four proxies and 4 of 32 provisioned workgroups active. Each cell is an endpoint. Outlined cells are wave-0 endpoints, which carry the requests. Only skewed round-robin spreads both the active and the outlined endpoints across all proxies.
TABLE II: Hardware configuration of the evaluation cluster.
RTT ( μs )
Push ( μs )
Mops/s
Idle
W =1,024
W =256
poll
Queue
p50
p99
W =1
own
shared
W =1
own
shared
(ns)
Raw store (floor)
1.92
1.96
0.12
0.12
—
—
—
—
—
KIWI queue
1.92
1.98
0.12
0.12
—
10.8
392.0
—
0.87
RCCL GIN
3.86
4.46
1.32
6.04
7.72
1.0
39.1
4.8
20.9
NVSHMEM †
2.68
2.84
0.92
14.92
11.84
1.0
55.7
38.5
1.67
TABLE III: GPU-to-CPU queue comparison on one MI300X. W is the number of producer waves. RTT is the unloaded GPU–CPU–GPU round trip, push is the p50 device-side cost of one push, and idle poll is the host cost of polling one empty queue. Own gives each wave its own queue; shared puts all waves on one queue. † Re-implemented on AMD GPUs.
Figure 8
Fig. 7: Expert-parallel dispatch-plus-combine latency on two to eight nodes; lower is better.
Fig. 8: vLLM TPOT for GLM-5.3; lower is better. N P N D denotes N prefill and N decode nodes. Each node has eight GPUs.
Fig. 9: KIWI-EP ablation ladder on eight nodes. Bars give the dispatch-plus-combine latency of each variant normalized to GDA; lower is better. Each variant adds one design choice to the previous one. Labels give the speedup of KIWI-EP over GDA.
Communication has become a bottleneck in distributed training and inference of large models. Overlapping communication with computation at the granularity of kernels, on separate streams, reduces only part of this communication cost. Fused kernels often have better performance by transmitting each output tile as soon as it is produced, but existing fused kernels are largely confined to a single NVLink domain. We present mKernel, a library of multi-GPU, multi-node fused kernels that overlap computation, intra-node NVLink communication, and inter-node RDMA at tile granularity. mKernel partitions the streaming multiprocessors (SMs) of a persistent kernel into compute and communication roles, and an on-GPU controller tunes the SM partition adaptively at run time, since the best SM partition varies with the kernel and the input shape. It structures data movement hierarchically so that data traversing the inter-node network is minimized. Finally, it drives the network from the GPU through a lightweight command queue and host proxy implemented directly on RDMA verbs, which allows the same kernels to run on any network backend (e.g. InfiniBand and on AWS EFA); we observe, surprisingly, that GPUDirect Async (IBGDA) yields little additional benefit over host-assisted GPU-initiated communication. We implement five kernels spanning tensor, sequence, and expert parallelism. On two 16-GPU H200 clusters, mKernel achieves speedups of up to 1.72x on GEMM+AllReduce and 1.88× on Ring Attention.
Machine learning systems specialize computation to the workload and hardware, but communication remains in generic libraries whose device code and transport decisions are not visible to high-level compilers. We present Obaleth, a compiler that makes GPU-initiated communication part of a kernel's intermediate representation, co-optimizes it alongside computation, and lowers it to specialized transport code. Obaleth plans completions around buffer lifetimes and optimizes memory ordering, synchronization, and batching across NVIDIA and AMD GPUs. On H100 GPUs, Obaleth sends small messages up to 3.3× faster, and on 256 GPUs runs all-to-all 1.25× and 2.3× faster than NVSHMEM and NCCL GIN, with much smaller memory and register use. In vLLM, it serves Qwen3-235B on 16 B200 GPUs and answers requests up to 1.29× faster than with DeepEP.
Expert parallelism (EP) enables inference of large Mixture-of-Experts (MoE) models by placing their experts across multiple GPUs, but requires substantial communication between GPUs at every MoE layer. As contemporary MoE models activate more experts per token, this communication accounts for a growing fraction of inference time. The cost becomes particularly pronounced on PCIe-based consumer GPU systems, where all inter-GPU transfers traverse CPU memory. However, existing MoE-specialized EP communication libraries assume that direct GPU-to-GPU access is available, largely overlooking consumer GPUs. Therefore, most LLM frameworks instead rely on NCCL, whose CPU-staged communication incurs redundant PCIe transfers and competes with expert computation for GPU resources, limiting their overlap. We present ThunderEP, a novel communication design for such systems that removes the relay hops of traditional ring algorithm, moves data through DMA engines to avoid compute resource contention, and minimizes synchronization latency by reducing the polling overhead of completion flags in CPU memory. We integrate the proposed design into vLLM and evaluate it on three widely used MoE models. Experiments on two PCIe systems equipped with RTX 4090 and RTX 5090 GPUs show that ThunderEP achieves average speedups of 2.00× and 1.53× over NCCL for dispatch and combine, respectively, and up to 1.66× end-to-end speedup over state-of-the-art MoE inference frameworks.
Jaehwan Lee, Sangmin Lee, Chaewon Kim +2
Dept. of Computer Science and Engineering Seoul National University · Dept. of Computer Science and Engineering Graduate School of Data Science Seoul National University