Modern GPU workloads increasingly rely on device-initiated communication to let GPU kernels communicate directly without synchronizing with the host at kernel boundaries. Existing libraries commonly favor placing transport control on GPU threads, an approach known as GPUDirect Async (GDA), and treat reverse offload to host proxy threads as a slower fallback. We argue that reverse offload can outperform GDA and should be a first-class communication path. We further propose exposing reverse offload as a programming interface that lets users define workload-specific communication operations. We present KIWI, a programmable reverse-offload framework that achieves sub-microsecond one-way device-to-host latency and hundreds of millions of operations per second. We address practical challenges of scaling reverse offload, including contention on shared queues, polling overhead, and load imbalance across proxies. KIWI-P2P, a SHMEM-like library built on KIWI, outperforms state-of-the-art GDA implementations in both latency and bandwidth. More importantly, KIWI-EP redesigns low-latency mixture-of-experts dispatch and combine kernels around workload-defined operations, lowering expert-parallel communication latency by up to 2.3x over the best existing backend on AMD GPUs and time per output token in vLLM by up to 29%. The source code of KIWI is available at https://github.com/AMDResearch/kiwi.
Figures & tables
Fig. 1: GDA versus reverse offload. (a) With GDA, GPU threads drive the NIC themselves. (b) With reverse offload, the GPU hands each request to a CPU proxy, which drives the NIC. In both, the NIC moves payload data directly between GPU memories.
Fig. 2: Traditional reverse offload versus KIWI. (a) Traditional reverse offload hides the proxy and the NIC behind a fixed device API; all GPU threads share one queue, typically served by a single proxy thread. (b) KIWI exposes a single call, invoke , that runs user-defined callbacks on the CPU. Each wave owns its queue, several proxy threads poll the queues, and callbacks drive the NIC through any host library, such as LCI.
Fig. 3: KIWI’s SPSC device-to-host queue. P is the producer index, C the consumer index, and C′ the wave’s cached copy of C . Each 8-byte slot pairs 4 bytes of a descriptor with a 4-byte signal written in the same store, so those bytes are valid once their signal is. Blue slots are published but not yet popped.
Path
Latency ( μs )
Throughput (Mops/s)
KIWI-P2P (KIWI + LCI)
5.23
2.88
Flag + LCI
5.25
2.97
Flag + ibverbs
5.24
3.40
TABLE I: Eight-byte single-PUT latency and throughput between two nodes. All paths use GPU buffers and receiver-GPU polling. In the baselines, the GPU notifies a host thread through a pinned host flag, and the host thread posts the puts.
Fig. 4: Endpoint-to-proxy assignment with four proxies and 4 of 32 provisioned workgroups active. Each cell is an endpoint. Outlined cells are wave-0 endpoints, which carry the requests. Only skewed round-robin spreads both the active and the outlined endpoints across all proxies.
TABLE II: Hardware configuration of the evaluation cluster.
RTT ( μs )
Push ( μs )
Mops/s
Idle
W =1,024
W =256
poll
Queue
p50
p99
W =1
own
shared
W =1
own
shared
(ns)
Raw store (floor)
1.92
1.96
0.12
0.12
—
—
—
—
—
KIWI queue
1.92
1.98
0.12
0.12
—
10.8
392.0
—
0.87
RCCL GIN
3.86
4.46
1.32
6.04
7.72
1.0
39.1
4.8
20.9
NVSHMEM †
2.68
2.84
0.92
14.92
11.84
1.0
55.7
38.5
1.67
TABLE III: GPU-to-CPU queue comparison on one MI300X. W is the number of producer waves. RTT is the unloaded GPU–CPU–GPU round trip, push is the p50 device-side cost of one push, and idle poll is the host cost of polling one empty queue. Own gives each wave its own queue; shared puts all waves on one queue. † Re-implemented on AMD GPUs.
Figure 8
Fig. 7: Expert-parallel dispatch-plus-combine latency on two to eight nodes; lower is better.
Fig. 8: vLLM TPOT for GLM-5.3; lower is better. N P N D denotes N prefill and N decode nodes. Each node has eight GPUs.
Fig. 9: KIWI-EP ablation ladder on eight nodes. Bars give the dispatch-plus-combine latency of each variant normalized to GDA; lower is better. Each variant adds one design choice to the previous one. Labels give the speedup of KIWI-EP over GDA.
Dept. of Computer Science and Engineering Seoul National University · Dept. of Computer Science and Engineering Graduate School of Data Science Seoul National University