The DMA Streaming Framework: Kernel-Level Buffer Orchestration for High-Performance AI Data Paths

Marco Graziano

cs.ARFeb 26, 2026

The DMA Streaming Framework: Kernel-Level Buffer Orchestration for High-Performance AI Data Paths

Authors: Marco Graziano

Organizations: Graziano Labs Corp. Palo Alto, CA, USA

Abstract

AI transport libraries move bytes efficiently, but they commonly assume that buffers are already correctly allocated, placed, shared, registered, and safe under completion and teardown pressure. This paper presents dmaplane, a Linux kernel module that makes this missing layer explicit as buffer orchestration. dmaplane exposes a stable kernel UAPI via /dev/dmaplane and composes ring-based command channels, DMA buffer lifecycle management, dma-buf export for cross-device sharing, a kernel-space RDMA engine, NUMA-aware allocation and verification, credit-based flow control, low-overhead observability, and GPU memory integration via PCIe BAR pinning. We evaluate orchestration sensitivity with measurements of NUMA cross-node penalties at DRAM scale, completion-safe flow control under sustained RDMA load, and GPU BAR mapping tiers versus cudaMemcpy. We also demonstrate end-to-end disaggregated inference by transferring KV-cache chunks between two machines using RDMA WRITE WITH IMMEDIATE and reconstructing tensor views on the receiver. RDMA measurements use Soft-RoCE; we distinguish measured results from provider-independent properties by construction.

Figures & tables

Capability	dmaplane	nvidia-peermem	NCCL	Mooncake	TransferEngine
Buffer allocation (coherent, page-backed)	Yes	No	internal	No	No
NUMA-aware placement	Yes	No	topology detection	mentioned	mentioned
dma-buf export or import	Yes	importer only	No	No	No
MR registration (kernel-space)	Yes	Yes (GPU BAR only)	internal	via libibverbs	via libibverbs
Credit-based flow control	Yes	No	per-channel credits	implicit in batching	per-connection
GPU BAR pinning and mapping	Yes	Yes	via CUDA	via CUDA	via CUDA

Table 1. Capability positioning across the buffer orchestration and transport stack.

Figure 1. dmaplane block diagram. dmaplane block diagram.

Figure 2. dma-buf export and per importer attachment mapping. dma-buf export and per importer attachment mapping.

Figure 3. RDMA resource hierarchy. RDMA resource hierarchy.

Figure 4. Timeline of send CQ credits and receive window credits in a WRITE WITH IMMEDIATE pipeline. Timeline diagram showing a sender posting WRITE WITH IMMEDIATE chunks bounded by send credits and receive window credits, the receiver polling the receive completion queue and reposting receives, and the sender polling the send completion queue to reclaim credits.

Phase	Time
Tokenization	1.2 ms
Prefill forward pass	45.3 ms
KV-cache consolidation	0.8 ms
KV-cache transfer	52.1 ms
KV-cache reconstruction	0.003 ms
—————————	——–

Table 2. Disaggregated inference pipeline timing (two EC2 ‘g5.xlarge‘, Soft-RoCE).

Metric	Value
Sustained throughput (10 s, max_credits=64)	1,037 MB/s average
Per-second window range	1,047 to 1,087 MB/s
Window spread	3.8%
CQ overflows	0
Credit stalls	0
Stress configuration (max_credits=4, high=3, low=1)

Table 3. RDMA streaming and flow control results (Soft-RoCE loopback).

Buffer size	Node 0 to 0	Node 0 to 1	Node 1 to 0	Node 1 to 1	Cross-node penalty
1 MB	13,284	13,218	12,976	13,051	< 1% (fits in cache)
64 MB	6,778	5,577	5,013	6,095	18% symmetric

Table 4. Cross node memcpy bandwidth on AWS ‘c5.metal‘. Values are MB/s.

Access method	Host to GPU (MB/s)	GPU to host (MB/s)	Section
UC BAR (ioremap)	44	6	Accompanying artifact
WC BAR (ioremap_wc, 4 MB)	10,097	107	Accompanying artifact
cudaMemcpy (16 MB)	12,552	13,124	Accompanying artifact
GPU RDMA loopback (rxe, 64 KB)	n/a	~20	Accompanying artifact
GPU RDMA cross-machine (1 Gigabit)	n/a	11.6	Accompanying artifact

Table 5. GPU memory access hierarchy (RTX 5000 Ada). Cross-machine values are over 1 Gigabit Ethernet to a remote NUC.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Subsystem	Test entry point	Pass condition
Device and ioctl dispatch	not specified	Module loads, /dev/dmaplane present, ioctls return expected codes.
Rings and workers	userspace stress harness	No kernel warnings, no lockdep issues, no data corruption, clean unload.
DMA buffers and mmap	not specified	Create buffer, mmap, write and verify, munmap, destroy succeeds.
dma-buf export	not specified	Export returns dma-buf fd, attach and detach paths do not leak, release callback fires.
RDMA setup and SEND and RECV	ioctl benchmarks	QP setup succeeds, SEND and RECV completes, CQs polled without errors.
MR registration	ioctl MR tests	MR registration returns lkey and rkey when requested, deregistration releases mappings.

Appendix

Table 6. Test suite map by subsystem.

Deploying RDMA for Distributed Inference and Training At Scale
Oct 8, 2026Yibo Huang, Yi Cui, Yukai Huang +12Communication-Efficient Distributed Training Distributed Inference
Fudan University · ByteDance Inc. · University of Michigan
mKernel: Fast Multi-GPU, Multi-Node Fused Kernels
Sep 11, 2026Ziming Mao, Yihan Zhang, Shawn Wei Chew +5GPU Kernel Optimization Communication-Efficient Distributed Training
UC Berkeley · UC Davis · UCLA +1
Maia 200: A Software Defined Dataflow System for Large-scale AI Acceleration
Aug 25, 2026Sherry Xu, Marco Heddes, Jackson Peng +14AI Accelerator Inference High-Performance Computing
Microsoft Corporation

Oct 8, 2026J/K move · Enter open · S save

Deploying RDMA for Distributed Inference and Training At Scale

Yibo Huang, Yi Cui, Yukai Huang, Huaping Zhou+11

Fudan University · ByteDance Inc. · University of Michigan

Communication-Efficient Distributed Training Distributed Inference

Open paper AskPDF arXiv

RDMA can reduce communication overhead in distributed machine learning systems, but realizing these benefits in production requires more than replacing TCP. Variable-size messages, concurrent connections, bursty incasts, and service failures introduce challenges that hardware offload alone cannot address. This paper reports our experience deploying ByteExpress, a software communication layer over hardware RDMA, in ByteDance's parameter-server-based inference and training systems. We present three lessons from the deployment: efficient RDMA requires buffer management across application and transport layers; scalable communication requires coordinated management of connections, receive capacity, and outstanding data; and service availability requires recovery above the transport layer. ByteExpress combines an RDMA-aware allocator, zero-copy transfers, QP multiplexing, receive-credit flow control, a software send window, and RPC-assisted recovery. In production inference, ByteExpress reduces mean and P99 model parameter fetch latency by 35.7% and 43.7%, respectively, and total CPU usage by 10.69% compared with kernel TCP. In training, ByteExpress reduces P99 model parameter push latency by up to 74.0% compared with TCP, with no PFC pauses observed at runtime.

Abstract

Explore similar work

Deploying RDMA for Distributed Inference and Training At Scale

mKernel: Fast Multi-GPU, Multi-Node Fused Kernels

Maia 200: A Software Defined Dataflow System for Large-scale AI Acceleration