XM-Sparse: High-Throughput Sparse-Attention LLM Serving over Scale-up Memory Pools
Organizations: Netpreme · The Hong Kong University of Science and Technology · National University of Singapore
Abstract
Sparse attention has been increasingly adopted by production LLMs for long-context inference, yet serving systems must still retain the full KV history, making GPU HBM capacity a key limit on serving concurrency. Existing systems offload the KV cache to host memory, but suffer from the limited bandwidth of PCIe and the limited capacity of host memory. Emerging scale-up memory pools (e.g., X-Mem) relieve both limits and open new opportunities for sparse-attention offloading. In this paper, we first conduct an in-depth analysis and reveal that, even with scale-up memory pools, decode throughput remains bounded by HBM capacity, as hot KV buffers and the fully resident indexer key (K-index) cache occupy substantial HBM. Moreover, the two caches exhibit distinct access patterns. Based on these insights, we present XM-Sparse, the first sparse-attention offloading system over scale-up memory expansion. The core idea of XM-Sparse is to offload both caches aggressively while matching each with an I/O mechanism tailored to its access pattern. Specifically, for dynamically selected KV entries, XM-Sparse fuses TMA-based retrieval with attention in a tile-level pipeline to hide on-demand retrieval latency. For predictable K-index accesses, it prefetches pages via SM-efficient GPU-initiated DMA and coordinates prefetching with latency-critical operations to avoid interfering with decoding. We have implemented XM-Sparse atop SGLang, and our experiments on DeepSeek-V3.2 and GLM-5.3 with an emulated scale-up memory pool show that XM-Sparse delivers up to 4.98 and 4.52 higher decode throughput than PCIe-based ECHO and HiSparse, respectively, and up to 1.34 over a naive integration on the same platform, across 16K-1M contexts.
Figures & tables
| Approach | BW/GPU | Capacity (scope) |
| PCIe 5.0 16 | 64 GB/s | 2 TB (8-GPU server) |
| CXL 2.0 16 | 64 GB/s | Tens of TB (pool) |
| NVLink-C2C (GB200) | 225 GB/s | 480 GB (2 GPUs) |
| Scale-up pool (e.g., X-Mem) | 900 GB/s | Tens of TB (node) |
| Model | DeepSeek-V3.2 | GLM-5.3 |
| Parameters | 671B | 744B |
| Layers | 61 | 78 |
| K-index cross-layer reuse | No | Yes |
| Top- tokens | 2048 | 2048 |
| Optimization | Metric | Off | On (change) |
| Fused kernel | Mean TBT (ms) | 25.93 | 21.86 ( 15.7%) |
| K-index offload | Throughput (tokens/s/GPU) | 1437.9 | 1849.3 ( 28.6%) |
KV-Cache Offloading
KV-cache offloading moves language models’ attention state from GPU memory to CPU, SSD, or remote tiers to expand context and serving capacity.
See all 47 papers