FlashOffloading: Low-Latency KV Cache Offloading for Long-Context LLM Serving
Organizations: Huawei Technologies Co., Ltd · Huawei Ascend Computing Technology Development · Shanghai Jiao Tong University
Abstract
In large language model (LLM) serving systems, the key-value (KV) cache grows linearly with sequence length and concurrency, so long-context inference easily hits the high-bandwidth memory (HBM) capacity wall and results in low concurrency. Offloading the KV cache to host dynamic random-access memory (DRAM) frees HBM space and raises concurrency, but the additional cache management and host-to-device (H2D) load overhead inflates inference latency, higher concurrency fails to translate into effective throughput. We present FlashOffloading, a host-DRAM-centric inference architecture that fully offloads the KV cache in both the prefill and decode phases. FlashOffloading builds rolling HBM cache for prefill, a hot cache for decode, a reverse sparse index with zero extra miss-lookup overhead, and a compute--load overlapped kernel, optimizing KV offloading latency across the entire pipeline and delivering throughput gains under a time-per-output-token (TPOT) service-level objective (SLO). In experiments deploying GLM-5.2 with multi-token prediction (MTP3) speculative decoding on an Ascend910C prefill--decode (PD)-disaggregated platform, FlashOffloading improves prefill throughput by up to 1.38 and decode throughput by up to 1.91 under a 30 ms TPOT constraint. FlashOffloading is integrated into an open-source serving framework: https://github.com/vllm-project/vllm-ascend/tree/releases/v0.30.0.
Figures & tables
| Seq. Length | Prefill HBM usage | Decode HBM usage | Latency | Total Throughput | |||||||||||||
| Input | Output | Offload Mode | Client BS | weight (GB) | KV (GB) | act. (GB) | max batched toks. | weight (GB) | KV (GB) | KV usage (GB) | BS per NPU | P99 TE2E (s) | P99 TTFT (s) | Avg. TPOT (ms) | Avg. ITL (ms) | Input | Output |
| 64k | 1 | none | 192 | 18.86 | 36.75 | 1.67 | 16384 | 25.36 | 28.40 | – | – | 93.8 | 93.8 | – | – | 13101 | – |
| 64k | 1 | P-only | 192 | 18.86 | 9.04 | 13.26 | 131072 | 25.36 | 28.40 | – | – | 86.4 | 86.4 | – | – | 14222 | – |
| 64k | 8192 | P-only | 192 | 18.86 | 9.04 | 13.26 | 131072 | 25.36 | 28.40 | 24.36 | 4 | 459.3 | 320.7 | 16.9 | 63.6 | 3832 | 3787 |
| 64k | 8192 | P+D | 192 | 18.86 | 9.04 | 13.26 | 131072 | 25.36 | 27.22 | 21.79 | 12 | 344.6 | 96.7 | 30.2 | 110.6 | 12707 | 6358 |
| 128k | 1 | none | 96 | 18.86 | 36.75 | 1.67 | 16384 | 25.36 | 28.40 | – | – | 110.4 | 110.4 | – | – | 11130 | – |
KV-Cache Offloading
KV-cache offloading moves language models’ attention state from GPU memory to CPU, SSD, or remote tiers to expand context and serving capacity.
See all 47 papers