An LLM serving engine sizes its key-value (KV) cache once, at startup, permanently setting aside a reserve for the worst-case prefill activation. During decode-dominant phases that reserve sits idle, yet it cannot be handed to the KV pool because it is exactly the memory a large prefill needs. We ask whether this reserve is reclaimable, and build a mechanism to test it. Our elastic KV cache lends the reserve to the KV pool during decode and returns it before prefill, driven by the scheduler's one-step-ahead view of the next batch. It is pure userspace on the CUDA virtual-memory path: two physical handles mapped into one contiguous virtual range per layer, so the attention kernel is unchanged and no driver patch is required. It decommits in a few milliseconds and recommits in tens of milliseconds, works with CUDA graphs and prefix caching, and never triggers an out-of-memory event. A static commit of the same memory is unsafe, crashing on prefill bursts, which makes the dynamic toggle necessary. Having built the mechanism, we test the premise it rests on and report an honest negative result. It only pays off if a small prefill chunk size badly hurts prefill latency. In a controlled experiment injecting long prompts into a live decode load, that penalty is small (median time-to-first-token differs by about 1% between chunk sizes of 8192 and 32768 tokens), because prefill is compute bound and decode consumes only about one token per sequence per step. Simply lowering max_num_batched_tokens recovers more KV than the controller does, at nearly equal latency. The reserve also dilutes under tensor parallelism, from 16% of KV at TP1 to 2.7% at TP4. We state precisely when reclaiming the reserve could still help, and release the mechanism as a reusable userspace elastic-VMM allocator.
Figures & tables
Figure 1: The prefill-activation reserve. KV capacity shrinks as the prefill chunk grows, because the engine reserves more memory for the prefill activation peak. The gap is about 16% of KV at a 32768-token chunk (Qwen2.5-7B, TP1, util=0.9).
Figure 2: Raising gpu_memory_utilization captures stranded headroom but plateaus at the startup guard (0.99 refuses to start). This is a different pool from the activation reserve.
Figure 3: Toggle latency for the 3 GiB reserve across 28 layers. The contiguous-VA mechanism commits in about 30 ms and decommits in about 3 ms, versus 180 ms for a naive whole-handle commit.
Figure 4: Static reclamation of the full reserve recovers the capacity but OOMs on a prefill burst. Only the dynamic toggle captures the full reserve safely.
Mode
TTFT median (s)
TTFT max (s)
KV capacity (K tok)
A: chunk 8192
4.97
6.07
375
B: chunk 32768
4.91
5.79
314
Elastic (auto)
4.90
5.76
364
Table 1: Foreground long-prompt TTFT under a live decode load, and KV capacity. Mode A (small chunk) wins on both axes. It has the most KV, and its TTFT is within about 1% of the others.
Figure 5: Foreground TTFT for six long prompts injected into a live decode load. The three modes nearly overlap; a small prefill chunk barely raises TTFT.
Figure 6: The KV-versus-latency frontier. Mode A (small chunk) has the most KV at about the same median TTFT, so it wins on both axes and the elastic cache has little to buy back.
Figure 7: Decode-only fraction of wall time by regime. Windows far exceed the recommit cost except under sustained saturation, and the larger chunk has more availability.
Figure 8: The reserve dilutes under tensor parallelism, from 16% of KV at TP1 to as little as 2.7% at TP4. Big models need high TP, so their reserve is small.
Large language model (LLM) serving is increasingly constrained by the GPU memory consumed by key-value (KV) caches. Existing compression, eviction, and offloading techniques alleviate this pressure, but serving runtimes typically treat only the configured target KV representation as execution-ready. Under memory pressure, this target-only contract can turn KV shortage into request stalls and preemptions. We present ElasticKV, a mixed-fidelity KV runtime built on the observation that target fidelity need not gate execution. ElasticKV introduces a compact intermediate KV state, making fidelity a runtime-managed execution property. To realize this state in a paged serving runtime, ElasticKV combines (i) a pair-structured layout that turns fidelity reduction into reusable GPU capacity, (ii) a dual-mode attention backend that directly consumes the compact state while preserving the native target-only path, and (iii) pressure-aware fidelity management that adapts KV fidelity to memory pressure. Our extensive evaluation across diverse workloads, model families and scales, and GPU platforms demonstrates the effectiveness and generality of ElasticKV. Under high concurrency, ElasticKV achieves 3.8-4.0× lower time-to-first-token (TTFT) and 9.1× lower P90 TTFT than vLLM while preserving generation quality.
Reusable prefix key-value (KV) caches can outgrow GPU memory in large language model (LLM) serving. A memory-semantic flash hierarchy offers SSD-backed capacity with a limited fast tier, but a logical KV hit is not necessarily ready for GPU retrieval. Demand staging exposes SSD latency, whereas immediate staging can reserve fast-tier capacity long before retrieval begins. We present TempoKV, a timing-aware resource-commitment layer that separates early knowledge of reuse from the acquisition of staging resources. It records reusable-KV hits as metadata-only claims and requests commitment when the runtime-estimated time until retrieval falls to the storage-estimated time needed to make KV resident and protected against eviction. These estimates adapt to runtime progress and staging state, while commitment remains subject to available protected capacity. We implement TempoKV in vLLM and LMCache on an SSD-backed CXL memory device without changing request scheduling. Across two models and three prefix cache ratios, TempoKV reduces protected fast-tier byte-time per request by 63-91% versus immediate staging while retaining much of the serving benefit of advance staging. In a fast-tier capacity sweep, output throughput and p95 time to first token (TTFT) remain nearly unchanged as capacity decreases from 100 to 25 GiB. Compared with unmodified LMCache's Device-DAX L1 configuration, TempoKV reduces p95 TTFT by up to 48.0% and increases output throughput by up to 27.8%.
Large language model serving faces a critical memory bottleneck: the KV cache grows with sequence length and batch size. PagedAttention uses fixed-size memory blocks to reduce allocator-level fragmentation, but recent KV eviction algorithms operate at a token granularity finer than block-level management. This mismatch causes intra-block fragmentation, leaving a large fraction of allocated KV memory unreclaimable. We present vToken, a lightweight token-level virtualization layer that decouples logical token liveness from physical block placement. vToken maintains a stable logical token view through token-table indirection and realizes physical reclamation by repacking live tokens asynchronously. The design preserves PagedAttention kernels and CUDA Graph compatibility. We implement vToken in vLLM and evaluate it with H2O, Random, and Scissorhands across models. Compared with a paired Naive-Evict baseline, vToken reduces retained KV blocks per request by 27.2%--72.3% and improves SLA-constrained throughput by up to 1.37×. Under a constrained active-KV budget, it extends the maximum feasible concurrency by up to 2×, while reducing the per-policy integration footprint from 500+ lines to under 50.
Yuanhang Gao, Xiangrui Yang, Yuanfeng Chen +4
National University of Defense Technology · Changsha, China · Peking University +1