GPU high bandwidth memory is scarce and expensive, and KV caches consume much of it as chats, agent loops, and document question answering accumulate state. Systems such as Mooncake, LMCache, FlexGen, InfiniGen, and AttentionStore extend GPU memory with CPU DRAM and SSD. The harder question is which blocks belong in each tier, when to move or evict them, and whether prefetching helps. We study these choices in a discrete event simulator spanning GPU HBM, CPU DRAM, and SSD, calibrated against a random forest execution time predictor. We compare recency, reuse frequency, predicted reuse, and an EWMA predictor with prefetch lookahead across chat, agent, and document question answering workloads. Tiering supports 73.02 times more concurrent sessions per GPU and lowers cost per session by 62.04 times. These gains come from tier capacities of 1 plus 8 plus 64, not placement policy. Decode is compute bound at batch size one in our setup, so placement barely affects throughput. It mainly changes PCIe migration traffic and time to first token. Recency produces 2.30 times less migration traffic than reuse frequency for chat. Reuse frequency performs best for agents and document question answering. The existing predicted reuse policy is byte identical to recency, making its agent recommendation effectively recency. A genuine EWMA predictor changes behavior but still ranks behind reuse frequency on the workloads prediction was expected to help. Prefetching does not justify its bandwidth cost. Across the policy and cache size grid, even an oracle with knowledge of future requests never beats no prefetch on migration traffic. Workload specific placement can reduce data movement, but the predicted reuse and prefetch recommendations are not supported as implemented.
Figures & tables
Fig. 1 : Study overview. (1) GPU HBM is small and expensive, but long-lived sessions accumulate a KV cache that overflows it. (2) The KV cache can be tiered across GPU HBM, CPU DRAM, and SSD, but where each block should live, and when to move it, is an unsettled policy question. (3) We build a tiered placement layer with reuse-aware promotion and prefetch and study which policy wins on chat, agent, and document-QA workloads.
Fig. 2 : The opening: as the number of concurrent long-lived sessions grows, the total KV cache overflows GPU HBM. Tiering the cache into CPU DRAM and SSD extends the effective capacity by orders of magnitude at a fraction of the cost per gigabyte; Table III quantifies this on the workloads studied. Conceptual schematic.
Fig. 3 : Why the policy is workload-specific: a workload’s reuse structure sets which placement policy minimizes fetch traffic (Fig. 5 ); heavy prefix reuse favours reuse-frequency, while recency-driven growth favours recency. Note the capacity and cost wins of tiering (Table III ) do not scale with reuse; they track tier capacity. Conceptual schematic.
Fig. 4 : The tiered KV-cache placement layer. KV blocks live across three tiers: GPU HBM (small, fast, hot blocks), CPU DRAM (medium, warm blocks), and SSD (large, cheap, cold blocks). A placement policy decides, per block, the tier and when to promote, demote, or evict, from access signals (recency, reuse frequency, predicted reuse). A prefetch engine moves blocks up the hierarchy ahead of the decode step that needs them, hiding transfer latency. The inset contrasts cost per gigabyte and bandwidth across the tiers.
Fig. 5 : PCIe migration traffic (GB, lower is better) over the placement-policy by workload grid; the boxed cell is the measured best policy per workload. Throughput is omitted because it is flat across policies at batch one (compute-bound decode); migration bytes is the quantity placement actually controls. predicted_reuse is byte-identical to lru on every workload, so the two share the chat winner. Source: results/metrics_all.csv .
Workload
Reuse pattern
Stresses
Long multi-turn chat
recency, growing context
HBM capacity
Agent loop
revisits recent context
promotion, prefetch
Repeated document QA
heavy prefix reuse
cross-request reuse
TABLE I : Workloads, chosen for their reuse structure.
TABLE III : Sessions per GPU and cost per session, HBM-only versus tiered (HBM + DRAM + SSD), at tier shape 1:8:64 . The gain equals the tier’s capacity multiple, not a policy effect. Source: results/ratio_sweep.csv .
Fig. 6 : Sessions per GPU, tiered (HBM + DRAM + SSD) versus HBM-only, across the working-set-to-HBM ratio sweep; the dotted line marks the default 1:8:64 shape. The tiered/HBM-only ratio is flat at ≈ 73.02 × throughout, which is exactly the tier’s capacity multiple; hit rate does not enter it. Source: results/ratio_sweep.csv .
Fig. 7 : Per-tier hit rate (HBM, DRAM, SSD, and the recomputed miss), by workload, for the winning policy. DRAM absorbs most of what overflows HBM; SSD is a shallow backstop. Source: results/metrics_all.csv .
Workload
Recommended
Measured verdict
Document QA
Reuse-frequency + prefetch
Policy holds (1.63 × less traffic); prefetch does not pay (Sec. V-E )
Agent loop
Predicted-reuse + prefetch
Does not hold as written: predicted-reuse ≡ recency; a genuine EWMA predictor finishes second; reuse-frequency wins here; prefetch does not pay
Long chat
Recency (LRU)
Holds, scenario-dependent: for the roadmap’s own case (one isolated long chat), every eviction policy is byte-identical, so cheap LRU is correct by default and capacity is the only lever; under concurrent chats recency genuinely wins (2.30 × less traffic, best TTFT)
Tight HBM
Deep SSD + prefetch
Sessions/GPU win holds (10.77 × , Sec. V-G ); deep SSD matches the recompute SLO on document QA but does not beat it on latency, and never meets it on chat/agent; prefetch does not pay
Capacity
Cost-latency frontier
A genuine frontier exists (Sec. V-F ): it leans SSD-heavy, spanning 8.38 × in cost for at most 10.1% latency
TABLE IV : The roadmap’s practitioner decision table (left) beside the measured verdict (right). “Holds” is judged on PCIe migration traffic, the quantity placement controls at batch one.
Fig. 8 : Per-token decode latency split into compute and fetch stall, by policy; the DRAM/SSD share of the stall is attributed by hit rate (approximate). Compute dominates and stall is < 1% of TPOT, so tiered fetches are hidden behind compute. Source: results/metrics_all.csv .
Fig. 9 : The real cost-latency frontier ( lru policy, one colour per workload), sweeping every (HBM, DRAM, SSD) composition at a fixed budget T=192 blocks; frontier points connected, dominated points faded. Cost per 1k sessions is a pure function of composition (verified in code, independent of workload/policy) and spans 8.38 × ; P99 TTFT moves at most 10.1% of its ceiling across that same span, so the frontier is nearly vertical and leans toward the cheap, SSD-heavy end. Source: results/e11_cost_frontier.csv .
Fig. 10 : Tight HBM (32 blocks) and fixed DRAM (64 blocks), SSD depth swept 8 to 1024 blocks: P99 TTFT and sessions/GPU vs. SSD depth (log scale), one panel per workload. The SLO (dashed) is each workload’s own gpu_only baseline P99 TTFT; the crossover (dotted), where present, marks the shallowest SSD depth that meets it. Chat and agent never cross; document QA crosses at depth 64 and matches the SLO with 0.000 ms headroom at the deepest point. Source: results/e10_capacity.csv .
Key-value (KV) cache memory management is the primary bottleneck limiting throughput and cost-efficiency in large-scale GPU inference serving. Current systems suffer from three compounding inefficiencies: (1) the absence of unified KV cache sizing across all attention architectures--particularly multi-head latent attention (MLA), which is unsupported in general-purpose frameworks, resulting in up to 57x memory over-provisioning; (2) confinement of KV cache to a single memory tier (GPU HBM) despite the availability of a rich hierarchy spanning CPU DRAM, CXL-attached memory, NVMe via GPUDirect Storage, RDMA fabric, and parallel filesystems; and (3) reactive eviction policies that discard reusable state, forcing redundant recomputation. We present a unified system that addresses all three problems. Our architecture-variant-aware sizing engine computes exact memory requirements per attention type, enabling up to 7.4x higher batch sizes. A six-tier memory hierarchy extends effective KV cache capacity from 40 GB to over 38 TB per node while maintaining sub-millisecond time-to-first-token (TTFT) for hot entries. A Bayesian reuse predictor with Beta conjugate priors over 16 (block-type, transition-type) pairs achieves 70-84% cache hit rates, combined with EMA-scored head-granular eviction and RoPE-aware prefetching. Component-level validation on trace replay using ShareGPT, LMSYS-Chat-1M, and agentic workloads demonstrates 70-84% cache hit rates. Analytical projections combining validated component behavior with published hardware specifications indicate 1.4-2.1x projected TTFT reduction, 1.7-2.9x throughput improvement, and 47% cost reduction compared to state-of-the-art baselines.
Reusable prefix key-value (KV) caches can outgrow GPU memory in large language model (LLM) serving. A memory-semantic flash hierarchy offers SSD-backed capacity with a limited fast tier, but a logical KV hit is not necessarily ready for GPU retrieval. Demand staging exposes SSD latency, whereas immediate staging can reserve fast-tier capacity long before retrieval begins. We present TempoKV, a timing-aware resource-commitment layer that separates early knowledge of reuse from the acquisition of staging resources. It records reusable-KV hits as metadata-only claims and requests commitment when the runtime-estimated time until retrieval falls to the storage-estimated time needed to make KV resident and protected against eviction. These estimates adapt to runtime progress and staging state, while commitment remains subject to available protected capacity. We implement TempoKV in vLLM and LMCache on an SSD-backed CXL memory device without changing request scheduling. Across two models and three prefix cache ratios, TempoKV reduces protected fast-tier byte-time per request by 63-91% versus immediate staging while retaining much of the serving benefit of advance staging. In a fast-tier capacity sweep, output throughput and p95 time to first token (TTFT) remain nearly unchanged as capacity decreases from 100 to 25 GiB. Compared with unmodified LMCache's Device-DAX L1 configuration, TempoKV reduces p95 TTFT by up to 48.0% and increases output throughput by up to 27.8%.
An LLM serving engine sizes its key-value (KV) cache once, at startup, permanently setting aside a reserve for the worst-case prefill activation. During decode-dominant phases that reserve sits idle, yet it cannot be handed to the KV pool because it is exactly the memory a large prefill needs. We ask whether this reserve is reclaimable, and build a mechanism to test it. Our elastic KV cache lends the reserve to the KV pool during decode and returns it before prefill, driven by the scheduler's one-step-ahead view of the next batch. It is pure userspace on the CUDA virtual-memory path: two physical handles mapped into one contiguous virtual range per layer, so the attention kernel is unchanged and no driver patch is required. It decommits in a few milliseconds and recommits in tens of milliseconds, works with CUDA graphs and prefix caching, and never triggers an out-of-memory event. A static commit of the same memory is unsafe, crashing on prefill bursts, which makes the dynamic toggle necessary. Having built the mechanism, we test the premise it rests on and report an honest negative result. It only pays off if a small prefill chunk size badly hurts prefill latency. In a controlled experiment injecting long prompts into a live decode load, that penalty is small (median time-to-first-token differs by about 1% between chunk sizes of 8192 and 32768 tokens), because prefill is compute bound and decode consumes only about one token per sequence per step. Simply lowering max_num_batched_tokens recovers more KV than the controller does, at nearly equal latency. The reserve also dilutes under tensor parallelism, from 16% of KV at TP1 to 2.7% at TP4. We state precisely when reclaiming the reserve could still help, and release the mechanism as a reusable userspace elastic-VMM allocator.