cs.LGOct 22, 2026

Gandalf: Flash Shall Not Die! Increasing Flash Storage Lifetime for KV Cache Offloading in LLM Serving

Authors: Maharani A. P. Irawan, Kuntai Du, Jiayi Yao, Dongjoo Seo, Yuyang Huang, Haryadi S. Gunawi, Junchen Jiang

Organizations: The University of Chicago, USA · TensorMesh Inc., USA · Samsung Semiconductor, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

KV caching reduces the serving cost of large language models (LLMs), but the growing KV cache footprint is increasingly exceeds GPU and host-memory capacity. Therefore, people use SSD to expand the KV cache storage capacity with high I/O bandwidth. However, in production LLM workload, such as multi-round conversation and LLM agents, the total amount of KV cache writes significantly exceeds SSD capacity, shortening SSD lifetime from years to months. To improve SSD lifetime, a natural solution is to reduce the KV cache writes by gating the KV cache writes via historical access patterns. However, that significantly lowers the KV cache hit rate (and increase the LLM serving cost) since KV caches from different attention types might have identical historical access pattern but different future access patterns. Our insight is that, to reduce KV cache writes without hurting KV cache hit rate, one need to not only consider the access pattern, but also application-level information such as the attention type of KV cache. Leveraging this observation, we propose Gandalf , the first SSD KV cache offloading system with SSD life time awareness. Gandalf prioritize the admission of full-attention KV cache into SSD over the cache generated by other types of attention, which significantly increases SSD lifetime without sacrificing cache hit rate. Our evaluation shows that Gandalf extends the SSD lifetime by 7.3X on average, while having negligible impact on cache hit rate across multiple workload traces and wide range of hardware capacity. Gandalf design can be extended to other flash-based storage, such as High Bandwidth Flash (HBF).

Figures & tables

Figure 1. Using historical access pattern creates a trade-off between cache hit rate and SSD lifetime. Gandalf maintains cache hit rate while increasing SSD lifetime.
Type2nd-round3rd-roundFuture roundsReuse count
Full attn\ding 51\ding 51\ding 51NroundsN_{\mathrm{rounds}}
Linear attn\ding 51\ding 55\ding 551
Table 1. Reuse pattern of first round’s KV caches in multi-round conversation
WorkloadModelKV cache size (TB)SSD lifetime (y)
Claude Code cc-traces-062126Qwen3.5-27B980.4
 Kimi-K32400.2
Claude Code cc-traces-060826Qwen3.5-27B810.4
 Kimi-K31990.2
Table 2. Under KV cache offloading workloads, SSDs may last less than a year—far short of their five-year warranty.
Figure 2. Under same acquisition budget, SSD is more optimal to be used for KV cache offloading workload, which is reuse-heavy.
Figure 3. Existing systems trades off SSD lifetime and cache hit rate.
Figure 4. Gandalf overview. Gandalf admits new KV caches to host memory and, when it is full, chooses between dropping a DRAM cache or moving it to SSD based on SSD write and replacement-cache recomputation costs.
Figure 5. CPU/SSD capacity ratio to obtain maximum SSD lifetime improvement.
Figure 6. In Claude Code WEKA trace, most linear-attention states are reused once or never.
  Required DWPD
TraceModel1TB4TB
cc-traces-weka-062126Qwen3.5-27B34.48.60
 Kimi-K348.212.0
qwen_traceA_blksz_16Qwen3.5-27B13032.4
 Kimi-K327468.5
aiperf-100r-500o-noshareQwen3.5-27B27869.4
Table 3. The traces writes exceed the SSD’s workload DWPD rating.
Figure 7. SSD lifetime against cache miss ratio for Gandalf and the baselines on all traces with Kimi-K3 model, 100 GB of DRAM and a 1 TB SSD. Gandalf , LRU-K, HotPrefix and Marconi are lines through their hyperparameters ( Gandalf ’s α\alpha , LRU-K’s KK , HotPrefix’s hotness frequency, and Marconi’s α\alpha ).
Figure 8. Gandalf achieves lower cache miss ratio while achieving higher SSD lifetime under different DRAM/SSD capacity ratios. We fix the total capacity of DRAM and SSD to 4TB to make sure the cache miss ratio and lifetime is comparable across different setups.
Figure 9. Gandalf improves baselines in wide range of hybrid models having different linear-to-full cache size ratios.
Figure 10. Gandalf SSD admission results in higher fraction of SSD writes that contributes to cache hits at least once. Unlike baselines, Gandalf cache hit is stable across time.
Figure 11. A bigger portion of linear state cache hit in Gandalf comes from DRAM, while most full attention KV cache hit comes from SSD.
Figure 12. The gain from Gandalf comes from both reducing the amount of writes to the SSD (left) and increasing the Read/Write ratio (right).
Figure 13. Gandalf increases SSD lifetime while maintaining low cache load latency.
Figure 14. Average cost per request, recomputation plus SSD wear, on ClaudeCode with Kimi-K3, 100 GB of DRAM and a 1 TB SSD. Gandalf has the lowest cost once the SSD wear is counted.
More in this topic

KV-Cache Offloading

KV-cache offloading moves language models’ attention state from GPU memory to CPU, SSD, or remote tiers to expand context and serving capacity.

See all 42 papers

Explore similar work

CardsList
Sep 28, 2026

TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash

Reusable prefix key-value (KV) caches can outgrow GPU memory in large language model (LLM) serving. A memory-semantic flash hierarchy offers SSD-backed capacity with a limited fast tier, but a logical KV hit is not necessarily ready for GPU retrieval. Demand staging exposes SSD latency, whereas immediate staging can reserve fast-tier capacity long before retrieval begins. We present TempoKV, a timing-aware resource-commitment layer that separates early knowledge of reuse from the acquisition of staging resources. It records reusable-KV hits as metadata-only claims and requests commitment when the runtime-estimated time until retrieval falls to the storage-estimated time needed to make KV resident and protected against eviction. These estimates adapt to runtime progress and staging state, while commitment remains subject to available protected capacity. We implement TempoKV in vLLM and LMCache on an SSD-backed CXL memory device without changing request scheduling. Across two models and three prefix cache ratios, TempoKV reduces protected fast-tier byte-time per request by 63-91% versus immediate staging while retaining much of the serving benefit of advance staging. In a fast-tier capacity sweep, output throughput and p95 time to first token (TTFT) remain nearly unchanged as capacity decreases from 100 to 25 GiB. Compared with unmodified LMCache's Device-DAX L1 configuration, TempoKV reduces p95 TTFT by up to 48.0% and increases output throughput by up to 27.8%.

Jay H. Park, Hyungjun Kim, Dong Kim

Samsung Semiconductor

KV-Cache OffloadingKV-Cache ManagementLLM Inference AccelerationPrefix Caching
AskPDF
Jun 1, 2026

Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving

Large Language Model (LLM) inference relies on key-value (KV) caches to avoid redundant attention computation. While approximate KV cache retention techniques reduce memory usage by sacrificing model accuracy, lossless approaches instead evict KV cache blocks from GPU memory and reconstruct them on demand to preserve exact outputs. Existing lossless KV cache management systems primarily base eviction decisions on access frequency or positional heuristics, without considering how different KV cache blocks affect the execution efficiency of GPU attention kernels. In this paper, we propose AsymCache, a computation-latency-aware KV cache management system for LLM inference that explicitly aligns cache residency decisions with GPU attention kernel performance, including three key components: Multi-Segment Attention (MSA) for efficient non-contiguous KV context processing, a cache eviction policy that jointly optimizes hit rate and position-aware recomputation cost, and an adaptive chunking scheduler for high hardware utilization. Experiments show that AsymCache reduces TTFT by up to 1.90-2.03x and time-per-output-token (TPOT) by 1.62-1.71x over latest baselines, confirming the effectiveness of the method in common workloads and validating its design goal of balancing computational efficiency with cache hit rate. Moreover, the low-level design of AsymCache allows seamless integration into agent serving systems such as Continuum, where it further reduces average job latency by up to 18.1%.

Chunan Shi, Yilei Chen, Yilin Chen +2

Peking University Beijing, China

KV-Cache OffloadingLLM InferenceKV CachingKV-Cache ManagementLLM Inference Acceleration
AskPDF
Sep 20, 2026

Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Management

NAND-backed storage offers the capacity needed to scale LLM prefix caching, but its block I/O path incurs CPU cache contention and host-DRAM staging in addition to NAND latency. Our characterization shows that these interface costs persist even with DRAM as the storage medium, motivating CXL-SSDs for byte-addressable access to NAND-backed capacity. Surprisingly, however, a stock CXL-SSD remains about 3×\times slower than local DRAM and no faster than an NVMe SSD, while generic prefetching provides little benefit. We present LM-CXD, a CXL-SSD specialized for LLM prefix caching. LM-CXD bridges the semantic gap between the serving engine, which knows which KV chunks will be consumed, and the device, which controls their placement and movement. It makes KV chunks device-visible I/O units, exposes NAND-to-DRAM progress to the serving engine, and uses device DRAM as a GPU-accessible buffer. LM-CXD further coordinates request scheduling with windowed prefetching and pipelines layerwise KV movement with GPU computation to hide NAND latency under limited device DRAM. Across five LLM models, LM-CXD reduces average TTFT over a stock CXL-SSD by up to 2.6×\times with compute asynchronous prefetching and 4.03×\times with layerwise prefetching, achieving TTFT within 1.5×\times of local DRAM on average.

Hyunsun Chung, Taewan Noh, Minji Kim +3

Sogang University · Samsung Electronics Co. · ETRI

LLM Inference AccelerationKV-Cache OffloadingKV CachingLLM Inference Scheduling
AskPDF