cs.LGOct 22, 2026

Gandalf: Flash Shall Not Die! Increasing Flash Storage Lifetime for KV Cache Offloading in LLM Serving

Authors: Maharani A. P. Irawan, Kuntai Du, Jiayi Yao, Dongjoo Seo, Yuyang Huang, Haryadi S. Gunawi, Junchen Jiang

Organizations: The University of Chicago, USA · TensorMesh Inc., USA · Samsung Semiconductor, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

KV caching reduces the serving cost of large language models (LLMs), but the growing KV cache footprint is increasingly exceeds GPU and host-memory capacity. Therefore, people use SSD to expand the KV cache storage capacity with high I/O bandwidth. However, in production LLM workload, such as multi-round conversation and LLM agents, the total amount of KV cache writes significantly exceeds SSD capacity, shortening SSD lifetime from years to months. To improve SSD lifetime, a natural solution is to reduce the KV cache writes by gating the KV cache writes via historical access patterns. However, that significantly lowers the KV cache hit rate (and increase the LLM serving cost) since KV caches from different attention types might have identical historical access pattern but different future access patterns. Our insight is that, to reduce KV cache writes without hurting KV cache hit rate, one need to not only consider the access pattern, but also application-level information such as the attention type of KV cache. Leveraging this observation, we propose Gandalf , the first SSD KV cache offloading system with SSD life time awareness. Gandalf prioritize the admission of full-attention KV cache into SSD over the cache generated by other types of attention, which significantly increases SSD lifetime without sacrificing cache hit rate. Our evaluation shows that Gandalf extends the SSD lifetime by 7.3X on average, while having negligible impact on cache hit rate across multiple workload traces and wide range of hardware capacity. Gandalf design can be extended to other flash-based storage, such as High Bandwidth Flash (HBF).

Figures & tables

Figure 1. Using historical access pattern creates a trade-off between cache hit rate and SSD lifetime. Gandalf maintains cache hit rate while increasing SSD lifetime.
Type2nd-round3rd-roundFuture roundsReuse count
Full attn\ding 51\ding 51\ding 51NroundsN_{\mathrm{rounds}}
Linear attn\ding 51\ding 55\ding 551
Table 1. Reuse pattern of first round’s KV caches in multi-round conversation
WorkloadModelKV cache size (TB)SSD lifetime (y)
Claude Code cc-traces-062126Qwen3.5-27B980.4
 Kimi-K32400.2
Claude Code cc-traces-060826Qwen3.5-27B810.4
 Kimi-K31990.2
Table 2. Under KV cache offloading workloads, SSDs may last less than a year—far short of their five-year warranty.
Figure 2. Under same acquisition budget, SSD is more optimal to be used for KV cache offloading workload, which is reuse-heavy.
Figure 3. Existing systems trades off SSD lifetime and cache hit rate.
Figure 4. Gandalf overview. Gandalf admits new KV caches to host memory and, when it is full, chooses between dropping a DRAM cache or moving it to SSD based on SSD write and replacement-cache recomputation costs.
Figure 5. CPU/SSD capacity ratio to obtain maximum SSD lifetime improvement.
Figure 6. In Claude Code WEKA trace, most linear-attention states are reused once or never.
  Required DWPD
TraceModel1TB4TB
cc-traces-weka-062126Qwen3.5-27B34.48.60
 Kimi-K348.212.0
qwen_traceA_blksz_16Qwen3.5-27B13032.4
 Kimi-K327468.5
aiperf-100r-500o-noshareQwen3.5-27B27869.4
Table 3. The traces writes exceed the SSD’s workload DWPD rating.
Figure 7. SSD lifetime against cache miss ratio for Gandalf and the baselines on all traces with Kimi-K3 model, 100 GB of DRAM and a 1 TB SSD. Gandalf , LRU-K, HotPrefix and Marconi are lines through their hyperparameters ( Gandalf ’s α\alpha , LRU-K’s KK , HotPrefix’s hotness frequency, and Marconi’s α\alpha ).
Figure 8. Gandalf achieves lower cache miss ratio while achieving higher SSD lifetime under different DRAM/SSD capacity ratios. We fix the total capacity of DRAM and SSD to 4TB to make sure the cache miss ratio and lifetime is comparable across different setups.
Figure 9. Gandalf improves baselines in wide range of hybrid models having different linear-to-full cache size ratios.
Figure 10. Gandalf SSD admission results in higher fraction of SSD writes that contributes to cache hits at least once. Unlike baselines, Gandalf cache hit is stable across time.
Figure 11. A bigger portion of linear state cache hit in Gandalf comes from DRAM, while most full attention KV cache hit comes from SSD.
Figure 12. The gain from Gandalf comes from both reducing the amount of writes to the SSD (left) and increasing the Read/Write ratio (right).
Figure 13. Gandalf increases SSD lifetime while maintaining low cache load latency.
Figure 14. Average cost per request, recomputation plus SSD wear, on ClaudeCode with Kimi-K3, 100 GB of DRAM and a 1 TB SSD. Gandalf has the lowest cost once the SSD wear is counted.
More in this topic

KV-Cache Offloading

KV-cache offloading moves language models’ attention state from GPU memory to CPU, SSD, or remote tiers to expand context and serving capacity.

See all 42 papers

Explore similar work

CardsList

  1. TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash

    Sep 28, 2026Jay H. Park, Hyungjun Kim, Dong KimKV-Cache OffloadingKV-Cache Management

    Samsung Semiconductor

  2. Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving

    Jun 1, 2026Chunan Shi, Yilei Chen, Yilin Chen +2KV-Cache OffloadingLLM Inference

    Peking University Beijing, China

  3. Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Management

    Sep 20, 2026Hyunsun Chung, Taewan Noh, Minji Kim +3LLM Inference AccelerationKV-Cache Offloading

    Sogang University · Samsung Electronics Co. · ETRI