KV Cache Optimization Strategies for Scalable and Efficient LLM Inference

Yichun Xu; Navjot K. Khaira; Tejinder Singh

cs.LGMar 20, 2026

KV Cache Optimization Strategies for Scalable and Efficient LLM Inference

Authors: Yichun Xu, Navjot K. Khaira, Tejinder Singh

Organizations: Dell Technologies, Hopkinton, MA 01748, USA · Dell Technologies, Santa Clara, CA 95054, USA

Abstract

The key-value (KV) cache is a foundational optimization in Transformer-based large language models (LLMs), eliminating redundant recomputation of past token representations during autoregressive generation. However, its memory footprint scales linearly with context length, imposing critical bottlenecks on GPU memory capacity, memory bandwidth, and inference throughput as production LLMs push context windows from thousands to millions of tokens. Efficient KV cache management has thus become a first-order challenge for scalable LLM deployment. This paper provides a systematic review of recent KV cache optimization techniques, organizing them into five principal directions: cache eviction, cache compression, hybrid memory solutions, novel attention mechanisms, and combination strategies. For each category we analyze the underlying mechanisms, deployment trade-offs, and empirical performance across memory reduction, throughput, and model accuracy metrics. We further map techniques to seven practical deployment scenarios, including long-context single requests, high-throughput datacenter serving, edge devices, multi-turn conversations, and accuracy-critical reasoning, providing actionable guidance for practitioners selecting among competing approaches. Our analysis reveals that no single technique dominates across all settings; instead, the optimal strategy depends on context length, hardware constraints, and workload characteristics, pointing toward adaptive, multi-stage optimization pipelines as a promising direction for future research.

Figures & tables

Panel of Figure 1 — Figure 1: Autoregressive generation, at each step the new token (orange) attends to all prior tokens (cyan). Without caching, keys and values for every past token would be recomputed from scratch at each step. The KV cache avoids this by storing and reusing them.

Panel of Figure 2 — Figure 2: Data-flow of the KV cache within a single transformer layer. Input token $x_{t}$ fans into three projections; $K_{t}$ and $V_{t}$ are appended to their respective caches (teal); $Q_{t}$ attends over the full caches to produce output $o_{t}$ . Cache size grows as $O(T)$ per head per layer.

Panel of Figure 3 — Figure 3: KV cache memory as a function of context length for three LLaMA-2 model variants under fp16 precision. Dashed lines mark GPU VRAM limits; dotted lines mark model parameter memory. At 128K tokens, a 7B model’s KV cache ( $\approx$ 64 GB) exceeds the capacity of an A100 GPU, illustrating the memory bottleneck that motivates KV cache optimization. Values computed as $2\times L\times H_{\text{kv}}\times d_{h}\times 2$ bytes per token; 70B uses GQA with 8 KV heads.

Panel of Figure 4 — Figure 4: Causal self-attention weight matrix for “The apple tastes sweet.” visualised with the Viridis colormap (dark purple = low, yellow = high). Gray cells are causally masked future tokens. Each row sums to 1 (post-softmax). Query “sweet” concentrates 65% of its attention on “apple”, demonstrating that KV entries carry highly non-uniform importance, the core premise of attention-score-driven eviction methods such as H 2 O and SnapKV.

Panel of Figure 5 — Figure 5: Taxonomy of KV cache optimization techniques surveyed in this paper, organized into five major categories.

Technique	Optimization Goal	Tradeoffs	Representative Methods	Good for
Cache Eviction	Memory Footprint, Decoding Latency	Accuracy and quality	H 2 O, SnapKV, NACL, InfiniPot, Hashevict, MorphKV, RocketKV, KVzip, Ada-KV	Long-context single requests, Edge devices, Minimal model modification
Cache Compression	KV cache size, Throughput	Dequantization and reconstruction overhead, Accuracy	KIVI, MiniCache, PALU, KVQuant	Bandwidth-bound workloads, Edge devices, Ultra-long contexts
Hybrid memory solution	TTFT in long-context, System efficiency	Hardware requirement, complexity in management	PagedAttention, InfiniGen, LayerKV, INF2, KVPR, Oneiros, CLO	High - throughput serving, Multi-tenant datacenter serving
New Attention Calculation	computational complexity, Inference Speed	Accuracy, model retraining	Linear, Log-Linear, Local Linear, KIMI Linear	Agentic/Long-horizon Tasks
Combination Methods	Throughput/Latency Balance	Complexity in design and management	FlexGen, Q-Hitter, ShadowKV, TailorKV	Consumer-Grade Hardware and Diverse Workloads, balanced performance

Table 1: Comparison of KV Cache optimization techniques

Figure 6: Upper plots illustrate symbolic plots of an attention map deploying different KV cache policies in LLM generation. Lower right: contrasts their accuracy-memory trade-off. Left: the overview of H 2 O framework [ 1 ] .

Figure 7: The graph shows the simplified workflow of SnapKV, where the orange area represents the cluster of features per head selected by SnapKV. These features are then used to form new Key-Value pairs concatenated with the features in the observation window. Together, the selected prefix and observation windows constitute the new KV cache utilized for the generation. [ 2 ] .

Method	Mechanism	Phase	Overview
H 2 O [ 1 ]	Evict top K noncritical tokens	Decoding	Retains a balance of Heavy - Hitter tokens (H2) and the most recent tokens
SnapKV [ 2 ]	Retain critical tokens and their context	After Prefill	Identifies critical attention features by voting from observation window and uses pooling for clustering
NACL [ 9 ]	Hybrid Proxy and Random Eviction	Prefill	Performs a single, global eviction during encoding
InfiniPot [ 10 ]	Continual Context Distillation	Prefill	Enables processing of infinite context within a fixed memory budget
HASHEVICT [ 11 ]	Attention - Free LSH Ranking	Decoding	Uses Locality - Sensitive Hashing (LSH) and Hamming Distance to estimate token importance
MorphKV [ 13 ]	Correlation - aware selection	Decoding	Maintains a constant - sized cache for long - response tasks while eliminating early - token bias

Table 2: Summary of KV Cache eviction techniques

Figure 8: Definition of per-token and per-channel quantization. $X\in\mathbb{R}^{l_{\text{prompt}}\times d}$ is the key/value cache, where $l_{\text{prompt}}$ is the number of tokens and $d$ is the number of channels. $z_{X}$ is the zero-point, and $s_{X}$ is the scaling factor.. [ 5 ] .

Figure 9: Palu’s low-rank projection method for KV-cache reduction. A weight matrix $\mathbf{W}$ of linear projection is decomposed into two low-rank matrices. Input $\mathbf{X}$ is down-projected to a latent representation $\mathbf{H}$ , which is cached. $\mathbf{Y}$ can be reconstructed from $\mathbf{H}$ using the up-projection matrix $\mathbf{B}$ . [ 19 ] .

Method	Mechanism	Granularity	Outlier handle
KIVI [ 5 ]	Asymmetric Quantization	Key : Per-channel; Value : Per-token	Keeps a small amount of residual cache in full precision
KVQuant [ 20 ]	Non-Uniform Quantization (NUQ)	Key : Per-channel (Pre-RoPE); Value : Per-token	Isolates the top 1% of outliers into a separate sparse fp16 representation
MiniCache [ 21 ]	Cross-Layer KV Merge	NA	Keeps highly distinct unmergeable state pairs in full precision
PALU [ 19 ]	Hidden-dimension compression with Low-Rank Projection and Reconstruction	Per-token per head group latent vector	Assigns higher ranks to critical layers

Table 3: Cache Compression Methods Comparison Table

Figure 10: vLLM system overview [ 22 ] .

Figure 11: Operation flow of the prefetching module of InfiniGen. [ 23 ] .

Method	Offload destination	Mechanism	Key optimization
Paged Attention [ 22 ]	CPU Memory (DRAM)	Paging the KV cache, inspired by virtual memory	Reducing memory fragmentation and enabling cache sharing
InfiniGen [ 23 ]	CPU Memory (DRAM)	Attention speculation and prefetching	Reducing PCIe transfer volume by only fetching essential data
LayerKV [ 24 ]	CPU Memory (DRAM)	Layer - wise management and SLO - aware scheduler to balance GPU/CPU residency	Reducing TTFT by minimizing queuing delays
INF2 [ 25 ]	Host Memory + NVMe SSDs (CSDs)	Attention - Near Storage (ANS) via Computational Storage Devices (CSDs)	High throughput for long - context models by leveraging internal storage bandwidth
KVPR [ 26 ]	CPU Memory (DRAM)	Overlapping partial cache recomputation on the GPU with asynchronous transfer	Minimizing GPU idle time caused by limited PCIe bandwidth
Oneiros [ 27 ]	CPU Memory (DRAM)	Parameter remapping to repurpose GPU memory from inactive models into KV cache space	Optimizing multi - tenant serving by reclaiming memory from idle models

Table 4: Hybrid Memory Solutions Comparison Table

Figure 12: Standard linear attention (top) vs. loglinear attention (bottom). The input consists of query, key, and value vectors [ 30 ] .

Method	Mechanism	Training Time Complexity	Decoding Time Complexity per step	Decoding Space Complexity (Total Memory)	Features
Softmax	Scaled Dot - Product Attention & Multi - Head Attention	$O(T^{2})$	$O(T)$	$O(T)$	High expressivity, high cost
Linear	Replaces softmax with a linear dot - product of kernel feature maps	$O(T)$	$O(1)$	$O(1)$	Limited expressivity and low cost
Log Linear	Replaces the fixed - size hidden state with a logarithmically growing set of hidden states	$O(T\log T)$	$O(\log T)$	$O(\log T)$	Balances efficiency and expressiveness
Local Linear	Use Local Linear Regression to perform query - specific local linear fitting	$O(T^{2})$	$\sim O(T)$	$O(T)$	Superior bias–variance trade - off; high cost
KIMI Linear	Hybrid architecture (KDA + Multi - Head Latent Attention)	Mainly $O(T)$ due to KDA	$O(1)$	$O(1)$	Outperforms full attention, but requires hybridization (e.g., 3:1 ratio with full attention) to maintain global information flow

Table 5: Attention Variants – Mechanisms, Complexities, and Features

Figure 13: During Pre-filling, ShadowKV offloads the value cache to the CPU while maintaining a low-rank key cache, landmarks, and outliers on the GPU. During decoding, it employs landmarks for sparse attention. [ 35 ] .

Figure 14: System overview of TailorKV. Offline identification categorizes the layers into quantization-friendly and sparsity-friendly. For quantization-friendly layers, we employ aggressive static quantization. For sparsity-friendly layers, we dynamically retrieve Top-K tokens. Critical current query and critical key cache represent the outliers in the query and key cache, respectively. [ 36 ] .

Technique	Memory	Speedups	Accuracy loss	Tradeoffs
H 2 O [ 1 ]	Up to 5–10 $\times$ memory reduction	Up to 29 $\times$ throughput improvement; $\leq$ 1.9 $\times$ lower latency compare to FlexGen	Comparable to baseline	accumulated-attention bias; risk of heavy - hitter loss
SnapKV [ 2 ]	8.2 $\times$ memory efficiency	3.6 $\times$ generation speedup	Comparable to baseline	Does not optimize prefill; cannot extend inherent model context limits
NACL [ 9 ]	Up to 5 $\times$ KV reduction	O(1) single-shot eviction; major prefill simplification	95% performance retention	Proxy-token selection heuristic; limited ultra-long behavior
InfiniPot [ 10 ]	Tested with 4k to 1M tokens	“Performance regardless of context length”	Consistent performance regardless of context length and optimizing memory use	Fixed compression ratio may not be optimal for all data types
HASHEVICT [ 11 ]	30–70% compression	1.5–2 $\times$ prefill speedup against baseline methods such as H 2 O, Scissorshands, and 17 $\times$ prefill/2 $\times$ decoding speed against FastGen	Decent performance except at very small budgets (e.g., 10% capacity)	Irreversible eviction; quality drops sharply at 10% cache budget
MorphKV [ 13 ]	Up to 5 $\times$ over Full-Attention.	Up to 4.68 $\times$ faster than SnapKV	Comparable to SnapKV	Performance is sensitive to hyperparameters like window size and fusion functions

Table 6: Comparison of KV Cache Optimization Techniques

KV-Cache Management

Key-value (KV) cache management reduces and coordinates Transformer attention state to make long-context inference faster and more memory-efficient.

See all 199 papers

Apr 6, 2026

Comparative Characterization of KV Cache Management Strategies for LLM Inference

Efficient inference with Large Language Models (LLMs) increasingly relies on Key-Value (KV) caches to store previously computed key and value vectors at each layer. These caches are essential to minimize redundant computation during autoregressive token generation, lowering computational complexity from quadratic to linear. However, the growth of KV caches has posed significant system-level challenges, particularly as model sizes increase, context lengths grow, and concurrent requests compete for limited memory resources. Even though several recent frameworks for KV cache management have emerged, their comparative trade-offs in memory consumption and inference performance have not been fully understood, especially under varying request sizes and model configurations. In this work, we conduct an empirical study of three state-of-the-art KV cache management frameworks: vLLM, InfiniGen, and H2O. These frameworks employ techniques such as tensor offloading, token eviction heuristics, and speculative scheduling to balance memory usage and performance. We evaluate their performance in terms of a range of metrics such as latency, throughput, and memory usage across a spectrum of key parameters including request rates, model sizes, and sparsity levels. Our results pinpoint the conditions for each framework to perform the best, revealing the most suitable selection and configuration of KV cache strategies under memory and performance constraints.

Oteo Mamo, Olga Kogiou, Hyunjin Yi +1

Florida State University, Tallahassee, FL, USA

KV-Cache Management LLM Inference Efficiency LLM Inference KV-Cache Offloading KV-Cache Eviction

Ask PDF

Mar 27, 2026

TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference

Key-value (KV) caching is critical for efficient inference in large language models (LLMs), yet its memory footprint scales linearly with context length, resulting in a severe scalability bottleneck. Existing approaches largely treat KV states as equally important across time, implicitly assuming uniform precision and accessibility. However, this assumption contrasts with human memory systems, where memories vary in clarity, recall frequency, and relevance with temporal proximity.Motivated by this insight, we propose TTKV, a KV cache management framework that maps the human memory system onto the KV cache. TTKV partitions the KV cache into temporal tiers with heterogeneous capacity and precision. The design addresses three aspects: (1) Tier Layout, decoupling fast and slow memory using HBM and DRAM; (2) Tier Content, assigning more recent KV states to faster, higher-precision tiers based on temporal proximity; and (3) Tier Interaction, employing block-wise streaming attention to overlap communication and computation when accessing slow tiers. Experiments show that TTKV reduces cross-tier traffic by 5.94x on 128K-context tasks, achieving up to 76% latency reduction and 2x throughput improvement over strong baselines.

Gradwell Dzikanyanga, Weihao Yang, Hao Huang +4

Harbin Institute of Technology, Shenzhen · Guangzhou University

KV-Cache Management Long-Context Language Model Inference LLM Inference Acceleration KV-Cache Offloading

Ask PDF

Apr 12, 2026

IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs

Key-Value (KV) cache plays a crucial role in accelerating inference in large language models (LLMs) by storing intermediate attention states and avoiding redundant computation during autoregressive generation. However, its memory footprint scales linearly with sequence length, often leading to severe memory bottlenecks on resource-constrained hardware. Prior work has explored offloading KV cache to the CPU while retaining only a subset on the GPU, but these approaches often rely on imprecise token selection and suffer performance degradation in long-generation tasks such as chain-of-thought reasoning. In this paper, we propose a novel KV cache management strategy, IceCache, which integrates semantic token clustering with PagedAttention. By organizing semantically related tokens into contiguous memory regions managed by a hierarchical, dynamically updatable data structure, our method enables more efficient token selection and better utilization of memory bandwidth during CPU-GPU transfers. Experimental results on LongBench show that, with a 256-token budget, IceCache maintains 99% of the original accuracy achieved by the full KV cache model. Moreover, compared to other offloading-based methods, IceCache attains competitive or even superior latency and accuracy while using only 25% of the KV cache token budget, demonstrating its effectiveness in long-sequence scenarios. The code is available on our project website at https://yuzhenmao.github.io/IceCache/.

Yuzhen Mao, Qitong Wang, Martin Ester +1

Simon Fraser University, BC, Canada · Harvard University, Cambridge, USA

KV-Cache Management LLM Inference Acceleration KV-Cache Offloading Long-Context Language Model Inference

Ask PDF

Abstract

KV-Cache Management

Explore similar work

Comparative Characterization of KV Cache Management Strategies for LLM Inference

TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference

IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs