The KV Cache Is the New Memory Wall

Tejinder Singh

cs.DCSep 25, 2026

The KV Cache Is the New Memory Wall

Authors: Tejinder Singh

Organizations: Dell Technologies, Santa Clara, CA 95054, USA

Abstract

Autoregressive LLM inference at long context is bounded by memory bandwidth, not arithmetic throughput, and the binding resource shifts from model weights to the Key-Value (KV) cache as sequence length grows. For Llama-3-70B in BF16, the 140 GB weight footprint exceeds the 80 GB HBM of a single accelerator, and one 128k-token sequence adds 42 GB of KV cache. Techniques that compress, evict, page, share, or offload KV state have proliferated, but reported gains use inconsistent workloads, hardware, and quality metrics, preventing cross-paper comparison. This SoK paper unifies the field analytically, with a protocol that strictly separates derived and reported claims. We derive closed-form arithmetic intensity as a decaying function of context length, parameterized by hardware topology for NVIDIA H100, NVIDIA B200, and AMD MI300X, including per-die bandwidth partitioning and the crossover lengths where KV traffic overtakes weight traffic. We classify the literature into five domains, quantization, token eviction, KV paging, prefix caching, and heterogeneous tiering, evaluating one method per domain at 128k context under a single protocol. The central finding is a three-regime structure: below a hardware-specific crossover, weight traffic dominates and KV compression yields negligible speedup; beyond it, KV traffic dominates and each domain trades quality for bandwidth savings approaching the roofline bound. Paging and prefix sharing are lossless but address capacity, not bandwidth. Quantization and eviction cut bandwidth directly, with degradation that accelerates below 4-bit precision and turns discontinuous for eviction on position-sensitive tasks. Tiering converts the bandwidth wall into an interconnect problem bounded by PCIe or NVLink rather than HBM. We close with design rules for selecting a compression domain given hardware, context length, and quality budget.

Figures & tables

Panel of Figure 1 — Figure 1: KV cache footprint per sequence versus context length for the Llama-3 family in BF16, computed from Eq. ( 1 ) [ 2 ] . Horizontal reference lines mark single-device HBM capacity. A Llama-3-70B sequence crosses one 80 GB H100 at 244k tokens of context and a Llama-3.1-405B sequence crosses it at 155k. At $10^{6}$ tokens the 70B and 405B caches reach 328 GB and 516 GB, exceeding the aggregate HBM of four and six H100 devices respectively.

Model	Attn.	$L$	$n_{q}$	$n_{kv}$	$d_{h}$	MB/token
Llama-2-13B [ 22 ]	MHA	40	40	40	128	0.82 MB
Mixtral 8x7B [ 23 ]	GQA	32	32	8	128	0.13 MB
Llama-3-8B [ 2 ]	GQA	32	32	8	128	0.13 MB
Llama-3-70B [ 2 ]	GQA	80	64	8	128	0.33 MB
Llama-3.1-405B [ 2 ]	GQA	126	128	8	128	0.52 MB
DeepSeek-V2 [ 8 ]	MLA	60	128	576 †	—	0.07 MB

Table 1: Per-token KV-cache footprint of representative architectures in BF16, computed from Eq. ( 1 ). † MLA stores a compressed latent vector of

(d_{c}+d_{r})=576

dimensions per layer instead of full key-value pairs.

Panel of Figure 2 — Figure 2: Five-domain taxonomy of KV-cache mitigation techniques. Each domain is annotated with the factor it scales in the unified footprint model of Eq. ( 4 ). Quantization and eviction trade quality for bytes. Paging and prefix caching are lossless and address capacity and duplication. Tiering relocates bytes across the memory hierarchy and shifts the bottleneck from HBM to interconnect bandwidth.

Domain	Mechanism	Scales	Lossless	Representative methods
Quantization	Store keys and values at $w_{q}<w$	$w_{q}$	No	KIVI [ 9 ] , KVQuant [ 10 ] , QServe [ 24 ]
Token eviction	Retain fraction $r$ of $s$ positions	$r$	No	StreamingLLM [ 11 ] , H2O [ 12 ] , SnapKV [ 13 ] , PyramidKV [ 26 ]
KV paging	Fixed-size block allocation	$\phi$	Yes	PagedAttention [ 14 ]
Prefix caching	Deduplicate shared prefixes	$\sigma$	Yes	RadixAttention [ 15 ] , LMCache [ 18 ]
Memory tiering	Migrate blocks across HBM, DRAM, NVMe	$\lambda$	Yes ∗	FlexGen [ 16 ] , CacheGen [ 17 ]

Table 2: Five-domain taxonomy of KV-cache mitigation. Each domain scales one factor of Eq. ( 4 ) or relocates bytes across tiers. ∗ Transport codecs such as CacheGen optionally quantize, making the link lossy by choice.

Panel of Figure 3 — Figure 3: Roofline with decode operating points for Llama-3-70B on H100, computed from Eq. ( 7 ) and Table 3 . Solid line. H100 roofline. Dashed and dotted lines. B200 aggregate and per-die rooflines, showing the bandwidth penalty when a sequence is pinned to one die. Marked ceilings are BF16 dense compute rates. Decode points sit between sixfold and more than two-hundredfold below the ridge across batch sizes and context lengths. Only prefill reaches the compute-bound regime. The prefill point uses a 4k-token prompt at 70 percent MFU, matching Figure 9 .

Device	Package	HBM	$\beta_{\mathrm{agg}}$	$\beta_{\mathrm{part}}$	BF16	FP8	$I^{*}$
H100 SXM	Monolithic	80 GB	3.35 TB/s	3.35 TB/s	989.5	1979	295
B200	2 dies, NV-HBI 10 TB/s	192 GB	8.0 TB/s	4.0 TB/s	2250	4500	281
MI300X	8 XCDs, Infinity Fabric	192 GB	5.3 TB/s	0.66 TB/s	1307	2615	247

Table 3: Hardware specification of the three benchmarked accelerators [ 5 , 20 , 21 ] . Compute figures are dense tensor core rates in TFLOP/s. Per-partition bandwidth is aggregate bandwidth divided by die or stack count. Ridge point computed against BF16 dense compute. MI300X on-package fabric bandwidth is not publicly disclosed. Its inter-GPU Infinity Fabric runs at 896 GB/s.

Panel of Figure 4 — Figure 4: Package topology of the three benchmarked accelerators [ 5 , 20 , 21 ] . H100 presents a monolithic die with uniform HBM access. B200 splits compute and HBM across two dies joined by NV-HBI at 10 TB/s, so a KV cache pinned to one die is served at 4 TB/s while striped pages reach the 8 TB/s aggregate. MI300X partitions compute across eight XCD chiplets with a 662.5 GB/s per-stack bandwidth slice; its on-package Infinity Fabric bandwidth is not publicly disclosed. The per-panel decode ceilings $R_{\infty}$ of Eq. ( 11 ) quantify the placement penalty: pinning costs $2\times$ on B200 and $8\times$ on MI300X, making KV page placement a first-order performance decision.

Model	$b{=}1$	$b{=}8$	$b{=}32$	$b{=}128$	$b{=}256$
Llama-3-8B	122.5k	15.3k	3.8k	957	479
Llama-3-70B	427.2k	53.4k	13.4k	3.3k	1.7k
Llama-3.1-405B	1.57M	196.2k	49.0k	12.3k	6.1k

Table 4: Traffic crossover context length

s^{*}(b)

in tokens from Eq. ( 12 ), BF16 weights and cache. Above the tabulated value, per-step KV traffic exceeds weight traffic.

Panel of Figure 5 — Figure 5: Traffic crossover context length $s^{*}(b)$ from Eq. ( 12 ), where per-step KV bytes equal weight bytes. Above each curve the decode step is KV bound. Production batch sizes of 32 to 256 push the Llama-3-70B crossover from 13.4k down to 1.7k tokens, so most long-context serving already operates in the KV-bound regime. The dashed line marks 128k context.

Context $s=128$ k, below crossover $s^{*}=427.2$ k.
Configuration	$w_{kv}$	KV (GB)	$T_{\mathrm{step}}$ (ms)	tok/s	Speedup	AI
BF16 baseline	16 bit	41.9	22.7	44.0	1.00 $\times$	2.6
INT8 cache	8 bit	21.0	20.1	49.7	1.13 $\times$	3.0
INT4 cache	4 bit	10.5	18.8	53.2	1.21 $\times$	3.2
2-bit cache	2 bit	5.2	18.2	55.1	1.25 $\times$	3.3
2-bit plus $r=0.5$ eviction	2 bit	2.6	17.8	56.1	1.28 $\times$	3.3

Table 5: Worked roofline bounds for Llama-3-70B on B200 at batch one with striped pages, derived from Eqs. ( 5 ), ( 6 ), and ( 7 ). Derived quantities, not measurements.

T_{\mathrm{step}}=B/\beta

with

\beta=8.0

TB/s; absolute times scale with realized bandwidth and ratios do not. Speedup relative to the BF16 baseline at the same context. The 128k block fits on one B200 (

181.9

\leq 192

GB). At 512k the BF16 and INT8 configurations exceed single-device capacity (307.8 and 223.9 GB against 192 GB) and are realized by sharding across two devices; the INT4 and 2-bit rows fit on one B200, and all speedup ratios are invariant to the shard count.

ID	Pattern	$b$	$s$	Binding constraint
W1	Single-sequence decode	1	128k	Latency, weight-dominated
W2	Batched decode	32	128k	Throughput, KV-dominated
W3	8 requests, 96k shared prefix plus 32k unique	8	128k	Capacity and TTFT
W4	Reuse after 10-minute gap	1	128k	Recompute versus fetch

Table 6: Benchmark workloads. W1 and W2 isolate the batch-size lever. W3 isolates prefix sharing. W4 isolates reuse across sessions.

Method	$c$	KV (GB)	W1	W2	Lossless	Quality delta (as reported)
BF16 paged baseline [ 14 ]	1 $\times$	41.9	1.00 $\times$	1.00 $\times$	—	reference
KIVI, 2-bit [ 9 ]	8 $\times$	5.2	1.25 $\times$	4.82 $\times$	No	$+0.03$ to $0.3$ ppl. LongBench $\approx$ baseline
KVQuant, 3-bit [ 10 ]	5.3 $\times$	7.9	1.23 $\times$	3.78 $\times$	No	$<0.1$ ppl. LongBench near baseline
H2O, $r{=}0.5$ [ 12 ]	2 $\times$	21.0	1.13 $\times$	1.83 $\times$	No	97 to 99% task retention reported
SnapKV, $r{=}0.25$ [ 13 ]	4 $\times$	10.5	1.21 $\times$	3.12 $\times$	No	96 to 98% LongBench retention reported
PagedAttention [ 14 ]	1 $\times$	41.9	1.00 $\times$	1.00 $\times$	Yes	None. Waste cut from 60 to 80% to $<4$ %

Table 7: SoK comparison matrix. Llama-3-70B, B200 with striped pages, 128k context. Byte factors and footprints are exact from Eq. ( 4 ). Decode speedups are derived from Eq. ( 14 ). Quality deltas are within-paper reported values on the benchmarks named and are not cross-comparable. † Total stored across the 8 requests of W3, versus 335.5 GB without sharing.

Panel of Figure 6 — Figure 6: KV footprint at 128k context per quantization level, Llama-3-70B, with quality deltas as reported by KIVI [ 9 ] and KVQuant [ 10 ] . Eight-bit storage is free. Four-bit is near-free with per-channel keys and outlier isolation. Two-bit holds only for structure-aware layouts. Naive round-to-nearest at 2-bit loses more than one perplexity point. Each halving of $w_{q}$ doubles the intensity floor of Eq. ( 8 ). Bar labels give the footprint, the reported quality delta, and the derived W2 decode speedup of Eq. ( 14 ) at byte factor $c=16/w_{q}$ , matching Table 7 .

Panel of Figure 7 — Figure 7: Reported retention-quality behavior of token eviction. Aggregate long-context benchmarks tolerate $r=0.25$ with 96 to 99 percent retention for heavy-hitter and observation-window policies [ 12 , 13 ] . Position-sensitive retrieval degrades discontinuously under window eviction [ 11 ] . Curves are representative within-paper values and are task dependent. Top-row labels give the KV footprint at 128k context for Llama-3-70B, $41.9\,r$ GB from Eq. ( 4 ).

Panel of Figure 8 — Figure 8: Allocation efficiency of KV memory. Reservation-based serving strands 60 to 80 percent of cache memory in over-reservation and fragmentation [ 14 ] . The category split of 25, 45, and 30 percent is illustrative within that reported range. PagedAttention bounds waste to the partially filled final block, measured below 4 percent [ 14 ] .

Panel of Figure 9 — Figure 9: Time to first token versus prefix cache hit rate for a 128k-token request on B200, derived from the exact prefill count of Eq. ( 18 ) at 70 percent MFU (Appendix A ). The weight-matmul term falls linearly in hit rate and the attention term as $1-h^{2}$ , so warm TTFT drops faster than linearly. The shaded band marks hit rates reported by RadixAttention on sharing-heavy workloads [ 15 ] . On workload W3 the shared 96k prefix is encoded once for 8 requests; a warm request encodes its 32k suffix against the resident prefix, a derived $8.8$ s versus $25.0$ s cold, or $2.8\times$ lower TTFT, and stored bytes fall $2.91\times$ , at zero quality cost.

Panel of Figure 10 — Figure 10: Time to make a 128k-token KV cache available to a decode worker. Recompute uses the exact prefill count of Eq. ( 18 ) at 70 percent MFU on B200 (Appendix A ); transfer values move the $41.9$ GB cache of Eq. ( 1 ) over PCIe Gen5 x16 at 64 GB/s or NVLink at 900 GB/s per direction. The codec rows apply the CacheGen transport compression factor of roughly $4\times$ [ 17 ] . Fetching beats recomputing by $38\times$ to $2{,}100\times$ . The advantage vanishes when decode itself must read remote memory, because PCIe sits $125\times$ below B200 HBM bandwidth [ 18 ] .

Panel of Figure 11 — Figure 11: Speedup versus retained quality at W2 (batched decode, 128k context, H100). Speedups are derived from Eq. ( 14 ). Quality values are representative within-paper reported numbers on differing task suites [ 9 , 10 , 12 , 13 ] , plotted on a normalized retention axis for orientation rather than as a protocol-aligned comparison. The dashed frontier runs through the non-dominated quantization points and the composed 2-bit plus eviction configuration at $6.6\times$ . Lossless domains (paging, prefix sharing, tiering) do not appear because they incur no quality cost.

Panel of Figure 12 — Figure 12: Domain profiles across six capability dimensions, scored 0 to 5 from the matrix of Table 7 and the analysis of Sections 3 and 4. Byte reduction and decode speedup reflect Eq. ( 14 ) at W2. Quality safety scores five for lossless domains. No domain dominates. Only the composed design space covers the hexagon.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Kernel	FLOPs	Bytes moved
QKV projection	$2dd_{h}(n_{q}+2n_{kv})$	$dd_{h}(n_{q}+2n_{kv})\,w_{p}$
Scores $qK^{\top}$	$2sn_{q}d_{h}$	$sn_{kv}d_{h}\,w_{kv}$
Softmax	$3n_{q}s$	$0$ (on chip)
Value contraction	$2sn_{q}d_{h}$	$sn_{kv}d_{h}\,w_{kv}$
Output projection	$2dd_{h}n_{q}$	$dd_{h}n_{q}\,w_{p}$
MLP, SwiGLU	$6dd_{ff}$	$3dd_{ff}\,w_{p}$

Appendix

Table 8: Per-layer FLOP and byte accounting for one decode step, one sequence, context length

s

. Softmax is counted exactly in Eq. ( 16 ) and omitted in the main text as lower order. KV bytes assume a single cache read; the write of the new token’s key and value is a

1/s

correction treated in Eq. ( 17 ).

Model	$d$	$d_{ff}$	$L\cdot P_{\mathrm{layer}}$	Embeddings	Reconstructed	$P$ used
Llama-3-8B	4096	14336	$6.98\times 10^{9}$	$1.05\times 10^{9}$	$8.03\times 10^{9}$	$8.03\times 10^{9}$
Llama-3-70B	8192	28672	$68.45\times 10^{9}$	$2.10\times 10^{9}$	$70.55\times 10^{9}$	$70.0\times 10^{9}$
Llama-3.1-405B	16384	53248	$401.5\times 10^{9}$	$4.20\times 10^{9}$	$405.7\times 10^{9}$	$405\times 10^{9}$

Appendix

Table 9: Parameter-count reconstruction.

L\cdot P_{\mathrm{layer}}

from Eq. ( 15 ); embeddings count the input and output projections at

|\mathcal{V}|\,d

. Derived quantities in the paper use the rounded

P

of the last column.

Symbol	Meaning	Running value
$L$	Layers	80
$d$	Hidden size	8192
$d_{h}$	Head dimension, $d/n_{q}$	128
$n_{q}$	Query heads per layer	64
$n_{kv}$	Key-value heads per layer	8
$g$	GQA group factor, $n_{q}/n_{kv}$	8

Appendix

Table 10: Model and attention symbols.

Symbol	Meaning	Running value
$s$	Context length, decimal tokens	128,000
$s_{0}$	Prefill length	128,000
$b$	Decode batch size	1–32
$c$	KV byte factor of a technique	1–16
$r$	Eviction retention fraction	0.25–1
$\sigma$	Prefix sharing factor	1–2.91

Appendix

Table 11: Workload, cost, and hardware symbols. References give the defining equation.

ID	$b$	Structure	Metrics	Binding equations
W1	1	No sharing	$T_{\mathrm{step}}$ , tok/s	Eqs. ( 6 ), ( 12 )
W2	32	No sharing	tok/s, speedup	Eqs. ( 7 ), ( 14 )
W3	8	96k shared prefix $+$ 32k unique	TTFT, stored bytes	Eqs. ( 4 ), ( 19 )
W4	1	Full reuse after 10-minute gap	fetch versus recompute	Eq. ( 19 ) and link rates of Section 2

Appendix

Table 12: Full workload parameterization. All workloads use Llama-3-70B (Table 1 ) on B200 (Table 3 ) at 128k context.

KV Caching

Key-value (KV) caching reuses Transformer states to speed generation while managing memory and data movement.

See all 281 papers

Sep 30, 2026

PatchKV: Weight-Space Compensation of KV Cache

Long-context inference with Large Language Models (LLMs) is bottlenecked by the linearly growing memory of the key-value (KV) cache. Existing compression methods reduce the cache through token eviction or approximation, but degrade sharply at aggressive compression budgets. We propose PatchKV, a training-free framework that compensates KV cache compression methods by carrying part of the context in the model's weights. PatchKV pairs an off-the-shelf compressed KV cache with a context-specific weight patch, which is computed once at context-loading time and served for downstream queries for the context. The weight patch is derived in closed form via ridge regression, by aligning the block-wise activations of context-derived reference query tokens under the full cache and the compressed cache. Once merged into the model, the patch leaves the forward graph and per-query inference cost unchanged in the single-context, multi-query setting. Across long-context QA (SCBench with up to 170K tokens, SQuAD, NIAH) and math (GSM8K) benchmarks on three model architectures, PatchKV consistently improves cache compression methods, suggesting an alternative direction to compensate them at aggressive budgets.

Chanryeol Lee, Chanhyuk Lee, Yeonwoo Choi +2

KAIST

Long-Context Language Model Inference LLM Inference Acceleration KV-Cache Compression

Ask PDF

Feb 19, 2026

ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs

Large Language Models (LLMs) are increasingly deployed in scenarios demanding ultra-long context reasoning, such as agentic workflows and deep research understanding. However, long-context inference is constrained by the KV cache, a transient memory structure that grows linearly with sequence length and batch size, quickly dominating GPU memory usage. Existing memory reduction techniques, including eviction and quantization, often rely on static heuristics and suffer from degraded quality under tight budgets. In this paper, we propose ARKV, a lightweight and adaptive framework that dynamically allocates precision levels to cached tokens based on per-layer attention dynamics and token-level importance. During a short prefill phase, ARKV estimates the original quantization (OQ) ratio of each layer by computing statistical scores such as attention entropy, variance and kurtosis. During decoding, tokens are assigned to one of three states, Original (full precision), Quantization (low precision), or Eviction, according to a fast heavy-hitter scoring strategy. Our experiments on LLaMA3 and Qwen3 models across diverse long- and short-context tasks demonstrate that ARKV preserves ~97% of baseline accuracy on long-context benchmarks while reducing KV memory usage by 4x, with minimal throughput loss. On short-context tasks, ARKV matches full-precision baselines; on GSM8K math reasoning, it significantly outperforms uniform quantization. These results highlight the practical viability of ARKV for scalable LLM deployment, offering fine-grained, data-driven memory control without retraining or architectural modifications. The source code and artifacts can be found in: https://github.com/Large-scale-Sustainable-Computing-LSC/ARKV

Jianlong Lei, Shashikant Ilager

Multiscale Networked Systems (MNS) Group, Informatics Institute University of Amsterdam, The Netherlands

KV-Cache Management Long-Context Language Model Inference KV-Cache Quantization KV-Cache Eviction

Ask PDF

May 8, 2026

RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache

Large language models (LLMs) have shown strong performance across diverse tasks, but their inference with long input contexts is bottlenecked by memory size and bandwidth. The Key-Value (KV) cache size grows linearly with sequence length and needs to be re-read from off-chip high-bandwidth memory (HBM) to on-chip memory at every decoding step, resulting in memory-bound inference. Existing methods reduce the cache by either eviction or quantization, but typically treat the two in isolation. In this paper, we cast KV cache compression as a rate-distortion problem, under which eviction and quantization are two end-points of the same bit allocation scheme. This exposes the need to optimize them jointly, motivating our method, RDKV (Rate-Distortion KV cache compression). RDKV derives the weight of each token or channel from the distortion that compression induces on the attention computation. Based on these weights, it assigns each token or channel a bit-width ranging from full precision down to zero bits guided by reverse water-filling, applied once after the prefilling stage. Experiments on LongBench, RULER, and InfiniteBench show that RDKV outperforms the best evaluated baseline by 9.1% on average. On LongBench it recovers 97.81% of full-cache accuracy with only 2.48% cache retention. Compared with full-cache FlashAttention-2 decoding, it achieves 4.5x decode speedup and 1.9x peak memory reduction with 128K context length, while maintaining comparable performance.

Junkai Zhang, Hang Guo, Luca Benini +1

ETH Zurich · Tsinghua University

KV Caching Long-Context Language Model Inference KV-Cache Eviction Rate-Distortion Optimization KV-Cache Compression KV-Cache Quantization

Ask PDF

Abstract

KV Caching

Explore similar work

PatchKV: Weight-Space Compensation of KV Cache

ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs

RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache