The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inference

Kaleem Ullah Qasim; Jiashu Zhang; Muhammad Kafeel Shaheen; Razan Alharith; Heying Zhang

cs.LGMar 20, 2026

The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inference

Authors: Kaleem Ullah Qasim, Jiashu Zhang, Muhammad Kafeel Shaheen, Razan Alharith, Heying Zhang

Organizations: School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu, 611756, China

Abstract

The key-value (KV) cache is widely treated as essential state in transformer inference, and a large body of work engineers policies to compress, evict, or approximate its entries. We prove that this state is entirely redundant: keys and values at every layer are deterministic projections of the residual stream, and recomputing them from a single residual vector per token incurs exactly zero reconstruction error, not approximately, but bit-identically. We verify this across six models from four architecture families (135M to 4B parameters). Cross-task residual patching at every layer produces D_KL = 0 between patched and original output distributions, confirming that the residual stream satisfies a Markov property and is the sole information-carrying state. Removing the cache entirely and recomputing from scratch yields token-identical output under greedy decoding on all models tested. We build on this result with KV-Direct, a bounded-memory inference scheme that checkpoints residual vectors (5 KB per token on Gemma 3-4B) instead of full KV pairs (136 KB), recomputing keys and values on demand. Over 20 conversation turns, KV-Direct holds peak memory at 42 MB while the standard cache grows past 103 MB. Against five eviction baselines (H2O, StreamingLLM, SnapKV, TOVA, window-only), KV-Direct maintains 100% token match at every cache budget; all baselines degrade to 5-28%. A per-operation latency analysis shows recomputation runs up to 5x faster than reading cached tensors at moderate batch sizes. Code is available at https://github.com/Kaleemullahqasim/KV-Direct.

Figures & tables

Figure 1: Three inference regimes compared. (Left) Standard KV cache: stores all K/V pairs, memory grows as

O(T)

with sequence length. (Centre) Sliding window eviction: bounds memory to the last

B

tokens but permanently discards evicted KV entries, yielding 5–28% token match and high KL divergence. (Right) KV-Direct \xspace : evicted KV entries are replaced by residual stream checkpoints (5 KB/token for Gemma3-4B), from which exact K and V are recomputed on the fly, achieving bounded memory with 100% token match and

D_{\mathrm{KL}}\approx 0

								Per-token memory
Model	Family	$L$	$n_{\text{kv}}$	$d_{\text{head}}$	$d$	Quant	Attn	KV cache $\to$	KV-Direct \xspace	$\rho$	Saving
SmolLM2-135M [ 45 ]	LLaMA	30	3	64	576	Full	G	22.5 KB $\to$	1.1 KB	$20.0\times$	$\blacktriangledown$ 95%
Qwen2.5-0.5B [ 46 ]	Qwen2	24	2	64	896	4-bit	G	12.0 KB $\to$	1.8 KB	$6.9\times$	$\blacktriangledown$ 85%
Qwen3-0.6B	Qwen3	28	8	128	1024	Full	G	112.0 KB $\to$	2.0 KB	$56.0\times$	$\blacktriangledown$ 98%
DS-R1-Distill-1.5B [ 47 ]	DeepSeek	28	2	128	1536	Full	G	28.0 KB $\to$	3.0 KB	$9.3\times$	$\blacktriangledown$ 89%
Qwen2.5-1.5B	Qwen2	28	2	128	1536	4-bit	G	28.0 KB $\to$	3.0 KB	$9.3\times$	$\blacktriangledown$ 89%

Table 1: Model architectures and per-token memory footprint under standard KV caching vs. KV-Direct \xspace (bfloat16,

b{=}2

bytes). KV-Direct \xspace stores one residual vector (

d_{\text{hidden}}\cdot b

bytes) instead of

2L

KV vectors, yielding

\rho=2Ln_{\text{kv}}d_{\text{head}}/d_{\text{hidden}}

compression (Eq. 25 ). Attn: G = global, S = sliding window. All models use pre-norm (RMSNorm) and RoPE.

\blacktriangledown

denotes memory reduction by KV-Direct \xspace .

Model	$L$	Attn	$\max\|\Delta K\|$	$\max\|\Delta V\|$
SmolLM2-135M	30	Full	0.00	0.00
Qwen2.5-0.5B	24	Full	0.00	0.00
Qwen3-0.6B	28	Full	0.00	0.00
DS-R1-1.5B	28	Full	0.00	0.00
Qwen2.5-1.5B (4-bit)	28	Full	0.00	0.00
Gemma 3-4B (Global)	5	Global	0.00	0.00

Table 2: KV reconstruction error across six models and four architecture families. Max absolute difference between cached and recomputed K/V over all layers. Every full-attention architecture achieves exact zero . For Gemma’s sliding-window layers, V remains zero; K is non-zero due to window-relative RoPE.

Model	Match	Cache	Recomp.	Speed
		(s)	(s)
SmolLM2-135M	30/30	0.11	0.19	$1.7\times$
Qwen2.5-0.5B	30/30	0.18	0.34	$1.9\times$
Qwen3-0.6B	30/30	0.20	0.40	$2.0\times$
DS-R1-1.5B	30/30	0.41	0.70	$1.7\times$
Qwen2.5-1.5B (4-bit)	30/30	0.20	0.70	$3.5\times$

Table 3: Generation comparison: standard KV-cached decoding vs. full recomputation from scratch. All models use greedy (argmax) decoding and produce identical output under both methods.

Model	Method	HellaSwag	PPL
SmolLM2-135M	Full cache	39.4	26.46
	KV-Direct \xspace	39.4	26.46
	Window-128	—	38.2
	Window-64	39.4	65.3
	Window-32	—	135.4
Qwen2.5-0.5B	Full cache	41.0	24.63

Table 4: Downstream task evaluation. HellaSwag 0-shot accuracy (%,

N{=}500

) and WikiText-2 perplexity. KV-Direct \xspace is independently measured via layer-by-layer recompute (not copied from full cache). On all five standard-attention models, KV-Direct \xspace matches full-cache outputs exactly.

Figure 2: Proportional square visualisation of per-token memory anatomy. The outer grey square represents the full KV cache footprint; the inner blue square represents the residual stream checkpoint, sized proportionally by area. The visual disparity between the two directly encodes the memory inflation ratio (shown in red above each model).

Figure 3: Multi-turn inference evaluation. (a) Memory growth over 20 conversation turns: standard KV cache grows to 103 MB while KV-Direct \xspace stabilises at 42 MB. (b) Latency per turn: both methods track nearly identically, confirming zero inference penalty from residual checkpointing. (c) Per-token memory across all six models: the KV cache costs $7$ – $27\times$ more than a single residual checkpoint.

Figure 4: Performance matrix across seven methods, five cache budgets, and two models. Top row: Token match percentage (higher is better; darker blue $=$ higher match). Bottom row: KL divergence from the full-cache output distribution (lower is better; blue $=$ near-zero divergence, red $=$ high divergence). KV-Direct \xspace and full KV cache achieve 100% token match and $\approx$ 0 KL divergence at every budget, while all five eviction baselines degrade severely (5–28% match, KL 7–14). The blue-bordered row highlights KV-Direct \xspace .

Figure 5: Effective rank of $\mathbf{M}^{(h)}=\mathbf{W}_{q}^{(h)}{\mathbf{W}_{k}^{(h)}}^{\top}$ at 90% spectral energy across three models. Each dot is one KV head at one layer. Colour : rank as a fraction of $d_{\text{head}}$ (blue $=$ compressible, red $=$ near full rank). Size : same fraction (larger $=$ higher rank). Dashed outlines on Gemma mark global-attention layers. Layer 0 consistently shows near-rank-1 heads across all models, consistent with the BOS-focus phenomenon [ 8 ] . Rank heterogeneity is visible both within and across architectures.

Figure 6: Token match (%) and KL divergence vs. KV projection rank $r$ on two models. At full rank ( $r{=}64$ ), both models achieve 100% match. Truncating to $r{=}32$ (50% of $d_{\text{head}}$ , capturing $>$ 95% spectral energy) causes catastrophic degradation: 5–15% match and $D_{\mathrm{KL}}>10$ . The shaded region marks ranks where lossy compression fails.

Figure 7: Recompute-to-cache-read latency ratio across five model architectures. Each curve traces one model as the eviction batch size $N$ increases from 1 to 500 tokens. The teal-shaded region marks where recomputation is faster than cache reading. All models cross below parity by $N{=}50$ .

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Cross-task residual patching across all layers for four models. Each block represents one layer where the recipient’s residual stream is replaced with the donor’s. All tested layers produce $D_{\mathrm{KL}}=0.0$ across all four architectures, confirming that the residual stream is a sufficient Markov state at every depth.

Figure 9: KV budget sweep across six window sizes ( $B\in\{8,16,32,64,128,256\}$ ) with 250-token generation. (a) Window-only token match degrades sharply as the cache budget shrinks, while KV-Direct \xspace maintains 100% match at all budgets. (b) Averaged across budgets, window-only caching achieves 16–23% match versus KV-Direct \xspace ’s perfect recovery.

Transformer Inference

Transformer inference studies how trained Transformer models produce outputs efficiently, accurately, and securely across hardware and context lengths.

See all 57 papers

A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
Jul 14, 2026Rahul Krishnan, Volker SchulzTucker Decomposition KV Caching
Universität Trier · Fachbereich IV, Mathematik, Universität Trier
VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
May 17, 2026Jiayi Yao, Samuel Shen, Kuntai Du +7KV-Cache Offloading KV Caching
University of Chicago · Tensormesh Inc. · Samsung Semiconductor +1
How Much Cache Does Reasoning Need? Depth-Cache Tradeoffs in KV-Compressed Transformers
Apr 20, 2026Xiao WangTransformer Inference KV Caching

Jul 14, 2026J/K move · Enter open · S save

A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation

Rahul Krishnan, Volker Schulz

Universität Trier · Fachbereich IV, Mathematik, Universität Trier

Tucker Decomposition KV Caching Low-Rank Matrix Decomposition Tensor Decomposition Low-Rank Compression Low-Rank Approximation+2

Open paper AskPDF arXiv

The key-value (KV) cache is the dominant memory bottleneck in long-context language model inference. Existing compression methods apply low-rank factorization or quantization independently, without jointly allocating rank and precision under a shared storage budget. We introduce JoLT, a training-free compressor that treats grouped prefill caches as fourth-order tensors and applies partial Tucker decomposition along the token and feature modes, the two axes that carry low-rank structure, while leaving the head and layer modes intact. A rotated low-bit quantizer captures the truncation residual, and a single Lagrangian dual allocates per-group Tucker ranks and residual bit-widths under a global byte constraint. FlashJoLT replaces the exact token-mode SVD with a randomized approximation that matches JoLT within the free zone at a fraction of the compression cost, and a fused Triton decode kernel evaluates attention directly over the stored factors without materializing dense KV tensors. Across five models from four architecture families, covering multi-head attention, grouped-query attention, and mixture-of-experts architecture, JoLT achieves 2 - 3x compression with less than 0.2% perplexity degradation, without retraining. On RULER at 64K context with LLaMA-3.1-8B, retrieval accuracy remains near-lossless through 3x and declines by only 0.90 and 2.40pp at 4x and 5x, respectively. JoLT demonstrates that tensor-aware low-rank decomposition and quantized residuals, unified under a single storage budget, achieve near-lossless KV-cache compression across diverse model architectures without retraining.

Abstract

Transformer Inference

Explore similar work

A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation

VeriCache: Turning Lossy KV Cache into Lossless LLM Inference

How Much Cache Does Reasoning Need? Depth-Cache Tradeoffs in KV-Compressed Transformers