Tensor Decomposition of Transformer Key-Value Caches: Spectral Structure and Format Comparison

Rahul Krishnan; Volker Schulz

math.NASep 23, 2026

Tensor Decomposition of Transformer Key-Value Caches: Spectral Structure and Format Comparison

Authors: Rahul Krishnan, Volker Schulz

Organizations: Universität Trier · Fachbereich IV, Mathematik, Universität Trier

Abstract

The key-value (KV) cache of autoregressive transformers can be viewed as a fourth-order tensor spanning attention heads, tokens, features, and grouped layers. We measure the singular-value spectra of all four mode unfoldings on Mistral-7B-v0.3 and LLaMA-2-13B and compare four standard tensor decompositions: Tucker, CP, tensor train, and t-SVD, at matched storage. The spectra partition the four axes into two classes. The token and feature modes carry low-rank structure, particularly for keys. The head and layer modes are nearly full-rank and resist compression at any practical error level. Among the four decompositions, Tucker achieves the lowest reconstruction error at every compression ratio from $2\times$ to $5\times$ , because it can leave the full-rank modes untouched. Comparisons with two-dimensional unfolding baselines show that the preferred representation differs between keys and values: 2D methods achieve lower key error, while four-way Tucker achieves lower value error at matched storage. A mode-pinning theorem certifies the full-rank preservation from the measured spectra alone. Two further spectral properties affect the compressible modes without touching the full-rank ones: values reach a higher error floor than keys at every ratio, and post-RoPE keys lose $41\%$ - $64\%$ of their pre-RoPE compressibility on both models.

Figures & tables

Tensor	Mode	$\sigma_{1}/\sigma_{\min}$	$r$ for $\leq 10\%$	$r$ for $\leq 20\%$	$L_{k}(n_{k}{-}1)$
K	heads ( $8$ )	$1.4$	$8/8$	$8/8$	$0.275$ – $0.298$
K	tokens ( $1024$ )	$0.4$ – $2.9$ M	$225/1024$	$89/1024$	$3{\times}10^{-7}$ – $2{\times}10^{-6}$
K	features ( $128$ )	$28.2$	$100/128$	$69/128$	$0.012$ – $0.013$
V	heads ( $8$ )	$1.4$	$8/8$	$8/8$	$0.278$ – $0.296$
V	tokens ( $1024$ )	$13$ – $550$ K	$576/1024$	$382/1024$	$5{\times}10^{-7}$ – $2{\times}10^{-5}$
V	features ( $128$ )	$2.7$	$126/128$	$118/128$	$0.056$ – $0.059$

Table 1: Per-mode spectral properties of the Mistral-7B layer-

15

cache (

T=1024

), from the mode-

k

SVD of each unfolding of the pre-RoPE key tensor

\mathcal{K}

and the value tensor

\mathcal{V}

. Ranges are min-to-max over

5

independent draws.

L_{k}(n_{k}{-}1)

is the tail energy from Definition 3.1 : values above

\varepsilon

indicate an index-like mode at level

\varepsilon

Ratio	K	V	K	V	K	V	K	V
	Tucker		CP		t-SVD		TT
$2\times$	0.087	0.279	$0.098$	$0.301$	$0.147$	$0.427$	$0.224$	$0.552$
$3\times$	0.125	0.381	$0.155$	$0.436$	$0.204$	$0.543$	$0.303$	$0.674$
$4\times$	0.153	0.451	$0.194$	$0.518$	$0.243$	$0.615$	$0.349$	$0.734$
$5\times$	0.176	0.503	$0.224$	$0.574$	$0.268$	$0.655$	$0.387$	$0.779$

Table 2: Format comparison on Mistral-7B-v0.3 (GQA,

n_{h}=8

), per-tensor storage budget : mean relative Frobenius error over all

32

layers and

3

prompts at

T=1024

. Lowest K and V errors per ratio in bold.

Ratio	K	V	K	V	K	V
	Tucker		t-SVD		TT
$2\times$	0.089	0.232	$0.167$	$0.426$	$0.278$	$0.608$
$3\times$	0.127	0.328	$0.222$	$0.536$	$0.373$	$0.729$
$4\times$	0.153	0.392	$0.258$	$0.604$	$0.434$	$0.785$
$5\times$	0.173	0.437	$0.279$	$0.641$	$0.495$	$0.824$

Table 3: Format comparison on LLaMA-2-13B (MHA,

n_{h}=40

), per-tensor storage budget : mean relative Frobenius error over all

40

layers and

3

prompt draws at

T=1024

. CP omitted (see text). Lowest K and V errors per ratio in bold.

Ratio	K Tucker	K SVD	V Tucker	V SVD
$2\times$	$0.140$	0.139	0.155	$0.421$
$3\times$	0.184	$0.199$	0.251	$0.534$
$4\times$	0.207	$0.242$	0.325	$0.604$

Table 4: Tucker versus per-head SVD on Mistral-7B-v0.3 (GQA,

n_{h}=8

), all

32

layers at

T=1024

, matched storage,

3

-seed mean.

		Palu		xKV		Tucker-3D		Tucker-4D
Model	Ratio	K	V	K	V	K	V	K	V
Mistral-7B	$2\times$	0.096	0.312	0.099	0.202	0.147	0.179	0.134	0.143
	$3\times$	0.138	0.417	0.133	0.301	0.190	0.282	0.179	0.236
	$4\times$	0.168	0.486	0.158	0.369	0.217	0.357	0.201	0.308
LLaMA-2-13B	$2\times$	0.119	0.325	0.113	0.186	0.142	0.138	0.145	0.125
	$3\times$	0.162	0.423	0.151	0.291	0.192	0.227	0.196	0.218

Table 5: Cross-layer K/V split, joint K/V storage budget : Tucker (3D and 4D, fixed groups of

L_{g}=4

layers) versus Palu and xKV,

T=1024

3

-seed mean.

		$2\times$		$3\times$		$4\times$
Model	Arm	K	V	K	V	K	V
Mistral ( $n_{h}=8$ )	free ( $r_{h}=8$ , $r_{L}=4$ )	$0.134$	$0.143$	$0.179$	$0.236$	$0.201$	$0.308$
	$r_{L}=3$	$0.393$	$0.318$	$0.417$	$0.380$	$0.432$	$0.430$
	$r_{L}=2$	$0.611$	$0.524$	$0.620$	$0.553$	$0.628$	$0.581$
	$r_{h}=n_{h}/2=4$	$0.667$	$0.677$	$0.670$	$0.681$	$0.674$	$0.687$
	$r_{h}=n_{h}/4=2$	$0.839$	$0.847$	$0.839$	$0.847$	$0.840$	$0.847$

Table 6: Head- and layer-rank ablation for Tucker-4D,

L_{g}=4

fixed groups,

T=1024

, pre-RoPE, residual-free,

3

-seed mean.

Forced mode	$r_{k}$	Keys			Values
		$L_{k}(r_{k})$	$E_{\mathrm{group}}$	Ratio	$L_{k}(r_{k})$	$E_{\mathrm{group}}$	Ratio
layer	$3$	$0.460$	$0.469$	$1.02$	$0.410$	$0.420$	$1.03$
layer	$2$	$0.676$	$0.677$	$1.00$	$0.621$	$0.621$	$1.00$
head	$4$	$0.666$	$0.668$	$1.00$	$0.672$	$0.672$	$1.00$
head	$2$	$0.839$	$0.839$	$1.00$	$0.843$	$0.843$	$1.00$

Table 7: Mode-

k

tail-energy lower bound

L_{k}(r_{k})

versus measured Tucker-4D group-level error

E_{\mathrm{group}}

, Mistral,

T=1024

, target

2\times

3

-seed mean over

8

groups.

Ratio	Mistral pre	Mistral post	gap%	LLaMA pre	LLaMA post	gap%
$2\times$	$0.1307$	$0.2117$	$+62\%$	$0.1413$	$0.2311$	$+64\%$
$3\times$	$0.1763$	$0.2687$	$+52\%$	$0.1922$	$0.2906$	$+51\%$
$4\times$	$0.1982$	$0.2957$	$+49\%$	$0.2171$	$0.3182$	$+47\%$
$6\times$	$0.2257$	$0.3335$	$+48\%$	$0.2403$	$0.3440$	$+43\%$
$8\times$	$0.2433$	$0.3626$	$+49\%$	$0.2545$	$0.3598$	$+41\%$

Table 8: Cross-architecture RoPE penalty on keys, Mistral-7B-v0.3 and LLaMA-2-13B,

T=1024

; gap is the relative increase in pooled key error from pre to post.

Explore similar work

A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
Jul 14, 2026Rahul Krishnan, Volker SchulzKey-Value Cache Compression Key-Value Cache
Universität Trier · Fachbereich IV, Mathematik, Universität Trier
The risk of KV cache compression
Jul 1, 2026Lukas Haverbeck, Carmen Amo Alonso, Andres Felipe Posada-Moreno +2Key-Value Cache Compression Cache
RWTH Aachen University · Stanford University
Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility
May 13, 2026Gergely Szilvasy, Manuel Faysse, Maria Lomeli +5Key-Value Cache Compression Key-Value Cache
Meta FAIR · MICS, CentraleSupélec

Jul 14, 2026, cs.LGJ/K move · Enter open · S save

A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation

Rahul Krishnan, Volker Schulz

Universität Trier · Fachbereich IV, Mathematik, Universität Trier

Key-Value Cache Compression Key-Value Cache Large Language Model Compression Cache Data Compression Methods Transformer Architectures+1

Open paper AskPDF arXiv

The key-value (KV) cache is the dominant memory bottleneck in long-context language model inference. Existing compression methods apply low-rank factorization or quantization independently, without jointly allocating rank and precision under a shared storage budget. We introduce JoLT, a training-free compressor that treats grouped prefill caches as fourth-order tensors and applies partial Tucker decomposition along the token and feature modes, the two axes that carry low-rank structure, while leaving the head and layer modes intact. A rotated low-bit quantizer captures the truncation residual, and a single Lagrangian dual allocates per-group Tucker ranks and residual bit-widths under a global byte constraint. FlashJoLT replaces the exact token-mode SVD with a randomized approximation that matches JoLT within the free zone at a fraction of the compression cost, and a fused Triton decode kernel evaluates attention directly over the stored factors without materializing dense KV tensors. Across five models from four architecture families, covering multi-head attention, grouped-query attention, and mixture-of-experts architecture, JoLT achieves 2 - 3x compression with less than 0.2% perplexity degradation, without retraining. On RULER at 64K context with LLaMA-3.1-8B, retrieval accuracy remains near-lossless through 3x and declines by only 0.90 and 2.40pp at 4x and 5x, respectively. JoLT demonstrates that tensor-aware low-rank decomposition and quantized residuals, unified under a single storage budget, achieve near-lossless KV-cache compression across diverse model architectures without retraining.

Abstract

Explore similar work

A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation

The risk of KV cache compression

Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility