Kronecker Product Attention: Memory-Efficient KV Compression via Structured Factorization

Bahareh Nikpour; Hossein Rajabzadeh; Maryam Dialameh; Vahid Partovi Nia; Parsa Omidi

cs.LGOct 8, 2026

Kronecker Product Attention: Memory-Efficient KV Compression via Structured Factorization

Authors: Bahareh Nikpour, Hossein Rajabzadeh, Maryam Dialameh, Vahid Partovi Nia, Parsa Omidi

Organizations: Huawei Technologies, Toronto, Canada · University of Waterloo, Waterloo, Canada · Huawei Noah’s Ark Lab

Abstract

Efficient autoregressive generation is increasingly limited by the memory footprint of the key-value (KV) cache, which grows linearly with sequence length, batch size, and model depth. Existing approaches such as Multi-Query Attention (MQA), Grouped-Query Attention (GQA), Multi-Head Latent Attention (MLA), and Tensor Product Attention (TPA) reduce this cost through different trade-offs, but often sacrifice expressivity, require additional positional-encoding machinery, or impose restrictive low-rank structure. We propose Kronecker Product Attention (KPA), which represents key and value activations as sums of Kronecker products of compact, token-dependent matrix factors. KPA strictly generalizes TPA: rank-one vector-outer-product factorization arises as a degenerate Kronecker split, while non-degenerate matrix factors can represent richer block-structured patterns. We further show that a structured rotary embedding operator matched to the Kronecker representation preserves the relative-position property of RoPE while rotating only the compact factors, removing the need for a decoupled RoPE branch as in MLA. Empirically, KPA matches or exceeds TPA, MHA, MQA, GQA, and MLA across four model scales and nine downstream benchmarks under identical training conditions, while achieving up to 10x KV cache compression. The Kronecker head split is a tuneable hyperparameter that allows practitioners to navigate the compression-accuracy trade-off at design time without architectural changes.

Figures & tables

Figure 1: Overview of Multi-Head Attention (MHA), Tensor Product Attention (TPA), and Kronecker Product Attention (KPA, ours). MHA projects

x

directly into dense

Q,K,V

. TPA and KPA instead factor each projection into two components, combined via an outer product (

\times

) for TPA or a Kronecker product (

\otimes

) for KPA. RoPE is applied only to

Q

and

K

(never

V

), and rotates only the

B

factor in both methods (Theorem 2.4 formalizes this for KPA);

A

is cached unrotated. Since TPA’s

A,B

are vectors, their outer product is a flat, unstructured matrix; since KPA’s

A,B

are matrices, their Kronecker product is block-structured with a two-level head/feature hierarchy (shown by the internal grid lines). Only

K,V

factors enter the KV cache;

Q

is not cached. The token index

t

is omitted throughout except on the final

Q_{t},K_{t},V_{t}

, for clarity.

Method	ARC-E	ARC-C	BoolQ	HellaSw.	OBQA	PIQA	W.G.	MMLU	SciQ	Avg.
MHA	59.51	29.52	59.60	45.68	34.20	68.82	53.43	23.33	76.90	50.11
MQA	57.62	31.91	59.45	45.69	35.40	69.31	53.51	26.47	74.60	50.44
GQA	58.67	31.48	58.29	45.45	35.20	68.50	54.46	24.58	76.50	50.35
MLA	56.65	29.52	57.83	46.05	34.60	69.42	52.80	24.62	79.70	50.13
TPA	58.38	31.57	59.39	46.83	37.00	70.02	54.06	25.52	79.90	51.41
KPA $h{=}(4{\times}4)$	64.23	30.72	59.66	44.09	34.40	70.08	54.38	26.79	85.30	52.18

Table 1: Zero-shot evaluation on nine benchmarks (FineWeb-Edu,

\sim

50B tokens), 353M-parameter models. Best score per column in bold . HellaSw. = HellaSwag, W.G. = WinoGrande.

Method	ARC-E	ARC-C	BoolQ	HellaSw.	OBQA	PIQA	W.G.	MMLU	SciQ	Avg.
MHA	59.93	33.62	61.93	50.63	36.00	71.06	55.41	22.87	81.20	52.52
MQA	60.73	33.62	57.34	50.09	37.00	69.97	55.49	25.30	79.60	52.13
GQA	61.66	34.30	58.72	49.85	38.40	71.16	53.75	25.23	77.60	52.30
MLA	63.55	32.85	60.95	51.72	38.80	70.51	55.01	24.55	81.90	53.32
TPA	63.22	35.58	60.03	51.26	36.80	71.44	55.56	24.77	79.60	53.14
KPA $h{=}(4{\times}5)$	68.56	35.92	60.09	50.28	36.80	72.85	55.33	25.26	85.90	54.55

Table 2: Zero-shot evaluation on nine benchmarks (FineWeb-Edu,

\sim

50B tokens), 773M-parameter models. Best score per column in bold . HellaSw. = HellaSwag, W.G. = WinoGrande.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Scale	Config	$C_{\mathrm{KPA}}$
353M	$h{=}(4{\times}4)$	$8.0\times$
353M	$h{=}(2{\times}8)$	$6.4\times$
353M	$h{=}(8{\times}2)$	$6.4\times$
773M	$h{=}(4{\times}5)$	$8.9\times$
773M	$h{=}(2{\times}10)$	$6.7\times$
773M	$h{=}(10{\times}2)$	$6.7\times$

Appendix

Table 3: KV cache compression factors for evaluated KPA configurations.

Method	ARC-E	ARC-C	BoolQ	HellaSw.	OBQA	PIQA	W.G.	MMLU	SciQ	Avg.
MHA	64.81	35.41	61.90	54.32	37.20	72.74	55.80	25.44	82.80	54.49
MQA	64.10	36.01	62.26	54.38	39.00	72.58	56.43	23.70	81.90	54.48
GQA	63.68	35.92	60.46	54.17	38.40	73.56	56.27	24.77	81.70	54.33
MLA	64.14	35.92	60.12	53.60	39.20	72.25	55.17	24.71	81.60	54.08
TPA	66.71	36.52	61.38	54.03	40.40	72.52	56.83	24.49	82.20	55.01
KPA $h{=}(5{\times}5)$	70.62	38.74	63.26	54.60	41.00	73.72	57.20	24.38	85.50	56.56

Appendix

Table 4: Zero-shot evaluation of 1.5B-parameter models (FineWeb-Edu,

\sim

50B tokens). Baselines from Zhang et al. [2025] . Best score in each column is bold .

Method	ARC-E	ARC-C	BoolQ	HellaSw.	OBQA	PIQA	W.G.	MMLU	SciQ	Avg.
MHA	50.63	26.96	59.39	36.18	32.00	64.96	51.85	23.40	70.30	46.19
MQA	49.62	25.34	55.72	35.94	31.40	64.85	51.30	23.37	68.70	45.14
GQA	48.70	25.68	56.15	35.58	31.40	64.91	51.62	23.12	68.20	45.04
MLA	50.21	26.71	58.01	36.25	32.80	64.69	50.59	24.67	71.90	46.20
TPA	51.26	27.39	57.00	36.68	32.80	64.47	49.72	24.61	72.00	46.21
KPA $h{=}(3{\times}4)$	52.10	27.30	57.58	35.90	32.20	65.10	50.80	24.50	73.80	46.59

Appendix

Table 5: Zero-shot evaluation of 125M-parameter models (FineWeb-Edu,

\sim

50B tokens). Baselines from Zhang et al. [2025] . Best score in each column is bold .

Explore similar work

Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility
May 13, 2026Gergely Szilvasy, Manuel Faysse, Maria Lomeli +5Self-Attention KV Caching
Meta FAIR · MICS, CentraleSupélec
Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection
Feb 16, 2026Hengshuai Yao, Xing Chen, Ahmed Murtadha +1KV-Cache Compression LLM Compression
Sapient Intelligence · Department of Computing Science, University of Alberta
AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD
Oct 3, 2026Sara Abdali, Jongwoo Ko, Pashmina CameronKV-Cache Compression Low-Rank Compression
Microsoft Applied Sciences Group (ASG)

May 13, 2026J/K move · Enter open · S save

Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility

Gergely Szilvasy, Manuel Faysse, Maria Lomeli, Matthijs Douze+4

Meta FAIR · MICS, CentraleSupélec

Self-Attention KV Caching Long-Context Language Model Inference Memory-Efficient Optimization KV-Cache Management Transformer Attention+3

Open paper AskPDF arXiv

Under modern test-time compute and agentic paradigms, language models process ever-longer sequences. Efficient text generation with transformer architectures is increasingly constrained by the Key-Value cache memory footprint and bandwidth. To address this limitation, we introduce Self-Pruned Key-Value Attention (SP-KV), a mechanism designed to predict future KV utility in order to reduce the size of the long-term KV cache. This strategy operates at a fine granularity: a lightweight utility predictor scores each key-value pair, and while recent KVs are always available via a local window, older pairs are written in the cache and used in global attention only if their predicted utility surpasses a given threshold. The LLM and the utility predictor are trained jointly end-to-end exclusively through next-token prediction loss, and are adapted from pretrained LLM checkpoints. Rather than enforcing a fixed compression ratio, SP-KV performs dynamic sparsification: the mechanism adapts to the input and typically reduces the KV cache size by a factor of

3

10\times

, longer sequences often being more compressible. This leads to vast improvements in memory usage and decoding speed, with little to no degradation of validation loss nor performance on a broad set of downstream tasks. Beyond serving as an effective KV-cache reduction mechanism, our method reveals structured layer- and head-specific sparsity patterns that we can use to guide the design of hybrid local-global attention architectures.

Abstract

Explore similar work

Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility

Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection

AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD