BELT : BEhavior-Aligned Low-Rank KV Cache Compression with Learned Top-k Rank Allocation

Yaoyang Liu; Shuwei Shen; Jincheng Tang; Jinguang Liu; Jinbao Xue; Hailong Yang; Ke Zhang; Yinjun Wu

cs.LGOct 13, 2026

BELT : BEhavior-Aligned Low-Rank KV Cache Compression with Learned Top-k Rank Allocation

Authors: Yaoyang Liu, Shuwei Shen, Jincheng Tang, Jinguang Liu, Jinbao Xue, Hailong Yang, Ke Zhang, Yinjun Wu

Organizations: Peking University · Tencent · BeiHang University

Abstract

Long-context inference makes the key-value (KV) cache a major memory bottleneck for large language models. We propose BELT, a low-rank KV cache compression framework. Its key-side objective targets attention distributions rather than raw keys or pre-softmax logits, yielding a quadratic surrogate with a closed-form low-rank solution. A learned global rank allocator uses differentiable soft selection during training and hard selection at deployment to meet a user-specified component budget. Experiments demonstrate improved model quality under aggressive compression and increased decoding throughput at comparable accuracy. Evaluated across three LLMs and multiple benchmarks, BELT achieves up to 70% low-rank KV cache compression while outperforming the strongest baseline by up to 23.3 points on LongBench, and up to 17.8x overall KV cache reduction when combined with quantization. With Triton-based GPU kernels and a quantized cache, BELT delivers up to 2.7x speedup for the attention module and 3.3x end-to-end decoding throughput.

Figures & tables

Figure 1: Left: Full attention, where each query attends to all context tokens. Middle: State-of-the-art low-rank compression (e.g., vanilla SVD) retains only the dominant KV eigenvector $\mu$ , reducing the query–key product from $q^{\top}k$ to $(q^{\top}\mu)(\mu\top k)$ . Since $\mu$ is derived solely from the input-context tokens and ignores subsequent queries, this distorts the attention distribution and can yield an incorrect answer. Right: The dominant eigen-vector derived by ours aligns the query and key decompositions, preserving the downstream attention output and yielding the correct answer.

Figure 2: Main results at 50% (dark) and 70% (light) KV cache compression. Each row connects one method’s two rates; the dashed line marks the uncompressed model. All axes point right toward better results; perplexity is reversed and logarithmic, with out-of-range values printed at the left end. Numbers beside BELT denote 50%/70% results. RULER uses 16K tokens, exceeding LLaMA-2-7B’s 4K context. A3 is not applicable to Qwen3-32B, whose query/key normalization lies between the projections and RoPE. Per-task results are in Appendix H.4 .

Method	Comp (%)	Perplexity $\downarrow$		Zero-Shot Accuracy (%) $\uparrow$
Method	Comp (%)	Wiki2	C4	OBQA	PIQA	ARC-e	ARC-c	Hella	Wino	Avg.
ZS-SVD	50	7.72	14.70	43.20	77.53	77.90	49.66	74.66	72.61	65.93
ZS-SVD	70	9.34	22.51	35.60	71.98	65.36	38.99	64.90	65.51	57.06
STAR-KV	50	8.31	14.82	40.20	76.66	76.01	47.61	71.65	66.06	63.03
STAR-KV	70	9.73	18.64	37.20	74.16	70.24	41.89	65.71	61.64	58.48
Direct FT	50	8.66	17.56	37.20	75.95	72.43	43.34	68.56	59.67	59.53

Table 1: Fine-tuning the low-rank projection matrix on LLaMA-3.1-8B-Instruct. Bold marks the best result at each compression rate.

Context Len.	PyTorch SDPA	STAR-KV	Palu	ERC-SVD	BELT	Gain (BELT)
1K	2091 (128)	3092 (256)	2485 (256)	855 (256)	4859 (1024)	2.32 $\times$
2K	1044 (64)	1681 (128)	1340 (128)	456 (128)	2935 (512)	2.81 $\times$
4K	530 (32)	867 (64)	688 (64)	234 (64)	1642 (256)	3.10 $\times$
8K	271 (16)	437 (32)	343 (32)	117 (32)	869 (128)	3.21 $\times$
16K	137 (8)	211 (16)	174 (16)	58 (16)	447 (64)	3.26 $\times$

Table 3: End-to-end generation throughput (tokens/s). The gray text in brackets denotes batch size. Baselines use 50% KV cache compression and BELT uses 70% with STAR-KV.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Latency breakdown of the attention module in Figure 3 , averaged over all 32 layers. Arrows give the speedup over SDPA; the ERC-SVD bars at 16K and 32K are truncated.

Model	Comp (%)	Perplexity $\downarrow$		Zero-Shot Accuracy (%) $\uparrow$
Model	Comp (%)	Wiki2	C4	OBQA	PIQA	ARC-e	ARC-c	Hella	Wino	Avg.
MPT-7B	0	7.70	9.99	42.60	79.22	74.66	41.81	76.15	69.14	63.93
Ours	50	7.83	10.76	41.20	78.78	73.23	40.36	74.83	67.72	62.69
Ours	70	8.34	12.06	38.80	76.44	68.98	39.08	71.13	66.06	60.08

Appendix

Table 4: BELT on MPT-7B, which encodes position with ALiBi instead of RoPE. Perplexity is measured on 2048-token windows.

Method	Compressed cache	Comp (%)	Perplexity $\downarrow$		Zero-Shot Accuracy (%) $\uparrow$
Method	Compressed cache	Comp (%)	Wiki2	C4	OBQA	PIQA	ARC-e	ARC-c	Hella	Wino	Avg.
LLaMA-3.1-8B-Inst	–	0	6.75	11.10	43.00	79.92	81.82	55.12	79.20	73.56	68.77
Ours (50%)	Keys and values	50.0	7.10	12.99	44.20	79.60	79.63	53.92	77.67	71.82	67.81
	Keys only	30.3	6.90	11.99	44.20	80.30	81.40	54.78	78.37	73.24	68.72
	Values only	19.7	6.97	12.17	44.80	79.22	80.89	54.52	78.10	73.56	68.52
Ours (70%)	Keys and values	70.0	8.54	17.51	41.80	76.66	73.32	47.01	72.70	68.75	63.37

Appendix

Table 5: Compressing only the keys or only the values on LLaMA-3.1-8B-Instruct. Each variant takes the BELT checkpoint at 50% or 70% compression, keeps its low-rank projections for the compressed cache and the original projections for the other cache, and is not retrained. Comp is the resulting KV cache compression.

Method	Comp (%)	Perplexity $\downarrow$		Zero-Shot Accuracy (%) $\uparrow$
Method	Comp (%)	Wiki2	C4	OBQA	PIQA	ARC-e	ARC-c	Hella	Wino	Avg.
LLaMA-3.1-8B-Inst	0	6.75	11.10	43.00	79.92	81.82	55.12	79.20	73.56	68.77
Static	50	7.66	14.92	42.20	78.35	76.73	50.77	74.44	72.53	65.84
Static	70	14.33	38.64	34.80	71.06	62.12	34.56	55.05	61.64	53.20
Layer-wise search	50	7.52	14.65	42.40	78.51	78.58	53.33	75.62	73.01	66.91
Layer-wise search	70	11.57	32.02	37.20	72.80	68.77	41.04	61.84	65.19	57.81

Appendix

Table 6: Rank allocation on LLaMA-3.1-8B-Instruct with the key and value bases of BELT fixed. All methods share the same KV cache budget; STAR-KV stops once its budgets are met and uses 1.5% and 0.3% fewer ranks at 50% and 70% compression. Bold marks the best result at each compression rate.

Model	Calibration data	Perplexity $\downarrow$		Zero-Shot Accuracy (%) $\uparrow$
Model	Calibration data	Wiki2	C4	OBQA	PIQA	ARC-e	ARC-c	Hella	Wino	Avg.
Qwen3-32B	–	7.02	11.98	45.80	80.90	84.43	60.84	82.61	73.16	71.29
Ours (50%)	WikiText-2	7.01	12.35	45.40	80.25	84.18	64.08	82.50	75.06	71.91
	FineWeb-Edu	7.11	12.10	46.00	80.58	85.10	63.57	82.23	74.19	71.94
	C4	7.10	12.07	47.20	80.09	84.72	62.29	82.06	73.72	71.68

Appendix

Table 7: Sensitivity of BELT to the calibration corpus on Qwen3-32B at 50% KV cache compression. Each corpus supplies both the calibration set of the key and value bases and the rank-allocation training data. WikiText-2 is the corpus of the main experiments (Table 9 ).

Kronecker truncation error	0.015933
Taylor approximation error	-0.004157
$\epsilon$	6.6e-5
$\|\mathcal{L}_{K}(\widehat{W}_{K}-\mathcal{L}_{K}^{\approx}(\widehat{W}_{K})\|$	0.011775

Appendix

Table 8: Evaluation of the bounds used in Theorem 2 at

\widehat{W_{K}}^{*}

Model	Comp (%)	Perplexity $\downarrow$		Zero-Shot Accuracy (%) $\uparrow$
Model	Comp (%)	Wiki2	C4	OBQA	PIQA	ARC-e	ARC-c	Hella	Wino	Avg.
LLaMA-2-7B	0	5.12	7.04	44.00	77.64	76.26	46.16	75.98	69.22	64.88
ASVD	50	6.02	8.22	43.80	76.22	72.77	42.66	72.18	67.40	62.51
ASVD	70	6.95	9.48	40.80	74.54	70.08	38.65	69.15	64.64	59.64
ERC-SVD	50	5.78	8.27	43.60	75.95	71.76	42.24	73.17	68.51	62.54
ERC-SVD	70	6.62	9.46	38.20	72.14	62.50	36.86	67.78	62.51	56.66

Appendix

Table 9: Zero-shot evaluation on commonsense reasoning benchmarks using LM-Eval-Harness. We report accuracy (%) under different KV cache compression rates, together with perplexity on WikiText-2 and C4. A3 does not apply to Qwen3-32B. Bold marks the best result at each compression rate.

Model	Comp (%)	Qasper	QMSum	TriviaQA	MultiQA	TREC	MultiNews	VCSum	Avg.
LLaMA-2-7B	0	7.37	20.28	89.02	20.68	68.00	11.55	8.14	32.15
ASVD	50	1.84	16.51	79.32	22.62	58.00	4.51	4.50	26.76
ASVD	70	2.54	12.62	67.37	15.69	44.50	5.24	4.96	21.85
ERC-SVD	50	4.86	17.49	84.46	20.73	57.00	6.75	2.81	27.73
ERC-SVD	70	1.00	14.47	65.76	11.44	40.50	6.95	1.60	20.25
SVD-LLM	50	2.86	18.34	83.75	17.97	56.00	3.97	0.02	26.13

Appendix

Table 10: LongBench evaluation under different KV cache compression rates. A3 does not apply to Qwen3-32B. Bold marks the best result at each compression rate.

Model	Comp (%)	MK1	MK2	MQ	MV	S1	S2	S3	FWE	SQ	Avg.
LLaMA-3.1-8B-Inst	0	100.00	100.00	99.95	99.25	100.00	100.00	100.00	95.00	71.90	96.23
ASVD	50	99.40	97.80	99.60	98.15	100.00	99.80	98.60	73.33	60.57	91.92
ASVD	70	44.60	2.60	4.30	7.10	40.60	60.40	21.00	22.33	25.85	25.42
ERC-SVD	50	97.80	94.80	97.85	95.95	99.80	100.00	99.40	87.40	62.52	92.84
ERC-SVD	70	79.80	36.80	70.70	73.85	97.40	88.00	18.80	6.93	34.98	56.36
SVD-LLM	50	98.80	96.40	96.10	95.95	99.80	99.60	99.60	84.20	65.45	92.88

Appendix

Table 11: RULER evaluation at a context length of 16K under different KV cache compression rates. LLaMA-2-7B, whose context window is 4K tokens, is not evaluated, and A3 does not apply to Qwen3-32B. MK1/MK2: NIAH multi-key 1/2; MQ: NIAH multi-query; MV: NIAH multi-value; S1–S3: NIAH single 1–3; FWE: frequent words extraction; SQ: SQuAD QA. Bold marks the best result at each compression rate.

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 269 papers

PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression
Aug 24, 2026Zizhong Wang, Jieying Wang, Zhao Zhang +1KV-Cache Compression Low-Rank Compression
EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction
Mar 24, 2026Shiyu Ji, Yixuan Wang, Yijun Liu +2KV-Cache Compression LLM Inference Acceleration
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, China
NestedKV: Nested Memory Routing for Long-Context KV Cache Compression
May 26, 2026Hong Chen, Xiang Liu, Yubo Gao +5KV Caching Long-Context Language Modeling
The Hong Kong University of Science and Technology (Guangzhou) · Jimei University

Aug 24, 2026J/K move · Enter open · S save

PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression

Zizhong Wang, Jieying Wang, Zhao Zhang, Jiajia Li

KV-Cache Compression Low-Rank Compression Long-Context Language Model Inference

Open paper AskPDF arXiv

Long-context inference in large language models (LLMs) is increasingly limited by the memory required for the key-value (KV) cache. KV cache compression addresses this problem by reducing the storage cost of previous tokens. Among existing approaches, low-rank compression is particularly attractive because it represents every token in reduced dimensions. Previous low-rank methods typically derive fixed projection spaces from model weights, construct fixed spaces from calibration activations, or construct a shared basis over a broad cache region. Such representations may not capture detailed but important information. We partition each per-head KV cache into fixed-length logical pages and observe substantial low-rank structure within individual pages. Based on this observation, we propose PuzzleKV, a training- and calibration-free method that treats each completed page as an independent compression unit. PuzzleKV decomposes pages within each layer and KV head, computes attention directly over dense and factorized pages, and incrementally compresses newly eligible pages during autoregressive decoding. Experiments across models, context lengths, and benchmarks demonstrate the effectiveness of PuzzleKV under matched storage budgets. At approximately 60% of the original KV cache storage, PuzzleKV achieves more than 96% of Full KV performance across both evaluated models and all benchmark settings, with substantial gains over Global SVD on RULER and competitive performance on LongBench. To achieve a more aggressive compression ratio, PuzzleKV can be further combined with quantization while retaining more than 93% of Full KV performance using only 18.7% of the original storage.

Abstract

KV-Cache Compression

Explore similar work

PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression

EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction

NestedKV: Nested Memory Routing for Long-Context KV Cache Compression