BasisKV: Value Latents for Communication and Key Routing

Lixun Zhang; Zishan Shao; Kangning Cui; Yu Liang; Fan Yang; Yiran Chen

cs.AINov 2, 2026

BasisKV: Value Latents for Communication and Key Routing

Authors: Lixun Zhang, Zishan Shao, Kangning Cui, Yu Liang, Fan Yang, Yiran Chen

Organizations: Duke University · Wake Forest University

Abstract

Long-context large language model (LLM) inference is increasingly constrained by the cost of maintaining an ever-growing KV-cache. Low-rank K/V projections provide a promising way to compress the KV-cache with minimal quality loss, but prior work has largely optimized KV-cache compression in isolation, leaving distributed inference, especially with tensor parallelism where attention computation is partitioned across multiple GPUs, largely underexplored. We make two observations: (i) tensor-parallel communication is an important cost when deploying low-rank KV-cache compression in distributed inference; and (ii) Value compression can be associated with the output projection, allowing the same low-rank Value representation to reduce both Value-cache width and tensor-parallel communication. Based on these observations, we propose BasisKV, a low-rank KV-cache compression method for communication-intensive, long-context LLM inference. BasisKV jointly optimizes the low-rank factors across per-GPU attention partitions against the complete attention output, rather than fitting each partition in isolation. The compressed Value cache is further reused to help identify which Key pages should be accessed: it predicts part of the Key information needed for routing, while a Key-specific residual stores what cannot be recovered from Value. The original Keys are then retrieved only for the selected pages to compute exact QK scores. Across multiple LLM families and system settings, BasisKV achieves a better quality--efficiency tradeoff than existing low-rank baselines, reaching up to 1.42x end-to-end speedup under 8-GPU tensor-parallel inference. Our code is available at https://github.com/alexz949/BasisKV

Figures & tables

Figure 1: Overview of BasisKV. BasisKV compresses each TP-local attention output into a low-rank Value latent, communicates the source-specific latents with AllGather, and reconstructs the complete output after communication. The same Value latent is retained in the cache and reused for Key routing where a Key-specific residual provides the remaining routing information.

Figure 2: Perplexity at matched TP traffic. Qwen3-8B-Base, WikiText-2, TP4. Gray dotted line: dense quality.

			PPL $\downarrow$		General MCQ $\uparrow$							MCQ $\uparrow$	Comm. $\downarrow$
Model	Budget	Method	WikiT.	C4	BoolQ	ARC-E	ARC-C	WinoG.	PIQA	HellaS.	OB	Avg.	Relative
Qwen3-8B Base	Dense		7.00	9.17	0.831	0.802	0.569	0.728	0.794	0.786	0.420	0.704	1.00
	75%	PaLU M-LRD	8.66	10.75	0.794	0.740	0.522	0.719	0.779	0.768	0.394	0.674	1.00
		PaLU G-LRD	8.24	10.40	0.793	0.771	0.556	0.718	0.796	0.787	0.408	0.690	1.00
		BasisKV	7.28	9.34	0.818	0.788	0.544	0.730	0.793	0.772	0.424	0.696	0.375
	50%	PaLU M-LRD	14.84	16.87	0.763	0.626	0.421	0.670	0.739	0.700	0.356	0.611	1.00

Table 1: Value compression across models. PaLU M-LRD uses per-head decomposition; G-LRD groups 4 heads. Comm. is

W_{O}

traffic relative to dense. Bold marks the best compressed result within each budget.

Figure 3: TP inference. Llama-3.1-8B-Instruct and Qwen3-32B,

8\times

L40S; 4K-token prompts, 128 output tokens. Left: end-to-end speedup; right: TTFT. V96/V64 retain Value widths of 96/64.

		RULER $\uparrow$											LongBench $\uparrow$
Model	Method	S1	S2	MK1	MK2	MQ	MV	QA1	QA2	VT	FWE	Avg.	NQA	MQA	HQA	MQue	DRead	GRep	SAM	PRetr	LCC	Avg.
Llama-3.1 8B-Instruct	Full-K	100.00	100.00	99.00	83.00	99.00	95.00	74.00	48.00	76.80	52.67	82.75	28.06	50.25	52.23	31.32	32.34	32.76	44.50	99.50	52.22	47.02
	Loki	79.00	96.00	97.00	18.00	55.50	53.25	48.00	32.00	25.00	48.33	55.21	27.72	48.48	52.04	31.23	29.54	32.36	46.06	99.50	51.95	46.54
	ShadowKV	99.00	99.00	98.00	61.00	97.00	91.75	74.00	48.00	28.40	58.00	75.41	29.22	51.06	52.64	31.49	31.12	30.99	43.74	99.00	51.18	46.72
	LRQK	100.00	100.00	99.00	76.00	99.00	93.25	75.00	47.00	53.60	48.67	79.15	27.90	50.00	52.13	31.46	30.63	31.00	45.37	99.50	51.61	46.62
	BasisKV	100.00	100.00	98.00	78.00	99.25	94.50	75.00	47.00	64.80	46.67	80.32	29.05	51.02	52.51	30.70	28.95	31.47	44.90	99.50	54.05	46.91

Table 2: Sparse attention across models. RULER (10 tasks) and LongBench (9 tasks). Avg. is the unweighted task mean. Bold marks the best sparse score.

	Attention block			Full-model decoding
Context	Dense	BasisKV	$\times$	Dense	BasisKV	$\times$
24K	5.27	5.17	1.02	28.63	28.33	1.01
32K	6.70	5.85	1.15	30.11	29.00	1.04
48K	9.61	7.31	1.31	33.02	30.43	1.09
64K	12.58	8.70	1.45	36.01	31.93	1.13
96K	18.33	11.77	1.56	41.78	34.98	1.19

Table 3: Sparse decoding across context lengths. TP1/B1, one L40S; median ms over three trials.

\times

: speedup over dense.

Figure 4: Decode throughput and GPU memory. Llama-3.1-8B-Instruct (left) and Qwen3-8B (right), TP8 on

8\times

L40S. Memory: peak per-rank PyTorch allocation during decode. Circles/squares: BasisKV/dense; solid/dashed: 64K/128K.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

State	Persistent width	Full-context routing	Selected attention
Compressed Value	$r_{V}$	$r_{b}$	$r_{V}$
Residual code	$r_{q}$	$r_{q}$	–
Exact Key	$d_{h}$	–	$d_{h}$

Appendix

Table 4: Per-token state and logical read width for one physical KV head.

			General MCQ $\uparrow$
Budget	Method	PPL $\downarrow$	BoolQ	ARC-E	ARC-C	WinoG.	PIQA	HellaS.	OB	Avg.
Dense		7.0025	0.831	0.802	0.569	0.728	0.794	0.786	0.420	0.704
75%	PaLU M	9.0649	0.752	0.682	0.480	0.704	0.766	0.755	0.394	0.647
	PaLU G4	9.0087	0.790	0.745	0.533	0.708	0.787	0.793	0.414	0.681
	BasisKV	7.1181	0.821	0.792	0.543	0.734	0.790	0.766	0.422	0.695
50%	PaLU M	10.9782	0.481	0.537	0.378	0.665	0.718	0.631	0.388	0.543

Appendix

Table 5: WikiText-2 calibration robustness. PPL and 7-task MCQ at matched Value rank; bold marks the best compressed method per rank.

		Accuracy (%) $\uparrow$
Model	Calibration	GSM8K	MATH500	MBPP+	IFEval
Qwen3.5-9B	Dense	93.25	91.00	72.75	83.92
	C4	86.81	75.60	51.59	69.32
	Mixed	89.23	83.00	69.84	77.82
	Reasoning	88.93	85.60	72.49	72.64
Qwen3-32B	Dense	$89.08$	79.60	45.77	29.21

Appendix

Table 6: Calibration-domain transfer. Full-test-set task scores (%); bold marks the best calibration corpus within each model. Dense rows are reference scores, not calibration variants.

Figure 5: ALS initialization and convergence. Relative MSE (left) and WikiText-2 perplexity (right) at R64 and R96. The dotted line marks six ALS sweeps.

				General MCQ $\uparrow$
Rank	Method	Rel. MSE $\downarrow$	PPL $\downarrow$	BoolQ	ARC-E	ARC-C	WinoG.	PIQA	HellaS.	OB	Avg.
R64	Group-SVD	0.16806	9.2547	0.801	0.717	0.464	0.728	0.777	0.710	0.388	0.655
R64	BasisKV S6	0.13681	8.6747	0.799	0.747	0.493	0.718	0.783	0.727	0.400	0.667
R96	Group-SVD	0.06932	7.3783	0.806	0.808	0.529	0.724	0.791	0.759	0.416	0.690
R96	BasisKV S6	0.05523	7.2679	0.799	0.811	0.569	0.721	0.793	0.768	0.412	0.696

Appendix

Table 7: Group-SVD versus BasisKV. WikiText-2 PPL and seven-task MCQ at R64/R96; bold marks the better method per rank and metric.

Method	S1	S2	S3	MK1	MK2	MQ	MV	VT	FWE	QA1	QA2	Avg.
Full-K	100.00	99.00	99.00	99.00	91.00	97.75	89.75	66.80	65.67	76.00	45.00	84.45
Loki	80.00	94.00	0.00	92.00	41.00	55.75	56.50	20.40	40.67	53.00	26.00	50.85
ShadowKV	96.00	99.00	96.00	98.00	84.00	93.25	87.50	26.60	54.33	78.00	45.00	77.97
LRQK	100.00	99.00	99.00	99.00	88.00	97.50	91.25	44.40	44.33	77.00	45.00	80.41
BasisKV B16R16	100.00	100.00	99.00	99.00	75.00	97.00	88.50	56.80	56.33	76.00	42.00	80.88

Appendix

Table 8: C4-only routing robustness on RULER-128K. Llama-3.1-8B-Instruct with uniform V96; all methods are evaluated on 11 tasks with 100 prompts per task. Full-K uses the same compressed V96 checkpoint. BasisKV B16R16 is fitted using 32 purely C4 128K windows.

Figure 6: Key predictability across layers. Held-out centered explained pre-RoPE Key energy on Llama-3.1-8B-Instruct and Qwen3-8B.

Router	S1	S2	S3	MK1	MK2	MQ	MV	VT	FWE	QA1	QA2	Avg.
Full-K	100.00	100.00	100.00	96.67	86.67	98.33	94.17	78.00	57.78	60.00	36.67	82.57
B16	70.00	86.67	16.67	86.67	23.33	60.00	25.00	27.33	32.22	53.33	33.33	46.78
R16	100.00	100.00	96.67	96.67	50.00	97.50	95.83	64.67	43.33	60.00	36.67	76.48
R32	100.00	100.00	96.67	96.67	73.33	99.17	94.17	68.00	50.00	60.00	36.67	79.52
B16R16	100.00	100.00	96.67	96.67	76.67	99.17	95.83	66.00	43.33	63.33	36.67	79.48

Appendix

Table 9: Base and Residual contributions on RULER-128K. Llama-3.1-8B-Instruct with compressed V96. All methods use the same 30 prompts per task. Sparse routers use Page4, 2048 routed tokens and 64 recent tokens.

		LongBench $\uparrow$										RULER-128K $\uparrow$
Model	Page size	NQA	MQA	HQA	MQue	DRead	GRep	SAM	PRetr	LCC	Avg.	MK2	FWE	VT	QA1	QA2
	Full-K	28.06	50.25	52.23	31.32	32.34	32.76	44.50	99.50	52.22	47.02	83.00	52.67	76.80	74.00	48.00
	1	28.79	50.15	52.18	29.93	30.77	32.18	45.23	99.50	51.24	46.66	80.00	46.33	66.00	75.00	48.00
	4	29.05	51.02	52.51	30.70	28.95	31.47	44.90	99.50	54.05	46.91	78.00	46.67	64.80	75.00	47.00
	8	27.96	50.24	51.45	31.15	30.00	31.29	44.43	99.50	53.46	46.61	75.00	50.67	63.40	75.00	47.00
Llama-3.1 8B-Instruct	32	29.16	51.62	51.66	31.15	29.58	29.10	43.34	99.50	48.95	46.01	66.00	60.33	66.80	74.00	46.00

Appendix

Table 10: Page granularity across models and tasks. Both models use V96 and B16R16 routing. LongBench reports nine tasks and their unweighted mean over 1,543/1,549 samples for Llama/Qwen; RULER-128K reports five common tasks with 100 shared prompts per task. Higher scores are better. Shading marks the adopted Page4 configuration.

Method	NIAH							VT	FWE	QA1	QA2	Avg.
Method	S1	S2	S3	MK1	MK2	MQ	MV	VT	FWE	QA1	QA2	Avg.
Dense	100.00	100.00	99.00	99.00	89.00	98.75	94.25	76.20	74.67	78.00	46.00	86.81
ShadowKV	100.00	100.00	100.00	99.00	76.00	97.25	89.50	70.20	69.33	79.00	46.00	84.21
LRQK	100.00	100.00	100.00	99.00	83.00	98.50	92.25	64.20	58.00	80.00	46.00	83.72
Loki	84.00	97.00	0.00	98.00	31.00	45.25	48.00	39.00	33.67	64.00	34.00	52.17
BasisKV	100.00	100.00	100.00	99.00	80.00	97.75	88.50	73.00	69.67	80.00	47.00	84.99

Appendix

Table 11: RULER-128K with dense Values. Llama-3.1-8B-Instruct, original V128 and

W_{O}

; 100 shared prompts per task. Scores are percentages; the mean weights all 11 tasks equally. Bold marks the best sparse score; ties involving BasisKV highlight only BasisKV.

Model	Dataset	Avg. V rank	BF16 PPL $\downarrow$	KVQuant (4-bit) PPL $\downarrow$	$\Delta$ PPL
Qwen3-8B-Base	WikiText-2	96	7.283	7.338	+0.055
Qwen3-8B-Base	WikiText-2	64	8.259	8.277	+0.019
Llama-3.1-8B Instruct	C4	96	10.662	10.727	+0.064
Llama-3.1-8B Instruct	C4	64	12.454	12.595	+0.141

Appendix

Table 12: Value compression with KVQuant. 4-bit NUQ4 with outlier handling for Keys and Value latents, simulated by quantize–dequantize on each BF16 checkpoint. Ranks are average retained Value dimensions;

\Delta

PPL is relative to BF16.

Method	KV format	NQA	MQA	HQA	MQue	DRead	GRep	SAM	PRetr	LCC	Avg.
Full-K	BF16	28.06	50.25	52.23	31.32	32.34	32.76	44.50	99.50	52.22	47.02
	FP8	28.73	50.37	52.32	31.29	33.06	33.06	45.05	99.00	51.34	47.14
	KVQuant (4-bit)	28.62	50.05	52.78	31.49	31.87	32.63	45.37	99.50	52.83	47.24
BasisKV	BF16	29.05	51.02	52.51	30.70	28.95	31.47	44.90	99.50	54.05	46.91
	FP8	28.76	50.98	52.62	30.71	30.41	31.45	44.55	99.50	52.62	46.85
	KVQuant (4-bit)	28.14	49.79	52.81	30.52	28.83	31.48	43.96	99.50	53.43	46.50

Appendix

Table 13: Sparse routing with quantized KV caches. Llama-3.1-8B-Instruct, uniform V96, 1,543 shared LongBench samples. Full-K: full attention; BasisKV: B16R16 Page4. FP8: E4M3; KVQuant: NUQ4 with outlier handling. Avg.: unweighted nine-task mean; higher is better.

KV Cache Selection

See all 9 papers

BELT : BEhavior-Aligned Low-Rank KV Cache Compression with Learned Top-k Rank Allocation
Oct 13, 2026Yaoyang Liu, Shuwei Shen, Jincheng Tang +5KV-Cache Compression Low-Rank Compression
Peking University · Tencent · BeiHang University
PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression
Aug 24, 2026Zizhong Wang, Jieying Wang, Zhao Zhang +1KV-Cache Compression Low-Rank Compression
CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
Jun 23, 2026Xiaolin Lin, Jingcun Wang, Olga Kondrateva +3LLM Inference Efficiency KV Caching
Technical University of Darmstadt, Darmstadt, Germany · University of Notre Dame, Notre Dame, IN, USA · Technical University of Ilmenau, Ilmenau, Germany

Oct 13, 2026J/K move · Enter open · S save

BELT : BEhavior-Aligned Low-Rank KV Cache Compression with Learned Top-k Rank Allocation

Yaoyang Liu, Shuwei Shen, Jincheng Tang, Jinguang Liu+4

Peking University · Tencent · BeiHang University

KV-Cache Compression Low-Rank Compression Long-Context Language Model Inference LLM Inference Acceleration

Open paper AskPDF arXiv

Long-context inference makes the key-value (KV) cache a major memory bottleneck for large language models. We propose BELT, a low-rank KV cache compression framework. Its key-side objective targets attention distributions rather than raw keys or pre-softmax logits, yielding a quadratic surrogate with a closed-form low-rank solution. A learned global rank allocator uses differentiable soft selection during training and hard selection at deployment to meet a user-specified component budget. Experiments demonstrate improved model quality under aggressive compression and increased decoding throughput at comparable accuracy. Evaluated across three LLMs and multiple benchmarks, BELT achieves up to 70% low-rank KV cache compression while outperforming the strongest baseline by up to 23.3 points on LongBench, and up to 17.8x overall KV cache reduction when combined with quantization. With Triton-based GPU kernels and a quantized cache, BELT delivers up to 2.7x speedup for the attention module and 3.3x end-to-end decoding throughput.

Abstract

KV Cache Selection

Explore similar work

BELT : BEhavior-Aligned Low-Rank KV Cache Compression with Learned Top-k Rank Allocation

PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression

CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference