SlimKV: Joint Token-Feature KV Cache Compression with Reconstruction-Free Beacon Attention

Zihan Teng; Jiayu Zhao; Wentao Ren; Minhao Fan; Tianrui Ma; Song Chen; Weichen Liu

cs.LGOct 2, 2026

SlimKV: Joint Token-Feature KV Cache Compression with Reconstruction-Free Beacon Attention

Authors: Zihan Teng, Jiayu Zhao, Wentao Ren, Minhao Fan, Tianrui Ma, Song Chen, Weichen Liu

Organizations: University of Science and Technology of China · Nanyang Technological University · Tencent

Abstract

Long-context LLM serving is increasingly bottlenecked by KV-cache memory, especially in resource-constrained scenarios. Among existing KV-cache compression strategies, token-wise methods reduce cached states but risk information loss through eviction or condensation, while feature-wise methods reduce per-token KV dimensions but can require full-dimensional reconstruction to apply positional embedding, limiting decoding speedups. We introduce SlimKV, a question-agnostic joint token-feature KV-cache compression method. SlimKV uses low-rank-aware training to compress long contexts into beacon memory states with latent KV representations, together with layer-adaptive rank allocation. We further uncover a positional asymmetry: removing key-side RoPE affects beacon and raw tokens differently, with much smaller degradation for beacon tokens. Exploiting this asymmetry, SlimKV trains beacon KV projections under a K-RoPE-free constraint and enables latent-space attention during decoding, mitigating reconstruction latency. On LongBench, SlimKV outperforms baselines at 16x/32x compression and remains leading at 4x/8x, where it retains over 96% of the uncompressed model's score. Needle-in-a-Haystack confirms robustness across evidence positions, and efficiency evaluation shows up to 7.34x attention speedup and 3.38x end-to-end decoding speedup over the uncompressed model at 128K length.

Figures & tables

Figure 1: Comparison of low-rank KV-cache strategies. K-RoPE forces reconstruction, while its removal enables latent-space attention (LSA) but loses order information. SlimKV uses K-RoPE-free beacon tokens to retain semantics and positional cues in latent KV space.

Figure 2: LongBench scores under K-RoPE ablations, showing positional asymmetry between raw and beacon tokens. SlimKV remains close to the original model without beacon K-RoPE.

Figure 3: Overview of SlimKV. SlimKV trains low-rank beacon KV projections and caches only latent beacon KV states after prefill. During decoding, SlimKV performs latent-space attention over cached beacon memories, avoiding full-dimensional reconstruction.

Figure 4: Layer-wise rank allocation. K/V ranks of Beacon and raw weights in Llama-3.1-8B under 2

\times

, 4

\times

, and 8

\times

feature compression, selected by equal explained variance.

Figure 5: Attention maps under K-RoPE ablations. Beacon K-RoPE removal remains much closer to the K-RoPE-kept pattern than raw-token K-RoPE removal.

Variant	Attn. $\uparrow$	Block $\uparrow$	Lag JS $\downarrow$	Col. JS $\downarrow$
Beacon removed	0.9707	0.9982	0.0002	0.0030
Raw removed	0.7570	0.8922	0.0062	0.0349

Table 1: Similarity to the w/ K-RoPE setting.

Figure 6: Reconstruction accuracy from compressed beacon states on length-16,384 random sequences over 46 symbols, averaged over three seeds. SlimKV starts from 8

\times

total compression due to 4

\times

feature compression. Dashed line denotes random-guessing probability.

Comp.	Method	Llama-3.1-8B						Qwen2.5-14B
Comp.	Method	S-Doc	M-Doc	Summ.	Few	Code	Avg.	S-Doc	M-Doc	Summ.	Few	Code	Avg.
1 $\times$	Full KV	44.34	46.65	28.71	69.45	58.37	49.50	42.72	43.73	28.05	72.04	61.05	51.52
16 $\times$	KVZip	33.86	29.11	23.97	22.97	34.89	28.96	7.83	5.48	3.27	35.12	14.20	13.18
	PALU	13.18	9.80	12.75	37.86	21.04	18.92	17.77	14.18	21.40	59.40	25.75	27.70
	SnapKV	38.55	43.30	23.61	65.92	56.62	45.60	36.74	51.20	20.22	68.50	52.37	45.81
	Activation Beacon	34.72	36.28	25.34	65.01	65.14	45.30	35.76	38.31	25.24	63.66	57.77	44.15

Table 2: LongBench scores under high KV-cache compression. SlimKV t

n

m

n\times

token-wise,

m\times

feature-wise. The bold and underlined numbers denote the first and second rankings, respectively.

Backbone	Comp.	$\Delta_{\textbf{AB}}$	$\Delta_{\textbf{SlimKV-f4}}$	$\Delta$ Red.	Gap Red.
Llama-3.1-8B	16 $\times$	4.20	3.14	1.06	25.2%
Llama-3.1-8B	32 $\times$	5.83	4.75	1.08	18.5%
Qwen2.5-14B	16 $\times$	7.37	5.29	2.08	28.2%
Qwen2.5-14B	32 $\times$	8.76	6.43	2.33	26.6%

Table 3: LongBench gap reduction over Activation Beacon.

\Delta

denotes the drop from Full KV.

Panel of Figure 7 — Figure 7: Needle-in-a-Haystack evaluation. (a) SlimKV outperforms baselines across depths under 16 $\times$ and 32 $\times$ compression. (b) SlimKV maintains retrieval ability for information beyond Qwen2.5-14B’s context window (32K). Since higher ratios limit PALU’s retrieval accuracy, we select 8 $\times$ compression to better reveal the behavior.

Backbone	Comp.	KVZip	PALU	AB	SlimKV-f2
Llama-3.1-8B	4 $\times$	36.39	46.66	48.21	48.10
Llama-3.1-8B	8 $\times$	37.25	46.42	46.93	47.77
Qwen2.5-14B	4 $\times$	47.72	45.95	48.86	49.92
Qwen2.5-14B	8 $\times$	46.74	45.15	45.50	49.94

Table 4: LongBench averages at low compression. The averages of the uncompressed models are in Table 2 .

Method	Comp.	S-Doc	M-Doc	Summ.	Few-shot	Code	Avg.
Full KV	1 $\times$	42.87	51.02	25.60	71.26	60.75	50.30
SlimKV-f4	16 $\times$	36.87	40.43	26.00	68.24	58.87	46.08
SlimKV-f4	32 $\times$	34.14	38.16	24.77	67.85	63.09	45.60

Table 5: LongBench category scores on Qwen3-30B-A3B-Instruct-2507.

Comp.	Metric	Method	32K	64K	96K	128K
16 $\times$	Attn.	Recon	1.76 $\times$	2.81 $\times$	3.63 $\times$	5.34 $\times$
	Attn.	SlimKV	1.94 $\times$	3.81 $\times$	5.08 $\times$	7.34 $\times$
	E2E	Recon	1.13 $\times$	1.64 $\times$	2.13 $\times$	2.86 $\times$
	E2E	SlimKV	1.22 $\times$	2.04 $\times$	2.63 $\times$	3.38 $\times$
32 $\times$	Attn.	Recon	1.52 $\times$	2.35 $\times$	3.84 $\times$	4.92 $\times$
	Attn.	SlimKV	1.62 $\times$	2.54 $\times$	4.36 $\times$	6.09 $\times$

Table 6: Decoding speedup over the uncompressed model on Llama-3.1-8B-Instruct. Attn./E2E denote attention/per-token latency.

T-wise	F-wise	Train-R	Infer-R	Ada-r	Score
✓	✗	✓	✓	✗	43.67
✓	✗	✓	✗	✗	41.28
✓	✗	✗	✗	✗	41.95
✓	✓	✗	✗	✗	44.38
✓	✓	✗	✗	✓	44.75

Table 7: Ablation study on LongBench at 32

\times

compression with Llama-3.1-8B-Instruct.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Proj.	CR	Beacon	Raw	Spearman	MAE / Max
Llama	K	2	64.00	64.00	1.000	0.00 / 0
	K	4	32.00	32.00	0.996	0.06 / 1
	K	8	16.00	16.00	0.996	0.06 / 1
	V	2	64.00	64.00	0.990	0.50 / 1
	V	4	32.00	32.00	0.954	0.50 / 2
	V	8	16.00	16.00	0.928	0.38 / 2

Appendix

Table 8: Layer-wise rank consistency between Beacon and raw K/V projections.

Comp.	Method	32K		64K		96K		128K
Comp.	Method	Attn.	E2E	Attn.	E2E	Attn.	E2E	Attn.	E2E
1 $\times$	Full KV	11.84	13.28	23.97	25.40	34.93	36.32	46.64	48.08
16 $\times$	Recon	6.74	11.75	8.52	15.46	9.63	17.02	8.73	16.79
16 $\times$	SlimKV	6.11	10.93	6.29	12.47	6.87	13.79	6.35	14.24
32 $\times$	Recon	7.80	11.56	10.18	15.23	9.11	13.95	9.48	15.69
32 $\times$	SlimKV	7.31	10.81	9.43	14.51	8.01	12.90	7.65	13.78

Appendix

Table 9: Detailed decoding latency on Llama-3.1-8B-Instruct over 128 generated tokens. Attn. and E2E denote cumulative attention and end-to-end decoding latency. All values are in seconds.

Comp.	Method	32K	64K	96K	128K
TTFT (s)
16 $\times$	Recon	16.59	46.38	89.15	153.74
16 $\times$	SlimKV	16.16	44.80	87.12	152.57
32 $\times$	Recon	11.28	28.34	50.13	82.73
32 $\times$	SlimKV	10.92	27.77	48.62	79.96
Peak Memory (GB)

Appendix

Table 10: TTFT and peak allocated memory on Llama-3.1-8B-Instruct. TTFT is reported in seconds and memory in GB.

Figure 8: Complete Needle-in-a-Haystack heatmaps for Activation Beacon and SlimKV-f4, where f4 denotes 4 $\times$ feature-wise compression. Results are shown on Llama-3.1-8B-Instruct and Qwen2.5-14B-Instruct under 16 $\times$ and 32 $\times$ total compression. Each cell reports retrieval accuracy at a given context length and needle depth.

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 269 papers

SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
Jun 30, 2026Amirhossein Abaskohi, Giuseppe Carenini, Peter West +1KV-Cache Offloading KV Caching
University of British Columbia · Microsoft Research
DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression
Jul 7, 2026Anna Cordoba, Adam Puente Tercero, Nerea Angulo Hijo +4KV Caching Long-Context Language Model Inference
Instituto de Investigación en Visión Artificial
Minima-KV: Retention-Preserving KV Cache Compression with Mixed-Format Paged Attention
Aug 24, 2026Sergii Kozyrev, Davyd MaiborodaKV-Cache Compression LLM Inference Acceleration
Minima AI, Inc · Minima AI, Inc.

Jun 30, 2026J/K move · Enter open · S save

SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference

Amirhossein Abaskohi, Giuseppe Carenini, Peter West, Yuhang He

University of British Columbia · Microsoft Research

KV-Cache Offloading KV Caching Long-Context Language Model Inference KV-Cache Compression Semantic Caching

Open paper AskPDF arXiv

Large language models increasingly operate over long contexts, where the KV cache becomes a dominant memory bottleneck: its size grows linearly with sequence length and must be retained throughout decoding, making full GPU caching prohibitively expensive without compression. Existing KV cache compression methods struggle to balance efficiency with faithful context preservation. Token eviction discards information, while semantic grouping fixes compression decisions at prefill time; neither can recover token-level detail from a compressed span once it becomes relevant during generation. As a solution, we propose SeKV, a resolution-adaptive semantic KV cache that organizes context into entropy-guided semantic spans and stores them across a GPU-CPU memory hierarchy without discarding information. Each span keeps a lightweight summary vector on GPU for coarse routing and a low-rank SVD basis on CPU for on-demand token-level reconstruction. A trained zoom-in mechanism selectively expands query-relevant spans during decoding, enabling precise retrieval without materializing the full KV cache on GPU. SeKV enables adaptive token-level reconstruction while keeping the base LLM fully frozen and adding fewer than 0.05% trainable parameters. Across four benchmarks, SeKV improves over the strongest semantic compression baseline by 5.9% on average while reducing GPU memory by 53.3% versus full KV caching at 128K context. Code is available on https://github.com/AmirAbaskohi/SeKV.

Abstract

KV-Cache Compression

Explore similar work

SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference

DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression

Minima-KV: Retention-Preserving KV Cache Compression with Mixed-Format Paged Attention