StructKV: Preserving the Structural Skeleton for Scalable Long-Context Inference

Zhirui Chen; Peiyang Liu; Ling Shao

cs.CLApr 8, 2026

StructKV: Preserving the Structural Skeleton for Scalable Long-Context Inference

Authors: Zhirui Chen, Peiyang Liu, Ling Shao

Organizations: UCAS-Terminus AI Lab, University of Chinese Academy of Sciences, China · National Engineering Research Center for Software Engineering, Peking University

Abstract

As Large Language Models (LLMs) scale to support context windows exceeding one million tokens, the linear growth of Key-Value (KV) cache imposes severe memory capacity and bandwidth bottlenecks, constraining the efficiency of long-context inference. Existing compression approaches typically prioritize tokens based on local saliency metrics to decouple prefill computation from decoding memory. However, these methods often rely on local saliency snapshots at a specific layer, thereby systematically discarding tokens that act as global information hubs across the network depth but appear temporarily dormant at the specific layer selected for pruning. To address this limitation, we propose StructKV, a structure-aware KV cache compression framework that introduces three core innovations: First, Global In-Degree Centrality aggregates attention patterns across the network depth to identify global information hubs. Second, Dynamic Pivot Detection utilizes information-theoretic metrics to adaptively locate the optimal layer for compression. Finally, Structural Propagation and Decoupling separates the computational budget from the memory storage budget. Experimental results on the LongBench and RULER benchmarks demonstrate that StructKV effectively preserves long-range dependencies and retrieval robustness.

Figures & tables

Figure 1: Comparison of StructKV with state-of-the-art methods. (a) Accuracy retention under varying KV cache budgets. StructKV (red line) demonstrates superior robustness, maintaining near-lossless performance even at extreme compression rates, whereas baselines suffer significant degradation. (b) A holistic comparison showing that StructKV effectively resolves the trilemma of prefill speed, decoding latency, and long-context accuracy.

Figure 2: The Three-Phase Workflow of StructKV. Phase 1: The model processes full context while accumulating global centrality scores ( $\mathcal{C}_{global}$ ). Phase 2: The Auto-Detector triggers a structural phase transition. The context is filtered using $\mathcal{C}_{global}$ , decoupling the propagation budget ( $R_{struct}$ ) from the storage budget ( $R_{KV}$ ). Phase 3: Deep layers operate on the reduced "structural skeleton" for maximum speed.

			Single-Doc QA			Multi-Doc QA			Summarization			Few-shot			Synthetic		Code
Method	Prefill	KV	NrtvQA	Qasper	MF-en	HotpotQA	2Wiki	Musique	GovRep	QMSum	MNews	TREC	Trivia	SAMSum	PCount	PRe	Lcc	RB-P	Avg.
Full-context
Full-context	100%	100%	30.21	45.53	55.01	56.01	46.65	31.28	35.13	25.28	27.25	73.00	91.64	43.80	8.91	99.50	63.38	56.64	49.33
Decoding-Only
StreamingLLM	100%	10%	26.05	26.29	33.68	48.07	37.93	25.03	25.37	21.49	19.97	58.50	88.46	42.61	7.91	88.00	61.07	54.94	41.59
	100%	20%	27.99	28.89	34.42	51.21	41.03	24.43	28.34	22.08	22.16	64.00	89.96	42.27	8.66	85.00	62.02	56.85	43.08

Table 1: LongBench results on LLaMA-3.1-8B-Instruct.

Figure 3: Adaptability of Pivot Selection on Qwen-2.5 Series, Llama-3.1-8B-Instruct and Ministral-8B-Instruct.

Figure 4: Visualization of Dynamic Pivot Detection on Qwen-2.5-32B-Instruct.

Figure 5: Layer-wise Hidden State Fidelity.

Method	Prefill	KV	8K	16K	32K	64K	128K	Avg.
Reference
Full-context	100%	100%	90.1	95.0	83.4	85.5	76.3	86.0
Decoding-Only
StreamingLLM	100%	10%	15.0	21.5	24.4	16.9	15.0	18.6
H2O	100%	10%	27.1	-	-	-	-	-
SnapKV	100%	10%	75.6	76.8	72.9	75.0	67.7	73.6

Table 2: RULER Benchmark Results on LLaMA-3.1-8B-Instruct.

Figure 6: Decoupling Analysis: Computation vs. Memory Budget.

Decay ( $\lambda$ )	LongBench Avg.	$\Delta$	Interpretation
0.50	47.41	-1.20	Too aggressive decay
0.80	48.35	-0.26	Robust Zone
0.90 (Ours)	48.61	Ref	Optimal trade-off
0.95	48.42	-0.19	Robust Zone
1.00	48.03	-0.58	No decay (flat history)

Table 3: Sensitivity analysis of the decay factor

\lambda

on LongBench (LLaMA-3.1-8B, 10% KV budget).

Figure 7: Prefill Latency Breakdown (LLaMA-3.1-8B, 32K Context).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

			Single-Doc QA			Multi-Doc QA			Summarization			Few-shot			Synthetic		Code
Method	Prefill	KV	NrtvQA	Qasper	MF-en	HotpotQA	2Wiki	Musique	GovRep	QMSum	MNews	TREC	Trivia	SAMSum	PCount	PRe	Lcc	RB-P	Avg.
Full-context
Full-context	100%	100%	24.55	47.99	52.21	60.43	52.64	35.28	32.36	24.16	26.64	74.50	92.04	45.94	9.00	100.00	67.06	67.19	50.75
Decoding-Only
StreamingLLM	100%	10%	22.52	33.02	27.69	48.68	37.66	21.26	23.55	21.09	18.36	61.50	91.38	43.45	6.00	63.00	62.84	62.55	40.28
	100%	20%	23.25	35.91	28.26	51.46	41.13	24.08	26.88	21.38	21.43	67.50	91.88	44.65	5.00	72.00	63.17	63.21	42.57

Appendix

Table 4: LongBench results on Ministral-8B-Instruct.

			Single-Doc QA			Multi-Doc QA			Summarization			Few-shot			Synthetic		Code
Method	Prefill	KV	NrtvQA	Qasper	MF-en	HotpotQA	2Wiki	Musique	GovRep	QMSum	MNews	TREC	Trivia	SAMSum	PCount	PRe	Lcc	RB-P	Avg.
Full-context
Full-context	100%	100%	28.79	44.79	51.77	58.38	46.11	29.23	33.46	24.32	25.91	73.50	88.93	47.92	8.83	100.00	62.03	66.83	49.43
Decoding-Only
StreamingLLM	100%	10%	25.85	44.79	49.10	50.73	44.92	24.83	33.20	23.20	25.73	72.00	89.41	47.73	7.91	85.00	61.06	56.75	46.39
	100%	20%	27.99	44.79	49.23	51.64	45.12	25.94	33.25	23.45	25.73	72.50	89.96	47.73	8.47	85.00	62.07	57.85	46.92

Appendix

Table 5: LongBench results on Qwen-2.5-7B-Instruct.

			Single-Doc QA			Multi-Doc QA			Summarization			Few-shot			Synthetic		Code
Method	Prefill	KV	NrtvQA	Qasper	MF-en	HotpotQA	2Wiki	Musique	GovRep	QMSum	MNews	TREC	Trivia	SAMSum	PCount	PRe	Lcc	RB-P	Avg.
Full-context
Full-context	100%	100%	31.85	48.22	56.53	63.41	50.16	34.82	36.54	27.83	29.41	76.55	91.52	49.83	12.53	100.00	63.85	70.57	52.73
Decoding-Only
StreamingLLM	100%	10%	28.53	46.82	52.14	54.21	48.33	28.45	35.12	25.26	27.44	74.05	90.52	48.63	9.54	88.00	61.25	60.56	48.68
	100%	20%	30.12	47.14	52.86	55.63	48.85	29.54	35.82	25.93	27.91	75.04	91.02	48.96	10.25	88.00	62.18	61.24	49.41

Appendix

Table 6: LongBench results on Qwen-2.5-14B-Instruct.

			Single-Doc QA			Multi-Doc QA			Summarization			Few-shot			Synthetic		Code
Method	Prefill	KV	NrtvQA	Qasper	MF-en	HotpotQA	2Wiki	Musique	GovRep	QMSum	MNews	TREC	Trivia	SAMSum	PCount	PRe	Lcc	RB-P	Avg.
Full-context
Full-context	100%	100%	34.21	50.87	58.43	67.54	53.92	38.65	38.76	30.21	31.89	79.82	92.74	53.15	17.56	100.00	69.87	75.43	55.82
Decoding-Only
StreamingLLM	100%	10%	30.84	49.12	54.37	57.65	51.23	32.18	37.15	27.89	29.56	77.34	91.56	50.89	13.45	90.50	66.78	65.43	51.62
	100%	20%	32.56	49.87	55.43	59.12	52.34	33.56	37.98	28.45	30.23	78.12	92.15	51.67	14.32	90.50	67.98	66.87	52.57

Appendix

Table 7: LongBench results on Qwen-2.5-32B-Instruct.

Figure 8: Needle-in-a-Haystack results on LLaMA-3.1-8B-Instruct with 10% KV retention rate.

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 267 papers

SGD-KV: Summarization Guided KV Cache Compression
Sep 3, 2026Zeyu Liu, Woomin Song, Xuandi Fu +5KV Caching Long-Context Language Model Inference
USC, USA · KAIST, Korea · Amazon AGI, USA
ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference
Feb 8, 2026Jitai Hao, Qiang Huang, Yaowei Wang +2Self-Attention KV Caching
Harbin Institute of Technology (Shenzhen), China
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
Jun 30, 2026Amirhossein Abaskohi, Giuseppe Carenini, Peter West +1KV-Cache Offloading KV Caching
University of British Columbia · Microsoft Research

Sep 3, 2026J/K move · Enter open · S save

SGD-KV: Summarization Guided KV Cache Compression

Zeyu Liu, Woomin Song, Xuandi Fu, Sai Muralidhar Jayanthi+4

USC, USA · KAIST, Korea · Amazon AGI, USA

KV Caching Long-Context Language Model Inference KV-Cache Management LLM Inference Acceleration KV-Cache Compression

Open paper AskPDF arXiv

Large language models (LLMs) face severe memory bottlenecks in long-context inference due to the linearly growing size of key-value (KV) caches. Existing KV cache compression techniques typically rely on simple heuristics, overlooking the distinct functional roles of different attention heads. We present SGD-KV (Summarization-Guided KV Cache Compression), a head-aware framework that leverages a novel chunk-summarization diagnostic task to systematically identify and prioritize attention heads specialized in hierarchical information aggregation. Experiments on Qwen2.5-7B-1M and Qwen3-32B across diverse long-context benchmarks demonstrate that SGD-KV achieves state-of-the-art performance with contexts up to 1M tokens, while reducing KV cache memory usage by up to 75%. Our findings show that strategically allocating the KV cache budget based on the summarization score distribution of attention heads yields a superior efficiency-accuracy trade-off for long-context inference.

Abstract

KV-Cache Compression

Explore similar work

SGD-KV: Summarization Guided KV Cache Compression

ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference

SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference