Distance-KV: Exploiting Relative Distance for Efficient Long-Context Inference

Xianpeng Shang; Canbin Huang; Jiang Li; Tian Lan; Qianyi Cai; Xiaojun Quan; Xiangdong Su

cs.LGSep 26, 2026

Distance-KV: Exploiting Relative Distance for Efficient Long-Context Inference

Authors: Xianpeng Shang, Canbin Huang, Jiang Li, Tian Lan, Qianyi Cai, Xiaojun Quan, Xiangdong Su

Organizations: Inner Mongolia University · Shenzhen Loop Area Institute · Sun Yat-sen University · Kyoto University ·

Abstract

The memory usage and decoding latency of LLM inference grow rapidly with context length. To reduce these costs, key-value (KV) cache compression methods selectively retain cached states based on token importance or differences in attention patterns across heads. However, we discover that retrieval capability varies substantially with relative distance, even within the same attention head. To exploit this structure, we introduce Distance-KV, which learns a static KV retention pattern over the joint space of layers, attention heads, and relative distances. The pattern is learned offline with the language model frozen and reused across inputs to prune and compact the KV cache without online importance scoring. Across three backbone models and four long-context benchmarks, Distance-KV consistently achieves the best overall performance among competing KV cache compression methods, exceeding the strongest compression baseline by up to 9.3 points on RULER at 128K. On Llama-3.1-8B-Instruct at 128K, Distance-KV reduces KV cache memory by 65.4% and achieves a $1.66\times$ decoding speedup relative to Dense. Together, these results identify relative distance as an important structural dimension for understanding how LLMs retrieve information over long contexts and for designing more efficient inference methods.

Figures & tables

Figure 1: Retrieval capability varies across attention heads and distances. Results are shown for Llama-3.1-8B-Instruct. (a) Per-head-normalized retrieval score@5 for all query heads, sorted by peak distance. (b) Raw retrieval score@5 profiles of two example heads. (c) Mean retrieval score@5 at near and far distances for every query head; the diagonal indicates equal scores.

Figure 2: Overview of Distance-KV. (a) Retrieval-guided offline learning produces a model- and budget-specific binary pattern with one query-head-by-distance matrix per layer while keeping the language model frozen. (b) After full-context prefill, the pattern is mapped to the current context and applied once to prune and pack its KV cache; the same pattern is reused across inputs.

Figure 3: Retrieval accuracy of Llama-3.1-8B-Instruct on multi-needle NIAH across context lengths and needle positions. Values in parentheses report the mean accuracy over the full evaluation grid.

Figure 4: RULER task-group scores across context lengths. Results are shown for (a) Llama-3.1-8B-Instruct and (b) Qwen2.5-7B-Instruct from 4K to 128K tokens. For task groups containing multiple tasks, we report the mean score within each group.

Model	Method	LongBench
Model	Method	S-Doc	M-Doc	Summ.	Few.	Synth.	Code	Avg.
Llama-2-7B- [-1pt]32K-Instruct	Dense	12.32	12.59	15.18	56.90	7.19	62.64	25.47
	StreamingLLM	8.42	9.12	13.51	36.13	3.11	57.71	18.74
	DuoAttention	13.82	11.57	14.74	48.24	5.61	54.48	22.82
	MoA	11.77	10.90	13.75	55.86	9.16	61.21	24.72
	KVzip-CI	8.85	9.15	11.73	30.47	2.25	50.36	16.59

Table 1: LongBench performance across three backbone models. Scores are averaged within each of the six official task categories, and Avg. denotes the overall average across all 21 datasets. Bold indicates the best KV cache compression method in each column.

Llama-3.1-8B-Instruct
Method	String retrieval			Semantic retrieval				Global			Multi-task		Avg.
	KV	P–S	VT	Repo	En.QA	Zh.QA	Choice	ICL	Summ.	Math	S+N	R+KV	Avg.
Dense	80.4	41.6	47.6	50.5	29.6	20.5	69.4	43.3	40.4	21.8	61.2	68.9	49.8
StreamingLLM	0.2	0.2	0.0	0.2	5.7	6.4	26.2	72.2	25.3	5.4	30.3	0.2	14.8
DuoAttention	58.4	19.6	43.0	42.1	25.9	16.9	65.1	58.9	35.8	9.4	49.7	45.5	40.0
MoA	26.2	13.0	11.4	20.5	16.2	15.6	66.8	61.9	31.4	9.8	42.5	14.8	27.4

Table 2: SCBench multi-turn performance on two backbone models across 12 tasks grouped into four capability categories. All scores are reported on a 0–100 scale, with higher values indicating better performance. Avg. denotes the unweighted mean of the four category averages, and bold indicates the best KV cache compression method in each column.

Figure 5: Inference efficiency across context lengths. (a) Physical KV cache size and (b) per-token decoding latency for Llama-3.1-8B-Instruct on one NVIDIA H100 GPU with batch size 1.

Variant	NIAH				VT	CWE	FWE	QA	Avg.
Variant	Single	M-Key	M-Value	M-Query	VT	CWE	FWE	QA	Avg.
Head-only	73.8	25.5	84.8	87.2	2.6	0.0	59.9	22.9	44.5
Distance-only	10.0	4.4	16.3	13.8	3.6	0.1	55.8	16.0	12.7
Distance-shuffled	84.7	36.9	90.1	92.3	33.6	0.1	64.5	32.8	54.7
Distance-KV	95.9	45.5	93.2	95.5	61.4	0.2	70.3	41.8	63.7

Table 3: Ablation results on RULER at 128K for Llama-3.1-8B-Instruct. Single, M-Key, and QA average three, three, and two constituent tasks, respectively; Avg. is the mean over all 13 tasks. Bold indicates the best result in each column. All variants are evaluated at the same training step.

Figure 6: Learned Distance-KV patterns across three backbone models. All patterns use a 20% logical query-head retention ratio. (a) Mean retention rate by layer. (b) Fraction of query heads that retain only a subset of relative-distance bins. (c) Mean retention rate by relative distance. Gray dashed lines in (a) and (c) indicate the 20% retention ratio.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Hard Concrete temperature $\beta$	$2/3$
Stretch bounds $(\gamma,\zeta)$	$(-0.1,1.1)$
Initialization center $\mu$	$0.7$
Initialization slope $a$	$6$
Probability bounds $[p_{\min},p_{\max}]$	$[0.001,0.999]$
Optimizer	AdamW

Appendix

Table 4: Shared hyperparameters for offline pattern learning.

Single-document QA
Task	Dense	Streaming	Duo	MoA	KVzip-CI	Distance
Task	Dense	LLM	Attention	MoA	KVzip-CI	KV
NarrativeQA	4.89	4.14	5.00	4.33	6.79	3.64
Qasper	12.30	7.22	15.04	12.32	6.91	11.59
MultiFieldQA-en	20.97	15.93	23.07	20.83	17.00	21.93
MultiFieldQA-zh	11.14	6.38	12.19	9.60	4.69	13.18

Appendix

Table 5: Task-level LongBench results on Llama-2-7B-32K-Instruct. Avg. is the unweighted mean over all 21 tasks. Bold indicates the best KV cache compression method in each row.

Single-document QA
Task	Dense	Streaming	Duo	MoA	KVzip-CI	Distance
Task	Dense	LLM	Attention	MoA	KVzip-CI	KV
NarrativeQA	29.74	17.81	28.88	28.38	28.87	31.40
Qasper	45.01	21.15	41.08	44.50	42.62	44.09
MultiFieldQA-en	56.18	30.05	53.64	44.92	48.22	55.29
MultiFieldQA-zh	63.39	26.83	59.47	56.21	54.27	62.77

Appendix

Table 6: Task-level LongBench results on Llama-3.1-8B-Instruct. Avg. is the unweighted mean over all 21 tasks. Bold indicates the best KV cache compression method in each row.

Single-document QA
Task	Dense	Streaming	Duo	MoA	KVzip-CI	Distance
Task	Dense	LLM	Attention	MoA	KVzip-CI	KV
NarrativeQA	26.85	11.29	22.92	22.83	15.45	22.00
Qasper	43.30	18.34	40.38	42.87	30.47	41.27
MultiFieldQA-en	52.09	32.85	49.53	40.34	36.96	50.82
MultiFieldQA-zh	61.62	29.80	60.62	55.62	50.86	61.54

Appendix

Table 7: Task-level LongBench results on Qwen2.5-7B-Instruct. Avg. is the unweighted mean over all 21 tasks. Bold indicates the best KV cache compression method in each row.

Single NIAH
Task	Dense	Streaming	Duo	MoA	KVzip-CI	Distance
Task	Dense	LLM	Attention	MoA	KVzip-CI	KV
Single 1	100.00	0.80	99.60	51.80	0.00	100.00
Single 2	98.60	2.20	97.60	72.20	0.00	95.60
Single 3	99.60	4.40	97.80	51.40	0.00	99.40
Multi-key NIAH

Appendix

Table 8: Task-level RULER results at 128K on Llama-3.1-8B-Instruct. Task Avg. averages all 13 tasks, while Group Avg. averages the eight task groups used in the main-text evaluation. Bold indicates the best KV cache compression method in each row.

Single NIAH
Task	Dense	Streaming	Duo	MoA	KVzip-CI	Distance
Task	Dense	LLM	Attention	MoA	KVzip-CI	KV
Single 1	89.80	0.80	100.00	75.80	100.00	100.00
Single 2	27.00	3.00	98.40	73.60	90.40	99.20
Single 3	20.00	3.00	91.80	74.20	85.60	96.60
Multi-key NIAH

Appendix

Table 9: Task-level RULER results at 128K on Qwen2.5-7B-Instruct. Task Avg. averages all 13 tasks, while Group Avg. averages the eight task groups used in the main-text evaluation. Bold indicates the best KV cache compression method in each row.

Figure 7: Additional efficiency results on Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct. (a)–(b) Physical KV cache size and per-token decoding latency on Qwen2.5-7B-Instruct. (c)–(d) Peak allocated GPU memory during decoding on both models. All measurements use a single H100 GPU with a batch size of 1.

Single NIAH
Task	Head-	Distance-	Distance-	Fully	Distance-
Task	only	only	shuffled	shuffled	KV
Single 1	81.00	14.80	74.00	19.00	93.60
Single 2	73.20	10.00	87.20	13.00	95.60
Single 3	67.20	5.20	93.00	7.00	98.60
Multi-key NIAH

Appendix

Table 10: Task-level structural ablation results on RULER at a context length of 128K using Llama-3.1-8B-Instruct. All variants are matched in physical KV retention, and Avg. denotes the mean over all 13 tasks. Bold indicates the best result in each row.

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 271 papers

FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference
Jul 7, 2026Anna Córdoba, Adam Puente Tercero, Nerea Angulo Hijo +4KV Caching Long-Context Language Model Inference
Instituto de Investigación en Visión Artificial
ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference
Feb 8, 2026Jitai Hao, Qiang Huang, Yaowei Wang +2Self-Attention KV Caching
Harbin Institute of Technology (Shenzhen), China
StructKV: Preserving the Structural Skeleton for Scalable Long-Context Inference
Apr 8, 2026Zhirui Chen, Peiyang Liu, Ling ShaoKV-Cache Compression Long-Context Language Model Inference
UCAS-Terminus AI Lab, University of Chinese Academy of Sciences, China · National Engineering Research Center for Software Engineering, Peking University

Jul 7, 2026J/K move · Enter open · S save

FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference

Anna Córdoba, Adam Puente Tercero, Nerea Angulo Hijo, Mar Linares Tercero+3

Instituto de Investigación en Visión Artificial

KV Caching Long-Context Language Model Inference Memory-Efficient Optimization LLM Inference Acceleration KV-Cache Compression

Open paper AskPDF arXiv

Long-context LLM inference is increasingly limited by the memory and bandwidth cost of KV caches, yet aggressive compression can remove the layer-specific evidence needed for retrieval and multi-step reasoning. We introduce FreqDepthKV, an inference-time cache compression method that factorizes adjacent-layer KV states into shared low-frequency depth components and sparse high-frequency residuals. A lightweight online probe assigns attention heads to shared-depth, residual-depth, or exact cache modes according to their contribution to reconstruction-sensitive attention logits, allowing the compression policy to adapt to prompt structure without retraining. Across long-context question answering, needle retrieval, summarization, and code generation benchmarks, FreqDepthKV preserves task accuracy under substantially smaller cache budgets. With a 32k-token prefill window, FreqDepthKV reaches 58.3 Exact Match, 63.0 F1, 32.5 ROUGE-L, and 48.1 pass@1, closely matching full KV while outperforming prior compressed-cache methods. It also improves decoding throughput to 70.4 tokens/s, reduces TTFT to 2.06 seconds, and lowers peak KV memory to 6.2 GB, achieving a 3.9x effective compression ratio.

Abstract

KV-Cache Compression

Explore similar work

FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference

ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference

StructKV: Preserving the Structural Skeleton for Scalable Long-Context Inference