KVSculpt: KV Cache Compression as Distillation

Bo Jiang; Sian Jin

cs.LGMar 29, 2026

KVSculpt: KV Cache Compression as Distillation

Authors: Bo Jiang, Sian Jin

Organizations: Temple University

Abstract

KV cache compression is critical for efficient long-context LLM inference. Approaches that reduce the per-pair footprint -- quantization and low-rank decomposition -- are orthogonal to those that reduce the sequence length of the cache. Along the sequence-length dimension, existing methods range from pure eviction -- selecting which KV pairs to keep -- to merging, which combines similar pairs into fewer ones. Both remain anchored to the original cache entries. We propose KVSculpt, which moves to the other end of this spectrum: instead of selecting or combining original pairs, we optimize a smaller set of unconstrained KV pairs in continuous embedding space to preserve each layer's attention behavior. Keys are optimized via L-BFGS and values are solved in closed form via least squares, alternating every few steps. On top of this, we introduce adaptive budget allocation, which uses a cheap pilot compression run to redistribute the compression budget across layers and KV heads based on per-component difficulty. On Qwen2.5-1.5B-Instruct with 2048-token contexts, KVSculpt reduces KL divergence by 3.5-4.1x compared to Select+Fit -- attention-score eviction with least-squares value fitting -- across compression ratios r in {0.3, 0.5, 0.7}. Adaptive allocation provides an additional 1.3x KL reduction at no extra inference cost. Analysis reveals that compression difficulty is highly non-uniform: per-layer pilot MSE varies by up to 100x across layers, and the two KV heads within a single layer can differ by up to 467x -- demonstrating that fine-grained budget allocation is essential.

Figures & tables

Panel of Figure 1 — Figure 1: Three paradigms for KV cache sequence-length reduction. (a) Eviction selects a subset of original KV pairs. (b) Merge combines similar pairs, modifying values but remaining anchored to original positions. (c) KVSculpt distills the compress zone into $k$ unconstrained pairs (orange) freely optimized in $\mathbb{R}^{d}$ via L-BFGS (keys) and least squares (values). All three keep the retain zone (green) intact. The key distinction is the degree of freedom: from discrete subset to weighted combination to fully continuous optimization.

Method	$r{=}0.3$	$r{=}0.5$	$r{=}0.7$
Random	$2.50$	$1.80$	$1.54$
Attn Score	$2.54\mathrm{e}{\text{-}1}$	$2.04\mathrm{e}{\text{-}1}$	$1.39\mathrm{e}{\text{-}1}$
Select+Fit	$2.33\mathrm{e}{\text{-}1}$	$1.86\mathrm{e}{\text{-}1}$	$1.25\mathrm{e}{\text{-}1}$
Joint Opt	$2.24\mathrm{e}{\text{-}1}$	$1.80\mathrm{e}{\text{-}1}$	$1.15\mathrm{e}{\text{-}1}$
KVSculpt	$\mathbf{5.75\mathrm{e}{\text{-}2}}$	$\mathbf{4.63\mathrm{e}{\text{-}2}}$	$\mathbf{3.58\mathrm{e}{\text{-}2}}$
vs. Select+Fit	$4.1\times$	$4.0\times$	$3.5\times$

Table 1: KL divergence (

\downarrow

) on 5 sequences (mean). KVSculpt consistently outperforms all eviction baselines by a wide margin.

Seq	Select+Fit	KVSculpt	Improvement
0 (easy)	$1.28\mathrm{e}{\text{-}1}$	$8.1\mathrm{e}{\text{-}3}$	$15.8\times$
1 (hard)	$1.39\mathrm{e}{\text{-}1}$	$1.17\mathrm{e}{\text{-}1}$	$1.2\times$
2 (hard)	$2.61\mathrm{e}{\text{-}1}$	$1.43\mathrm{e}{\text{-}1}$	$1.8\times$
3 (easy)	$3.92\mathrm{e}{\text{-}1}$	$9.7\mathrm{e}{\text{-}3}$	$40.5\times$
4 (easy)	$2.46\mathrm{e}{\text{-}1}$	$9.6\mathrm{e}{\text{-}3}$	$25.5\times$
Mean	$2.33\mathrm{e}{\text{-}1}$	$5.75\mathrm{e}{\text{-}2}$	$4.1\times$

Table 2: Per-sequence KL at

r{=}0.3

. KVSculpt achieves near-lossless compression on easy sequences and remains superior on hard ones.

Layer	Adam	L-BFGS	Oracle	Gap
L2	$1.78\mathrm{e}{\text{-}3}$	$1.05\mathrm{e}{\text{-}4}$	$9.67\mathrm{e}{\text{-}5}$	$8\%$
L14	$7.97\mathrm{e}{\text{-}2}$	$1.28\mathrm{e}{\text{-}3}$	$1.20\mathrm{e}{\text{-}3}$	$7\%$
L26	$1.14\mathrm{e}{\text{-}1}$	$1.20\mathrm{e}{\text{-}3}$	$1.18\mathrm{e}{\text{-}3}$	$2\%$

Table 3: Per-layer MSE (

r{=}0.3

). L-BFGS (1 restart) is

17

–

95\times

better than Adam and within 2–8% of the 100-restart oracle, indicating near-optimal per-layer solutions.

Strategy	Near attn cos	Far attn cos
Bootstrap (last 128)	0.950	0.737
Uniform spread	0.948	0.759
Random sample	0.948	0.756
$k$ -means centroids	0.947	0.756
Farthest-point	0.925	0.758

Table 4: Query sampling strategies. Uniform spread provides the best overall trade-off, especially at far horizons.

Seq	Uniform KL	$\\|\mathbf{V}\\|^{0.5}$ vs. Uni.
0	$8.1\mathrm{e}{\text{-}3}$	$1.05\times$
1	$1.17\mathrm{e}{\text{-}1}$	$\mathbf{0.75\times}$
2	$1.43\mathrm{e}{\text{-}1}$	$1.73\times$
3	$9.7\mathrm{e}{\text{-}3}$	$0.84\times$
4	$9.6\mathrm{e}{\text{-}3}$	$1.23\times$
Mean	$5.75\mathrm{e}{\text{-}2}$	$1.27\times$

Table 5: Static allocation (

\|\mathbf{V}\|^{0.5}

) is unstable across sequences. Multipliers are relative to uniform allocation (

r{=}0.3

Seq	Uniform	Pilot-MSE 0.5	vs. Uniform
0	$8.14\mathrm{e}{\text{-}3}$	$8.93\mathrm{e}{\text{-}3}$	$1.10\times$
1	$1.170\mathrm{e}{\text{-}1}$	$7.01\mathrm{e}{\text{-}2}$	$\mathbf{0.60\times}$
2	$1.433\mathrm{e}{\text{-}1}$	$1.208\mathrm{e}{\text{-}1}$	$\mathbf{0.84\times}$
3	$9.67\mathrm{e}{\text{-}3}$	$8.64\mathrm{e}{\text{-}3}$	$\mathbf{0.89\times}$
4	$9.59\mathrm{e}{\text{-}3}$	$7.77\mathrm{e}{\text{-}3}$	$\mathbf{0.81\times}$
Mean	$5.75\mathrm{e}{\text{-}2}$	$4.31\mathrm{e}{\text{-}2}$	$\mathbf{0.75\times}$

Table 6: Pilot-MSE 0.5 allocation wins 4/5 sequences with 25% mean KL reduction (

r{=}0.3

Seq	Mean ratio	Max ratio	Layer
0	$21.5\times$	$467\times$	L0
1	$5.5\times$	$31\times$	L15
2	$6.1\times$	$33\times$	L15
3	$3.7\times$	$17\times$	L15
4	$6.2\times$	$56\times$	L15

Table 7: Head MSE asymmetry: mean and max ratio of per-head pilot MSE within the same layer (

r{=}0.3

Seq	Per-layer	Per-layer+head	$\Delta$
0	$0.82\times$	$\mathbf{0.81\times}$	$-1\%$
1	$1.37\times$	$\mathbf{1.15\times}$	$-16\%$
2	$0.88\times$	$\mathbf{0.84\times}$	$-5\%$
3	$0.87\times$	$\mathbf{0.86\times}$	$-1\%$
4	$1.77\times$	$\mathbf{0.75\times}$	$-58\%$
Mean	$1.05\times$	$\mathbf{0.92\times}$	$-8\%$

Table 8: Per-head allocation always improves over per-layer alone. All numbers relative to uniform (

r{=}0.3

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Method	$r{=}0.1$	$r{=}0.3$	$r{=}0.5$	$r{=}0.7$
Random	$3.43{\pm}1.21$	$2.60{\pm}0.45$	$2.06{\pm}0.62$	$1.83{\pm}0.57$
Attn Score	$.322{\pm}.105$	$.198{\pm}.084$	$.137{\pm}.065$	$.086{\pm}.056$
Select+Fit	$.322{\pm}.107$	$.185{\pm}.076$	$.129{\pm}.057$	$.080{\pm}.050$
Joint Opt	$.311{\pm}.113$	$.182{\pm}.073$	$.121{\pm}.058$	$.073{\pm}.047$

Appendix

Table 9: KL divergence (mean

\pm

std, 20 sequences) at representative ratios. Joint Opt marginally improves over Select+Fit; both are far behind KVSculpt .

$\alpha$	KL	vs. Uniform	$k_{\text{min}}$	$k_{\text{max}}$
0 (uniform)	$1.170\mathrm{e}{\text{-}1}$	ref	358	358
0.3	$1.175\mathrm{e}{\text{-}1}$	$1.00\times$	277	480
0.5	$\mathbf{8.83\mathrm{e}{\text{-}2}}$	$\mathbf{0.75\times}$	231	578
0.7	$1.328\mathrm{e}{\text{-}1}$	$1.14\times$	191	689
1.0	$9.27\mathrm{e}{\text{-}2}$	$0.79\times$	140	880

Appendix

Table 10: Dampening exponent

\alpha

for

\|\mathbf{V}\|^{\alpha}

per-layer allocation (Seq 1,

r{=}0.3

\alpha{=}0.5

is optimal; higher values over-allocate to outlier layers.

KV-Cache Management

Key-value (KV) cache management reduces and coordinates Transformer attention state to make long-context inference faster and more memory-efficient.

See all 193 papers

Jun 13, 2026

Exploring a Layer-Wise Design Space for KV Cache Eviction

KV cache eviction methods typically use a single retention-rule family throughout a model, making eviction-method identity a model-level design choice. Yet Transformer layers differ substantially in their attention behavior, representations, and sensitivity to compression, suggesting that a uniform rule may overlook useful layer-wise structure. This raises a basic question: should eviction methods themselves vary across layers? We investigate this question by composing existing eviction methods across Transformer layers and systematically exploring the resulting layer-wise design space. Using simple offline profiles, we construct fixed routes and study how their quality varies with method placement and cache budget. On LongBench, heterogeneous routing improves performance on a majority of tasks over homogeneous policies at the same cache budget. Even when method counts are held fixed, the profile-guided placement ranks second among 100 evaluated assignments, demonstrating that routing quality depends strongly on where methods are placed. Moreover, the same fixed route outperforms the best of nine standalone baselines across all five tested cache budgets. Together, these results establish layer-wise method composition as an exploitable, placement-sensitive design dimension for KV cache compression.

Chao Fei, Kaihua Liang, Hanzhi Hu +4

King Abdullah University of Science and Technology · Fudan University

KV Caching KV-Cache Eviction KV-Cache Compression

Ask PDF

Mar 24, 2026

EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction

The increasing memory demand of the Key-Value (KV) cache poses a significant bottleneck for Large Language Models (LLMs) in long-context applications. Existing low-rank KV compression methods reduce this footprint by modifying model projections, limiting the flexibility to switch back to standard full-cache inference when sufficient memory is available. In this paper, we propose EchoKV, a flexible KV cache compression framework that supports on-demand transitions from full KV caching to compressed caching. Unlike traditional compression-decompression paradigms, EchoKV utilizes a lightweight network to reconstruct the discarded KV components from a partial subset, exploiting intrinsic inter-layer and intra-layer similarities among attention heads. We further introduce a lightweight two-stage fine-tuning strategy, requiring only a few minutes on a single A100 GPU for a 7B model. Experimental results on LongBench and RULER demonstrate that EchoKV consistently outperforms existing methods across multiple compression ratios and backbone models while preserving the throughput of full-cache inference in short-context scenarios.

Shiyu Ji, Yixuan Wang, Yijun Liu +2

Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, China

KV-Cache Compression LLM Inference Acceleration

Ask PDF

Apr 22, 2026

LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction

Long-context inference in Large Language Models (LLMs) is bottlenecked by the linear growth of Key-Value (KV) cache memory. Existing KV cache compression paradigms are fundamentally limited by heuristics: heuristic budgeting relies on statistical priors rather than task objectives, causing resource misallocation, while heuristic selection relies on coupled query-key interactions or static inductive biases (e.g., attention sinks). To address this limitation, we introduce LKV (Learned KV Eviction), which formulates KV compression as an end-to-end differentiable optimization problem. LKV integrates LKV-H to learn task-optimized global budgets, and LKV-T to derive intrinsic KV importance without materializing attention matrices. This design bypasses heuristic proxies, strictly aligning compression with task objectives. Extensive evaluations demonstrate that LKV achieves state-of-the-art performance on both LongBench and RULER benchmarks at high compression rates. In particular, on LongBench, LKV achieves near-lossless performance with only 15% KV cache retention. Crucially, our analysis identifies learned budgeting as the dominant driver of fidelity, demonstrating that data-driven allocation is essential to overcome the limitations of hand-crafted heuristics.

Enshuai Zhou, Yifan Hao, Chao Wang +7

University of Science and Technology of China, Hefei, China · State Key Lab of Processors, Institute of Computing Technology, CAS, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

KV Caching Long-Context Language Model Inference Memory-Efficient Optimization KV-Cache Eviction KV-Cache Management KV-Cache Compression

Ask PDF

Abstract

KV-Cache Management

Explore similar work

Exploring a Layer-Wise Design Space for KV Cache Eviction

EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction

LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction