KVSculpt: KV Cache Compression as Distillation

Bo Jiang; Sian Jin

cs.LGMar 29, 2026

KVSculpt: KV Cache Compression as Distillation

Authors: Bo Jiang, Sian Jin

Organizations: Temple University

Abstract

KV cache compression is critical for efficient long-context LLM inference. Approaches that reduce the per-pair footprint -- quantization and low-rank decomposition -- are orthogonal to those that reduce the sequence length of the cache. Along the sequence-length dimension, existing methods range from pure eviction -- selecting which KV pairs to keep -- to merging, which combines similar pairs into fewer ones. Both remain anchored to the original cache entries. We propose KVSculpt, which moves to the other end of this spectrum: instead of selecting or combining original pairs, we optimize a smaller set of unconstrained KV pairs in continuous embedding space to preserve each layer's attention behavior. Keys are optimized via L-BFGS and values are solved in closed form via least squares, alternating every few steps. On top of this, we introduce adaptive budget allocation, which uses a cheap pilot compression run to redistribute the compression budget across layers and KV heads based on per-component difficulty. On Qwen2.5-1.5B-Instruct with 2048-token contexts, KVSculpt reduces KL divergence by 3.5-4.1x compared to Select+Fit -- attention-score eviction with least-squares value fitting -- across compression ratios r in {0.3, 0.5, 0.7}. Adaptive allocation provides an additional 1.3x KL reduction at no extra inference cost. Analysis reveals that compression difficulty is highly non-uniform: per-layer pilot MSE varies by up to 100x across layers, and the two KV heads within a single layer can differ by up to 467x -- demonstrating that fine-grained budget allocation is essential.

Figures & tables

Panel of Figure 1 — Figure 1: Three paradigms for KV cache sequence-length reduction. (a) Eviction selects a subset of original KV pairs. (b) Merge combines similar pairs, modifying values but remaining anchored to original positions. (c) KVSculpt distills the compress zone into $k$ unconstrained pairs (orange) freely optimized in $\mathbb{R}^{d}$ via L-BFGS (keys) and least squares (values). All three keep the retain zone (green) intact. The key distinction is the degree of freedom: from discrete subset to weighted combination to fully continuous optimization.

Method	$r{=}0.3$	$r{=}0.5$	$r{=}0.7$
Random	$2.50$	$1.80$	$1.54$
Attn Score	$2.54\mathrm{e}{\text{-}1}$	$2.04\mathrm{e}{\text{-}1}$	$1.39\mathrm{e}{\text{-}1}$
Select+Fit	$2.33\mathrm{e}{\text{-}1}$	$1.86\mathrm{e}{\text{-}1}$	$1.25\mathrm{e}{\text{-}1}$
Joint Opt	$2.24\mathrm{e}{\text{-}1}$	$1.80\mathrm{e}{\text{-}1}$	$1.15\mathrm{e}{\text{-}1}$
KVSculpt	$\mathbf{5.75\mathrm{e}{\text{-}2}}$	$\mathbf{4.63\mathrm{e}{\text{-}2}}$	$\mathbf{3.58\mathrm{e}{\text{-}2}}$
vs. Select+Fit	$4.1\times$	$4.0\times$	$3.5\times$

Table 1: KL divergence (

\downarrow

) on 5 sequences (mean). KVSculpt consistently outperforms all eviction baselines by a wide margin.

Seq	Select+Fit	KVSculpt	Improvement
0 (easy)	$1.28\mathrm{e}{\text{-}1}$	$8.1\mathrm{e}{\text{-}3}$	$15.8\times$
1 (hard)	$1.39\mathrm{e}{\text{-}1}$	$1.17\mathrm{e}{\text{-}1}$	$1.2\times$
2 (hard)	$2.61\mathrm{e}{\text{-}1}$	$1.43\mathrm{e}{\text{-}1}$	$1.8\times$
3 (easy)	$3.92\mathrm{e}{\text{-}1}$	$9.7\mathrm{e}{\text{-}3}$	$40.5\times$
4 (easy)	$2.46\mathrm{e}{\text{-}1}$	$9.6\mathrm{e}{\text{-}3}$	$25.5\times$
Mean	$2.33\mathrm{e}{\text{-}1}$	$5.75\mathrm{e}{\text{-}2}$	$4.1\times$

Table 2: Per-sequence KL at

r{=}0.3

. KVSculpt achieves near-lossless compression on easy sequences and remains superior on hard ones.

Layer	Adam	L-BFGS	Oracle	Gap
L2	$1.78\mathrm{e}{\text{-}3}$	$1.05\mathrm{e}{\text{-}4}$	$9.67\mathrm{e}{\text{-}5}$	$8\%$
L14	$7.97\mathrm{e}{\text{-}2}$	$1.28\mathrm{e}{\text{-}3}$	$1.20\mathrm{e}{\text{-}3}$	$7\%$
L26	$1.14\mathrm{e}{\text{-}1}$	$1.20\mathrm{e}{\text{-}3}$	$1.18\mathrm{e}{\text{-}3}$	$2\%$

Table 3: Per-layer MSE (

r{=}0.3

). L-BFGS (1 restart) is

17

–

95\times

better than Adam and within 2–8% of the 100-restart oracle, indicating near-optimal per-layer solutions.

Strategy	Near attn cos	Far attn cos
Bootstrap (last 128)	0.950	0.737
Uniform spread	0.948	0.759
Random sample	0.948	0.756
$k$ -means centroids	0.947	0.756
Farthest-point	0.925	0.758

Table 4: Query sampling strategies. Uniform spread provides the best overall trade-off, especially at far horizons.

Seq	Uniform KL	$\\|\mathbf{V}\\|^{0.5}$ vs. Uni.
0	$8.1\mathrm{e}{\text{-}3}$	$1.05\times$
1	$1.17\mathrm{e}{\text{-}1}$	$\mathbf{0.75\times}$
2	$1.43\mathrm{e}{\text{-}1}$	$1.73\times$
3	$9.7\mathrm{e}{\text{-}3}$	$0.84\times$
4	$9.6\mathrm{e}{\text{-}3}$	$1.23\times$
Mean	$5.75\mathrm{e}{\text{-}2}$	$1.27\times$

Table 5: Static allocation (

\|\mathbf{V}\|^{0.5}

) is unstable across sequences. Multipliers are relative to uniform allocation (

r{=}0.3

Seq	Uniform	Pilot-MSE 0.5	vs. Uniform
0	$8.14\mathrm{e}{\text{-}3}$	$8.93\mathrm{e}{\text{-}3}$	$1.10\times$
1	$1.170\mathrm{e}{\text{-}1}$	$7.01\mathrm{e}{\text{-}2}$	$\mathbf{0.60\times}$
2	$1.433\mathrm{e}{\text{-}1}$	$1.208\mathrm{e}{\text{-}1}$	$\mathbf{0.84\times}$
3	$9.67\mathrm{e}{\text{-}3}$	$8.64\mathrm{e}{\text{-}3}$	$\mathbf{0.89\times}$
4	$9.59\mathrm{e}{\text{-}3}$	$7.77\mathrm{e}{\text{-}3}$	$\mathbf{0.81\times}$
Mean	$5.75\mathrm{e}{\text{-}2}$	$4.31\mathrm{e}{\text{-}2}$	$\mathbf{0.75\times}$

Table 6: Pilot-MSE 0.5 allocation wins 4/5 sequences with 25% mean KL reduction (

r{=}0.3

Seq	Mean ratio	Max ratio	Layer
0	$21.5\times$	$467\times$	L0
1	$5.5\times$	$31\times$	L15
2	$6.1\times$	$33\times$	L15
3	$3.7\times$	$17\times$	L15
4	$6.2\times$	$56\times$	L15

Table 7: Head MSE asymmetry: mean and max ratio of per-head pilot MSE within the same layer (

r{=}0.3

Seq	Per-layer	Per-layer+head	$\Delta$
0	$0.82\times$	$\mathbf{0.81\times}$	$-1\%$
1	$1.37\times$	$\mathbf{1.15\times}$	$-16\%$
2	$0.88\times$	$\mathbf{0.84\times}$	$-5\%$
3	$0.87\times$	$\mathbf{0.86\times}$	$-1\%$
4	$1.77\times$	$\mathbf{0.75\times}$	$-58\%$
Mean	$1.05\times$	$\mathbf{0.92\times}$	$-8\%$

Table 8: Per-head allocation always improves over per-layer alone. All numbers relative to uniform (

r{=}0.3

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Method	$r{=}0.1$	$r{=}0.3$	$r{=}0.5$	$r{=}0.7$
Random	$3.43{\pm}1.21$	$2.60{\pm}0.45$	$2.06{\pm}0.62$	$1.83{\pm}0.57$
Attn Score	$.322{\pm}.105$	$.198{\pm}.084$	$.137{\pm}.065$	$.086{\pm}.056$
Select+Fit	$.322{\pm}.107$	$.185{\pm}.076$	$.129{\pm}.057$	$.080{\pm}.050$
Joint Opt	$.311{\pm}.113$	$.182{\pm}.073$	$.121{\pm}.058$	$.073{\pm}.047$

Appendix

Table 9: KL divergence (mean

\pm

std, 20 sequences) at representative ratios. Joint Opt marginally improves over Select+Fit; both are far behind KVSculpt .

$\alpha$	KL	vs. Uniform	$k_{\text{min}}$	$k_{\text{max}}$
0 (uniform)	$1.170\mathrm{e}{\text{-}1}$	ref	358	358
0.3	$1.175\mathrm{e}{\text{-}1}$	$1.00\times$	277	480
0.5	$\mathbf{8.83\mathrm{e}{\text{-}2}}$	$\mathbf{0.75\times}$	231	578
0.7	$1.328\mathrm{e}{\text{-}1}$	$1.14\times$	191	689
1.0	$9.27\mathrm{e}{\text{-}2}$	$0.79\times$	140	880

Appendix

Table 10: Dampening exponent

\alpha

for

\|\mathbf{V}\|^{\alpha}

per-layer allocation (Seq 1,

r{=}0.3

\alpha{=}0.5

is optimal; higher values over-allocate to outlier layers.

KV-Cache Management

Key-value (KV) cache management reduces and coordinates Transformer attention state to make long-context inference faster and more memory-efficient.

See all 193 papers

Exploring a Layer-Wise Design Space for KV Cache Eviction
Jun 13, 2026Chao Fei, Kaihua Liang, Hanzhi Hu +4KV Caching KV-Cache Eviction
King Abdullah University of Science and Technology · Fudan University
EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction
Mar 24, 2026Shiyu Ji, Yixuan Wang, Yijun Liu +2KV-Cache Compression LLM Inference Acceleration
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, China
LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction
Apr 22, 2026Enshuai Zhou, Yifan Hao, Chao Wang +7KV Caching Long-Context Language Model Inference
University of Science and Technology of China, Hefei, China · State Key Lab of Processors, Institute of Computing Technology, CAS, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Jun 13, 2026J/K move · Enter open · S save

Exploring a Layer-Wise Design Space for KV Cache Eviction

Chao Fei, Kaihua Liang, Hanzhi Hu, Hongcheng Guo+3

King Abdullah University of Science and Technology · Fudan University

KV Caching KV-Cache Eviction KV-Cache Compression

Open paper AskPDF arXiv

KV cache eviction methods typically use a single retention-rule family throughout a model, making eviction-method identity a model-level design choice. Yet Transformer layers differ substantially in their attention behavior, representations, and sensitivity to compression, suggesting that a uniform rule may overlook useful layer-wise structure. This raises a basic question: should eviction methods themselves vary across layers? We investigate this question by composing existing eviction methods across Transformer layers and systematically exploring the resulting layer-wise design space. Using simple offline profiles, we construct fixed routes and study how their quality varies with method placement and cache budget. On LongBench, heterogeneous routing improves performance on a majority of tasks over homogeneous policies at the same cache budget. Even when method counts are held fixed, the profile-guided placement ranks second among 100 evaluated assignments, demonstrating that routing quality depends strongly on where methods are placed. Moreover, the same fixed route outperforms the best of nine standalone baselines across all five tested cache budgets. Together, these results establish layer-wise method composition as an exploitable, placement-sensitive design dimension for KV cache compression.

Abstract

KV-Cache Management

Explore similar work

Exploring a Layer-Wise Design Space for KV Cache Eviction

EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction

LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction