Beyond Sparse Weights: When Is Attention Compressible?

Chiwun Yang; Xiaoyu Li

cs.LGAug 21, 2026

Beyond Sparse Weights: When Is Attention Compressible?

Authors: Chiwun Yang, Xiaoyu Li

Organizations: City University of Hong Kong · University of New South Wales

Abstract

KV-cache compression is often justified by attention maps with a few large weights. This is incomplete: large weights may not contain most of the mass, omitted values can cancel, and preserving the attention output may not preserve the task. We separate these questions. Global score gaps -- not threshold counts -- determine how many tokens are needed to retain a target mass. For a realized row, the weighted sum of omitted values is the exact missing statistic. A controlled retrieval--aggregation model explains when truncation helps and when it hurts. These results motivate CertKV, a training-free compressor that reserves one tail-summary slot per head and allocates the rest by value dispersion. Under matched budgets, CertKV is top-two in seven of nine LongBench-v2 settings, remains in the leading compressed tier on 128K RULER, and realizes a ten-fold cache budget in a packed Llama prototype. Compressibility depends on the mass, values, future queries, and task -- not on a sparse-looking map alone.

Figures & tables

Panel of Figure 1 — Figure 1: Finite-range count sparsity does not yield a transferable mass budget. (a) Within fixed head types, $\hat{k}(0.1A_{(1)})$ grows sublinearly over the measured lengths. (b) A profile that beats a shuffled control on five of six anchors at fixed length still misses the measured $128$ K budget by $>1.5\times$ when fitted at $4$ K. (c) $A_{(1)}$ can fall while normalized count stays bounded. Intervals are 95% document bootstraps ( $B=5000$ ); (b,c) use an equal-model median. Gemma is a $4$ K descriptive point.

Panel of Figure 2 — Figure 2: Task utility depends on regime. (a) Exact and top- $k$ exchange order at $\alpha^{\star}=0.50$ in the controlled mixture; the real-task observable does not predict a change (1500 trials/ $\alpha$ , $B=5000$ , 95% trial bootstrap). (b) Five-seed paired- $t$ differences are $0.10{\pm}0.08$ at $4\times$ and $0.17{\pm}0.13$ at $8\times$ ; $16/32\times$ include zero. This test is separate from the controlled theorem.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Cross-modal clean two-phase universality is not established. Per-input/layer/head gap-slope estimates are shown on a log axis; the dashed line is the unit-slope boundary and dotted lines are descriptive two-Gaussian means. Neither modality meets the clean two-mode criterion because the fitted mixtures have no sufficiently deep between-mean trough. Brackets are

95\%

percentile sensitivity intervals from resampling the four structured inputs (seed

0

B=5000

). Exact Hartigan dip statistics are reported without iid head-level

p

-values.

Figure 4: Count-sublinearity on real books, with resampling intervals (real Project Gutenberg ; fraction of heads with sublinear large-weight count). Error bars are two-sided

95\%

Student-

t

intervals over six document/position resampling replicates (seed base

0

, eight documents), not six independent training runs.

Figure 5: Real hidden states sit outside the clean distributional regimes ( Sections 3 and 3.6 ; three architectures). (a,b) Effective rank and excess kurtosis, with seed-

0

B=5000

95\%

corpus-bootstrap bands over three trained corpora. (c) Condensed-head fraction with corpus-then-head bootstrap; the single-corpus untrained condition is omitted, and identical PG19/code inputs are not treated as independent corpora.

Figure 6: Sparsity accompanies learned alignment in a controlled from-scratch replication ( Qwen3 -architecture byte-LM). (a) Median kept fraction, real versus shuffled text; (b) next-token accuracy. Curves show mean

{\pm}

sample standard deviation over seeds

0,1,2

; the single-checkpoint legacy control is omitted.

Figure 7: Per-head distributions by static type. Left/middle: violins show the finite observed distributions of per-head count

\hat{k}

and top weight

A_{(1)}

at the longest context; black points and bars are the model-balanced center and seed-

0

B=5000

95\%

percentile document-bootstrap interval. Right: head-type composition per anchor with the same document-bootstrap intervals. The types summarize static profiles and should not be read as functional head labels.

Figure 8: Sparse-head growth recurs in two controlled architectures. Sparse-head fraction (kept-fraction

\rho_{\eta=0.1}\leq 0.1

) versus training step for two from-scratch byte-level language models ( GPT-NeoX -architecture,

3.22

M parameters; Qwen3 -architecture,

4.26

M parameters; three seeds, solid

=

mean, band

=\pm

sample standard deviation). On real text both trajectories cross the

15\%

guide ( GPT-NeoX to

{\sim}93\%

, Qwen3 to

{\sim}75\%

of heads sparse); the i.i.d.-random-byte controls (dashed) are nonzero but remain lower.

Method	Persistent KV	Decode peak
Full cache ( $1\times$ )	$8.59$ GB	$23.11$ GiB
CertKV ( $10\times$ )	$0.86$ GB	$16.34$ GiB

Appendix

Table 6: Physical memory relative to the full cache. The

64

K CertKV row uses the exact

10\times

live-slot budget.

Budget	Method	Single-Doc	Multi-Doc	ICL	Dialogue	Code	Structured	Avg. (503-wtd.)
$2\times$	Full cache	$30.3{\pm}6.9$	$32.0{\pm}8.0$	$23.5{\pm}9.3$	$15.4{\pm}11.5$	$30.0{\pm}12.0$	$6.1{\pm}7.6$	$26.8{\pm}3.9$
$2\times$	SnapKV	$\mathbf{34.3{\pm}6.9}$	$28.0{\pm}8.0$	$\mathbf{22.2{\pm}9.3}$	$\underline{15.4{\pm}11.5}$	$\underline{28.0{\pm}12.0}$	$\underline{9.1{\pm}10.6}$	$\mathbf{27.0{\pm}3.9}$
$2\times$	StreamingLLM	$32.6{\pm}6.9$	$\underline{28.8{\pm}8.0}$	$19.8{\pm}8.6$	$\mathbf{17.9{\pm}11.5}$	$\underline{28.0{\pm}12.0}$	$\mathbf{18.2{\pm}13.6}$	$\mathbf{27.0{\pm}3.9}$
$2\times$	PyramidKV	$20.6{\pm}6.0$	$19.2{\pm}6.8$	$19.8{\pm}8.6$	$\mathbf{17.9{\pm}11.5}$	$4.0{\pm}5.0$	$\mathbf{18.2{\pm}13.6}$	$18.1{\pm}3.3$
$2\times$	Ada-KV	$\underline{33.1{\pm}6.9}$	$\mathbf{29.6{\pm}8.0}$	$\underline{21.0{\pm}8.6}$	$\underline{15.4{\pm}11.5}$	$\mathbf{30.0{\pm}12.0}$	$3.0{\pm}4.5$	$26.6{\pm}3.9$
$2\times$	CertKV	$\underline{33.1{\pm}6.9}$	$\mathbf{29.6{\pm}8.0}$	$\mathbf{22.2{\pm}9.3}$	$\underline{15.4{\pm}11.5}$	$\mathbf{30.0{\pm}12.0}$	$3.0{\pm}4.5$	$\underline{26.8{\pm}3.9}$

Appendix

Table 7: Llama-3.1-8B LongBench-v2 by domain. Accuracy (%) is estimate

{\pm}

95% within-domain bootstrap half-width; Avg. is 503-weighted.

Budget	Method	Single-Doc	Multi-Doc	ICL	Dialogue	Code	Structured	Avg. (503-wtd.)
$2\times$	Full cache	$32.6{\pm}6.9$	$34.4{\pm}8.8$	$32.1{\pm}9.9$	$33.3{\pm}15.4$	$34.0{\pm}13.0$	$33.3{\pm}15.2$	$33.2{\pm}4.1$
$2\times$	SnapKV	$\mathbf{34.3{\pm}6.9}$	$\mathbf{33.6{\pm}8.8}$	$27.2{\pm}9.9$	$\underline{28.2{\pm}14.1}$	$\mathbf{34.0{\pm}13.0}$	$\underline{33.3{\pm}15.2}$	$\mathbf{32.4{\pm}4.1}$
$2\times$	StreamingLLM	$\underline{33.7{\pm}6.9}$	$28.0{\pm}8.0$	$\mathbf{29.6{\pm}9.9}$	$12.8{\pm}10.3$	$\underline{30.0{\pm}12.0}$	$\underline{33.3{\pm}15.2}$	$29.6{\pm}4.0$
$2\times$	PyramidKV	$27.4{\pm}6.9$	$26.4{\pm}8.0$	$\underline{28.4{\pm}9.9}$	$23.1{\pm}12.8$	$26.0{\pm}12.0$	$\mathbf{45.5{\pm}18.2}$	$28.0{\pm}3.9$
$2\times$	Ada-KV	$32.0{\pm}6.9$	$\underline{30.4{\pm}8.0}$	$\mathbf{29.6{\pm}9.9}$	$\mathbf{33.3{\pm}15.4}$	$\mathbf{34.0{\pm}13.0}$	$\underline{33.3{\pm}15.2}$	$\underline{31.6{\pm}4.1}$
$2\times$	CertKV	$33.1{\pm}6.9$	$28.8{\pm}8.0$	$\underline{28.4{\pm}9.9}$	$\underline{28.2{\pm}14.1}$	$\mathbf{34.0{\pm}13.0}$	$\underline{33.3{\pm}15.2}$	$31.0{\pm}4.1$

Appendix

Table 8: Qwen3-8B LongBench-v2 by domain. Accuracy (%) is estimate

{\pm}

95% within-domain bootstrap half-width; Avg. is 503-weighted.

Budget	Method	Single-Doc	Multi-Doc	ICL	Dialogue	Code	Structured	Avg. (503-wtd.)
$2\times$	Full cache	$29.1{\pm}6.9$	$30.4{\pm}8.0$	$29.6{\pm}9.9$	$28.2{\pm}14.1$	$18.0{\pm}11.0$	$24.2{\pm}15.2$	$28.0{\pm}3.9$
$2\times$	SnapKV	$30.9{\pm}6.9$	$\underline{29.6{\pm}8.0}$	$25.9{\pm}9.9$	$\underline{23.1{\pm}12.8}$	$\underline{20.0{\pm}11.0}$	$18.2{\pm}13.6$	$27.2{\pm}3.9$
$2\times$	StreamingLLM	$\underline{32.6{\pm}6.9}$	$\mathbf{30.4{\pm}8.0}$	$\mathbf{29.6{\pm}9.9}$	$\mathbf{25.6{\pm}14.1}$	$18.0{\pm}11.0$	$\underline{21.2{\pm}13.6}$	$\mathbf{28.8{\pm}3.9}$
$2\times$	PyramidKV	$21.7{\pm}6.3$	$26.4{\pm}8.0$	$24.7{\pm}9.3$	$12.8{\pm}10.3$	$\mathbf{24.0{\pm}12.0}$	$18.2{\pm}13.6$	$22.7{\pm}3.7$
$2\times$	Ada-KV	$31.4{\pm}6.9$	$\mathbf{30.4{\pm}8.0}$	$\underline{27.2{\pm}9.9}$	$20.5{\pm}12.8$	$\underline{20.0{\pm}11.0}$	$\underline{21.2{\pm}13.6}$	$27.8{\pm}3.9$
$2\times$	CertKV	$\mathbf{33.1{\pm}6.9}$	$\underline{29.6{\pm}8.0}$	$\underline{27.2{\pm}9.9}$	$17.9{\pm}11.5$	$18.0{\pm}11.0$	$\mathbf{24.2{\pm}15.2}$	$\underline{28.0{\pm}3.9}$

Appendix

Table 9: Mistral-7B LongBench-v2 by domain. Accuracy (%) is estimate

{\pm}

95% within-domain bootstrap half-width; Avg. is 503-weighted.

KV-Cache Compression

Key-value cache compression reduces the memory and bandwidth needed to retain attention states during long-context inference.

See all 257 papers

ChampionKV: Head-Wise Adaptive Budget Allocation for KV Cache Compression
Oct 13, 2026Yuechuan Chen, Yanda Guo, Xiaoxi Liu +4KV-Cache Compression KV-Cache Management
Sun Yat-sen University · The Chinese University of Hong Kong · University of Electronic Science and Technology of China +2
ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression
Jul 31, 2026Yuhang Zhan, Lisi Chen, Shuo ShangSoftmax Attention KV Caching
University of Electronic Science and Technology of China
KV-COBRA: KV Cache Compression via Co-Optimized Bit-Rank Allocation
Sep 21, 2026Sihyeon Ha, Jaeho Lee, Yo-Seb JeonLow-Rank Compression KV-Cache Compression
Pohang University of Science and Technology (POSTECH)

Oct 13, 2026J/K move · Enter open · S save

ChampionKV: Head-Wise Adaptive Budget Allocation for KV Cache Compression

Yuechuan Chen, Yanda Guo, Xiaoxi Liu, Weilin Ruan+3

Sun Yat-sen University · The Chinese University of Hong Kong · University of Electronic Science and Technology of China+2

KV-Cache Compression KV-Cache Management Long-Context Language Modeling

Open paper AskPDF arXiv

Some observation-window-based KV cache compressors use prompt attention to identify important tokens but allocate uniform or predefined budgets across heads, overlooking input-dependent differences in budget demand. We propose ChampionKV, an input-adaptive head-wise budget allocator for long-context KV cache compression. It uses the maximum aggregated observation-window attention score as a lightweight empirical proxy for attention concentration, assigning larger budgets to heads with dispersed attention and smaller budgets to concentrated heads. An abnormal-head mask and a floor-budget mechanism improve robustness under weak attention responses. ChampionKV plugs into compressors such as SnapKV and PyramidKV without changing their token-selection procedures. On LongBench and RULER, both Champion variants consistently improve over their SnapKV and PyramidKV base compressors; the improvements are largest and statistically significant on Llama3.1-70B-Instruct, especially under tight budgets, whereas results on the tested 7B/8B models remain competitive but mixed. ChampionKV does not uniformly outperform Ada-KV, whose global cross-head Top-K selection retains an advantage in several settings. Controlled analysis further shows improved retained attention mass over the corresponding base compressors at negligible overhead.

Abstract

KV-Cache Compression

Explore similar work

ChampionKV: Head-Wise Adaptive Budget Allocation for KV Cache Compression

ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression

KV-COBRA: KV Cache Compression via Co-Optimized Bit-Rank Allocation