As context windows in LLMs scale to 100K+ tokens, the key-value (KV) cache becomes the dominant memory bottleneck, with recent methods claiming 80-90% savings and minimal benchmark degradation. We argue these evaluations miss a structural issue: attention is not just storage but routing, and retaining KV pairs does not guarantee semantic accessibility. We propose a physics-inspired view of KV compression as a controlled perturbation of token-level routing, distinguishing retention, accessibility, and utilization. Using synthetic tasks probing multi-entity tracking, disambiguation, coreference, and multi-hop reasoning, we find that moderate compression degrades internal representations with little accuracy loss, revealing redundancy; all models exhibit a sharp hallucination safety cliff near 90% compression, correlated with spikes in Global Eviction Ratio (GER), suggesting a phase transition in semantic reachability; and architectures differ in routing dynamics, with LLaMA showing early consensus and late diversification, and Qwen showing funnel-like late convergence, leading to distinct resilience profiles. Beyond erasure, we identify representational rigidity, where excessive head-level consensus collapses routing flexibility despite token survival. These results suggest sparse token-route structures govern compression tolerance, reframing KV compression as a structural probe of attention geometry and linking long-context scalability to sparsity and the lottery ticket hypothesis in self-attention.
Figures & tables
Figure 1: Comparison of traditional versus synthetic evaluation frameworks for KV cache compression. Existing benchmarks (left) primarily report coarse-grained task accuracy, whereas our synthetic, controlled framework (right) probes structural reachability, routing collapse, and semantic fragility under compression.
Name
Passage Length (words)
# Queries
Total Passages
Total Queries
LLM Generated
Base task
175 ± 25
6
200
1199
Yes
Knowledge manipulation
32
13
4000
52000
No
Multi presence
500 ± 100
10
100
1000
Yes
Multi entity
500 ± 100
10
100
1000
Yes
Long context
1100 ± 200
10
100
1000
Yes
Coreference
44
9
8000
72000
No
Table 1: Structural summary of the synthetic dataset suite used for controlled KV compression evaluation.
Model
Setup
Base
Know
Multi-P
Multi-E
Long
Coref
Hops
LLaMA-3 8B
AGN
70.05
91.28
79.64
80.33
45.93
75.63
31.54
AWR
69.86
92.09
77.66
81.93
45.54
60.90
31.25
Qwen-2.5 7B
AGN
68.80
92.20
73.63
77.46
38.68
78.89
27.13
AWR
69.63
90.93
71.88
79.38
40.20
52.00
26.35
Table 2: Base (0% compression) F1 scores across datasets.
Figure 2: Base task performance across compression levels. Localized spikes under moderate compression suggest sparse substructure effects. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 27 of Appendix B .
Figure 3: Knowledge manipulation results. Qwen exhibits a more gradual rate of degradation under compression, particularly in the question-aware setting.
Figure 5: Multi presence forward vs. reverse asymmetry.
Figure 6: Multi entity results showing reduced directional asymmetry. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 28 of Appendix B .
Figure 7: Long context degradation across compression levels.
Figure 9: Individual hop breakdown showing instability in intermediate semantic links.
Figure 10: Answer-type tag performance in Base task. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 29 of Appendix B .
Figure 11: Answer-type tag behavior in Multi entity. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 30 of Appendix B .
Figure 13: Question-type tags for Base task. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 31 of Appendix B .
Figure 15: Multi entity question-type behavior shows a similar trend to Base task but with much smoother curves in question agnostic. The question aware setup has a far more jagged curve with distinct points of significant drop off. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 32 of Appendix B .
Figure 16: Consensus across layers. LLaMA exhibits early stabilization and late specialization; Qwen exhibits early exploration and late consolidation. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 33 of Appendix B .
Figure 17: LLaMA layerwise consensus under compression. Early agreement persists, with depth-wise diversification supporting parallel feature channels. Corresponding results for LLaMA-3.2 3B are shown in Figure 34 of Appendix B .
Figure 18: Qwen layerwise consensus under compression. Diversity persists deeper and collapses late, consistent with a consolidation-heavy tail. Corresponding results for Qwen-2.5 3B and Qwen-2.5 14B are shown in Figure 35 of Appendix B .
Figure 19: LLaMA attention heatmaps under compression. Chunk pruning removes contiguous segments; Ada pruning is more irregular. At 0.9, global route destruction dominates. Corresponding results for LLaMA-3.2 3B are shown in Figure 37 of Appendix B .
Figure 20: Qwen attention heatmaps under compression. Policy differences are visible at low compression but are overwhelmed at 0.9 by large-scale pruning. Corresponding results for Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 38 and Figure 39 of Appendix B .
Figure 21: Eviction rates vs. compression. Pruning is broadly question-agnostic unless explicitly conditioned. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 42 of Appendix B .
Figure 22: Error rates vs. compression. Hallucination exhibits a cliff near α≈0.9 , with “unknown” behavior more erratic under AdaKV. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 43 of Appendix B .
Figure 23: Compression susceptibility χ=∂α∂H peaks sharply near α≈0.9 , consistent with a critical transition in the error landscape.
Figure 24: GER correlates strongly with hallucination rate, indicating that global route deletion is a primary driver of catastrophic failure. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 44 of Appendix B .
Figure 25: Base task probing across compression. Some tags remain decodable; others collapse early and may partially recover under question-aware setups. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 40 of Appendix B .
Figure 26: Multi-Entity probing across compression. Question awareness helps selectively rather than uniformly, suggesting interaction between representational geometry and pruning alignment. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 41 of Appendix B .
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Context
Question
Response
Base task
Gonzalo Batistuta, an Argentine forward, is widely regarded as one of the greatest football players of all time…
What is Gonzalo Batistuta’s nationality?
Argentine
Knowledge man.
Cora Delaine was born on December 11, 1954. They studied in Daegu Global Science University…
What is Cora Delaine’s first name?
Cora
Multi presence
The Astralis Spire stands tall as one of the most remarkable achievements of modern architecture and engineering. Rising high above the skyline of its bustling city…
What was the main reason for constructing Astralis Spire?
To improve television and radio broadcasting disrupted by urban development
Multi entity
Harland Kane is one of the most influential authors of contemporary horror, suspense, and supernatural fiction. Over a career spanning decades, Kane has…
Which novel by Kane is set in a haunted hotel?
The Shadow of the Pines
Long context
During the mid-Cretaceous period, the Riverback Sailbacks were formidable semi-aquatic predators, inhabiting river systems, floodplains, and coastal wetlands. Known for their elongated jaws, conical teeth, and large…
Where was Kaelin Vireo from?
The riverine fossil station of Thaloris
Coreference
Julian Foster was born on September 7, 1962 in Aleppo, Aleppo Governorate, Syria. He is male. He studied in Tianjin Harbor University in Podgorica, Podgorica, Montenegro…
What does he work as?
Tour Guide
Appendix
Table 3: Sample contexts and inputs for each task along with the ground truth answer.
Tag
Context
Question
Response
Person
Joao Fernandes, widely considered one of the greatest footballers of all time…
What is the name of the person described in the passage?
Joao Fernandes
Thing
Gonzalo Batistuta, an Argentine forward, is widely regarded as one of the greatest…
What is Gonzalo Batistuta’s nationality?
Argentine
Organization
Mookie Blaylock, a rock band from Northport formed in Riverbend in 1990…
What is the subject of the given passage?
Mookie Blaylock
Creature
The Void Stalker is a fictional endoparasitoid extraterrestrial species that serves as the main…
What kind of species is the Void Stalker?
An endoparasitoid extraterrestrial species
Location
The Stones of Pharaohs are a complex of monumental structures located on the Giza Plateau…
In which city are The Stones of Pharaohs located?
Cairo
Numerals
The Thistlewood Vale Witch Trials represent a chilling episode in early American history…
How many individuals were convicted and executed during the trials?
20
Appendix
Table 4: Examples of answer-type tags.
Tag
Context
Question
Response
Standard
The Basalt Bloom is an immense igneous province, covering a vast expanse of central India. It formed through a series of massive volcanic eruptions…
Where is the Basalt Bloom located?
Central India
Manipulated
Xena Bell was born on November 20, 1968. They studied in Shanghai Maritime University…
What is Xena Bell’s last name?
Bell
Part
Breezeflare is characterized by unusual sounds during breathing caused by airflow obstruction in the airways. The condition is commonly observed…
Nyla Soren is recognized for leading what type of initiatives?
Community workshops, public education
Appendix
Table 5: Examples of question-type tags
Task
Direction
Context
Question
Response
Mutli presence
FWD
The series known as Chrono Nexus is widely regarded as one of the most compelling science fiction narratives in Japanese animation. Created by Itsuki Aoyama…
Who created Chrono Nexus?
Itsuki Aoyama
REV
Few creatures in Earth’s history capture the imagination quite like the Thunderfang Predator, one of the largest and most fearsome carnivorous dinosaurs to have ever lived…
Which predator had serrated teeth as long as a human hand?
Thunderfang Predator
Multi entity
FWD
In the distant waters of the Andaman Islands lies the infamous Iron Shadows Prison. Constructed by the British in the late 19th century, this prison became a…
Who was the nationalist imprisoned in Iron Shadows and known as Veer?
Vinayak Damodar Savarkar
REV
Shinroku is a popular Japanese manga and anime series known for its unique blend of comedy, action, and science fiction. Set in an alternate Edo period where…
Hana Tanaka appears as a companion in which story?
Shinroku
Appendix
Table 6: Examples of forward and reverse questions
Figure 27: Base task performance across compression levels for 3B and 14B models.
Figure 28: Multi entity forward and reverse asymmetry is reduced even when the model size is changed, showing invariance to the size of the model.
Figure 29: Answer-type results for Base task closely mirror the results from the LLaMA 3 8B and Qwen 2.5 7B models, especially in their weak points such as the Events tag.
Figure 30: Answer type tag behaviour in Multi entity for the 3B and 14B models shows similar trends as seen in LLaMA 3 8B and Qwen 2.5 7B.
Figure 31: Base task question-type results continue to show a similar trend with smaller and larger models, as seen in the 7B and 8B models.
Figure 32: Multi entity results for LLaMA 3.2 3B show similar jagged points in question-aware settings, while the performance almost turns into a smooth linear pattern in Qwen 2.5 3B and 14B models.
Figure 33: Consensus scores tend to follow a similar trend with the only major changes being Qwen 3B’s slightly more linear increase rather than the slightly parabolic increase seen in 7B and 14B models.
Figure 34: Layerwise consensus under compression for LLaMA 3.2 3B shows a strikingly similar convergence very early on with depth-wise diversification.
Figure 35: Layerwise consensus under compression for Qwen 2.5 3B and 14B. The 3B model has persistent diversity even after several layers, occasionally failing to converge, while 14B remains stable early on but shows the same funnel-like trend upon convergence.
Figure 36: Compression 50% for LLaMA 3 8B and Qwen 2.5 7B.
Figure 37: Similar smooth pruning in Chunk and irregular removal in Ada is seen in LLaMA 3.2 3B as well.
Figure 38: The non-contiguous nature of pruning is far less diminished in the Qwen 2.5 3B model due to its use of only two heads across all layers.
Figure 39: The use of eight heads in Qwen 2.5 14B shows similar trends to the LLaMA 3 family of models, with eviction patterns being fairly similar in their nature.
Figure 40: Probing results still show drastic inconsistency as compression as the organization and creature tags still suffer.
Figure 41: Question-aware and agnostic compression still induce fairly different understandings despite the different model sizes and architectures involved.
Figure 42: Eviction trends still remain agnostic to downward tasks even when changes to the scale of the model are seen.
Figure 43: Smaller models (increasing unknown errors in AdaKV persist fragility but retain the similar hallucination cliff as compared to larger models (Qwen 2.5 14B). The trend of increased unknown errors in AdaKV is still present.
Figure 44: Correlation between GER and hallucination rate for all models.
Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware. Existing KV cache eviction methods typically apply heuristic token scoring over all heads in GQA-based LLMs. These methods ignore the different functionalities of attention heads, leading to the eviction of critical tokens and thus degrading the performance of LLMs. To address this issue, we propose CompressKV, a resource-efficient KV-cache compression framework for GQA-based LLMs. Instead of aggregating attention scores from all heads, CompressKV identifies Semantic Retrieval Heads (SRHs) that capture both the initial and final tokens of a prompt and semantically important mid-context evidence, and uses them to select tokens whose KV pairs should be retained. Furthermore, CompressKV allocates cache budgets across layers according to offline estimates of layer-wise eviction error. Experiments on LongBench and Needle-in-a-Haystack show that CompressKV consistently outperforms existing KV-cache eviction methods across memory budgets. Notably, it preserves over 97% of full-cache performance using only 3% of the KV cache on LongBench question-answering tasks and achieves 90% accuracy with just 0.7% KV storage on Needle-in-a-Haystack. These results demonstrate an improved resource--performance trade-off for long-context LLM inference. Our code is publicly available at: https://github.com/TUDa-HWAI/CompressKV
Xiaolin Lin, Jingcun Wang, Olga Kondrateva +3
Technical University of Darmstadt, Darmstadt, Germany · University of Notre Dame, Notre Dame, IN, USA · Technical University of Ilmenau, Ilmenau, Germany
The key-value (KV) cache is a primary capacity and bandwidth bottleneck in long-context LLM serving. We present Minima-KV, a retention-preserving hierarchy for mixed-format paged attention. Recent and protected Anchor pages remain in FP8, while older non-anchor pages move to packed TQ3; every live-request page remains addressable. Format-specific kernels compute partial attention states and combine them through a globally normalized online-softmax merge, enabling direct heterogeneous decode without a cache-sized dense shadow. Across separate, configuration-bound Qwen3.6-27B profiles on a single 96-GB NVIDIA RTX PRO 6000 Blackwell GPU, deployment accounting reports 18.3 KiB of attention KV per live token, corresponding to 3.50x compression relative to BF16 and 1.75x relative to FP8. A materializing quality profile matches its dense control on 16K RULER needle-in-a-haystack tasks. On the same 503-question LongBench v2 set, measured deltas are -0.80, -0.60, and -0.40 percentage points at 16K, 32K, and 64K. A separate single-pair direct-decode canary with two 59,008-token requests measures 3.625x active-KV compression and 0.9821x throughput relative to its control, routes all 16 full-attention layers without fallback, and retains no dense shadow. These results establish a practical mixed-format path for compressing long-context state without evicting live-request KV pages.
Reasoning capability has advanced rapidly in large language models (LLMs), leading to an increasing size of key-value (KV) cache in both prefilling and decoding stages. Existing KV cache compression methods mainly rely on attention weights to estimate token importance. While attention effectively captures contextual relevance, it overlooks complementary information-theoretic signals related to predictive uncertainty and token informativeness. In this paper, we revisit token importance from a forward-looking perspective and introduce Forward Influence, a metric that measures how compressed tokens affect future contexts. Our analysis reveals that tokens selected by attention scores mainly influence nearby contexts, whereas tokens associated with high predictive uncertainty exhibit substantially stronger influence on distant future contexts. Based on the observation, we propose InfoKV, an entropy-aware KV cache compression framework that incorporates information-theoretic signals. It combines token-level predictive uncertainty with layer-wise representation evolution and integrates the resulting entropy scores with attention scores during reasoning. Experiments on long-context reasoning benchmarks with Llama-3.1, Llama-3.2, and DeepSeek-R1 demonstrate that InfoKV consistently outperforms existing attention-based KV compression methods in both long prefilling and decoding scenarios.
Jushi Kai, Zhuiri Xiao, Alexandra Birch +1
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Jiao Tong University · School of Informatics, University of Edinburgh