cs.CLMar 2, 2026
SaveUnderstanding the Physics of Key-Value Cache Compression for LLMs through Attention Dynamics
Organizations: IIT Delhi, India
Abstract
As context windows in LLMs scale to 100K+ tokens, the key-value (KV) cache becomes the dominant memory bottleneck, with recent methods claiming 80-90% savings and minimal benchmark degradation. We argue these evaluations miss a structural issue: attention is not just storage but routing, and retaining KV pairs does not guarantee semantic accessibility. We propose a physics-inspired view of KV compression as a controlled perturbation of token-level routing, distinguishing retention, accessibility, and utilization. Using synthetic tasks probing multi-entity tracking, disambiguation, coreference, and multi-hop reasoning, we find that moderate compression degrades internal representations with little accuracy loss, revealing redundancy; all models exhibit a sharp hallucination safety cliff near 90% compression, correlated with spikes in Global Eviction Ratio (GER), suggesting a phase transition in semantic reachability; and architectures differ in routing dynamics, with LLaMA showing early consensus and late diversification, and Qwen showing funnel-like late convergence, leading to distinct resilience profiles. Beyond erasure, we identify representational rigidity, where excessive head-level consensus collapses routing flexibility despite token survival. These results suggest sparse token-route structures govern compression tolerance, reframing KV compression as a structural probe of attention geometry and linking long-context scalability to sparsity and the lottery ticket hypothesis in self-attention.
Figures & tables
Figure 1: Comparison of traditional versus synthetic evaluation frameworks for KV cache compression. Existing benchmarks (left) primarily report coarse-grained task accuracy, whereas our synthetic, controlled framework (right) probes structural reachability, routing collapse, and semantic fragility under compression.
| Name | Passage Length (words) | # Queries | Total Passages | Total Queries | LLM Generated |
|---|---|---|---|---|---|
| Base task | 175 25 | 6 | 200 | 1199 | Yes |
| Knowledge manipulation | 32 | 13 | 4000 | 52000 | No |
| Multi presence | 500 100 | 10 | 100 | 1000 | Yes |
| Multi entity | 500 100 | 10 | 100 | 1000 | Yes |
| Long context | 1100 200 | 10 | 100 | 1000 | Yes |
| Coreference | 44 | 9 | 8000 | 72000 | No |
Table 1: Structural summary of the synthetic dataset suite used for controlled KV compression evaluation.
| Model | Setup | Base | Know | Multi-P | Multi-E | Long | Coref | Hops |
|---|---|---|---|---|---|---|---|---|
| LLaMA-3 8B | AGN | 70.05 | 91.28 | 79.64 | 80.33 | 45.93 | 75.63 | 31.54 |
| AWR | 69.86 | 92.09 | 77.66 | 81.93 | 45.54 | 60.90 | 31.25 | |
| Qwen-2.5 7B | AGN | 68.80 | 92.20 | 73.63 | 77.46 | 38.68 | 78.89 | 27.13 |
| AWR | 69.63 | 90.93 | 71.88 | 79.38 | 40.20 | 52.00 | 26.35 |
Table 2: Base (0% compression) F1 scores across datasets.
Figure 2: Base task performance across compression levels. Localized spikes under moderate compression suggest sparse substructure effects. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 27 of Appendix B .
Figure 3: Knowledge manipulation results. Qwen exhibits a more gradual rate of degradation under compression, particularly in the question-aware setting.
Figure 4: Coreference performance across setups. Question-aware pruning significantly increases overconfident errors.
Figure 5: Multi presence forward vs. reverse asymmetry.
Figure 6: Multi entity results showing reduced directional asymmetry. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 28 of Appendix B .
Figure 7: Long context degradation across compression levels.
Figure 8: Hops task performance. Multi-step semantic chaining collapses rapidly under compression.
Figure 9: Individual hop breakdown showing instability in intermediate semantic links.
Figure 10: Answer-type tag performance in Base task. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 29 of Appendix B .
Figure 11: Answer-type tag behavior in Multi entity. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 30 of Appendix B .
Figure 12: Knowledge manipulation answer-type tag-level behavior.
Figure 13: Question-type tags for Base task. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 31 of Appendix B .
Figure 14: Knowledge manipulation question-type analysis.
Figure 15: Multi entity question-type behavior shows a similar trend to Base task but with much smoother curves in question agnostic. The question aware setup has a far more jagged curve with distinct points of significant drop off. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 32 of Appendix B .
Figure 16: Consensus across layers. LLaMA exhibits early stabilization and late specialization; Qwen exhibits early exploration and late consolidation. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 33 of Appendix B .
Figure 17: LLaMA layerwise consensus under compression. Early agreement persists, with depth-wise diversification supporting parallel feature channels. Corresponding results for LLaMA-3.2 3B are shown in Figure 34 of Appendix B .
Figure 18: Qwen layerwise consensus under compression. Diversity persists deeper and collapses late, consistent with a consolidation-heavy tail. Corresponding results for Qwen-2.5 3B and Qwen-2.5 14B are shown in Figure 35 of Appendix B .
Figure 19: LLaMA attention heatmaps under compression. Chunk pruning removes contiguous segments; Ada pruning is more irregular. At 0.9, global route destruction dominates. Corresponding results for LLaMA-3.2 3B are shown in Figure 37 of Appendix B .
Figure 20: Qwen attention heatmaps under compression. Policy differences are visible at low compression but are overwhelmed at 0.9 by large-scale pruning. Corresponding results for Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 38 and Figure 39 of Appendix B .
Figure 21: Eviction rates vs. compression. Pruning is broadly question-agnostic unless explicitly conditioned. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 42 of Appendix B .
Figure 22: Error rates vs. compression. Hallucination exhibits a cliff near , with “unknown” behavior more erratic under AdaKV. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 43 of Appendix B .
Figure 23: Compression susceptibility peaks sharply near , consistent with a critical transition in the error landscape.
Figure 24: GER correlates strongly with hallucination rate, indicating that global route deletion is a primary driver of catastrophic failure. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 44 of Appendix B .
Figure 25: Base task probing across compression. Some tags remain decodable; others collapse early and may partially recover under question-aware setups. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 40 of Appendix B .
Figure 26: Multi-Entity probing across compression. Question awareness helps selectively rather than uniformly, suggesting interaction between representational geometry and pruning alignment. Corresponding results for LLaMA-3.2 3B, Qwen-2.5 3B, and Qwen-2.5 14B are shown in Figure 41 of Appendix B .
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Context | Question | Response |
| Base task | Gonzalo Batistuta, an Argentine forward, is widely regarded as one of the greatest football players of all time… | What is Gonzalo Batistuta’s nationality? | Argentine |
| Knowledge man. | Cora Delaine was born on December 11, 1954. They studied in Daegu Global Science University… | What is Cora Delaine’s first name? | Cora |
| Multi presence | The Astralis Spire stands tall as one of the most remarkable achievements of modern architecture and engineering. Rising high above the skyline of its bustling city… | What was the main reason for constructing Astralis Spire? | To improve television and radio broadcasting disrupted by urban development |
| Multi entity | Harland Kane is one of the most influential authors of contemporary horror, suspense, and supernatural fiction. Over a career spanning decades, Kane has… | Which novel by Kane is set in a haunted hotel? | The Shadow of the Pines |
| Long context | During the mid-Cretaceous period, the Riverback Sailbacks were formidable semi-aquatic predators, inhabiting river systems, floodplains, and coastal wetlands. Known for their elongated jaws, conical teeth, and large… | Where was Kaelin Vireo from? | The riverine fossil station of Thaloris |
| Coreference | Julian Foster was born on September 7, 1962 in Aleppo, Aleppo Governorate, Syria. He is male. He studied in Tianjin Harbor University in Podgorica, Podgorica, Montenegro… | What does he work as? | Tour Guide |
Table 3: Sample contexts and inputs for each task along with the ground truth answer.
| Tag | Context | Question | Response |
|---|---|---|---|
| Person | Joao Fernandes, widely considered one of the greatest footballers of all time… | What is the name of the person described in the passage? | Joao Fernandes |
| Thing | Gonzalo Batistuta, an Argentine forward, is widely regarded as one of the greatest… | What is Gonzalo Batistuta’s nationality? | Argentine |
| Organization | Mookie Blaylock, a rock band from Northport formed in Riverbend in 1990… | What is the subject of the given passage? | Mookie Blaylock |
| Creature | The Void Stalker is a fictional endoparasitoid extraterrestrial species that serves as the main… | What kind of species is the Void Stalker? | An endoparasitoid extraterrestrial species |
| Location | The Stones of Pharaohs are a complex of monumental structures located on the Giza Plateau… | In which city are The Stones of Pharaohs located? | Cairo |
| Numerals | The Thistlewood Vale Witch Trials represent a chilling episode in early American history… | How many individuals were convicted and executed during the trials? | 20 |
Table 4: Examples of answer-type tags.
| Tag | Context | Question | Response |
|---|---|---|---|
| Standard | The Basalt Bloom is an immense igneous province, covering a vast expanse of central India. It formed through a series of massive volcanic eruptions… | Where is the Basalt Bloom located? | Central India |
| Manipulated | Xena Bell was born on November 20, 1968. They studied in Shanghai Maritime University… | What is Xena Bell’s last name? | Bell |
| Part | Breezeflare is characterized by unusual sounds during breathing caused by airflow obstruction in the airways. The condition is commonly observed… | Nyla Soren is recognized for leading what type of initiatives? | Community workshops, public education |
Table 5: Examples of question-type tags
| Task | Direction | Context | Question | Response |
|---|---|---|---|---|
| Mutli presence | FWD | The series known as Chrono Nexus is widely regarded as one of the most compelling science fiction narratives in Japanese animation. Created by Itsuki Aoyama… | Who created Chrono Nexus? | Itsuki Aoyama |
| REV | Few creatures in Earth’s history capture the imagination quite like the Thunderfang Predator, one of the largest and most fearsome carnivorous dinosaurs to have ever lived… | Which predator had serrated teeth as long as a human hand? | Thunderfang Predator | |
| Multi entity | FWD | In the distant waters of the Andaman Islands lies the infamous Iron Shadows Prison. Constructed by the British in the late 19th century, this prison became a… | Who was the nationalist imprisoned in Iron Shadows and known as Veer? | Vinayak Damodar Savarkar |
| REV | Shinroku is a popular Japanese manga and anime series known for its unique blend of comedy, action, and science fiction. Set in an alternate Edo period where… | Hana Tanaka appears as a companion in which story? | Shinroku |
Table 6: Examples of forward and reverse questions
Figure 27: Base task performance across compression levels for 3B and 14B models.
Figure 28: Multi entity forward and reverse asymmetry is reduced even when the model size is changed, showing invariance to the size of the model.
Figure 29: Answer-type results for Base task closely mirror the results from the LLaMA 3 8B and Qwen 2.5 7B models, especially in their weak points such as the Events tag.
Figure 30: Answer type tag behaviour in Multi entity for the 3B and 14B models shows similar trends as seen in LLaMA 3 8B and Qwen 2.5 7B.
Figure 31: Base task question-type results continue to show a similar trend with smaller and larger models, as seen in the 7B and 8B models.
Figure 32: Multi entity results for LLaMA 3.2 3B show similar jagged points in question-aware settings, while the performance almost turns into a smooth linear pattern in Qwen 2.5 3B and 14B models.
Figure 33: Consensus scores tend to follow a similar trend with the only major changes being Qwen 3B’s slightly more linear increase rather than the slightly parabolic increase seen in 7B and 14B models.
Figure 34: Layerwise consensus under compression for LLaMA 3.2 3B shows a strikingly similar convergence very early on with depth-wise diversification.
Figure 35: Layerwise consensus under compression for Qwen 2.5 3B and 14B. The 3B model has persistent diversity even after several layers, occasionally failing to converge, while 14B remains stable early on but shows the same funnel-like trend upon convergence.
Figure 36: Compression 50% for LLaMA 3 8B and Qwen 2.5 7B.
Figure 37: Similar smooth pruning in Chunk and irregular removal in Ada is seen in LLaMA 3.2 3B as well.
Figure 38: The non-contiguous nature of pruning is far less diminished in the Qwen 2.5 3B model due to its use of only two heads across all layers.
Figure 39: The use of eight heads in Qwen 2.5 14B shows similar trends to the LLaMA 3 family of models, with eviction patterns being fairly similar in their nature.
Figure 40: Probing results still show drastic inconsistency as compression as the organization and creature tags still suffer.
Figure 41: Question-aware and agnostic compression still induce fairly different understandings despite the different model sizes and architectures involved.
Figure 42: Eviction trends still remain agnostic to downward tasks even when changes to the scale of the model are seen.
Figure 43: Smaller models (increasing unknown errors in AdaKV persist fragility but retain the similar hallucination cliff as compared to larger models (Qwen 2.5 14B). The trend of increased unknown errors in AdaKV is still present.
Figure 44: Correlation between GER and hallucination rate for all models.