Although multimodal Large Language Models (MLLMs) excel in diverse tasks, their scalability remains limited by the memory and computational overhead of KV cache storage. Recent KV cache eviction approaches incorporate a cosine similarity-based diversity metric with importance metrics to selectively retain critical key-value pairs. However, cosine similarity involves normalization that discards magnitude information, and it often yields uniformly high similarity values across layers due to the anisotropy property of hidden representations. In our study ECOKV, we rigorously deconstruct the capabilities of existing diversity metrics. Moving beyond simple measurement, we propose a geometry-aware composite metric that jointly leverages Euclidean distance and cosine similarity to capture token diversity from complementary perspectives. Furthermore, we use these two metrics to estimate the redundancy level of each attention head, allowing adaptive weighting between diversity and importance scores during token selection. Finally, we demonstrate that the observation window commonly employed to preserve recent tokens can be substantially reduced, thereby allocating more cache capacity to informative tokens and yielding consistent improvements. Extensive experiments demonstrate that ECOKV achieves state-of-the-art performance under various compression ratios and can be seamlessly integrated with existing KV cache eviction methods. We further analyze the relationship between importance and diversity, and examine redundancy patterns across layers and attention heads.
Figures & tables
Figure 1: Distribution of layer-wise key cosine similarities in Qwen2-VL-7B-Instruct. Cosine similarity remains consistently high across all layers.
Figure 2: (a) Cosine similarity remains consistently high across layers. (b) PCA projection of layer 0 keys. Although the embeddings are not identical, their cosine is approximately 1.0.
Methods
DocVQA (%)
OCRBench (%)
TextVQA (%)
ChartQA (%)
TextCaps
256
128
64
256
128
64
256
128
64
256
128
64
256
128
64
LLaVA-NeXT-Mistral-7B
SnapKV
59.7
55.2
47.3
45.0
39.0
31.9
63.5
61.0
57.1
50.2
47.5
42.7
0.650
0.558
0.444
+ MixKV
61.7
58.1
48.8
49.9
44.7
36.1
65.2
64.3
60.1
50.8
47.7
43.6
0.708
0.659
0.514
+ ECOKV
62.7
60.8
56.9
49.7
48.4
43.7
65.5
65.1
63.0
52.5
51.5
48.8
0.792
0.721
0.678
Δbaseline
+1.0
+2.7
+8.1
-0.2
+3.7
+7.6
+0.3
+0.8
+2.9
+1.7
+3.8
+5.2
+0.084
+0.062
+0.164
Table 1: Performance comparison across benchmarks under different methods and models.
Methods
Budget Size
256
128
64
SnapKV
55.0
50.3
42.1
SnapKV + MixKV
56.2
51.7
42.8
SnapKV + ECOKV
58.0
56.1
51.3
PyramidKV
51.4
46.4
38.1
PyramidKV + MixKV
52.6
47.8
38.6
Table 2: Evaluation results on MP-DocVQA.
Figure 3: Performance under different combination weights α and β . Smaller window size is not applied to isolate gains from metric design.
KV cache eviction methods such as SnapKV and PyramidKV rank tokens solely by mean attention over a small observation window. We study a unified score, μi+λ1σi+λ2corr(i,S), adding attention dispersion across window queries and redundancy relative to selected tokens. For λ2<0, the score penalizes similarity to selected tokens as in maximal marginal relevance (MMR), without extra forward passes. To test whether this relevance-diversity balance should vary with depth, we compare fixed global coefficients with three-segment and quadratic profiles. Only these depth profiles are searched on a development split under a sinh reparameterization. On all 16 English LongBench datasets with Mistral-7B at a budget of 64 entries per layer, a single global diversification constant improves 13 of 16 datasets (macro +1.1); the gain holds at budget 32 and narrows at 128. Per-dataset search finds no detectable layer structure on most datasets; on passage retrieval it finds a large one: a mid-layer sign flip that rewards similarity and is worth +9.6 over the baseline at budget 64 and, without re-tuning, +13.2 over the global constant at budget 128. Ablations attribute the gain to the redundancy term; replaying every accepted search state on the held-out test set separates genuine structure from tuning noise.
Tianfang Xie, Wei Zhu
Georgia Institute of Technology Atlanta, GA 30332, USA · Zhangjiang Lab Shanghai, China
Multimodal Large Language Models face severe challenges in computational efficiency and memory consumption due to the substantial expansion of the visual KV cache when processing long visual contexts. Existing KV cache compression methods typically rely on the "persistence of importance" hypothesis to prune tokens. However, this approach proves fragile in multimodal settings due to two key issues: 1) Visual tokens display "deferred importance," initially exhibiting low salience but becoming pivotal during later decoding, which can lead to premature eviction. 2) Discrete pruning disrupts the inherent spatial continuity of visual cues. To address these challenges, we propose RetentiveKV, an entropy-driven KV cache optimization method that reformulates KV eviction from "discrete context truncation" to "continuous memory evolution" based on State Space Models. Our method leverages information entropy to quantify the information potential of low-attention tokens and integrates tokens scheduled for eviction into a continuous state space through entropy-guided state transitions, enabling their dynamic reactivation when semantic relevance arises during subsequent decoding. Extensive experiments on multimodal benchmarks demonstrate that RetentiveKV achieves 5.0 times KV cache compression and 1.5 times decoding acceleration.
Sihao Liu, YuFan Xiong, Zhonghua Jiang +2
Zhejiang University · Alibaba · Shanghai Institute for Advanced Study of Zhejiang University
KV cache eviction methods typically use a single retention-rule family throughout a model, making eviction-method identity a model-level design choice. Yet Transformer layers differ substantially in their attention behavior, representations, and sensitivity to compression, suggesting that a uniform rule may overlook useful layer-wise structure. This raises a basic question: should eviction methods themselves vary across layers? We investigate this question by composing existing eviction methods across Transformer layers and systematically exploring the resulting layer-wise design space. Using simple offline profiles, we construct fixed routes and study how their quality varies with method placement and cache budget. On LongBench, heterogeneous routing improves performance on a majority of tasks over homogeneous policies at the same cache budget. Even when method counts are held fixed, the profile-guided placement ranks second among 100 evaluated assignments, demonstrating that routing quality depends strongly on where methods are placed. Moreover, the same fixed route outperforms the best of nine standalone baselines across all five tested cache budgets. Together, these results establish layer-wise method composition as an exploitable, placement-sensitive design dimension for KV cache compression.
Chao Fei, Kaihua Liang, Hanzhi Hu +4
King Abdullah University of Science and Technology · Fudan University