Multimodal Large Language Models face severe challenges in computational efficiency and memory consumption due to the substantial expansion of the visual KV cache when processing long visual contexts. Existing KV cache compression methods typically rely on the "persistence of importance" hypothesis to prune tokens. However, this approach proves fragile in multimodal settings due to two key issues: 1) Visual tokens display "deferred importance," initially exhibiting low salience but becoming pivotal during later decoding, which can lead to premature eviction. 2) Discrete pruning disrupts the inherent spatial continuity of visual cues. To address these challenges, we propose RetentiveKV, an entropy-driven KV cache optimization method that reformulates KV eviction from "discrete context truncation" to "continuous memory evolution" based on State Space Models. Our method leverages information entropy to quantify the information potential of low-attention tokens and integrates tokens scheduled for eviction into a continuous state space through entropy-guided state transitions, enabling their dynamic reactivation when semantic relevance arises during subsequent decoding. Extensive experiments on multimodal benchmarks demonstrate that RetentiveKV achieves 5.0 times KV cache compression and 1.5 times decoding acceleration.
Figures & tables
Figure 1: (a) Visualization of Deferred-Critical Tokens: The presence of delayed activation patterns, where visually salient features remain dormant during early decoding phases, defies the static heavy-hitter assumption. This temporal misalignment triggers erroneous eviction in importance-based pruning. (b) Visualization of Pruning-Induced Spatial Discontinuity: We compare baseline and pruned attention maps. The instability of visual signals triggers excessive pruning (white voids). This fragmentation disrupts the spatial continuity of visual representations.
Figure 2: Entropy Shifts under KV Eviction.
Figure 3: Overview of the proposed RetentiveKV framework. The architecture coordinates efficient long-context multimodal reasoning through three core components: (1) An Entropy-Guided KV Retention Estimator that identifies deferred-critical tokens by analyzing layer-wise entropy shifts ( Ht ) and accumulated attention ( αt ); (2) An Entropy-Guided State Transition mechanism that absorbs evicted KV pairs into modality-adaptive state space; and (3) A Query-Conditioned State Retrieval module that dynamically fuses retrieved long-term context ( Os ) with local attention outputs via a learnable gating factor ( γt ) during autoregressive decoding.
Method
TextVQA
DocVQA
MathVista
MMStar
MMMU
BLINK
MMCoQA
ALFRED
LLaVA-v1.5-7B
Full Cache
62.30
61.24
60.40
62.52
43.34
64.04
35.50
16.32
H2O
60.20
56.42
59.30
61.66
42.39
62.62
28.00
14.49
SnapKV
61.00
58.52
59.10
62.13
42.33
63.14
30.50
14.57
SAINT
60.00
57.84
58.60
61.84
42.28
62.64
28.00
14.64
Meda
60.80
57.26
59.20
62.07
42.36
63.04
28.50
14.87
Table 1: Performance comparison of different KV compression (The best results are highlighted in bold .)
Method
Budget
Latency
Memory
Full Cache
100%
32.15 ms/token
2.24 GiB
RetentiveKV
50%
27.84 ms/token
1.18 GiB
35%
24.42 ms/token
0.84 GiB
20%
21.42 ms/token
0.46 GiB
5%
18.42 ms/token
0.23 GiB
Table 2: Comparison of decoding latency and GPU memory usage for different cache budgets.
Figure 4: Comparison results for various cache budgets.
EM
MA
QR
DocVQA
MMCoQA
✓
✓
58.70
60.91
✓
✓
60.20
61.55
✓
✓
59.80
61.38
✓
✓
✓
61.40
64.22
Table 3: Ablation study on key components of RetentiveKV.
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security · Hangzhou Dianzi University, Hangzhou, China