HybridKV: Hybrid KV Cache Compression for Efficient Multimodal Large Language Model Inference
Authors: Bowen Zeng, Feiyang Ren, Jun Zhang, Xiaoling Gu, Ke Chen, Lidan Shou, Huan Li
Organizations: The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security · Hangzhou Dianzi University, Hangzhou, China
Multimodal Large Language Models (MLLMs) have advanced unified reasoning over text, images, and videos, but their inference is hindered by the rapid growth of key-value (KV) caches. Each visual input expands into thousands of tokens, causing caches to scale linearly with context length and remain resident in GPU memory throughout decoding, which leads to prohibitive memory overhead and latency even on high-end GPUs. A common solution is to compress caches under a fixed allocated budget at different granularities: token-level uniformly discards less important tokens, layer-level varies retention across layers, and head-level redistributes budgets across heads. Yet these approaches stop at allocation and overlook the heterogeneous behaviors of attention heads that require distinct compression strategies. We propose HybridKV, a hybrid KV cache compression framework that integrates complementary strategies in three stages: heads are first classified into static or dynamic types using text-centric attention; then a top-down budget allocation scheme hierarchically assigns KV budgets; finally, static heads are compressed by text-prior pruning and dynamic heads by chunk-wise retrieval. Experiments on 11 multimodal benchmarks with Qwen2.5-VL-7B show that HybridKV reduces KV cache memory by up to 7.9× and achieves 1.52× faster decoding, with almost no performance drop or even higher relative to the full-cache MLLM.
Figures & tables
Figure 1: Left : We introduce HybridKV , a hybrid KV cache compression framework for efficient yet effective MLLM inference. HybridKV leverages head-level attention patterns (detailed in Section 2 ) to classify static and dynamic heads via text-guided signals, enabling hierarchical budget allocation with tailored pruning and retrieval strategies. Right : HybridKV outperforms existing counterparts including SnapKV Li et al. (2024b) , LOOK-M Wan et al. (2024) , MadaKV Li et al. (2025) and SparseMM Wang et al. (2025) on eight benchmarks using only 10% of the KV cache on Qwen2.5-VL-7B . Performance metrics are shown as a percentage relative to Full Cache .
Figure 2: Static vs. dynamic attention patterns in MLLM inference. The decode stage ( a , b ) strongly correlates with the corresponding prefill stage ( c , d ). (a, c) Static Heads : focus stably on a small set of tokens during decoding ( a ), consistent with high - sparsity text - centric attention ( Eq. 2 ) in prefill ( c ). (b, d) Dynamic Heads : shift focus across many tokens during decoding ( b ), consistent with low - sparsity distributions in prefill ( d ).
Figure 3: Cumulative attention distribution during prefill for selected heads in Qwen2.5-VL-7B . Each curve shows the fraction of attention captured by the top x% tokens across video tasks ( VATEX , NextQA , Video - ChatGPT ) and image tasks ( STD , SlideVQA , TextCaps ).
Figure 4: Overview of HybridKV . Section 3.1 : Heads are classified by text - centric sparsity into static (anchor, few tokens) and dynamic (wave, broad focus). Section 3.2 : A two-layer scheme allocates KV cache budgets first by head type and individual heads then. Section 3.3 : Hybrid compression integrates static pruning (drop uninformative tokens) with dynamic retrieval (reactivate tokens when needed) to optimize cache usage.
Method
CL-CH
DocVQA
MMCoQA
MM-QA
SlideVQA
STD
WebQA
WikiVQA
Average
Qwen2.5-VL-7B
Full Cache
43.07
97.50
66.50
75.00
83.50
29.79
76.50
92.50
70.55
SnapKV
37.41
97.50
54.00
73.00
83.00
29.02
71.00
89.50
66.80
LOOK-M
38.81
97.50
52.50
73.50
82.50
29.25
71.00
88.00
66.63
MadaKV
37.86
97.50
55.00
74.50
82.00
28.24
70.50
90.00
66.95
SparseMM
37.63
97.50
62.00
75.50
83.50
28.69
70.50
92.50
68.48
HybridKV
40.57
97.50
63.00
76.00
83.50
29.84
76.00
93.00
69.93
Table 1: Performance of five KV cache compression strategies on three MLLMs on image tasks. ROUGE-L Lin (2004) evaluates generated–reference consistency for CL-CH and STD , while exact match accuracy is used for other QA tasks. Higher values indicate better performance, and the best measures are highlighted in bold by default.
Method
VATEX
NextQA
Video-ChatGPT
BLEU-4
Meteor
ROUGE-L
CIDEr
Average
WUPS
CI
DO
CU
TU
CO
Average
Qwen2.5-VL-7B
Full Cache
0.2181
0.2209
0.4266
0.4628
0.3321
33.79
3.06
3.15
3.52
2.24
2.69
2.93
SnapKV
0.2146
0.2125
0.4240
0.4498
0.3252
33.29
2.90
2.96
3.40
1.95
2.53
2.75
LOOK-M
0.2180
0.2174
0.4280
0.4571
0.3301
33.07
2.92
2.97
3.38
2.05
2.49
2.76
MadaKV
0.2051
0.2157
0.4228
0.4306
0.3186
33.25
2.94
3.03
3.41
2.02
2.56
2.79
SparseMM
0.2133
0.2116
0.4227
0.4198
0.3169
33.84
2.90
2.95
3.37
1.99
2.52
2.75
Table 2: Performance of five KV cache compression strategies on three MLLMs on video tasks. Evaluation metrics (e.g., BLEU-4 and Meteor) are explained in Appendix C ; generally, higher measures indicate better performance.
Method
Budget
Accuracy
GPU Memory
Latency
(avg.)
(GB)
(ms/token)
Full Cache
100%
2.93
1.73
58.94
HybridKV
20%
2.88
0.40
42.08
HybridKV
10%
2.85
0.22
38.65
Table 3: KV cache GPU memory usage and decoding latency on Qwen2.5-VL-7B with Video-ChatGPT .
Figure 5: Scalability under different KV cache budgets on WebQA and VATEX . HybridKV outperforms all baselines across budgets, matching or exceeding full cache performance and maintaining high accuracy even under aggressive compression (e.g., 5% budget).
Figure 6: Illustration of hybrid KV cache compression.
θ
CL-CH
STD
WebQA
Average
0.75
39.06
29.53
72.50
47.03
0.80
39.49
29.41
72.50
47.13
0.85
39.87
29.51
75.00
48.13
0.90
40.57
29.84
76.00
48.83
0.95
39.13
29.69
76.00
48.27
Appendix
Table 6: Sensitivity study on the score threshold θ . For this analysis, the share coefficient is fixed at r=0.75 and the KV cache budget is 10%.
r
CL-CH
STD
WebQA
Average
0.1
39.26
29.37
75.00
47.88
0.25
39.87
29.65
75.50
48.34
0.50
40.03
29.75
75.50
48.43
0.75
40.57
29.84
76.00
48.83
1.00
40.12
29.68
74.00
47.93
1.25
39.02
29.44
74.50
47.65
Appendix
Table 7: Sensitivity study on the share coefficient r . For this analysis, the score threshold is fixed at θ=0.90 and the KV cache budget is 10%.
Figure 7: Case study of HybridKV on CL-CH using Qwen2.5-VL-7B . Other methods generate the wrong answer after KV cache compression, while HybridKV still retains the correct answer.
Figure 8: Case study of HybridKV on Video-ChatGPT using Qwen2.5-VL-7B . HybridKV enables the decode tokens to adaptively focus on critical visual regions, highlighted by red font and boxes, while the model with full cache attend to the noisy tokens, leading to degraded accuracy. This demonstrates that retaining a small yet salient subset of tokens during decoding can not only preserve but even enhance the model’s capabilities.