VisCache: Visual KV Cache Pruning for Efficient Vision Large Language Model Inference

Lyuke Wang; Zhuo Li; Guangxu Zhu

cs.CVAug 25, 2026

VisCache: Visual KV Cache Pruning for Efficient Vision Large Language Model Inference

Authors: Lyuke Wang, Zhuo Li, Guangxu Zhu

Organizations: Shenzhen International Center for Industrial and Applied Mathematics · Shenzhen Research Institute of Big Data · The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute

Abstract

While Vision Large Language Models (VLLMs) have achieved remarkable success in multimodal reasoning, their long-context inference remains prohibitively expensive due to the massive computation and memory overhead of visual Key-Value (KV) caches. Existing KV compression methods often apply uniform pruning across visual tokens and layers, leading to substantial information loss and degraded performance.To address this challenge, we propose VisCache, a plug-and-play framework for coarse-to-fine Visual KV Cache pruning without training, which consists of two synergistic stages. First, a lightweight VLM filters temporal redundancy by selectively forwarding semantically informative keyframes. Second, we introduce {PruneKV}, a surgical KV compression algorithm tailored to the attention dynamics of VLLMs. Unlike rigid pruning strategies, PruneKV adopts a parabolic layer-wise budget allocation together with an asymmetric update mechanism that selectively prunes keys while fusing values, thereby preserving critical contextual information. Extensive experiments demonstrate that VisCache substantially improves inference efficiency, achieving up to {2.35 $\times$ speedup} and significant memory reduction while maintaining competitive performance with only {19--28%} KV cache retention. VisCache consistently outperforms existing baselines, establishing a new Pareto frontier between efficiency and performance for long-context VLLM inference. Code is available at https://github.com/Wlklk/VisCache

Figures & tables

Figure 1: Visualization of different plug-and-play layer-wise KV cache compression methods. It illustrates that VisCache differs fundamentally from other baseline approaches in terms of layer-wise budgets.

Figure 2: Overview of VisCache. The framework operates in two synergistic stages: (I) A lightweight scout VLM performs prompt-aware keyframe filtering via MMR to eliminate temporal redundancy at the source; (II) During inference, the VLLM constructs the KV cache, which is subsequently refined by PruneKV through attention-aware, layer-wise compression.

Method	RR	FLOPs	FLOPs	ActCap	DREAM1K	NExTQA	ActQA	EgoSchema	Avg.
Method	RR	(T)	Ratio	ROUGE-L	ROUGE-L	Acc	Acc	Acc	Acc
Qwen2.5-VL-3B-Instruct
Full Cache	100%	14.80	100%	2.63	9.19	34.69	40.58	57.20	44.16
Q-Frame	40%	6.26	42%	2.42	5.86	39.52	39.42	46.40	41.78
PyramidKV	40%	1.99	13%	2.43	8.48	39.53	39.56	54.80	44.63
FastV	40%	3.11	21%	2.43	8.63	37.67	40.65	50.40	42.91

Table 1: Comparison of different KV cache compression methods on various VQA and VS datasets. Bold and underlined numbers indicate the best and second-best results, respectively.

Method	RR	AS	AP	AA	FA	UA	OE	OI	OS	MD	AL	ST	AC	MC	MA	SC	FP	CO	EN	ER	CI	Avg
Qwen2.5-VL-3B-Instruct
Full Cache	100%	69.0	66.8	72.0	26.4	72.4	85.0	69.7	31.3	50.5	37.5	76.0	48.5	62.5	89.0	45.5	27.0	47.1	36.5	26.8	58.0	56.6
Q-Frame	50%	49.5	63.3	68.5	23.4	64.3	80.5	59.6	31.3	46.0	37.5	77.0	36.4	56.5	78.0	40.9	24.5	52.9	35.0	37.0	61.5	51.2
PyramidKV	50%	65.5	66.8	54.0	24.9	46.4	77.5	55.6	43.8	39.0	38.0	51.0	42.4	62.5	87.0	45.5	20.5	52.9	32.0	28.8	53.0	49.4
FastV	50%	60.5	53.8	61.5	29.4	48.0	80.5	64.7	18.8	41.0	38.5	50.5	36.4	63.0	81.5	45.5	20.5	47.1	30.0	22.5	62.5	47.8
PDrop	50%	54.5	58.3	72.5	24.4	65.5	77.5	63.7	30.0	44.5	36.0	70.5	45.5	63.0	86.5	50.0	29.0	50.0	34.5	25.0	53.5	51.7

Table 2: Results on MVBench of different KV cache compression methods with

(p=0.75,q=0.67)

. The first row lists abbreviations of the subset names, from AS to CI (e.g., Action Sequence (AS)). Bold and underlined numbers indicate the best and second-best results, respectively.

Method	RR	Total Mem.	KV Cache	TPS
Full Cache	100%	5.57	0.06	10.4
PyramidKV	40%	4.98	0.02	9.7
FastV	40%	4.98	0.02	10.6
PDrop	40%	4.99	0.02	15.0
VisCache	40%	4.10	0.03	15.5
VisCache	28%	4.10	0.02	18.6

Table 3: Real-system inference efficiency on ActCap. Mem. : GPU memory in GB (Total = peak usage, KV = raw KV cache size). TPS : tokens per second (higher is better).

Figure 3: Comparison of average inference time of different baseline algorithms on Qwen2.5-VL-3B-Instruct at different KV cache compression rates for the VS task. The left, middle, and right columns report the average end-to-end (E2E) latency, time to first token (TTFT), and time per output token (TPOT), respectively.

Figure 4: Comparison of average inference time between full KV cache and VisCache under varying numbers of output tokens.

Budget	V Cache	ActCap	DREAM1K	NExTQA	ActQA	EgoSchema	Avg.
Strategy	Fusion	ROUGE-L	ROUGE-L	Acc	Acc	Acc	Acc
Full cache (upper bound)
Full Cache	–	2.82	7.87	60.71	46.47	65.20	57.46
Without V Cache Fusion
Fixed	$\times$	2.43	7.24	59.00	43.44	63.60	55.35
Arithmetic	$\times$	2.67	7.17	58.07	42.31	57.00	52.46

Table 4: Ablation study on budget allocation strategy and value cache fusion. We keep

\text{RR}=28\%

of the visual KV cache on Qwen2.5-VL-32B-Instruct across all variants. Fixed : uniform budget per layer. Arithmetic : arithmetic progression (PyramidKV Cai et al. (2024) ). Geometric : geometric progression across four layer blocks (PDrop Xing et al. (2024) ). Parabola : our proposed parabolic schedule (Equation 3 ). V Cache Fusion : asymmetric value fusion.

Methods	RR	Scout	PruneKV	DREAM1K	EgoSchema
				ROUGE-L	Acc
PDrop	40%	-	-	8.63	50.00
VisCache	40%	$\surd$	$\times$	8.48	46.20
VisCache	40%	$\times$	$\surd$	9.65	48.00
VisCache	40%	$\surd$	$\surd$	9.79	52.00
PDrop	28%	-	-	8.62	49.60

Table 5: Comparison of different KV cache compression methods on various VQA and VS datasets. Bold and underlined numbers indicate the best and second-best results, respectively.

Ranking strategy	RR	ActCap	DREAM1K	NExTQA	ActQA	EgoSchema	Avg. Acc.
		ROUGE-L	ROUGE-L	Acc.	Acc.	Acc.
Shared global ranking	28%	2.45	8.70	41.25	40.66	55.00	45.64
Layer-specific ranking	28%	2.35	8.73	41.20	39.82	51.75	44.26
Shared global ranking	19%	2.39	8.26	41.44	38.51	54.60	44.85
Layer-specific ranking	19%	2.29	7.95	41.24	37.41	53.20	43.95

Table 6: Ablation of shared global and layer-specific visual-token ranking at matched retention ratios on Qwen2.5-VL-3B-Instruct. Bold numbers indicate the best result under each retention ratio.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Jaccard similarity between key visual tokens selected by the small VLM with MMR and those selected from the sum of attention scores across Qwen2.5-VL-3B-Instruct layers, under different visual token compression rates. This figure highlights the consistency between the two token sets at varying compression levels.

Figure 6: Trade-off between inference accuracy and inference speed under different truncation layers of VLLM.

Figure 7: The relationship between fusion strength

\mu

and accuracy on EgoSchema with Qwen2.5-VL-32B-Instruct.

Fusion Strength $\mu$	Mean Accuracy	Variance
0.1	59.80	1.15
0.2	60.40	2.30
0.3	60.40	0.03
0.4	58.40	0.44
0.5	58.40	0.75
0.6	59.00	1.00

Appendix

Table 7: Mean accuracy and variance under different fusion strengths

\mu

on EgoSchema. Each setting is evaluated three times independently.

Method	RR	FLOPs (T)	FLOPs Ratio	ActCap	DREAM1K	NExTQA	ActQA	EgoSchema	Avg
Benchmark				ROUGE-L	ROUGE-L	Acc	Acc	Acc	Acc
Qwen3-VL-4B-Instruct
Full Cache	100%	5.08	100%	18.76	14.24	65.72	46.13	67.60	59.82
VisCache	28%	0.52	10.2%	18.66	14.34	62.90	45.35	60.22	56.16
VisCache	19%	0.51	10.0%	18.23	14.32	63.41	47.17	60.22	56.93

Appendix

Table 8: Results of applying VisCache on the Qwen3-VL-4B-Instruct architecture. VisCache maintains competitive inference performance with minimal degradation compared to using full cache.

Method	RR	FLOPs (T)	FLOPs Ratio	ActCap	DREAM1K	ActQA
Benchmark				ROUGE-L	ROUGE-L	Acc
LLaVA-OneVision-Qwen2-7b-ov-hf
Full Cache	100%	32.73	100%	5.04	14.45	38.36
VisCache	28%	8.62	26%	5.52	11.71	34.97

Appendix

Table 9: Results of applying VisCache on the LLaVA-OneVision architecture. It shows that VisCache maintains competitive inference performance across different VLLM architectures, exhibiting minimal degradation compared to using full cache.

Method	RR	Bits	ActCap	DREAM1K	NExtQA	ActQA	EgoSchema	Avg.
Benchmark			ROUGE-L	ROUGE-L	Acc	Acc	Acc	Acc
Qwen2.5-VL-3B-Instruct
Full Cache	100%	32	2.63	9.19	34.69	40.58	57.20	44.16
VisCache	28%	32	2.42	8.70	41.25	40.66	55.00	45.64
VisCache (FlatQuant)	28%	4	2.45	8.58	40.28	43.07	56.20	46.52
VisCache (KIVI)	28%	4	2.43	8.50	40.81	43.48	55.60	46.63

Appendix

Table 10: Comparison of combining VisCache with different KV cache quantization methods on various VQA and VS datasets.

Method	RR	ActCap	DREAM1K	NExTQA	ActQA	EgoSchema	Avg. Acc.
Benchmark		ROUGE-L	ROUGE-L	Acc	Acc	Acc	Acc
Full Cache	100%	2.63	9.19	34.69	40.58	57.20	44.16
Q-Frame	28%	2.37	9.69	40.98	38.60	46.60	42.06
PyramidKV	28%	2.37	7.34	38.54	40.21	50.00	42.92
FastV	28%	2.43	8.59	39.60	39.58	47.60	42.26
PDrop	28%	2.42	8.62	38.90	37.34	49.60	41.95

Appendix

Table 11: Comparison of different KV cache compression methods under the same retention ratio on Qwen2.5-VL-3B-Instruct. Bold and underlined numbers indicate the best and second-best results, respectively. Avg. Acc. denotes the average accuracy over NExTQA, ActQA, and EgoSchema.

Method	RR	FLOPs	FLOPs	DREAM1K	EgoSchema
Method	RR	(T)	Ratio	ROUGE-L	Acc
Qwen2.5-VL-32B-Instruct
Full Cache	100%	93.08	100%	7.87	65.20
Q-Frame	28%	41.47	45%	7.16	62.00
PDrop	28%	20.15	22%	5.16	64.00
VisCache	28%	10.75	12%	7.29	65.80

Appendix

Table 12: Comparison of representative KV cache compression methods under the same retention ratio on Qwen2.5-VL-32B-Instruct. Bold and underlined numbers indicate the best and second-best results, respectively.

Figure 8: Performance on DREAM1K and EgoSchema under different visual KV cache retention ratios. The dashed lines indicate the full cache performance for each benchmark.

Method	RR	DREAM1K	EgoSchema
		ROUGE-L	Acc
FastV	28%	8.59	47.60
PDrop	28%	8.62	49.60
Scout + FastV	28%	8.49	48.80
Scout + PDrop	28%	8.56	52.80
FastV	19%	8.13	52.00

Appendix

Table 13: Compatibility analysis of Scout-based Temporal Redundancy Filtering (TRF) with existing visual token pruning methods.

Method	RR	E2E Latency (s)	TTFT (s)	TPOT (ms)
DREAM1K
Full Cache	100%	12.24	4.42	118.81
VisCache	40%	9.55 (1.28 $\times$ )	2.05	119.06
VisCache	28%	6.33 (1.93 $\times$ )	3.45	45.79
VisCache	19%	5.20 (2.35 $\times$ )	2.23	45.60
EgoSchema

Appendix

Table 14: Detailed inference time analysis under different KV cache retention ratios. We report the end-to-end (E2E) latency, time to first token (TTFT), and time per output token (TPOT) on DREAM1K and EgoSchema benchmarks based on Qwen2.5-VL-3B-Instruct backbone. Our VisCache consistently reduces inference latency as the KV cache RR decreases, achieving up to 2.35

\times

E2E speedup while maintaining efficient generation performance.

Method	Scout	ActCap	DREAM1K	NExTQA	ActQA	EgoSchema
	Model	ROUGE-L	ROUGE-L	Acc	Acc	Acc
Full Cache	–	2.63	9.19	34.69	40.58	57.20
VisCache	CLIP	2.42	8.70	41.25	40.66	55.00
	BLIP	2.44	8.49	37.18	40.81	46.80
	OpenCLIP	2.43	8.51	41.56	39.78	53.40

Appendix

Table 15: Effect of different scout VLMs for temporal filtering. We keep

\text{RR}=28\%

of the visual KV cache on Qwen2.5-VL-32B-Instruct across all variants.

Figure 9: Comparison between Full Cache and VisCache on a pull-up activity video.

Figure 10: Comparison between Full Cache and VisCache on a movie scene video.

Explore similar work

Make Your LVLM KV Cache More Lightweight
May 1, 2026Xihao Chen, Yangyang Guo, Roger ZimmermannKV Caching Efficient VLM Inference
Integrative Sciences and Engineering Programme, National University of Singapore · School of Computing, National University of Singapore
KVCapsule: Efficient Sequential KV Cache Compression for Vision-Language Models with Asymmetric Redundancy
May 14, 2026Yingbing Huang, Tharun Adithya Srikrishnan, Steven K. Reinhardt +1Vision-Language Models Efficient VLM Inference
University of Illinois Urbana-Champaign · AMD
HybridKV: Hybrid KV Cache Compression for Efficient Multimodal Large Language Model Inference
Apr 7, 2026Bowen Zeng, Feiyang Ren, Jun Zhang +4KV-Cache Compression Efficient Multimodal Inference
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security · Hangzhou Dianzi University, Hangzhou, China

May 1, 2026J/K move · Enter open · S save

Make Your LVLM KV Cache More Lightweight

Xihao Chen, Yangyang Guo, Roger Zimmermann

Integrative Sciences and Engineering Programme, National University of Singapore · School of Computing, National University of Singapore

KV Caching Efficient VLM Inference Large Vision-Language Models KV-Cache Compression Visual Token Compression

Open paper AskPDF arXiv

Key-Value (KV) cache has become a de facto component of modern Large Vision-Language Models (LVLMs) for inference. While it enhances decoding efficiency in Large Language Models (LLMs), its direct adoption in LVLMs introduces substantial GPU memory overhead due to the large number of vision tokens processed during the prefill stage. To tackle this problem, we propose LightKV, a novel approach that reduces KV cache size by exploiting the redundancy among vision-token embeddings. Guided by text prompts, LightKV employs cross-modality message passing to aggregate informative messages across vision tokens and progressively compress them during prefill. This prompt-aware guidance distinguishes our method from prior vision-only compression strategies. We evaluate LightKV on eight open-source LVLMs across eight public benchmark datasets, e.g., MME and SeedBench. Experimental results demonstrate that with only 55% of the original vision tokens, LightKV (a) halves the vision-token KV cache size, (b) reduces computation by up to 40%, and (c) preserves general-purpose performance while significantly outperforming existing baselines.

Abstract

Explore similar work

Make Your LVLM KV Cache More Lightweight

KVCapsule: Efficient Sequential KV Cache Compression for Vision-Language Models with Asymmetric Redundancy

HybridKV: Hybrid KV Cache Compression for Efficient Multimodal Large Language Model Inference