Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects

Jun Zhang; Yicheng Ji; Feiyang Ren; Yihang Li; Bowen Zeng; Zonghao Chen; Ke Chen; Lidan Shou

cs.CLApr 7, 2026

Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects

Authors: Jun Zhang, Yicheng Ji, Feiyang Ren, Yihang Li, Bowen Zeng, Zonghao Chen, Ke Chen, Lidan Shou, +2 more

Organizations: The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security

Abstract

Large Vision-Language Models (LVLMs) enable sophisticated reasoning over images and videos, yet their inference is hindered by a systemic efficiency barrier known as visual token dominance. This overhead is driven by a multi-regime interplay between high-resolution feature extraction, quadratic attention scaling, and memory bandwidth constraints. We present a systematic taxonomy of efficiency techniques structured around the inference lifecycle, consisting of encoding, prefilling, and decoding. Unlike prior reviews focused on isolated optimizations, we analyze the end-to-end pipeline to reveal how upstream decisions dictate downstream bottlenecks, covering compute-bound visual encoding, the intensive prefilling of massive contexts, and the ''visual memory wall'' in bandwidth-bound decoding. By decoupling the efficiency landscape into the axes of shaping information density, managing long-context attention, and overcoming memory limits, this work provides a structured analysis of how isolated optimizations compose to navigate the trade-off between visual fidelity and system efficiency. The survey concludes by outlining four future frontiers supported by pilot empirical insights, including hybrid compression based on functional unit sensitivity, modality-aware decoding with relaxed verification, progressive state management for streaming continuity, and stage-disaggregated serving through hardware-algorithm co-design. Our literature repository is at https://github.com/SuDIS-ZJU/Efficient-LVLMs-Inference.

Figures & tables

Characteristic	Encoding Stage	Prefilling Stage	Decoding Stage
Primary Metric	TTFT	TTFT	TPOT
Bottleneck	Compute-Bound	Compute & Memory Bound	Memory-Bound
Arithmetic Intensity	High ( $\gg 1$ )	Medium	Low ( $\ll 1$ )
Complexity (FLOPs)	$\mathcal{O}(N_{p}\cdot D_{v}^{2})$	$\mathcal{O}((N_{v}+N_{t})^{2}\cdot D_{\mathcal{L}})$	$\mathcal{O}((N_{v}+N_{t})\cdot D_{\mathcal{L}})$
LVLM Challenge	High-res inputs ( $N_{p}\uparrow$ ) surge FLOPs	( $N_{v}\gg N_{t}$ ) causes quadratic spikes	Static visual KV cache saturates VRAM

Table 1: Hardware bottlenecks, arithmetic intensity, and complexity dynamics across the three inference stages.

Category	Method	Input Modality	Training-Free	Key Strategy & Insight
Diversity-Guided	G-Prune Jiang et al. (2025)	General	Yes	Similarity graph & information flow to retain representative tokens
	PACT Dhouib et al. (2025)	General	Yes	Distance-bounded clustering & merging redundant tokens
	DivPrune Alvar et al. (2025)	General	Yes	Max-Min diversity optimization for token subset selection
	CDPruner Zhang et al. (2025i)	General	Yes	Determinantal Point Processes (DPP) & conditional diversity
	DART Wen et al. (2025)	General	Yes	Pivot-based duplication pruning
	DyCoke Tao et al. (2025a)	Video	Yes	Plug-and-play module for temporal token merging

Table 2: Representative prefilling-side token compression methods, categorized by the optimization signal (Diversity or Attention) and input modality (general vision-language, video, and streaming video).

Granularity	Method	Scenario	Key Strategy & Insight
Token-Level	LOOK-M Wan et al. (2024b)	Static	Text-Prior Pruning : Prioritizes textual KVs; evicts visual tokens based on attention scores.
	Elastic Cache Liu et al. (2024d)	Static	Merging : Fuses less important KVs guided by distinct encoding/decoding metrics.
	FastCache Zhu et al. (2025a)	Serving	Self-supervised : Uses a lightweight modality-specific compressor to reduce overhead.
	Inf-MLLM Ning et al. (2024)	Streaming	Bias Adjustment : Maintains compact cache with adjustable attention bias for long-term dependency.
	SparseVILA Khaki et al. (2025)	Streaming	Decoupled Sparsity : Decouples query-agnostic pruning (prefill) and query-aware retrieval (decoding).
	ReKV Di et al. (2025)	Streaming	Retrieval : Offloads video chunks to external memory and selectively retrieves query-relevant KVs.

Table 3: Representative KV cache compression methods, categorized by operational granularity (token, layer, head, and bit) and inference scenario (static, streaming, and serving).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Paradigm	Optimal Workload	Key Mechanism	Performance Impact		Communication Cost
Paradigm	Optimal Workload	Key Mechanism	P99 TTFT	TPOT	Communication Cost
Stage-based	Long Video / Heavy Prefill	Inter-device Disaggregation	$\downarrow\downarrow$	$\uparrow$	High
(e.g., EPDServe)	(Prefill-dominant)	(Avoids Compute Blocking)	(Best Stability)	(Dedicated)	(Context/KV Transfer)
Modality-based	Balanced Multimodal	Inter-device Partitioning	$-$	$\uparrow\uparrow$	Medium
(e.g., ModServe)	(General VQA)	(Resource Specialization)	(Stable)	(Pipeline)	(Embedding Transfer)
Resource-based	Latency-Sensitive / Edge	SM-level Multiplexing	$\downarrow$	$\uparrow$	None

Appendix

Table 4: Qualitative comparison of LVLM serving paradigms, mapping architectural choices to workload characteristics. Legend:

\downarrow

\uparrow

denotes latency/throughput improvement; P99 TTFT indicates worst-case stability.

Metric	Category	Formulation	Definition
TTFT	Latency	$t_{\text{first}}-t_{\text{arr}}$	Time to First Token . The duration of the prefilling phase, measured from the request arrival time $t_{\text{arr}}$ to the first token generation $t_{\text{first}}$ .
TPOT	Latency	$\frac{t_{\text{end}}-t_{\text{first}}}{N}$	Time Per Output Token . The generation speed during decoding, measured from the first token $t_{\text{first}}$ to the last token generation $t_{\text{end}}$ , averaged over $N$ tokens.
SLO Attainment	Reliability	$\frac{1}{M}\sum_{i=1}^{M}\mathbb{I}(L_{r}\leq\tau)$	Service Level Objective Attainment . Proportion of total requests $M$ where the end-to-end latency $L_{r}$ of request $r$ satisfies the threshold $\tau$ . $\mathbb{I}(\cdot)$ is the indicator function.
Goodput	Throughput	$\frac{1}{T}\sum_{i=1}^{M}\mathbb{I}(L_{r}\leq\tau)$	Effective Throughput Rate . The number of requests per second that strictly satisfy the SLO constraint $\tau$ , calculated over the total serving duration $T$ .

Appendix

Table 5: Taxonomy of efficiency metrics, categorized by performance dimension (Latency, Reliability, Throughput).

Domain	Task Competency	Representative Benchmarks
	Multimodal Reasoning	MathVista Lu et al. (2023) , MMMU Yue et al. (2024) , MathVision Wang et al. (2024c) , DynaMath Zou et al. (2024) , LogicVista Xiao et al. (2024) , VPCT Shao et al. (2024) , MMMU-Pro Yue et al. (2025) , EMMA Hao et al. (2025) , SFE Zhou et al. (2025b) , ZeroBench Roberts et al. (2025) , WebQA Chang et al. (2022) , MultiModalQA Talmor et al. (2021)
Image	General Visual QA	HallusionBench Guan et al. (2024) , MMStar Chen et al. (2024c) , MMBench Liu et al. (2024b) , MUIRBench Wang et al. (2024a) , MMVP Zhang et al. (2024b) , VLMsAreBlind Rahmanzadehgervi et al. (2024) , VLMsAreBiased Vo et al. (2025) , SimpleVQA Cheng et al. (2025c) , MME-CC Zhang et al. (2025g) , MMCoQA Li et al. (2022) , SlideVQA Tanaka et al. (2023)
	Long-Context Understanding	DUDE Van Landeghem et al. (2023) , MMLongBench Wang et al. (2025f) , OminiDocBench Ouyang et al. (2025) , MileBench Song et al. (2024a)
	Long Video Understanding	CGBench Chen et al. (2024a) , LongVideoBench Wu et al. (2024a) , MLVU Zhou et al. (2025a) , LVBench Wang et al. (2024e) , ALLVB Tan et al. (2025) , VideoMME Fu et al. (2025) , VDC Chai et al. (2025)
Video	Knowledge & Reasoning	VideoMMMU Hu et al. (2025b) , MMVU Zhao et al. (2025) , VCRBench Qi et al. (2025) , VideoReasonBench Liu et al. (2025d) , VideoHolmes Cheng et al. (2025b) , Minerva Nagrani et al. (2025) , VideoSimpleQA Cao et al. (2025) , NExT-QA Xiao et al. (2021) , Video-ChatGPT Maaz et al. (2024)
	Motion & Perception	Countix Dwibedi et al. (2020) , TVBench Cores et al. (2024) , TempCompass Liu et al. (2024c) , TOMATO Shangguan et al. (2024) , MVBench Li et al. (2024) , EgoTempo Plizzari et al. (2025) , MotionBench Hong et al. (2025) , Vatex Wang et al. (2019)

Appendix

Table 6: Taxonomy of LVLM capability benchmarks, categorized by input domain (Image or Video) and chronological evolution from single-image perception to complex reasoning.

Method	Image				Video-ChatGPT
Method	SlideVQA	MMCoQA	WebQA	MM-QA	CI	DO	CU	TU	CO
Full Cache	83.50	66.50	76.50	75.00	3.06	3.15	3.52	2.24	2.69
LOOK-M	82.50	52.50	71.00	73.50	2.92	2.97	3.38	2.05	2.49
MadaKV	82.00	55.00	70.50	74.50	2.94	3.03	3.41	2.02	2.56
SparseMM	83.50	62.00	70.50	75.50	2.90	2.95	3.37	1.99	2.52
Hybrid	83.50	63.00	76.00	76.00	2.99	3.05	3.47	2.15	2.57

Appendix

Table 7: Performance of four KV cache compression on Qwen2.5-VL-7B across image and video tasks. Image tasks use exact match accuracy . For Video-ChatGPT, scores (ranging from 0 to 5) are generated by gpt-4o-mini across five dimensions: Correctness of Information (CI), Detail Orientation (DO), Contextual Understanding (CU), Temporal Understanding (TU), and Consistency (CO).

Method	Budget	Accuracy	GPU Memory	Latency
Method	Budget	(Avg.)	(GB)	(ms/token)
Full Cache	100%	2.93	1.73	58.94
Hybrid	20%	2.88	0.40	42.08
Hybrid	10%	2.85	0.22	38.65

Appendix

Table 8: KV cache GPU memory usage and decoding latency on Qwen2.5-VL-7B with Video-ChatGPT.

Target / Draft Model	Method	Video Detail Caption
		Main Object		Detail		Camera		Background		Average
		Acc(%)	Score	Acc(%)	Score	Acc(%)	Score	Acc(%)	Score	Acc(%)	Score	Ret.(%)
Qwen2.5-VL-32B / 7B	SpecVLM	34.52	2.11	31.12	1.93	28.59	1.78	30.85	1.91	31.27	1.93	100.0
	+Random Relaxation (50%)	31.85	1.94	32.63	2.00	28.37	1.82	25.13	1.70	29.50	1.87	95.6
	+Random Relaxation (75%)	31.23	1.97	31.87	1.90	26.22	1.73	25.63	1.75	28.74	1.84	93.6
Qwen2.5-VL-32B / 3B	SpecVLM	34.52	2.11	31.12	1.93	28.59	1.78	30.85	1.91	31.27	1.93	100.0

Appendix

Table 9: Performance metric using Qwen2.5-VL on three speculative decoding settings and Video Detail Caption benchmark. Ret.(%) refers to the performance retention ratio on average of accuracy and score, compared with autoregressive decoding.

Target / Draft Model	Method	Video Detail Caption
Target / Draft Model	Method	Mean Accepted Length	Speedup
Qwen2.5-VL-32B / 7B	SpecVLM	3.40	1.40 $\times$
	+Random Relaxation (50%)	5.55	2.04 $\times$
	+Random Relaxation (75%)	7.42	2.61 $\times$
Qwen2.5-VL-32B / 3B	SpecVLM	2.99	0.97 $\times$
	+Random Relaxation (50%)	5.30	1.64 $\times$

Appendix

Table 10: Efficiency metrics using Qwen2.5-VL on three speculative decoding settings and Video Detail Caption benchmark. Draft tokens per decoding step is set to 10. Decoding speedup is measured relative to autoregressive decoding.

Figure 4: Stage-wise bottleneck analysis of generic LVLM inference on NVIDIA A100. We illustrate the operational intensity of distinct inference phases against the hardware Roofline limits. The Decoding phase is strictly memory-bound (

\mathcal{I}_{a}\approx 1

), constrained by bandwidth. In contrast, the Visual Encoding phase represents a compute-bound workload (

\mathcal{I}_{a}\approx 1200

). The Prefilling phase (

\mathcal{I}_{a}\approx 160

) occupies the transitional region near the hardware’s ridge point (

\mathcal{I}_{ridge}\approx 153

FLOPs

/

Byte), utilizing both compute and memory resources efficiently.

Parameter	Value
Peak Performance ( $\pi_{peak}$ )	$312$ TFLOPS ( FP16 Tensor Core)
Memory Bandwidth ( $\beta_{mem}$ )	$2,039$ GB/s (HBM2e)
Ridge Point ( $\mathcal{I}_{ridge}$ )	$\approx 153$ FLOPs $/$ Byte

Appendix

Table 11: Hardware specifications for the NVIDIA A100 (80GB) used in the Roofline model.

Explore similar work

Mar 30, 2026

Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies

Although Large Vision Language Models (LVLMs) have demonstrated impressive multimodal reasoning capabilities, their scalability and deployment are constrained by massive computational requirements. In particular, the massive amount of visual tokens from high-resolution input data aggravates the situation due to the quadratic complexity of attention mechanisms. To address these issues, the research community has developed several optimization frameworks. This paper presents a comprehensive survey of the current state-of-the-art techniques for accelerating LVLM inference. We introduce a systematic taxonomy that categorizes existing optimization frameworks into four primary dimensions: visual token compression, memory management and serving, efficient architectural design, and advanced decoding strategies. Furthermore, we critically examine the limitations of these current methodologies and identify critical open problems to inspire future research directions in efficient multimodal systems.

Surendra Pathak, Bo Han

Computer Science Department, George Mason University, Fairfax, VA - 22030, USA

Efficient VLM Inference Visual Token Compression

Ask PDF

Mar 25, 2026

Attention-aware Inference Optimizations for Large Vision-Language Models with Memory-efficient Decoding

Large Vision-Language Models (VLMs) have achieved remarkable success in multi-modal reasoning, but their inference time efficiency remains a significant challenge due to the memory overhead during decoding, especially when the query and answer of VLMs consist of long sequences of visual and text tokens. This paper presents AttentionPack, an adaptive and attention-aware optimization framework tailored for large vision-language models with improving memory-efficiency during decoding, focusing on addressing the challenges due to the increased high number of visual inputs and interactions, particularly in long-context tasks with multiple high-resolution images or videos. AttentionPack is novel in two aspects: (i) We introduce a multi-head attention compaction method for economically storing key and value matrices by exploiting the implicit low-rank structure, and (ii) we develop a token-specific attention-aware decompression mechanism to reduce latency overhead. Experimental results on multiple benchmarks demonstrate that AttentionPack improves memory efficiency by up to 8x, enabling higher batch sizes and faster batch inference while preserving the model output quality or longer context lengths for superior retrieval performance. We also report the effectiveness of AttentionPack combined with eviction, quantization and kernel fusion, showing further efficiency gains for resource-limited environments.

Fatih Ilhan, Gaowen Liu, Ramana Rao Kompella +5

Georgia Institute of Technology · Cisco Research

Efficient VLM Inference Large Vision-Language Models KV-Cache Compression Long-Context VLMs

Ask PDF

Mar 17, 2026

Empirical Recipes for Efficient and Compact Vision-Language Models

Deploying vision-language models (VLMs) in resource-constrained settings demands low latency and high throughput, yet existing compact VLMs often fall short of the inference speedups their smaller parameter counts suggest. To explain this discrepancy, we conduct an empirical end-to-end efficiency analysis and systematically profile inference to identify the dominant bottlenecks. Based on these findings, we develop optimization recipes tailored to compact VLMs that substantially reduce latency while preserving accuracy. These techniques cut time to first token (TTFT) by 53% on InternVL3-2B and by 93% on SmolVLM-256M. Our recipes are broadly applicable across both VLM architectures and common serving frameworks, providing practical guidance for building efficient VLM systems. Beyond efficiency, we study how to extend compact VLMs with structured perception outputs and introduce the resulting model family, ArgusVLM. Across diverse benchmarks, ArgusVLM achieves strong performance while maintaining a compact and efficient design.

Jiabo Huang, Zhizhong Li, Sina Sajadmanesh +2

Sony AI.

Vision-Language Models Efficient VLM Inference

Ask PDF