Large Vision-Language Models (VLMs) have achieved remarkable success in multi-modal reasoning, but their inference time efficiency remains a significant challenge due to the memory overhead during decoding, especially when the query and answer of VLMs consist of long sequences of visual and text tokens. This paper presents AttentionPack, an adaptive and attention-aware optimization framework tailored for large vision-language models with improving memory-efficiency during decoding, focusing on addressing the challenges due to the increased high number of visual inputs and interactions, particularly in long-context tasks with multiple high-resolution images or videos. AttentionPack is novel in two aspects: (i) We introduce a multi-head attention compaction method for economically storing key and value matrices by exploiting the implicit low-rank structure, and (ii) we develop a token-specific attention-aware decompression mechanism to reduce latency overhead. Experimental results on multiple benchmarks demonstrate that AttentionPack improves memory efficiency by up to 8x, enabling higher batch sizes and faster batch inference while preserving the model output quality or longer context lengths for superior retrieval performance. We also report the effectiveness of AttentionPack combined with eviction, quantization and kernel fusion, showing further efficiency gains for resource-limited environments.
Figures & tables
Figure 1: The schematic of the workflow during inference. After the prefill phase, at each decoding step, we first compress the cache along combined heads. We perform attention-aware partial decompression before attention score computation.
Figure 2: Rank vs explained variance ratio without/with combining along head axis before compression for key and value vectors.
Figure 3: Visualization of compression and partial decompression.
Figure 4: Impact of attention-aware decompression. Each line represents the results when AttentionPack is applied for key (k), value (v) caches or both (kv). Every line has four dots with the size of each representing the ratio of visual tokens ( r1∈{0.125,0.25,0.375,0.5} ) decompressed with the full rank.
Figure 5: Total decode latency for 100 queries with LLaVA1.5-7B.
Figure 6: Decoding latency for 32 tokens with various batch sizes and sequence lengths using standard attention and with fused kernel implementation of AttentionPack. We observe up to 2.4x faster inference in single and batch query settings.
Figure 7: Visualization of scores (Eq. 1 ) at the first decoding step after processing the question on various examples from MSVD-QA, showing that the visual tokens closely related to the prompt have higher importance scores and will be decompressed with higher ranks.
Figure 8: Visualization of tracked attention-score statistics (Eq. 1 ) on various image QA pairs from OCR-VQA and A-OKVQA.
Figure 9: Total decoding latency for 100 queries across various context lengths (4/16 frames resulting in 1024/4096 visual tokens).
Figure 10: Accumulated attention scores of the first 250 tokens at the timestep T=500 from a random conversational text sample from MTBench [ 33 ] (for the first three layers, which are represented with different colors).
Large Vision-Language Models (LVLMs) enable sophisticated reasoning over images and videos, yet their inference is hindered by a systemic efficiency barrier known as visual token dominance. This overhead is driven by a multi-regime interplay between high-resolution feature extraction, quadratic attention scaling, and memory bandwidth constraints. We present a systematic taxonomy of efficiency techniques structured around the inference lifecycle, consisting of encoding, prefilling, and decoding. Unlike prior reviews focused on isolated optimizations, we analyze the end-to-end pipeline to reveal how upstream decisions dictate downstream bottlenecks, covering compute-bound visual encoding, the intensive prefilling of massive contexts, and the ''visual memory wall'' in bandwidth-bound decoding. By decoupling the efficiency landscape into the axes of shaping information density, managing long-context attention, and overcoming memory limits, this work provides a structured analysis of how isolated optimizations compose to navigate the trade-off between visual fidelity and system efficiency. The survey concludes by outlining four future frontiers supported by pilot empirical insights, including hybrid compression based on functional unit sensitivity, modality-aware decoding with relaxed verification, progressive state management for streaming continuity, and stage-disaggregated serving through hardware-algorithm co-design. Our literature repository is at https://github.com/SuDIS-ZJU/Efficient-LVLMs-Inference.
Jun Zhang, Yicheng Ji, Feiyang Ren +7
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security
Although Large Vision Language Models (LVLMs) have demonstrated impressive multimodal reasoning capabilities, their scalability and deployment are constrained by massive computational requirements. In particular, the massive amount of visual tokens from high-resolution input data aggravates the situation due to the quadratic complexity of attention mechanisms. To address these issues, the research community has developed several optimization frameworks. This paper presents a comprehensive survey of the current state-of-the-art techniques for accelerating LVLM inference. We introduce a systematic taxonomy that categorizes existing optimization frameworks into four primary dimensions: visual token compression, memory management and serving, efficient architectural design, and advanced decoding strategies. Furthermore, we critically examine the limitations of these current methodologies and identify critical open problems to inspire future research directions in efficient multimodal systems.
Surendra Pathak, Bo Han
Computer Science Department, George Mason University, Fairfax, VA - 22030, USA
While Vision Large Language Models (VLLMs) have achieved remarkable success in multimodal reasoning, their long-context inference remains prohibitively expensive due to the massive computation and memory overhead of visual Key-Value (KV) caches. Existing KV compression methods often apply uniform pruning across visual tokens and layers, leading to substantial information loss and degraded performance.To address this challenge, we propose VisCache, a plug-and-play framework for coarse-to-fine Visual KV Cache pruning without training, which consists of two synergistic stages. First, a lightweight VLM filters temporal redundancy by selectively forwarding semantically informative keyframes. Second, we introduce {PruneKV}, a surgical KV compression algorithm tailored to the attention dynamics of VLLMs. Unlike rigid pruning strategies, PruneKV adopts a parabolic layer-wise budget allocation together with an asymmetric update mechanism that selectively prunes keys while fusing values, thereby preserving critical contextual information. Extensive experiments demonstrate that VisCache substantially improves inference efficiency, achieving up to {2.35× speedup} and significant memory reduction while maintaining competitive performance with only {19--28%} KV cache retention. VisCache consistently outperforms existing baselines, establishing a new Pareto frontier between efficiency and performance for long-context VLLM inference. Code is available at https://github.com/Wlklk/VisCache
Lyuke Wang, Zhuo Li, Guangxu Zhu
Shenzhen International Center for Industrial and Applied Mathematics · Shenzhen Research Institute of Big Data · The Chinese University of Hong Kong, Shenzhen +1