ASAP: Attention-Shift-Aware Pruning for Efficient LVLM Inference

Surendra Pathak; Bo Han

cs.CVMar 15, 2026

ASAP: Attention-Shift-Aware Pruning for Efficient LVLM Inference

Authors: Surendra Pathak, Bo Han

Organizations: George Mason University, Fairfax, VA, USA

Abstract

While Large Vision-Language Models (LVLMs) demonstrate exceptional multi-modal capabilities, the quadratic computational cost of processing high-resolution visual tokens remains a critical bottleneck. Though recent token reduction strategies attempt to accelerate inference, such methods inadequately exploit attention values and fail to address token redundancy. More critically, they overlook the ``attention shift'' phenomenon inherent in LVLMs, which skews token attention scores. In this work, we propose ASAP, a novel training-free, KV-Cache-compatible pruning recipe that comprehensively addresses these limitations. First, we mitigate the attention shift by utilizing a dynamic bidirectional soft attention mask, ensuring the selection of genuinely informative tokens rather than naive attention-based selection. Second, we posit that high semantic redundancy within the token set degrades performance. We therefore introduce a weighted soft merging component that merges semantically similar tokens, preserving only the most feature-dense visual patches for subsequent layers. ASAP achieves virtually lossless compression of visual context, retaining 99.02% of the original LLaVA-NeXT-7B performance while aggressively slashing computational FLOPs by ~80%.

Figures & tables

Figure 1: ASAP integration within the LVLM architecture. The plug-and-play pruning module is embedded directly within the language model backbone, operating between standard decoder layers to compress the visual token sequence during the inference forward pass.

Panel of Figure 2 — Figure 2: Qualitative comparison of token selection. Retained tokens (top 128) are visible, while dropped tokens are darkened. FastV demonstrates a bottom spatial bias, missing the distant third car. ASAP successfully mitigates this bias, preserving critical background features to accurately count all cars.

Figure 3: The two-stage ASAP architecture. After decoder layer $L$ , a salience-guided bidirectional matrix retains top- $k$ salient visual tokens. A feature-based similarity module then consolidates redundant tokens via salience-weighted merging and a salvage mechanism. This dual-filtering pipeline passes a compact, feature-dense visual set to layer $L+1$ , strictly preserving all text and system prompts.

Method	FLOPs	Ratio	VQA ${}^{\scalebox{0.6}{\text{V2}}}$	GQA	SQA	VQA ${}^{\scalebox{0.6}{\text{Text}}}$	MME	MMB	MMB ${}^{\scalebox{0.6}{\text{CN}}}$	Avg
LLaVA-1.5-13B	3.817	100%	80.0	63.25	71.6	61.19	1529.93	67.71	63.60	100
Retain 192 tokens ( $\downarrow$ 66.67%)
FastV [ECCV’24]	1.259	32.99	75.42	58.67	70.64	57.23	1442.31	66.43	61.05	95.37
PruMerge+ [ICCV’25]	1.253	32.83	76.97	59.72	71.10	57.12	1451.31	66.32	62.54	95.94
SparseVLM [ICML’25]	1.253	32.83	78.31	57.80	73.08	51.93	1485.27	67.61	61.77	95.75
DivPrune [CVPR’25]	1.253	32.83	78.42	59.12	73.08	58.84	1482.01	65.95	59.02	96.68

Table 1: Performance comparison of ASAP with other pruning strategies on LLaVA-1.5-13B [ 31 ] . Ratio represents the FLOPs ratio (percentage of vanilla FLOPs retained). Avg refers to the average percentage performance across all benchmarks compared to the vanilla model. Best scores are highlighted in red .

Retain 960 tokens ( $\downarrow$ 66.67%)
Method	FLOPs	VQA ${}^{\scalebox{0.6}{\text{V2}}}$	GQA	VQA ${}^{\scalebox{0.6}{\text{Text}}}$	MMB	MMVet	Avg
LLaVA-NeXT-7B	20.825	81.2	62.9	59.6	65.8	40.0	100.0
FastV [ECCV’24]	6.477	80.33	61.98	59.32	65.12	39.4	98.89
VisPruner [ICCV’25]	6.458	80.0	62.20	58.81	65.29	38.4	98.26
ASAP (Ours)	6.258	81.19	62.29	59.67	66.41	40.1	100.06
Retain 640 tokens ( $\downarrow$ 77.78%)

Table 2: Performance comparison of ASAP with other pruning strategies on LLaVA-NeXT-7B [ 30 ] . Avg refers to the average percentage performance across all benchmarks compared to the vanilla model. Best scores are highlighted in red .

Method	FLOPs	Ratio (%)	GQA	SQA ${}^{\scalebox{0.6}{\text{IMG}}}$	POPE	MME	MMB ${}^{\scalebox{0.6}{\text{CN}}}$	Avg
InternVL 2.5 8B	12.575	100.0	63.40	97.81	90.59	1696.26	82.60	100.00
Retain 510 tokens
FastV [ECCV’24]	3.969	31.56	61.44	96.03	89.10	1614.42	80.32	97.17
ASAP (Ours)	3.969	31.56	62.31	97.07	90.08	1645.32	82.47	98.76
Retain 340 tokens
FastV [ECCV’24]	2.663	21.17	59.17	91.51	88.47	1603.51	79.33	95.02

Table 3: Performance comparison of ASAP with other pruning strategies on InternVL 2.5 8B. Avg refers to the average percentage performance across all benchmarks compared to the vanilla model. Best scores within each token budget are highlighted in red .

Figure 4: Ablation of ASAP modules on TextVQA and MME. FastV prunes naively using Top- $k$ selection. Bidirectional uses Salience-Guided Bidirectional attention values. ASAP adds Salience-Weighted Consolidation and Budget Reallocation. The progressive gains confirm that combining attention-driven selection with feature consolidation maximizes semantic density.

Retain 192 tokens
Method	# Token	FLOPs (T)	Percption score	Storage (MB)	GPU Memory (GB)	CUDA Time (ms)
LLaVA-1.5-7B	576	3.817	1506.46	321.39	14.33	38.27
FastV [ECCV’24]	192	1.259	1437.45	129.73	13.98	33.65
ASAP (Ours)	192	1.259	1466.19	116.64	13.96	35.13
Retain 128 tokens
FastV [ECCV’24]	128	0.836	1365.435	97.33	13.97	31.45

Table 4: System efficiency comparison on LLaVA-1.5-7B. We evaluate the computational cost and memory footprint during the prefill phase.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Impact of integrating SG-BiMask in existing pruning frameworks. Our SG-BiMask component consistently improves performance when integrated with the standard causal attention baseline (FastV) across all evaluated benchmarks.

Explore similar work

OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning
May 28, 2026Geng Li, Guohao Chen, Ting Chen +6Efficient VLM Inference Large Vision-Language Models
1Nanyang Technological University (NTU)
VPRune: Efficient Training-free Pre-LLM Visual Token Pruning
Sep 21, 2026Guangchuan Lv, Dianxing Shi, Dingjie FuEfficient VLM Inference Large Vision-Language Models
Northeastern University · Beihang University · Huazhong University of Science and Technology
ACPruner: Visual Token Pruning as Biased Attention Coverage Maximization in LVLMs
Sep 28, 2026Xu Li, Yuxuan Liang, Yi Zheng +6Efficient VLM Inference Large Vision-Language Models
College of Computer Science and Artificial Intelligence Fudan University, Shanghai, China

May 28, 2026J/K move · Enter open · S save

OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning

Geng Li, Guohao Chen, Ting Chen, Shilin Shan+5

1Nanyang Technological University (NTU)

Efficient VLM Inference Large Vision-Language Models Visual Token Pruning Visual Token Compression

Open paper AskPDF arXiv

Vision-language models (VLMs) rely on long visual token sequences for visual understanding, making the prefill stage expensive in both computation and memory. Most existing pruning methods follow an absolute-ranking paradigm, assigning importance scores to visual tokens and retaining a fixed top-K subset. In this work, we argue that this paradigm is fundamentally brittle: attention sinks distort token importance rankings, while image redundancy and query-dependent visual evidence make fixed token budgets unreliable across inputs. We propose OccamToken, a training-free framework that replaces absolute token ranking with register-anchored relative evidence testing. Instead of asking which tokens are globally important, OccamToken evaluates whether a visual token provides information beyond a register-based reference. Our key insight is that register tokens naturally absorb low-information attention patterns, making them a stable reference for identifying genuinely informative visual evidence. Based on this principle, OccamToken performs both image-adaptive redundancy pruning and query-adaptive relevance pruning through dynamic thresholds derived from register attention. Across LLaVA-NeXT, LLaVA-v1.5, and Qwen3-VL, OccamToken consistently improves the accuracy-efficiency trade-off without additional training. Notably, on LLaVA-NeXT, it reduces 2,880 visual tokens to approximately 40 while preserving over 93% of the original accuracy, enabling stable visual token compression even in the extreme 1.4% retention regime.

Abstract

Explore similar work

OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning

VPRune: Efficient Training-free Pre-LLM Visual Token Pruning

ACPruner: Visual Token Pruning as Biased Attention Coverage Maximization in LVLMs