VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions

Adrian Bulat; Alberto Baldrati; Ioannis Maniadis Metaxas; Yassine Ouali; Georgios Tzimiropoulos

cs.CVMar 24, 2026

VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions

Authors: Adrian Bulat, Alberto Baldrati, Ioannis Maniadis Metaxas, Yassine Ouali, Georgios Tzimiropoulos

Organizations: Samsung AI Cambridge · Technical University of Iasi · Queen Mary University of London

Abstract

Existing approaches for improving the efficiency of Large Vision-Language Models (LVLMs) are largely based on the concept of visual token reduction. This approach, however, creates an information bottleneck that impairs performance, especially on challenging tasks that require fine-grained understanding and reasoning. In this work, we challenge this paradigm by introducing VISion On Request (VISOR), a method that reduces inference cost without discarding visual information. Instead of compressing the image, VISOR improves efficiency by sparsifying the interaction between image and text tokens. Specifically, the language model attends to the full set of high-resolution visual tokens through a small, strategically placed set of attention layers: general visual context is provided by efficient cross-attention between text-image, while a few well-placed and dynamically selected self-attention layers refine the visual representations themselves, enabling complex, high-resolution reasoning when needed. Based on this principle, we first train a single universal network on a range of computational budgets by varying the number of self-attention layers, and then introduce a lightweight policy mechanism that dynamically allocates visual computation based on per-sample complexity. Extensive experiments show that VISOR drastically reduces computational cost while matching or exceeding state-of-the-art results across a diverse suite of benchmarks, and excels in challenging tasks that require detailed visual understanding.

Figures & tables

Figure 1: Efficiency comparison: FLOPs reduction vs acc. Notice that our approach is significantly more efficient while also retaining the performance on the harder datasets. See Sects. 3 and 5.1 for “easy”-“hard” definition.

Panel of Figure 2 — Figure 2: Cross-modality attention patterns across layers. We plot the proportion of attention scores allocated to three interaction types: text queries attending to image tokens (Query-to-Image), answer tokens attending to image tokens (Answer-to-Image), and answer tokens attending to query tokens (Answer-to-Query). For easy tasks like SQA, interaction is sparse and dominated by text-to-text attention. For hard tasks like DocVQA, the model attends to the image across the whole network.

Panel of Figure 3 — Figure 3: Evolution of visual representations across layers , measured by pairwise CKA similarity. For easy tasks (e.g., SQA), visual features remain largely static (high similarity across layers). For harder tasks (e.g., DocVQA), features are progressively refined.

Panel of Figure 4 — Figure 4: Accuracy sensitivity by dropping all vision tokens for different subsets of LLM layers. Left: Accuracy distribution on a dataset-by-dataset basis. Certain datasets (e.g., DocVQA, ChartQA) are particularly sensitive to reduced vision-language interactions. Right: we show how the layer-drop config. & accuracy correlate among datasets. Two clusters emerge: vision-sensitive (“hard”) (e.g., InfoVQA, OCRBench, etc.) and coarse vision (“easy”) (e.g., POPE, SQA, GQA, etc.) datasets.

Figure 5: Conceptual architecture of VISOR. Visual information is sparsely injected into the language stream via a few cross-attention and self-attention layers modelling text-image and image-image interactions. Cross-attention efficiently provides visual context to the text tokens without altering the visual representations. Self-attention, while more costly, refines the visual tokens, enabling subsequent cross-attention layers to access higher-level visual features. This design strikes a balance between efficiency and representational power.

Method	Easy									Hard					Avg. FLOPs Savings
Method	RWorldQA	SQA	GQA	MME	MSTAR	POPE	TextVQA	AI2D	Avg. (Easy)	ChartQA	OCRBench	InfoVQA	DocVQA	Avg. (Hard)	Avg. FLOPs Savings
LLaVA-OV [ 22 ]	54.0	67.2	58.3	60.6	40.6	88.4	66.0	56.7	61.5	60.9	58.8	40.0	68.7	57.1	$1.0\times$
Downsample [ 24 ]	52.8	67.4	57.4	62.4	40.0	86.1	58.4	56.7	60.2	49.3	47.9	27.3	49.5	43.5	$4.0\times$
VisionZip [ 50 ]	51.9	67.0	53.0	62.1	38.9	85.5	46.5	53.2	57.3	44.0	27.0	23.9	36.7	32.9	$5.7\times$
VisionZip † [ 50 ]	54.7	65.8	55.7	61.9	39.3	86.9	55.7	54.8	59.3	51.2	45.0	27.2	48.8	43.1	$5.7\times$
VisPruner [ 53 ]	53.3	65.7	53.2	60.1	38.1	85.9	47.8	53.7	57.2	44.0	28.3	25.4	42.7	35.1	$5.7\times$

Table 1: Comparison with state-of-the-art methods on various vision-language benchmarks. The metric used is accuracy for all datasets, except for MME where we report a score (higher is better; MME values are divided by 20 for normalization purposes).

approx. 4x token reduction
Method	FLOPs sav.	Easy	Hard
VISOR	$8.9\times$	63.6	58.4
VISOR-TR [2x]	$17.8\times$	63.3	57.8
VISOR-TR [4x]	$35.0\times$	63.1	56.2
VISOR+ VisionZip	$37.0\times$	63.3	55.3
VISOR+ VisPruner	$39.0\times$	63.5	55.9

Table 2: Accuracy comparison when combining VISOR with token reduction methods.

CA	Easy	Hard	SA	CA	Easy	Hard
6	63.3	51.8	2	8	63.2	55.1
8	63.4	52.0	4	8	63.3	56.9
10	63.5	52.1	7	8	63.9	58.3

Table 3: Acc. comparison across configurations and categories.

SA	CA	Universal	Easy	Hard
2	8	$\times$	63.1	55.1
2	8	✓	63.8	56.3
7	8	$\times$	63.6	58.3
7	8	✓	64.2	59.1

Table 4: Accuracy comparison between independently trained models and a universally trained model supporting multiple configurations. Both model variants use the same fixed configuration for all samples.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Performance heatmap for different configurations across datasets. Each row represents a configuration, and each column corresponds to a dataset. The color intensity indicates the relative accuracy achieved by that configuration on the respective dataset.

Method	RWQA	SQA	GQA	MME	MMSTAR	POPE	TxtVQA	AI2D	CQA	OCRB	InfoVQA	DocVQA
VISOR	54.6	75.3	61.8	60.5	40.1	87.6	67.8	61.5	65.2	61.8	37.6	68.9
VISOR	$8.4\times$	$8\times$	$12\times$	$8.3\times$	$10\times$	$12\times$	$6.3\times$	$12\times$	$8\times$	$6.6\times$	$6\times$	$6\times$
VISOR-TR	55.4	75.4	60.7	59.5	38.5	87.7	67.4	61.9	65.3	60.7	37.4	67.7
VISOR-TR	$24\times$	$16\times$	$24\times$	$17.7\times$	$17.7\times$	$24\times$	$14.5\times$	$24\times$	$16\times$	$15\times$	$12\times$	$12\times$

Appendix

Table 5: Per-dataset saving rates. We compare our method against state-of-the-art approaches using a shared LLaVA-OV (0.5B) backbone. For each method, the top row indicates the accuracy, while the bottom row shows the FLOPs savings relative to the baseline LLaVA-OV model. The metrics used are accuracy for most datasets, except for MME where we report a score (higher is better).

Figure 7: Layer configuration assignments made by the router for each test dataset.

Figure 8: Performance for various VISOR configurations.

Figure 9: VISOR oracle: smallest amount of layers that maximizes accuracy.

Figure 10: Efficiency comparison - number of FLOPS vs. vision sequence length.

Method	Easy									Hard
Method	RealWorldQA	SQA	GQA	MME	MSTAR	POPE	TextVQA	AI2D	Avg. (Easy)	ChartQA	OCRBench	InfoVQA	DocVQA	Avg. (Hard)
Qwen2-VL-2B	61.1	78.1	60.2	75.3	43.6	87.7	79.3	70.4	69.5	75.0	77.5	64.1	89.5	76.5
VISOR	60.0	82.1	60.8	75.1	49.2	88.9	76.1	75.0	70.9	78.1	76.0	55.9	87.3	74.3

Appendix

Table 6: Comparison on various vision-language benchmarks for Qwen2-VL-2B LVLM.

Method	Easy									Hard					Avg. FLOPs Savings
Method	RealWorldQA	SQA	GQA	MME	MSTAR	POPE	TextVQA	AI2D	Avg. (Easy)	ChartQA	OCRBench	InfoVQA	DocVQA	Avg. (Hard)	Avg. FLOPs Savings
LLaVA-OV [ 22 ]	54.0	67.2	58.3	60.6	40.6	88.4	66.0	56.7	61.5	60.9	58.8	40.0	68.7	57.1	$1.0\times$
+ $M^{3}$	53.6	75.1	59.1	63.6	41.5	88.1	65.5	62.2	63.6	62.9	54.0	34.7	58.0	52.4	$16.0\times$
+ VisPruner train.	53.1	69.2	56.7	60.9	36.9	86.7	55.4	56.3	59.4	52.4	44.2	28.2	52.9	44.4	$16.0\times$
VISOR-TR (Ours)	55.4	75.4	60.7	59.5	38.5	87.7	67.4	61.9	63.3	65.3	60.7	37.4	67.7	57.8	$18\times$

Appendix

Table 7: Comparison between token reduction vs intermittent attention (VISOR).

Method	RWQA	SQA	GQA	MME	MMSTAR	POPE	TxtVQA	AI2D	CQA	OCRB	InfoVQA	DocVQA
VISOR-TR	55.4	75.4	60.7	59.5	38.5	87.7	67.4	61.9	65.3	60.7	37.4	67.7
VISOR-TR	$24\times$	$16\times$	$24\times$	$17.7\times$	$17.7\times$	$24\times$	$14.5\times$	$24\times$	$16\times$	$15\times$	$12\times$	$12\times$
VISOR-TR-excl	55.5	75.6	61.6	59.2	38.3	88.3	66.9	62.0	65.0	61.1	37.2	67.6
VISOR-TR-excl	$14.5\times$	$15.5\times$	$9.8\times$	$11.4\times$	$20\times$	$9.8\times$	$12\times$	$14.5\times$	$16\times$	$14.1\times$	$12\times$	$12\times$

Appendix

Table 8: Results for the routing mechanism trained on the full train set (VISOR-TR), contrasted with training excluding samples from AI2D, DocVQA and GQA (VISOR-TR-excl).

Method	Easy									Hard					Avg. FLOPs Savings
Method	RealWorldQA	SQA	GQA	MME	MMSTAR	POPE	TextVQA	AI2D	Avg. (Easy)	ChartQA	OCRBench	InfoVQA	DocVQA	Avg. (Hard)	Avg. FLOPs Savings
LLaVA-OV [ 22 ]	57.3	75.6	58.6	63.7	40.5	88.1	69.9	60.3	64.3	64.2	60.3	46.7	76.6	62.0	$1.0\times$
Downsample [ 24 ]	53.6	76.9	58.4	62.5	39.2	86.1	59.5	60.0	62.0	46.7	48.0	29.1	52.6	44.1	$4.0\times$
VisionZip [ 50 ]	52.7	74.8	54.2	64.5	38.3	86.2	54.9	57.2	60.4	37.8	33.8	26.3	44.7	35.7	$5.7\times$
VisionZip † [ 50 ]	54.6	76.2	57.2	63.9	38.1	87.0	62.0	58.2	62.2	50.0	44.3	30.8	55.1	45.1	$5.7\times$
SparseVLM [ 56 ]	52.0	74.8	50.3	58.4	34.5	82.3	60.6	57.1	58.7	51.3	38.9	34.9	57.9	45.7	$4.5\times$

Appendix

Table 9: Comparison with state-of-the-art models on various vision-language benchmarks using a LLaVA-OV 1.5B backbone. The metrics used are accuracy for most datasets, except for MME where we report a score (higher is better). MME values are divided by 20.

Method	Training data %	RWQA	SQA	GQA	MME	MMSTAR	POPE	TxtVQA	AI2D	CQA	OCRB	InfoVQA	DocVQA
VISOR	100%	54.6	75.3	61.8	60.5	40.1	87.6	67.8	61.5	65.2	61.8	37.6	68.9
VISOR	50%	56.9	72.3	60.1	60.2	38.6	88.3	65.1	57.3	62.2	58.3	33.8	62.7

Appendix

Table 10: VISOR LLava-OV 0.5B results when training with different portions of the training data. We report performance when using the full training set (100%) and a reduced subset (50%).

Method	Vision Encoder	Easy									Hard					Avg. FLOPs Savings
Method	Vision Encoder	RealWorldQA	SQA	GQA	MME	MSTAR	POPE	TextVQA	AI2D	Avg. (Easy)	ChartQA	OCRBench	InfoVQA	DocVQA	Avg. (Hard)	Avg. FLOPs Savings
LLaVA-OV	SigLIP-400M	54.0	67.2	58.3	60.6	40.6	88.4	66.0	56.7	61.5	60.9	58.8	40.0	68.7	57.1	$1\times$
LLaVA-OV	FastVLM	52.5	67.4	56.3	58.4	35.8	87.7	61.3	51.5	58.9	57.9	47.6	32.5	51.1	47.3	$20\times$
VISOR	FastVLM	52.3	73.7	56.3	58.8	37.7	86.7	61.9	55.8	60.4	63.0	49.0	33.4	53.1	49.6	$60\times$

Appendix

Table 11: Comparison between LLaVA-OV 0.5B and VISOR with FastVLM vision encoder. Avg. FLOPs Savings refers only to LLM FLOPs and does not include the cost of vision encoding.

Method	MUIR	Blink	MMIU
LLaVA	26.8	40.4	34.2
VISOR	28.1	39.0	34.5

Appendix

Table 12: Multi-image performance comparison.

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 485 papers

ASAP: Attention-Shift-Aware Pruning for Efficient LVLM Inference
Mar 15, 2026Surendra Pathak, Bo HanEfficient VLM Inference Visual Token Pruning
George Mason University, Fairfax, VA, USA
Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
Apr 7, 2026Jun Zhang, Yicheng Ji, Feiyang Ren +7Efficient VLM Inference Large Vision-Language Models
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security
OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning
May 28, 2026Geng Li, Guohao Chen, Ting Chen +6Efficient VLM Inference Large Vision-Language Models
1Nanyang Technological University (NTU)

Mar 15, 2026J/K move · Enter open · S save

ASAP: Attention-Shift-Aware Pruning for Efficient LVLM Inference

Surendra Pathak, Bo Han

George Mason University, Fairfax, VA, USA

Efficient VLM Inference Visual Token Pruning LVLM Pruning Visual Token Compression

Open paper AskPDF arXiv

While Large Vision-Language Models (LVLMs) demonstrate exceptional multi-modal capabilities, the quadratic computational cost of processing high-resolution visual tokens remains a critical bottleneck. Though recent token reduction strategies attempt to accelerate inference, such methods inadequately exploit attention values and fail to address token redundancy. More critically, they overlook the ``attention shift'' phenomenon inherent in LVLMs, which skews token attention scores. In this work, we propose ASAP, a novel training-free, KV-Cache-compatible pruning recipe that comprehensively addresses these limitations. First, we mitigate the attention shift by utilizing a dynamic bidirectional soft attention mask, ensuring the selection of genuinely informative tokens rather than naive attention-based selection. Second, we posit that high semantic redundancy within the token set degrades performance. We therefore introduce a weighted soft merging component that merges semantically similar tokens, preserving only the most feature-dense visual patches for subsequent layers. ASAP achieves virtually lossless compression of visual context, retaining 99.02% of the original LLaVA-NeXT-7B performance while aggressively slashing computational FLOPs by ~80%.

Abstract

Efficient VLM Inference

Explore similar work

ASAP: Attention-Shift-Aware Pruning for Efficient LVLM Inference

Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects

OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning