Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding

Mincheol Kwon; Minseung Lee; Seonga Choi; Miso Choi; Kyeong-Jin Oh; Hyunyoung Lee; Cheonyoung Park; Yongho Song

cs.CVMar 24, 2026

Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding

Authors: Mincheol Kwon, Minseung Lee, Seonga Choi, Miso Choi, Kyeong-Jin Oh, Hyunyoung Lee, Cheonyoung Park, Yongho Song,

Organizations: Korea University · KT Corporation · Soongsil University · Kakao Mobility

Abstract

Large Vision-Language Models (LVLMs) have shown strong performance across various multimodal tasks by leveraging the reasoning capabilities of Large Language Models (LLMs). However, processing visually complex and information-rich images, such as infographics or document layouts, requires these models to generate a large number of visual tokens, leading to significant computational overhead. To address this, we propose PinPoint, a novel two-stage framework that first identifies instruction-relevant image regions and then refines them to extract fine-grained visual features for improved reasoning and efficiency. Central to our approach is the Instruction-Region Alignment, which localizes relevant regions using both visual input and textual instructions. We further introduce new annotations that provide richer ground-truth supervision for instruction-relevant regions across challenging VQA benchmarks: InfographicVQA, MultiPageDocVQA, and SinglePageDocVQA. Experimental results show that PinPoint not only achieves superior accuracy compared to existing methods but also reduces computational overhead by minimizing irrelevant visual tokens.

Figures & tables

Figure 1: Given an input instruction (e.g., “What are the average retirement savings for Americans?”), (a) a conventional approach aims to prune tokens based on attention weights; however, this method is often unreliable and can produce hallucinated outputs. (b) Our method, called PinPoint, first identifies instruction-relevant regions and then refines these regions to retain only the most relevant tokens, thereby improving the model’s factuality.

Figure 2: VQA performance increases with a larger proportion of instruction-relevant visual tokens. Model: LLaVA-NeXT [ 22 ] . Data: InfographicVQA [ 32 ] .

Figure 3: An overview of the proposed PinPoint architecture, which comprises two main stages: (1) Region Selection and (2) Region Refinement. Given a user instruction (e.g., “What symbol represents American identity?”), the Region Selection stage identifies adaptive top- $k$ instruction-relevant image regions (e.g., areas surrounding the Statue of Liberty) using region-level feature extraction via a sliding window approach. The Region Refinement stage then further processes these regions to extract fine-grained visual tokens, which are subsequently fed into a large language model (LLM) to generate the final answer (e.g., “The statue of liberty”).

Figure 4: Illustration for intra-image contrastive loss. Region features required to answer the given instruction are treated as positives, while all other regions are treated as negatives. The loss function pulls the positive pair $(E^{t},E^{v}_{\text{pos}})$ closer in the embedding space, while pushing away the negative pairs $(E^{t},E^{v}_{\text{neg},i})$ .

Figure 5: (a) Built upon existing VQA benchmarks, such as InfoVQA [ 32 ] , SPDocVQA [ 33 ] , and MPDocVQA [ 39 ] , we construct a new dataset using a pipeline that provides annotations of instruction-relevant regions likely to contain answers and supporting evidence. (b) Examples of the annotated regions. More examples are available in the supplemental material.

Model	Method	InfoVQA		SPDocVQA		MPDocVQA		GQA		Latency (ms)
	Method	ANLS $\uparrow$	FLOPs(T) (%) $\downarrow$	ANLS $\uparrow$	FLOPs(T) (%) $\downarrow$	ANLS $\uparrow$	FLOPs(T) (%) $\downarrow$	Acc $\uparrow$	FLOPs(T) (%) $\downarrow$	Latency (ms)
LLaVA -NeXT-7B	Vanilla [ 22 ]	0.2552	38.98 (100.0%)	0.6628	51.68 (100.0%)	0.3758	50.01 (100.0%)	0.7598	30.39 (100.0%)	569.5
	Random sampling	0.2387	24.62 (63.2%)	0.4888	27.70 (53.6%)	0.2988	26.66 (53.3%)	0.7484	19.26 (63.4%)	360.7
	ToMe [ 5 ]	0.1975	26.67 (68.4%)	0.3215	39.41 (76.3%)	0.2135	38.32 (56.6%)	0.7293	19.10 (62.9%)	315.1
	FastV [ 7 ]	0.2306	26.22 (67.3%)	0.6099	28.10 (54.4%)	0.3523	29.31 (58.6%)	0.7478	19.23 (65.9%)	386.0
	PDrop [ 43 ]	0.2335	26.00 (66.7%)	0.5507	26.93 (52.1%)	0.3637	30.82 (61.6%)	0.7436	20.33 (66.9%)	392.0

Table 1: Quantitative comparison with state-of-the-art methods on VQA benchmarks: InfoVQA [ 32 ] , SPDocVQA [ 33 ] , MPDocVQA [ 39 ] , and GQA [ 16 ] . We report Average Normalized Levenshtein Similarity (ANLS) scores, TFLOPs, and latency demonstrating that our proposed method consistently outperforms existing approaches while utilizing a comparable number of visual tokens. For reference, we also include results from a vanilla model that does not apply any token reduction strategy.

Figure 6: A qualitative comparison among FastV [ 7 ] , PyramidDrop [ 43 ] , and our proposed PinPoint reveals distinct behavioral differences. While FastV [ 7 ] and PyramidDrop [ 43 ] tend to prune tokens within the answer-relevant regions—often leading to hallucinatory or semantically inconsistent responses—our PinPoint effectively identifies and preserves the answer-critical regions, thereby yielding more accurate and contextually grounded predictions. Note that red indicates the ground-truth answer regions, blue denotes the sliding windows selected by our method, and purple highlights the image areas pinpointed as most relevant to the predicted answer.

Model	InfoVQA		SPDocVQA		MPDocVQA		GQA
	ANLS $\uparrow$	FLOPs(T) $\downarrow$	ANLS $\uparrow$	FLOPs(T) $\downarrow$	ANLS $\uparrow$	FLOPs(T) $\downarrow$	ANLS $\uparrow$	FLOPs(T) $\downarrow$
Vanilla [ 22 ]	0.2552	38.98 (100.0%)	0.6628	51.68 (100.0%)	0.3758	50.01 (100.0%)	0.7598	30.39 (100%)
ViCrop [ 48 ]	0.2547	147.39 (378.1%)	0.5941	168.71 (168.7%)	0.3661	171.86 (343.6%)	0.7497	137.58 (452.71%)
Ours	0.3024	25.48 (65.3%)	0.6472	28.44 (55.0%)	0.3866	28.28 (56.5%)	0.7608	17.96 (59.1%)
Ours $+$ Global	0.3075	33.09 (84.9%)	0.6732	37.34 (72.3%)	0.3976	37.20 (74.4%)	0.7704	26.64 (87.7%)

Table 2: Performance Analysis of Focused Region Selection. PinPoint effectively selects and provides answer-relevant regions to the LLM, resulting in higher accuracy with fewer tokens compared to the vanilla baseline [ 22 ] . Conversely, ViCrop [ 48 ] introduces irrelevant regions that act as noise, causing performance degradation despite a high computational cost.

Model	$\mathcal{L}_{intra}$	InfoVQA		SPDocVQA		MPDocVQA		GQA
	$\mathcal{L}_{intra}$	ANLS $\uparrow$	Region Acc. $\uparrow$	ANLS $\uparrow$	Region Acc. $\uparrow$	ANLS $\uparrow$	Region Acc. $\uparrow$	ANLS $\uparrow$	Region Acc. $\uparrow$
LLaVA -NeXT-7B	–	0.3011	82%	0.6465	96%	0.3811	85%	0.7465	93%
LLaVA -NeXT-7B	✓	0.3024	84%	0.6472	98%	0.3866	87%	0.7608	98%
Qwen2 -VL-7B	–	0.7010	92%	0.8799	96%	0.6595	90%	0.7600	97%
Qwen2 -VL-7B	✓	0.7140	95%	0.8977	98%	0.6723	94%	0.7624	98%

Table 3: Effect of the intra-image contrastive loss (

\mathcal{L}_{intra}

). Including the

\mathcal{L}_{intra}

loss component (

\checkmark

) consistently improves both ANLS and Region Accuracy scores compared to the baseline (–) across all models and datasets.

Region Refinement	InfoVQA		SPDocVQA		MPDocVQA		GQA
	ANLS	Region Acc.	ANLS	Region Acc.	ANLS	Region Acc.	ANLS	Region Acc.
–	0.2603	84%	0.6230	98%	0.3403	87%	0.7484	98%
✓	0.3024	84%	0.6472	98%	0.3866	87%	0.7608	98%

Table 4: Effect of Region Refinement Module. By capturing more fine-grained and discriminative visual cues, Region Refinement allows the model to significantly outperform the baseline.

Model	Trainable Params	Percent of Total
LLaVA-NeXT-7B [ 22 ]	99.4 M	1.40%
Qwen2-VL-7B [ 41 ]	72.6 M	0.86%

Table 5: Parameter Efficiency of PinPoint. The table details the number of trainable parameters (our PinPoint module) relative to the total parameters of the frozen base models. In both cases, the trainable parameters constitute less than 1.5% of the entire model, highlighting the high efficiency of our method.

Model	Vanilla	Ours
	Total	PinPoint	Total	Ratio
LLaVA-NeXT-7B	40.30	1.67	23.37	7.14%
Qwen2-VL-7B	52.30	5.68	25.44	22.30%

Table A1 : Computational Overhead of the PinPoint Module. The table reports FLOPs of the frozen model (Vanilla), FLOPs (T) with PinPoint, and the resulting ratio.

Model	InfoVQA	SPDocVQA	MPDocVQA	GQA
Vanilla [ 41 ]	0.7399	0.9359	0.7775	0.7687
Ours + Global	0.7560	0.9380	0.8071	0.7682

Table A2 : Performance Analysis of Focused Region Selection for Qwen2-VL [ 41 ] . PinPoint effectively identifies and supplies answer-relevant regions, thereby improving performance on VQA tasks. Metric: ANLS.

r

: 0.3.

Figure A1 : A qualitative comparison among LLaVA-NeXT [ 22 ] , FastV [ 7 ] , PyramidDrop [ 43 ] , and our PinPoint shows that the vanilla model [ 22 ] produces rich but often hallucinated descriptions, whereas FastV [ 7 ] and PyramidDrop [ 43 ] generate overly short outputs that miss key details. In contrast, PinPoint yields coherent, detailed descriptions with fewer hallucinations by leveraging instruction-relevant regions. Note that blue marks the windows selected by PinPoint, and purple indicates the pinpointed answer-relevant regions. In addition, light blue indicates well-grounded keywords, whereas light red denotes hallucinated keywords.

Method	InfoVQA	SPDocVQA	MPDocVQA	GQA	Avg. FLOPs(T)
Vanilla [ 22 ]	0.2552	0.6628	0.3758	0.7598	42.77
Honeybee [ 6 ]	0.2287	0.4451	0.2395	0.7460	27.80
Ours	0.3024	0.6472	0.3866	0.7608	25.04

Table A3 : Performance comparison of training-based efficiency methods. Our method outperforms Honeybee [ 6 ] across all evaluated datasets with lower computational cost. Model: LLaVA-NeXT [ 22 ]

Method	POPE			CHAIR		AMBER				Avg. FLOPs(T)
Method	Rand. ↑	Pop. ↑	Adv. ↑	CHAIR S ↓	CHAIR I ↓	CHAIR ↓	Cover ↑	Hal ↓	Cog ↓	Avg. FLOPs(T)
Vanilla	88.1	86.8	85.6	26.1	7.8	8.7	62.1	48.9	4.6	34.32
Ours	89.0	87.7	86.0	25.6	7.1	8.0	53.1	42.4	3.9	22.18

Table A4 : Hallucination evaluation on POPE [ 24 ] (discriminative) and CHAIR [ 34 ] / AMBER [ 40 ] (generative). PinPoint achieves the lowest hallucination rates across POPE [ 24 ] , CHAIR [ 34 ] , and AMBER [ 40 ] , outperforming in both discriminative and generative settings while maintaining strong reliability. Model: LLaVA-NeXT [ 22 ] .

Method	Training Data	MMMU	MMMU-Pro (standard 10)	TextVQA	Avg. FLOPs(T)
Vanilla [ 22 ]	-	0.3410 (100.0%)	0.1864 (100.0%)	0.5327 (100.0%)	39.92 (100.0%)
FastV [ 7 ]	Train-free	0.3356 (98.4%)	0.1927 (103.4%)	0.5102 (95.8%)	28.17 (70.6%)
PDrop [ 43 ]	Train-free	0.3411 (100.0%)	0.1871 (100.4%)	0.5240 (98.4%)	26.00 (65.1%)
PinPoint (Ours)	GQA	0.3500 (102.6%)	0.1990 (106.8%)	0.7293 (136.9%)	24.39 (61.1%)

Table A5 : Cross-dataset performance on general and real-world benchmarks. Despite being trained exclusively on the GQA [ 16 ] dataset, PinPoint demonstrates superior generalization and robustness across challenging real-world tasks, including MMMU [ 45 ] , MMMU-Pro [ 46 ] and TextVQA [ 38 ] . Our method significantly outperforms state-of-the-art train-free approaches while achieving the lowest computational cost (FLOPs).

Model	Setting	InfoVQA		SPDocVQA		MPDocVQA
	Setting	ANLS $\uparrow$	Region Acc. $\uparrow$	ANLS $\uparrow$	Region Acc. $\uparrow$	ANLS $\uparrow$	Region Acc. $\uparrow$
LLaVA-NeXT-7B	(a)	0.2980	82%	0.6457	97%	0.3836	86%
LLaVA-NeXT-7B	(b)	0.3024	84%	0.6472	98%	0.3866	87%
Qwen2-VL-7B	(a)	0.7017	93%	0.8843	96%	0.6572	90%
Qwen2-VL-7B	(b)	0.7140	95%	0.8977	98%	0.6723	94%

Table A6 : Effect of Encompass Supervision on Instruction–Region Alignment. compare two supervision settings for instruction-relevant regions: (a) a single, tightly bounded answer box, and (b) encompass regions that aggregate multiple reasoning boxes covering all supporting visual evidence. Using encompass supervision (b) consistently yields higher ANLS and Region Accuracy across all datasets and base models, showing that including contextual supporting regions leads to better instruction–image alignment.

Model	$\boldsymbol{r}$	InfoVQA			SPDocVQA			MPDocVQA			GQA
	$\boldsymbol{r}$	ANLS	Acc.	Cov.	ANLS	Acc.	Cov.	ANLS	Acc.	Cov.	ANLS	Acc.	Cov.
LLaVA-NeXT-7B	20%	0.2964	67%	36%	0.5724	83%	32%	0.3891	71%	36%	0.7219	85%	32%
	40%	0.3014	77%	55%	0.6270	93%	52%	0.3836	83%	56%	0.7508	93%	50%
	60%	0.3024	84%	71%	0.6472	98%	71%	0.3866	87%	72%	0.7608	98%	69%
Qwen2-VL-7B	20%	0.5882	69%	33%	0.7405	84%	30%	0.6094	73%	35%	0.7172	86%	30%
	40%	0.6649	87%	54%	0.8563	95%	54%	0.6578	86%	56%	0.7521	94%	51%

Table A7 : Ablation on Region Coverage Threshold

\boldsymbol{r}

. We vary the region coverage threshold

r

in the Region Selection stage and report answer accuracy (ANLS), region localization accuracy (Acc.), and region coverage (Cov.) for each dataset and base model. Across all settings, PinPoint maintains strong performance, with larger

r

values increasing coverage and generally yielding modest ANLS gains.

Dataset	Split	Annotation Count			Image Count	Image Pages	QA Pairs
	Split	answer	evidence	encompass	Image Count	Image Pages	QA Pairs
InfoVQA	Train	1.06	1.64	1.0	4,162	1.0	17,887
InfoVQA	Validation	1.09	1.53	1.0	500	1.0	2,801
SPDocVQA	Train	1.00	0.30	1.0	10,194	1.0	39,463
SPDocVQA	Validation	1.00	0.28	1.0	1,286	1.0	5,349
MPDocVQA	Train	1.00	1.05	1.0	5,131	9.1	36,230

Table A8 : Number of Bounding Box Annotations and QA Pairs per Dataset.

Category	Train (%)	Validation (%)
Visually Grounded	41.4	38.7
OCR-Extractable (Unique)	39.4	31.2
OCR-Extractable (Multiple)	12.6	21.9
Manual Annotation	6.6	8.2

Table A9 : Distribution of InfoVQA [ 32 ] Questions by Processing Type.

Category	Train (%)	Validation (%)
Visually Grounded	35.2	28.0
OCR-Extractable (Unique)	62.6	59.0
OCR-Extractable (Multiple)	2.2	12.9
Manual Annotation	0.0	0.1

Table A10 : Distribution of SPDocVQA [ 33 ] Questions by Processing Type.

Category	Train (%)	Validation (%)
Visually Grounded	39.0	57.4
OCR-Extractable (Unique)	52.2	34.0
OCR-Extractable (Multiple)	8.6	7.9
Manual Annotation	0.2	0.7

Table A11 : Distribution of MPDocVQA [ 39 ] Questions by Processing Type.

Figure A2 : Additional Qualitative Results for PinPoint on InfoVQA [ 32 ] large-size dataset. Red marks the ground-truth regions, blue shows the window selected by our method.

Figure A3 : Additional Qualitative Results for PinPoint on InfoVQA [ 32 ] medium-size dataset. Red marks the ground-truth regions, blue shows the windows selected by our method, and purple highlights the pinpointed answer-relevant areas.

Figure A4 : Additional Qualitative Results for PinPoint on SPDocVQA [ 33 ] dataset. Red marks the ground-truth regions, blue shows the windows selected by our method, and purple highlights the pinpointed answer-relevant areas.

Figure A5 : Additional Qualitative Results for PinPoint on GQA [ 16 ] dataset. Red marks the ground-truth regions, blue shows the windows selected by our method, and purple highlights the pinpointed answer-relevant areas.

Figure A6 : Qualitative examples of the PinPoint annotations on InfoVQA [ 32 ] . Purple boxes denote answer regions, pink boxes indicate supporting evidence regions, and yellow boxes represent encompass regions that jointly cover both answer and evidence, providing a more complete instruction-relevant area for supervision.

Figure A7 : Qualitative examples of the PinPoint annotations on SPDocVQA [ 33 ] . Purple boxes denote answer regions, pink boxes indicate supporting evidence regions, and yellow boxes represent encompass regions that jointly cover both answer and evidence, providing a more complete instruction-relevant area for supervision.

Figure A8 : Prompt for selecting a single bounding box among multiple candidates.

Figure A9 : prompt for generating a reasoning sentence required to derive the correct answer.

Figure A10 : Prompt for extracting and localizing the answer and supporting reasoning elements from OCR entries.

Figure A11 : Prompt for identifying instruction-relevant regions using a VLM

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 485 papers

Adaptive Visual Token Reduction for Accelerated Image Understanding
Oct 7, 2026Seyoung Jeong, Jong Pil Yun, Sang Jun LeeEfficient VLM Inference Visual Token Pruning
Jeonbuk National University, Jeonju, Republic of Korea · Korea Institute of Industrial Technology (KITECH), Incheon, Republic of Korea · Chung-Ang University, Seoul, Republic of Korea
Think with Structured Grounding: Perceptual Reinforcement Learning for Chart and Visual-Tabular Understanding
Aug 23, 2026Changjiang Jiang, Qiannian Zhao, Lei Xin +3Visually Grounded Reasoning Fine-Grained Visual Perception
Mohamed bin Zayed University of Artificial Intelligence · Nanjing University
Look Before You Reason: Implicit Visual Thinking for Efficient Multimodal Reasoning
Oct 19, 2026Zerui Chen, Changrui Chen, Fei Ni +1Multimodal Reasoning Efficient Multimodal Inference
MVP Lab, Imperial College London

Oct 7, 2026J/K move · Enter open · S save

Adaptive Visual Token Reduction for Accelerated Image Understanding

Seyoung Jeong, Jong Pil Yun, Sang Jun Lee

Jeonbuk National University, Jeonju, Republic of Korea · Korea Institute of Industrial Technology (KITECH), Incheon, Republic of Korea · Chung-Ang University, Seoul, Republic of Korea

Efficient VLM Inference Visual Token Pruning Visual Token Compression Visual Question Answering

Open paper AskPDF arXiv

Large Vision-Language Models achieve strong VQA performance, but processing high-resolution, information-rich images requires substantial computation, motivating visual token reduction. However, existing methods often prune individual tokens or rely on fixed-size cropping, limiting their ability to preserve spatially structured information such as horizontally or vertically elongated text. To address this limitation, we propose ReFIT, an instruction-guided visual token reduction framework for efficient LVLM inference. ReFIT consists of Relevance-Guided Window Reshaping (RWR) and Instruction-Guided Token Refinement (ITR), where RWR captures instruction-relevant regions by adapting to their spatial characteristics, while ITR further removes unnecessary visual tokens. Experiments on four VQA benchmarks demonstrate that ReFIT improves answer accuracy while reducing computational cost, and qualitative results demonstrate its effectiveness in localizing relevant regions and removing unnecessary visual information.

Abstract

Efficient VLM Inference

Explore similar work

Adaptive Visual Token Reduction for Accelerated Image Understanding

Think with Structured Grounding: Perceptual Reinforcement Learning for Chart and Visual-Tabular Understanding

Look Before You Reason: Implicit Visual Thinking for Efficient Multimodal Reasoning