Large Vision-Language Models (LVLMs) have shown strong performance across various multimodal tasks by leveraging the reasoning capabilities of Large Language Models (LLMs). However, processing visually complex and information-rich images, such as infographics or document layouts, requires these models to generate a large number of visual tokens, leading to significant computational overhead. To address this, we propose PinPoint, a novel two-stage framework that first identifies instruction-relevant image regions and then refines them to extract fine-grained visual features for improved reasoning and efficiency. Central to our approach is the Instruction-Region Alignment, which localizes relevant regions using both visual input and textual instructions. We further introduce new annotations that provide richer ground-truth supervision for instruction-relevant regions across challenging VQA benchmarks: InfographicVQA, MultiPageDocVQA, and SinglePageDocVQA. Experimental results show that PinPoint not only achieves superior accuracy compared to existing methods but also reduces computational overhead by minimizing irrelevant visual tokens.
Figures & tables
Figure 1: Given an input instruction (e.g., “What are the average retirement savings for Americans?”), (a) a conventional approach aims to prune tokens based on attention weights; however, this method is often unreliable and can produce hallucinated outputs. (b) Our method, called PinPoint, first identifies instruction-relevant regions and then refines these regions to retain only the most relevant tokens, thereby improving the model’s factuality.
Figure 2: VQA performance increases with a larger proportion of instruction-relevant visual tokens. Model: LLaVA-NeXT [ 22 ] . Data: InfographicVQA [ 32 ] .
Figure 3: An overview of the proposed PinPoint architecture, which comprises two main stages: (1) Region Selection and (2) Region Refinement. Given a user instruction (e.g., “What symbol represents American identity?”), the Region Selection stage identifies adaptive top- k instruction-relevant image regions (e.g., areas surrounding the Statue of Liberty) using region-level feature extraction via a sliding window approach. The Region Refinement stage then further processes these regions to extract fine-grained visual tokens, which are subsequently fed into a large language model (LLM) to generate the final answer (e.g., “The statue of liberty”).
Figure 4: Illustration for intra-image contrastive loss. Region features required to answer the given instruction are treated as positives, while all other regions are treated as negatives. The loss function pulls the positive pair (Et,Eposv) closer in the embedding space, while pushing away the negative pairs (Et,Eneg,iv) .
Figure 5: (a) Built upon existing VQA benchmarks, such as InfoVQA [ 32 ] , SPDocVQA [ 33 ] , and MPDocVQA [ 39 ] , we construct a new dataset using a pipeline that provides annotations of instruction-relevant regions likely to contain answers and supporting evidence. (b) Examples of the annotated regions. More examples are available in the supplemental material.
Model
Method
InfoVQA
SPDocVQA
MPDocVQA
GQA
Latency (ms)
ANLS ↑
FLOPs(T) (%) ↓
ANLS ↑
FLOPs(T) (%) ↓
ANLS ↑
FLOPs(T) (%) ↓
Acc ↑
FLOPs(T) (%) ↓
LLaVA -NeXT-7B
Vanilla [ 22 ]
0.2552
38.98 (100.0%)
0.6628
51.68 (100.0%)
0.3758
50.01 (100.0%)
0.7598
30.39 (100.0%)
569.5
Random sampling
0.2387
24.62 (63.2%)
0.4888
27.70 (53.6%)
0.2988
26.66 (53.3%)
0.7484
19.26 (63.4%)
360.7
ToMe [ 5 ]
0.1975
26.67 (68.4%)
0.3215
39.41 (76.3%)
0.2135
38.32 (56.6%)
0.7293
19.10 (62.9%)
315.1
FastV [ 7 ]
0.2306
26.22 (67.3%)
0.6099
28.10 (54.4%)
0.3523
29.31 (58.6%)
0.7478
19.23 (65.9%)
386.0
PDrop [ 43 ]
0.2335
26.00 (66.7%)
0.5507
26.93 (52.1%)
0.3637
30.82 (61.6%)
0.7436
20.33 (66.9%)
392.0
Table 1: Quantitative comparison with state-of-the-art methods on VQA benchmarks: InfoVQA [ 32 ] , SPDocVQA [ 33 ] , MPDocVQA [ 39 ] , and GQA [ 16 ] . We report Average Normalized Levenshtein Similarity (ANLS) scores, TFLOPs, and latency demonstrating that our proposed method consistently outperforms existing approaches while utilizing a comparable number of visual tokens. For reference, we also include results from a vanilla model that does not apply any token reduction strategy.
Figure 6: A qualitative comparison among FastV [ 7 ] , PyramidDrop [ 43 ] , and our proposed PinPoint reveals distinct behavioral differences. While FastV [ 7 ] and PyramidDrop [ 43 ] tend to prune tokens within the answer-relevant regions—often leading to hallucinatory or semantically inconsistent responses—our PinPoint effectively identifies and preserves the answer-critical regions, thereby yielding more accurate and contextually grounded predictions. Note that red indicates the ground-truth answer regions, blue denotes the sliding windows selected by our method, and purple highlights the image areas pinpointed as most relevant to the predicted answer.
Model
InfoVQA
SPDocVQA
MPDocVQA
GQA
ANLS ↑
FLOPs(T) ↓
ANLS ↑
FLOPs(T) ↓
ANLS ↑
FLOPs(T) ↓
ANLS ↑
FLOPs(T) ↓
Vanilla [ 22 ]
0.2552
38.98 (100.0%)
0.6628
51.68 (100.0%)
0.3758
50.01 (100.0%)
0.7598
30.39 (100%)
ViCrop [ 48 ]
0.2547
147.39 (378.1%)
0.5941
168.71 (168.7%)
0.3661
171.86 (343.6%)
0.7497
137.58 (452.71%)
Ours
0.3024
25.48 (65.3%)
0.6472
28.44 (55.0%)
0.3866
28.28 (56.5%)
0.7608
17.96 (59.1%)
Ours + Global
0.3075
33.09 (84.9%)
0.6732
37.34 (72.3%)
0.3976
37.20 (74.4%)
0.7704
26.64 (87.7%)
Table 2: Performance Analysis of Focused Region Selection. PinPoint effectively selects and provides answer-relevant regions to the LLM, resulting in higher accuracy with fewer tokens compared to the vanilla baseline [ 22 ] . Conversely, ViCrop [ 48 ] introduces irrelevant regions that act as noise, causing performance degradation despite a high computational cost.
Model
Lintra
InfoVQA
SPDocVQA
MPDocVQA
GQA
ANLS ↑
Region Acc. ↑
ANLS ↑
Region Acc. ↑
ANLS ↑
Region Acc. ↑
ANLS ↑
Region Acc. ↑
LLaVA -NeXT-7B
–
0.3011
82%
0.6465
96%
0.3811
85%
0.7465
93%
✓
0.3024
84%
0.6472
98%
0.3866
87%
0.7608
98%
Qwen2 -VL-7B
–
0.7010
92%
0.8799
96%
0.6595
90%
0.7600
97%
✓
0.7140
95%
0.8977
98%
0.6723
94%
0.7624
98%
Table 3: Effect of the intra-image contrastive loss ( Lintra ). Including the Lintra loss component ( ✓ ) consistently improves both ANLS and Region Accuracy scores compared to the baseline (–) across all models and datasets.
Region Refinement
InfoVQA
SPDocVQA
MPDocVQA
GQA
ANLS
Region Acc.
ANLS
Region Acc.
ANLS
Region Acc.
ANLS
Region Acc.
–
0.2603
84%
0.6230
98%
0.3403
87%
0.7484
98%
✓
0.3024
0.6472
0.3866
0.7608
Table 4: Effect of Region Refinement Module. By capturing more fine-grained and discriminative visual cues, Region Refinement allows the model to significantly outperform the baseline.
Model
Trainable Params
Percent of Total
LLaVA-NeXT-7B [ 22 ]
99.4 M
1.40%
Qwen2-VL-7B [ 41 ]
72.6 M
0.86%
Table 5: Parameter Efficiency of PinPoint. The table details the number of trainable parameters (our PinPoint module) relative to the total parameters of the frozen base models. In both cases, the trainable parameters constitute less than 1.5% of the entire model, highlighting the high efficiency of our method.
Model
Vanilla
Ours
Total
PinPoint
Total
Ratio
LLaVA-NeXT-7B
40.30
1.67
23.37
7.14%
Qwen2-VL-7B
52.30
5.68
25.44
22.30%
Table A1 : Computational Overhead of the PinPoint Module. The table reports FLOPs of the frozen model (Vanilla), FLOPs (T) with PinPoint, and the resulting ratio.
Model
InfoVQA
SPDocVQA
MPDocVQA
GQA
Vanilla [ 41 ]
0.7399
0.9359
0.7775
0.7687
Ours + Global
0.7560
0.9380
0.8071
0.7682
Table A2 : Performance Analysis of Focused Region Selection for Qwen2-VL [ 41 ] . PinPoint effectively identifies and supplies answer-relevant regions, thereby improving performance on VQA tasks. Metric: ANLS. r : 0.3.
Figure A1 : A qualitative comparison among LLaVA-NeXT [ 22 ] , FastV [ 7 ] , PyramidDrop [ 43 ] , and our PinPoint shows that the vanilla model [ 22 ] produces rich but often hallucinated descriptions, whereas FastV [ 7 ] and PyramidDrop [ 43 ] generate overly short outputs that miss key details. In contrast, PinPoint yields coherent, detailed descriptions with fewer hallucinations by leveraging instruction-relevant regions. Note that blue marks the windows selected by PinPoint, and purple indicates the pinpointed answer-relevant regions. In addition, light blue indicates well-grounded keywords, whereas light red denotes hallucinated keywords.
Method
InfoVQA
SPDocVQA
MPDocVQA
GQA
Avg. FLOPs(T)
Vanilla [ 22 ]
0.2552
0.6628
0.3758
0.7598
42.77
Honeybee [ 6 ]
0.2287
0.4451
0.2395
0.7460
27.80
Ours
0.3024
0.6472
0.3866
0.7608
25.04
Table A3 : Performance comparison of training-based efficiency methods. Our method outperforms Honeybee [ 6 ] across all evaluated datasets with lower computational cost. Model: LLaVA-NeXT [ 22 ]
Method
POPE
CHAIR
AMBER
Avg. FLOPs(T)
Rand. ↑
Pop. ↑
Adv. ↑
CHAIR S ↓
CHAIR I ↓
CHAIR ↓
Cover ↑
Hal ↓
Cog ↓
Vanilla
88.1
86.8
85.6
26.1
7.8
8.7
62.1
48.9
4.6
34.32
Ours
89.0
87.7
86.0
25.6
7.1
8.0
53.1
42.4
3.9
22.18
Table A4 : Hallucination evaluation on POPE [ 24 ] (discriminative) and CHAIR [ 34 ] / AMBER [ 40 ] (generative). PinPoint achieves the lowest hallucination rates across POPE [ 24 ] , CHAIR [ 34 ] , and AMBER [ 40 ] , outperforming in both discriminative and generative settings while maintaining strong reliability. Model: LLaVA-NeXT [ 22 ] .
Method
Training Data
MMMU
MMMU-Pro (standard 10)
TextVQA
Avg. FLOPs(T)
Vanilla [ 22 ]
-
0.3410 (100.0%)
0.1864 (100.0%)
0.5327 (100.0%)
39.92 (100.0%)
FastV [ 7 ]
Train-free
0.3356 (98.4%)
0.1927 (103.4%)
0.5102 (95.8%)
28.17 (70.6%)
PDrop [ 43 ]
Train-free
0.3411 (100.0%)
0.1871 (100.4%)
0.5240 (98.4%)
26.00 (65.1%)
PinPoint (Ours)
GQA
0.3500 (102.6%)
0.1990 (106.8%)
0.7293 (136.9%)
24.39 (61.1%)
Table A5 : Cross-dataset performance on general and real-world benchmarks. Despite being trained exclusively on the GQA [ 16 ] dataset, PinPoint demonstrates superior generalization and robustness across challenging real-world tasks, including MMMU [ 45 ] , MMMU-Pro [ 46 ] and TextVQA [ 38 ] . Our method significantly outperforms state-of-the-art train-free approaches while achieving the lowest computational cost (FLOPs).
Model
Setting
InfoVQA
SPDocVQA
MPDocVQA
ANLS ↑
Region Acc. ↑
ANLS ↑
Region Acc. ↑
ANLS ↑
Region Acc. ↑
LLaVA-NeXT-7B
(a)
0.2980
82%
0.6457
97%
0.3836
86%
(b)
0.3024
84%
0.6472
98%
0.3866
87%
Qwen2-VL-7B
(a)
0.7017
93%
0.8843
96%
0.6572
90%
(b)
0.7140
95%
0.8977
98%
0.6723
94%
Table A6 : Effect of Encompass Supervision on Instruction–Region Alignment. compare two supervision settings for instruction-relevant regions: (a) a single, tightly bounded answer box, and (b) encompass regions that aggregate multiple reasoning boxes covering all supporting visual evidence. Using encompass supervision (b) consistently yields higher ANLS and Region Accuracy across all datasets and base models, showing that including contextual supporting regions leads to better instruction–image alignment.
Model
r
InfoVQA
SPDocVQA
MPDocVQA
GQA
ANLS
Acc.
Cov.
ANLS
Acc.
Cov.
ANLS
Acc.
Cov.
ANLS
Acc.
Cov.
LLaVA-NeXT-7B
20%
0.2964
67%
36%
0.5724
83%
32%
0.3891
71%
36%
0.7219
85%
32%
40%
0.3014
77%
55%
0.6270
93%
52%
0.3836
83%
56%
0.7508
93%
50%
60%
0.3024
84%
71%
0.6472
98%
71%
0.3866
87%
72%
0.7608
98%
69%
Qwen2-VL-7B
20%
0.5882
69%
33%
0.7405
84%
30%
0.6094
73%
35%
0.7172
86%
30%
40%
0.6649
87%
54%
0.8563
95%
54%
0.6578
86%
56%
0.7521
94%
51%
Table A7 : Ablation on Region Coverage Threshold r . We vary the region coverage threshold r in the Region Selection stage and report answer accuracy (ANLS), region localization accuracy (Acc.), and region coverage (Cov.) for each dataset and base model. Across all settings, PinPoint maintains strong performance, with larger r values increasing coverage and generally yielding modest ANLS gains.
Dataset
Split
Annotation Count
Image Count
Image Pages
QA Pairs
answer
evidence
encompass
InfoVQA
Train
1.06
1.64
1.0
4,162
1.0
17,887
Validation
1.09
1.53
1.0
500
1.0
2,801
SPDocVQA
Train
1.00
0.30
1.0
10,194
1.0
39,463
Validation
1.00
0.28
1.0
1,286
1.0
5,349
MPDocVQA
Train
1.00
1.05
1.0
5,131
9.1
36,230
Table A8 : Number of Bounding Box Annotations and QA Pairs per Dataset.
Category
Train (%)
Validation (%)
Visually Grounded
41.4
38.7
OCR-Extractable (Unique)
39.4
31.2
OCR-Extractable (Multiple)
12.6
21.9
Manual Annotation
6.6
8.2
Table A9 : Distribution of InfoVQA [ 32 ] Questions by Processing Type.
Category
Train (%)
Validation (%)
Visually Grounded
35.2
28.0
OCR-Extractable (Unique)
62.6
59.0
OCR-Extractable (Multiple)
2.2
12.9
Manual Annotation
0.0
0.1
Table A10 : Distribution of SPDocVQA [ 33 ] Questions by Processing Type.
Category
Train (%)
Validation (%)
Visually Grounded
39.0
57.4
OCR-Extractable (Unique)
52.2
34.0
OCR-Extractable (Multiple)
8.6
7.9
Manual Annotation
0.2
0.7
Table A11 : Distribution of MPDocVQA [ 39 ] Questions by Processing Type.
Figure A2 : Additional Qualitative Results for PinPoint on InfoVQA [ 32 ] large-size dataset. Red marks the ground-truth regions, blue shows the window selected by our method.
Figure A3 : Additional Qualitative Results for PinPoint on InfoVQA [ 32 ] medium-size dataset. Red marks the ground-truth regions, blue shows the windows selected by our method, and purple highlights the pinpointed answer-relevant areas.
Figure A4 : Additional Qualitative Results for PinPoint on SPDocVQA [ 33 ] dataset. Red marks the ground-truth regions, blue shows the windows selected by our method, and purple highlights the pinpointed answer-relevant areas.
Figure A5 : Additional Qualitative Results for PinPoint on GQA [ 16 ] dataset. Red marks the ground-truth regions, blue shows the windows selected by our method, and purple highlights the pinpointed answer-relevant areas.
Figure A6 : Qualitative examples of the PinPoint annotations on InfoVQA [ 32 ] . Purple boxes denote answer regions, pink boxes indicate supporting evidence regions, and yellow boxes represent encompass regions that jointly cover both answer and evidence, providing a more complete instruction-relevant area for supervision.
Figure A7 : Qualitative examples of the PinPoint annotations on SPDocVQA [ 33 ] . Purple boxes denote answer regions, pink boxes indicate supporting evidence regions, and yellow boxes represent encompass regions that jointly cover both answer and evidence, providing a more complete instruction-relevant area for supervision.
Figure A8 : Prompt for selecting a single bounding box among multiple candidates.
Figure A9 : prompt for generating a reasoning sentence required to derive the correct answer.
Figure A10 : Prompt for extracting and localizing the answer and supporting reasoning elements from OCR entries.
Figure A11 : Prompt for identifying instruction-relevant regions using a VLM
Large Vision-Language Models achieve strong VQA performance, but processing high-resolution, information-rich images requires substantial computation, motivating visual token reduction. However, existing methods often prune individual tokens or rely on fixed-size cropping, limiting their ability to preserve spatially structured information such as horizontally or vertically elongated text. To address this limitation, we propose ReFIT, an instruction-guided visual token reduction framework for efficient LVLM inference. ReFIT consists of Relevance-Guided Window Reshaping (RWR) and Instruction-Guided Token Refinement (ITR), where RWR captures instruction-relevant regions by adapting to their spatial characteristics, while ITR further removes unnecessary visual tokens. Experiments on four VQA benchmarks demonstrate that ReFIT improves answer accuracy while reducing computational cost, and qualitative results demonstrate its effectiveness in localizing relevant regions and removing unnecessary visual information.
Seyoung Jeong, Jong Pil Yun, Sang Jun Lee
Jeonbuk National University, Jeonju, Republic of Korea · Korea Institute of Industrial Technology (KITECH), Incheon, Republic of Korea · Chung-Ang University, Seoul, Republic of Korea
Multimodal Large Language Models (MLLMs) capable of thinking with images often rely on external tools for fine-grained perception. However, this reliance introduces significant inference latency and fails to effectively resolve the spatial-structural gap-a fundamental challenge in text-dense and structurally relational visuals (e.g., charts and visual tables) where strict relative spatial arrangements bind textual elements. Without external tools, standard MLLMs struggle with such fine-grained visual reasoning tasks. To address these issues, we propose Think with Structured Grounding (TwSG), a novel fine-grained image perception framework designed to internalize complex images's tool-use capabilities within the model. TwSG distills the benefits of multi-step reasoning and micro-cropping into a single efficient forward pass during inference. Specifically, we use an MLLM to identify key regions guided by ground-truth answers, and then prompt a teacher model to generate high-quality visual question-answering (VQA) data. These fine-grained, region-based supervisory signals are subsequently distilled back into the full-image representation. Our training pipeline consists of two stages: (1) a cold-start supervised fine-tuning (SFT) phase using multi-turn data with focused area descriptions to foster complex reasoning and error recovery; and (2) a reinforcement fine-tuning (RFT) phase driven by a novel process reward mechanism, TL-GRPO, which encourages strategic reasoning. Extensive experiments across various MLLM architectures demonstrate that TwSG reduces inference latency while substantially improving accuracy and robustness, endowing models with native fine-grained region description and flexible reasoning capabilities.
Changjiang Jiang, Qiannian Zhao, Lei Xin +3
Mohamed bin Zayed University of Artificial Intelligence · Nanjing University
Vision-language models (VLMs) have advanced rapidly in multimodal reasoning, yet they remain limited on tasks requiring fine-grained understanding of small or easily overlooked image regions. Recent thinking with images approaches mitigate this limitation by allowing models to inspect images through operations such as zooming, cropping, or code execution. While effective, their multi-turn interactions with VLMs introduce redundant token computation and slow inference, and separate region inspection from the reasoning step that depends on it. To address these issues, we propose Look Before You Reason (LoRe), an implicit visual thinking framework for efficient visual reasoning within a single-turn VLM interaction. Given an image and a question, LoRe predicts the most relevant image region, aligns it with the corresponding image patch features, and strengthens the associated key-value (KV) cache entries, enabling subsequent decoding to directly exploit the selected visual cues without additional visual inputs or explicit tool calls. We further employ reinforcement learning to guide the VLM toward identifying informative image regions and leveraging them during reasoning. To support fine-grained evaluation of multimodal reasoning, we introduce COCO-ObjQA, a large-scale benchmark for object-centric multimodal reasoning. Experiments on three challenging multimodal benchmarks demonstrate that LoRe consistently improves VLM reasoning performance with low additional inference overhead.