Vision-language models (VLMs) fail many detail-centric questions for a concrete reason: the answer is visible in the image, yet lost after the image is compressed into a low-resolution global view. Allocating more visual tokens to every query improves some OCR and document cases, but it spends computation indiscriminately and can disturb tasks that rely on global context. We propose GapSight, a framework for learning visual re-reading: a VLM first takes a global glance, then selectively returns to a free-form region when the question calls for local evidence. The supervision comes from the target model's own failure signal. Offline, we compare answer loss or multiple-choice option margin under a global-only view and candidate crop-augmented views; crops that improve the target answer become model-specific review labels. A lightweight free-form crop router distills these labels into a one-shot inference policy that predicts whether to review, expected utility, and a continuous crop box from the global state. Across LLaVA-1.5-7B, InternVL2.5-8B, and Qwen2-VL-2B-Instruct, GapSight improves the Base no-zoom baseline on six benchmarks spanning OCR, documents, charts, infographics, VStarBench, and MME-RealWorld-Lite. On InternVL2.5-8B, GapSight raises the six-benchmark average from 52.25 to 64.29, above CropVLM (57.16), ViCrop (55.84), and ZoomRefine (54.43). Mechanism analyses show that the router rescues concrete wrong answers, adapts its action rate by task, and forms a favorable token-performance profile. These results position loss-gap supervision as a practical route to teaching VLMs when and where to look again.
Figures & tables
Figure 1: Overview of GapSight . Offline loss-gap mining probes the target VLM with a global view and candidate crop-augmented views, converting answer-loss or option-margin improvements into supervision for action, utility, and free-form box prediction. At inference time, the free-form crop router predicts from the global state whether to preserve the preview or inject one crop for visual re-reading.
Backbone
Method
TextVQA
DocVQA
ChartQA
InfoVQA
VStar
MME-Lite
Six Avg
Avg. tokens
LLaVA-1.5-7B
Base no-zoom
58.63
20.87
21.43
17.72
45.18
32.71
32.76±0.10
576.0
ZoomRefine
62.18
28.73
24.31
24.58
46.47
28.39
35.78±0.24
1151.2
ViCrop
59.04
27.18
21.76
22.94
51.96
33.88
36.13±0.22
1152.0
CropVLM
58.12
27.66
25.37
20.83
52.64
33.66
36.38±0.25
1151.3
GapSight
71.84
40.28
33.16
27.36
48.93
33.82
42.57±0.33
912.8
InternVL2.5-8B
Base no-zoom
73.74
55.84
63.61
39.42
51.24
29.66
52.25±0.10
256.0
Table 1: Per-benchmark results with external crop/zoom methods listed explicitly. Per-benchmark entries report mean scores over three seeds; TextVQA, DocVQA, ChartQA, and InfoVQA use 1,000-example evaluation subsets, VStarBench uses 191 examples, and MME-Lite uses 1,919 examples. Six Avg reports mean and standard deviation over the aggregate score. Bold marks the best score within each backbone and benchmark column.
Target
Source data
Rows
Pos.
Neg.
Ambig.
Answer NLL
TextVQA, DocVQA, ChartQA, InfoVQA
58,311
23,542
13,679
21,090
Option margin
GQA-derived A–D choices
80,000
21,491
5,496
53,013
Table 2: Mined label statistics. Answer-NLL labels use TextVQA, DocVQA, ChartQA, and InfographicVQA ( Singh et al. 2019 ; Mathew et al. 2021 ; Masry et al. 2022 ; Mathew et al. 2022 ) ; option-margin labels use GQA-derived A–D choice rows ( Hudson and Manning 2019 ) . Positive rows provide review actions and crop targets; ambiguous rows are not box-supervised.
Supervision
Text
Doc
Chart
Info
Avg.
Act.
Tokens
Random labels
78.5
57.5
66.5
53.5
64.00
60.2%
410.1
CLIP relevance
83.5
67.5
67.5
49.0
66.88
58.0%
404.5
loss-gap
87.26
78.11
67.63
53.27
71.57
59.3%
407.6
Table 3: Supervision-signal ablation on the four InternVL2.5-8B VQA benchmarks. The backbone, router, candidate bank, schedule, and evaluation are fixed; only the label teacher changes. Act. is action rate and Tokens is average visual-token use.
Figure 2: Token-performance comparison across LLaVA-1.5-7B, InternVL2.5-8B, and Qwen2-VL-2B-Instruct. Each point averages the method’s Six-Bench Avg and average visual-token usage across the three backbones. GapSight reaches the highest mean score while using fewer visual tokens than ZoomRefine, ViCrop, and CropVLM.
Figure 3: Cross-model transfer of mined crop utility. For each benchmark, rows indicate the model used to mine the crop and columns indicate the model used to evaluate it; L, I, and Q denote LLaVA, InternVL, and Qwen. Values are normalized by the target model’s self-mined crop utility. Off-diagonal values below 1.00 show that crops useful for one backbone often preserve only part of their utility for another backbone.
Benchmark
W → R
R → W
Net
Rescue
Harm
TextVQA
28
5
+23
59.6
3.3
DocVQA
44
7
+37
55.7
5.8
ChartQA
9
6
+3
13.6
4.5
InfographicVQA
26
9
+17
23.9
9.9
All
107
27
+80
35.6
5.4
Table 4: Benefit/harm transition analysis on the four InternVL2.5-8B VQA benchmarks. W → R counts Base no-zoom errors corrected by GapSight ; R → W counts regressions. Rescue and harm are percentages normalized by Base no-zoom wrong/right examples.
Benchmark
Score
Action
Avg. tokens
TextVQA
87.26
0.740
445.4
DocVQA
78.11
0.515
387.8
ChartQA
67.63
0.350
345.6
InfographicVQA
53.27
0.765
451.8
VStarBench
58.72
0.539
394.1
MME-Lite
40.76
0.256
321.5
Table 5: Task-level gate behavior for the InternVL2.5-8B reported operating point. The router reviews frequently on OCR/infographic tasks and is more conservative on MME-Lite.