The Answer Knows Where to Look: Zooming before MLLMs Answer under Visual-Token Budgets

Zhensheng Hu; Guanglong Xu; Guanqiang Hu

cs.CVOct 23, 2026

The Answer Knows Where to Look: Zooming before MLLMs Answer under Visual-Token Budgets

Authors: Zhensheng Hu, Guanglong Xu, Guanqiang Hu

Organizations: Integrated Hospital of Traditional Chinese Medicine, Southern Medical University, Guangzhou, China · Institute of Medical Engineering Interdisciplinary Research, Xinjiang Medical University, Urumqi, China · School of Mathematics and Statistics, Zhaoqing University, Guangdong 526061, China · Guangdong Kamfu Technology Co., Ltd., Foshan, China

Abstract

Multimodal large language models (MLLMs) read images through visual tokens, and under the small per-image budgets that efficient deployment requires, the fine text of documents, charts and infographics becomes illegible. Zooming in recovers the evidence, but the region must be chosen before the model answers. We find that MLLMs know where the evidence is, but not in a form that can be read before they answer: the answer tokens localize it from the middle layers on, whereas the attention available before answering does so only in the top layers, at a depth that varies across models, and diffusely. Supervising the pre-answer attention to point degrades reading, and the benefit of zooming is governed by each image's information gap. Building on these findings, we introduce ours, which lets an MLLM look where its answer would. A dedicated look token, processed in a single prefill and kept out of the answering pass, learns in hindsight to attend as the model's own correct answers do, extracting this knowledge into a concentrated pointer before any answer exists. Its hidden state then selects how to look, from no zoom to crops, square windows, windows that adapt to the pointer and foveated warps, supervised by the counterfactual outcome of every view. ours needs no box annotations, per-model layer search, external localizers or generation before the answer. On Qwen3-VL at three scales and five text-rich benchmarks, it surpasses the strongest baseline by 4.8-9.1 points at 98 visual tokens per view and attention-guided zooming by 11.1-17.0 points, while running faster than attention-guided and grounding-based zooming.

Figures & tables

Figure 1: The answer knows where to look (Qwen3-VL-4B, 196 visual tokens, V-CoT-Doc). (b) Before answering, the attention of the model’s middle layers barely touches the evidence (green box). (c) With the correct answer teacher-forced, the same layers attend to the evidence, but only once the answer is known. (d) Our <look> token concentrates on the evidence in a single prefill, and (e) the view it selects makes the value legible. Numbers give the fraction of attention inside the evidence box.

Figure 2: Where the model looks before and while answering (V-CoT-Doc, 196 tokens). In the middle layers, the attention before answering places little mass on the evidence, while the attention of the answer tokens already localizes it. Before answering, comparable localization appears only in the top layers (best band of four layers per model). Our <look> token, read from the middle layers, concentrates the most mass on the evidence before any answer is produced.

Figure 3: The information gap governs the value of looking again. Accuracy gain of a zoomed view (a

0.7

crop placed by the pre-answer attention of the base model) over the single view, against the information gap

\rho

(native over budget pixels). (a) Quintiles of

\rho

over 9,200 image–budget pairs per model (2,300 questions, 4 budgets). (b) Per benchmark (4B): the same images at budgets from 784 to 98 tokens.

Figure 4: Overview of LookZoom. (1) A dedicated <look> token is appended to the prompt and processed in a single prefill on the low-resolution view $v$ . Its attention to the image tokens tells where to look, and its hidden state tells how to look. (2) The selected view transform is applied to the full-resolution image and re-encoded within the same per-view budget. (3) The reader answers from the low-resolution view and the selected view; the <look> token never enters this pass. Training: the <look> attention is distilled from the answer-time attention of the model’s own correct rollouts (hindsight distillation), the action head is supervised by the counterfactual outcomes of every view, and the reader is adapted with GRPO.

				$B=196$						$B=98$	$B=392$
	Method	Training-free	Views	V-CoT-Doc	DocVQA	InfoVQA	ChartQA	TextVQA	Avg.	Avg.	Avg.
Qwen3-VL-2B	Single view	–	1	52.0	66.4	37.6	77.6	68.9	60.5	40.8	74.9
	AttWarp [ 5 ]	✓	1	54.7	64.3	42.3	77.4	73.0	62.3	40.9	76.6
	Ours (replace)	✗	1	68.3	72.0	53.4	78.0	76.3	69.6	55.6	77.6
	ViCrop [ 33 ]	✓	2	53.7	65.8	41.9	75.8	70.8	61.6	43.1	73.3
	Ground-then-crop	✓	2	68.3	73.1	46.4	74.6	73.2	67.1	51.0	76.5

Table 1: Main results under per-view visual-token budgets. Accuracy (%, official metrics) with at most

B

visual tokens per view; Views : views in the answering pass. Per-benchmark results use

B{=}196

; the last three columns average the five benchmarks at 196, 98 and 392 tokens. Baselines run on the base model, with the official settings where code is public; Ground-then-crop adds a crop of the evidence box named by the model, and GapSight † is our re-implementation (no public code), trained on the same images as our action head. Ours (replace) answers from the selected view alone. Best and second best per model in bold and underlined ;

\Delta

: Ours minus the best baseline.

Figure 5: Qualitative results (Qwen3-VL-4B, 196 tokens). Top: input with the window selected by LookZoom (red). Middle: the zoomed view as the model reads it, re-encoded within the budget. Bottom: answers (green: correct, red: wrong). The selected views make units, numbers and brand names legible that the single view misreads.

Method	Passes	Gen. first	Tokens	Lat. (s)	Acc.
Single view	1	✗	194	0.25	65.4
ViCrop [ 33 ]	3	✗	647	0.38	68.7
AttWarp [ 5 ]	2	✓	388	0.60	68.1
Ground-then-crop	2	✓	579	1.26	72.0
GapSight † [ 40 ]	2	✗	561	0.33	71.8
Ours (replace)	2	✗	388	0.34	73.2

Table 2: Cost on Qwen3-VL-4B at

B{=}196

in the accounting of AttWarp [ 5 ] : MLLM passes, generation before the final answer, visual tokens processed and latency per sample (batch size 1, one RTX 4090). Acc.: five-benchmark average. † Our re-implementation.

(a) Where to look: pointer driving the same view
Setting	98	196	392
Adapt. window, pre-answer, same layers	52.6	72.3	80.7
Adapt. window, pre-answer, best layers †	56.8	73.9	81.9
Adapt. window, <look> token	62.5	75.5	82.2
Fovea, pre-answer, same layers	48.9	70.4	80.4
Fovea, pre-answer, best layers †	51.5	72.6	81.5

Table 3: Ablations on Qwen3-VL-4B (five-benchmark average accuracy at 98 / 196 / 392 tokens); each block changes one component. Pre-answer: attention of the answer-start position at the layers of the <look> token or at the best top layers ( † 21–24, located with the evidence boxes of V-CoT-Doc).

	Evidence mass		Accuracy
Pointer training	196	392	Single	Adaptive
<look> token ( ours )	.229	.266	59.0	70.6
w/o sharpening	.152	.164	58.7	66.8
w/o outcome gate	.225	.262	59.1	70.2
Shared attention	.219	.251	58.4	69.5

Table 4: Training the pointer (Qwen3-VL-2B). Evidence mass of the pointer on V-CoT-Doc at 196 / 392 tokens, and five-benchmark accuracy averaged over 98 / 196 / 392 tokens for single-view reading and for the adaptive window placed by the pointer. Each row changes one element of training; the shared-attention variant trains the pre-answer attention instead of a <look> token.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Training questions	1,500 each from Visual CoT (DocVQA split), ChartQA, InfographicVQA
GRPO steps / prompts per step / rollouts $G$	200 / 16 / 8
Rollout sampling	temperature 1.0, top- $p$ 0.95, 48 tokens
Training budgets	uniform over $\{98,196,392\}$ tokens
Reader LoRA	rank 16, $\alpha{=}32$ , dropout 0.05, all linear layers
Reader learning rate	$10^{-5}$ (AdamW), gradient clip 1.0
Seed	0

Appendix

Table S1 : Hyperparameters (identical for the three model scales unless stated).

Benchmark	Source	Base	ViCrop	AttWarp
DocVQA (EM)	reported	77.3	82.5 (+5.2)	84.1 (+6.8)
DocVQA (EM)	ours	72.4	79.0 (+6.6)	81.5 (+9.1)
TextVQA (VQA, bin.)	reported	81.0	83.8 (+2.8)	84.7 (+3.7)
TextVQA (VQA, bin.)	ours	75.4	77.2 (+1.8)	76.7 (+1.3)

Appendix

Table S2 : Reproduction of the baselines with public code , each in a setting reported by its authors. Top: ViCrop and AttWarp, re-implemented line by line from the official code, in the setting of AttWarp [ 5 ] (Qwen2.5-VL-7B,

512^{2}

images). They reproduce the reported improvements over the base model on DocVQA and also improve TextVQA, by smaller margins; absolute numbers differ because we evaluate a 1,000-question subset with our prompt. Bottom: DeepEyes-7B [ 38 ] with the official checkpoint, prompt and tool loop on V* [ 28 ] at native resolution; the 25 of 191 answers that the official rules leave undecided are judged by Qwen2.5-VL-7B instead of the official Qwen2.5-72B.

Method	V-CoT-Doc	DocVQA	InfoVQA	ChartQA	TextVQA	Avg.	Visual tokens	Tool calls
$98\times 32^{2}$ pixels per image
Qwen2.5-VL-7B, single view	33.7	56.0	38.7	70.0	61.8	52.0	128	0.00
DeepEyes-7B, official prompt	22.7	38.1	25.8	33.0	29.6	29.8	199	1.68
DeepEyes-7B + answer-format instruction	30.7	47.8	32.0	51.4	49.1	42.2	192	1.58
$196\times 32^{2}$ pixels per image
Qwen2.5-VL-7B, single view	73.3	80.1	53.5	82.8	74.6	72.9	252	0.00

Appendix

Table S3: DeepEyes-7B under per-view budgets (accuracy, %). Every image of a trajectory, the input view and each crop, is capped at the same number of pixels as in our experiments (

B\times 32^{2}

; with the

28

-pixel tokens of Qwen2.5-VL this allows up to

128

256

512

tokens per image). Visual tokens are counted over all images of a trajectory. Its base model, Qwen2.5-VL-7B, is shown with a single view.

Region	98	196	392
LookZoom, 2B	0.70	0.76	0.74
LookZoom, 4B	0.68	0.73	0.74
LookZoom, 8B	0.81	0.82	0.79
Ground-then-crop, 2B	0.39	0.53	0.67
Ground-then-crop, 4B	0.33	0.40	0.65
Ground-then-crop, 8B	0.34	0.40	0.60

Appendix

Table S4: Pointing vs . naming: evidence inside the zoomed region (V-CoT-Doc, 300 questions). Fraction of the area of the evidence boxes that falls inside the region viewed at high resolution; questions without a window (no zoom or a foveated warp) count as 0. Ground-then-crop uses the named box enlarged by its own size on each side; GapSight † (our re-implementation), the crop of its router; for DeepEyes, the first crop of the trajectory.

Qwen3-VL-2B
Method	98	196	392	784
GapSight † , base reader	49.9	65.5	75.4	81.1
Our reader, GapSight’s crops	53.9	68.0	77.2	81.6
Ours	60.1	72.6	79.3	81.4
Qwen3-VL-4B
GapSight † , base reader	57.3	71.8	79.1	83.2

Appendix

Table S5: GapSight † with the same reader (five-benchmark average accuracy, %). The second row keeps GapSight’s decisions and crops but answers with our RL-adapted reader, so it differs from LookZoom only in where and how to look. Bold: best per scale and budget. † Our re-implementation.

Figure S1 : Accuracy across visual-token budgets (average over five benchmarks), including 784 tokens. The dashed line is the replace variant of LookZoom, which has the token cost of AttWarp. GapSight † : our re-implementation.

98 tokens
Method	V-CoT	Doc	Info	Chart	Text	Avg.
Single view	17.7	42.3	27.3	61.6	54.9	40.8
AttWarp [ 5 ]	19.7	32.4	29.9	61.8	60.9	40.9
Ours (replace)	50.0	58.2	36.0	66.6	67.1	55.6
ViCrop [ 33 ]	22.7	40.7	28.3	62.6	61.3	43.1
Ground-then-crop	42.3	52.6	32.0	63.2	64.9	51.0

Appendix

Table S6 : Full results of Qwen3-VL-2B at all budgets (accuracy, %). Ours (replace) answers from the selected view alone; † our re-implementation (Sec. B ). 784 tokens is the boundary regime with a small information gap.

98 tokens
Method	V-CoT	Doc	Info	Chart	Text	Avg.
Single view	18.7	49.3	33.4	66.4	60.1	45.6
AttWarp [ 5 ]	22.3	41.8	35.8	67.2	67.5	46.9
Ours (replace)	55.3	62.1	43.8	69.6	70.5	60.2
ViCrop [ 33 ]	31.3	51.5	41.1	69.6	69.0	52.5
Ground-then-crop	43.0	57.7	48.1	71.2	71.5	58.3

Appendix

Table S7 : Full results of Qwen3-VL-4B at all budgets (accuracy, %). Ours (replace) answers from the selected view alone; † our re-implementation (Sec. B ). 784 tokens is the boundary regime with a small information gap.

98 tokens
Method	V-CoT	Doc	Info	Chart	Text	Avg.
Single view	23.7	50.5	36.9	69.2	60.9	48.2
AttWarp [ 5 ]	25.7	41.2	39.1	65.6	67.5	47.8
Ours (replace)	57.3	63.8	47.3	75.8	69.4	62.7
ViCrop [ 33 ]	33.3	54.0	37.1	73.8	71.9	54.0
Ground-then-crop	46.3	61.9	53.2	75.6	73.4	62.1

Appendix

Table S8 : Full results of Qwen3-VL-8B at all budgets (accuracy, %). Ours (replace) answers from the selected view alone; † our re-implementation (Sec. B ). 784 tokens is the boundary regime with a small information gap.

Method	V-CoT	Doc	Info	Chart	Text	Avg.
Single view	56.0	70.7	45.8	80.0	73.7	65.3
ViCrop [ 33 ]	66.0	72.9	50.7	79.2	75.6	68.9
AttWarp [ 5 ]	60.3	69.8	50.2	80.5	78.6	67.9
Ground-then-crop	66.7	76.9	58.0	79.4	75.8	71.3
GapSight † [ 40 ]	73.0	78.4	53.9	77.7	72.9	71.2
Ours (replace)	71.3	77.4	57.3	79.8	77.5	72.6

Appendix

Table S9: Full evaluation sets (Qwen3-VL-4B,

B{=}196

; accuracy, %). Number of questions: V-CoT 300, Doc 5,349, Info 2,801, Chart 2,500, Text 5,000. V-CoT-Doc is evaluated in full in all experiments. † Our re-implementation.

	Tok.	Single	Random	Pre-ans.	<look>	GT
4B V-CoT-Doc	98	19.0	32.3	61.3	61.3	69.3
	196	57.7	60.7	80.7	80.0	84.0
	392	82.7	80.7	88.7	88.7	90.0
4B 5-avg.	98	45.6	49.1	62.4	62.9	–
	196	66.0	65.8	75.4	75.1	–
	392	79.3	77.3	81.2	81.1	–

Appendix

Table S10 : Is the zoomed view used? Accuracy when the same square window (

0.5\min(W,H)

) is placed at random, by the pre-answer attention, by the <look> token, or centered on the ground-truth evidence (upper bound; V-CoT-Doc only). Pointed windows are far better than random ones, so the model relies on the content of the zoomed view.

Qwen3-VL-2B
	98		196		392
Method	Rescue	Harm	Rescue	Harm	Rescue	Harm
Ours	37.9	8.0	39.9	7.1	33.4	5.6
Ours (replace)	35.4	14.9	38.1	9.7	31.2	7.0
Ground-then-crop	27.8	15.5	31.1	10.4	29.8	8.4
GapSight † [ 40 ]	24.8	16.1	22.2	6.9	15.5	4.8

Appendix

Table S11: What zooming fixes and what it breaks , relative to the single view of the base model (five benchmarks pooled, 2,300 questions per cell). Rescue: share of the single-view errors that become correct; harm: share of the single-view correct answers that become wrong (an answer counts as correct when its score is at least

0.5

; strict match on V-CoT-Doc). Bold: highest rescue and lowest harm per scale and budget. † Our re-implementation.

(a) Two views, adaptive window
Setting	98	196	392
2B, pre-answer	48.7	68.7	77.3
2B, <look> token	59.4	73.0	79.3
4B, pre-answer	52.6	72.3	80.7
4B, <look> token	62.5	75.5	82.2
8B, pre-answer	56.5	77.2	85.2

Appendix

Table S12: Pre-answer attention vs . <look> token as the pointer of the same view (five-benchmark average accuracy; pre-answer attention read from the layers of the <look> token). (a) Adaptive window in the two-view protocol, all scales. (b) Replace protocol on Qwen3-VL-4B: the zoomed view is the only view, so a misplaced window cannot be compensated by the low-resolution view.

Pointer training	196	392	$\Delta$ acc.
	Evidence mass		Reading
<look> token, untrained	.068	.057	–
+ reward of the selected views	.137	.145	$+0.0$
+ hindsight distillation	.152	.161	$0.0$
+ sharpened target	.221	.257	$+0.1$
Shared attention, supervised	.217	.247	$-0.7$

Appendix

Table S13: Training the pointer, development runs (Qwen3-VL-2B, zoomed view alone). Evidence mass of the pointer on V-CoT-Doc at 196 / 392 tokens, and change of single-view reading accuracy relative to the same RL fine-tuning without pointer losses (mean over 11 benchmark–budget settings). Training signals accumulate from top to bottom. The <look> token reaches the highest evidence mass without affecting reading, whereas supervising the shared attention degrades reading.

Figure S2 : Decisions of the action head (Qwen3-VL-4B, five benchmarks; 98, 196 and 392 tokens pooled into quintiles of the information gap

\rho

, as in Fig. 3 a). (a) Share of questions that receive a zoomed view. (b) Median area of the zoomed window relative to the image (foveated warps excluded).

Figure S3 : Per-layer evidence mass (V-CoT-Doc, 196 tokens; each layer averaged over heads, normalized and smoothed on its own). Red band: layers from which the <look> token is read; gray band: best four layers of the pre-answer attention, located with the evidence boxes.

Figure S4 : Further successes (Qwen3-VL-4B, 196 tokens). Top: input with the window selected by LookZoom (red) and, for V-CoT-Doc, the evidence boxes (green). Middle: the zoomed view as the model reads it. Bottom: answers (green: correct, red: wrong).

Figure S5 : Failure cases (Qwen3-VL-4B, 196 tokens), with the same layout as Fig. S4 .

Large Vision-Language Models

Large vision-language models (LVLMs) connect image and video perception with language generation, reasoning, and interactive tasks.

See all 404 papers

Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning
Jul 7, 2026Yake Wei, Yuan Wang, Fengyun Rao +2Vision-Language Grounding VLM Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China · Laboratory of Research on Large Models and Intelligent Governance · Tsinghua University +1
What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models
Feb 28, 2026Yingqi Fan, Junlong Tong, Anhao Zhao +1Multimodal Large Language Models Multimodal Model Interpretability
Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative · Shanghai Jiao Tong University +1
Region-Level Policy Optimization for Fine-grained MLLM Perception
Sep 17, 2026Yuheng Shi, Xiaohuan Pei, Minjing Dong +1Efficient VLM Inference Large Vision-Language Models
University of Sydney · City University of Hong Kong

Jul 7, 2026J/K move · Enter open · S save

Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning

Yake Wei, Yuan Wang, Fengyun Rao, Jing Lyu+1

Gaoling School of Artificial Intelligence, Renmin University of China · Laboratory of Research on Large Models and Intelligent Governance · Tsinghua University+1

Vision-Language Grounding VLM Reasoning Visually Grounded Reasoning Visual Grounding Fine-Grained Visual Perception

Open paper AskPDF arXiv

Recent advancements in Multimodal Large Language Models (MLLMs) have evolved from static perception to interleaved visual-language reasoning, often referred to as ``thinking with images''. A basic operation in this reasoning process is to zoom in on regions of interest (often represented with bounding boxes) to acquire finer visual details. In this paper, we propose Segmentation before Answering (SegAnswer), which shifts the unit of zoom-in from the popular bounding box to pixel-level segmentation mask. By employing fine-grained masks to isolate the target area from cluttered environments, segmented visual input yields a more precise region of interest, effectively filtering out redundant background and interfering objects. Furthermore, the discrete patches of segmented visual input align more seamlessly with how MLLMs structure visual tokens via positional embeddings. In experiments, we evaluate SegAnswer across diverse benchmarks, including high-resolution perception, general perception, and hallucination. It achieves consistent improvements and also exhibits considerable performance on segmentation tasks, validating its capability for reliable pixel grounding.

Abstract

Large Vision-Language Models

Explore similar work

Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning

What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models

Region-Level Policy Optimization for Fine-grained MLLM Perception