Attention maps cheaply show where a vision-language model (VLM) looks, but attention sinks make them unreliable for localization. Contrasting the question with a generic prompt suppresses sinks, yet only some layers localize the queried object. Existing methods find these layers with box annotations or fixed rules. We show that attention itself can identify them. A question changes how much a head attends to the image and where this attention goes. The two effects separate exactly, and the second, measured in bits on unlabeled questions, indicates which layers localize. WhereBits turns these where-to-look bits into a layer prior and combines it with per-head contrastive scores. Its three design constants were fixed once on labeled COCO data, and a new model or resolution needs only unlabeled questions. Across eight VLMs, WhereBits matches the pointing accuracy of box-tuned layers on COCO and RefCOCO and ranks target tokens better. This advantage holds on a held-out model family and at higher resolutions. Its COCO pointing accuracy is 0.72, against 0.22 for mean attention. In training-free zoom-in, its crops raise V* accuracy by 6.0 points under forced-choice evaluation.
Figures & tables
Figure 2: From attention rows to a question-specific map. (a) The inputs pair the image x or a gray image ∅ with the question t or a generic prompt g . (b,c) On a held-out COCO example (supplement), head-mean attention of Qwen3-VL-8B at layer 21 peaks in the top-left corner under both prompts (crosses). (d) Weighted per-head log-ratios at this layer peak on the queried cat (circle, target outlined). (e) Bits per layer for this question (bars) and the prior (line). (f) The final map over all layers. Heatmaps show within-image percentiles.
Figure 3: Unlabeled questions identify the layers whose maps localize. (a–c) On COCO, the layer scores from unlabeled calibration questions and the held-out pointing accuracy of each layer peak at similar depths. Both are normalized by their maximum, and ρ is their Spearman correlation. (d) Share of the 12 COCO and RefCOCO pairs in which a ViCrop layer tuned on n labeled questions reaches WhereBits (2,000 draws). (e) AUROC of the tuned layer for LLaVA-1.5-7B on COCO, with the 10th to 90th percentiles shaded.
Table 1: Held-out localization. Pointing accuracy (Pt.) and token AUROC, averaged over six models (five for V ∗ ). Bold marks the best label-free result. † Layer tuned with labels (COCO labels for V ∗ ). ‡ Reads the instruction and separator rows. § Added after the protocol was fixed.
Figure S1 : Where the peaks go (held-out COCO). (a) Share of questions whose peak lies at the map’s most frequent position. (b) Share of misses whose peak is a sink. (c) Hidden-state norm percentile of the peak token at 60% depth.
Figure S2 : Bits and per-layer pointing in every model. Blue: where-to-look bits per layer, ∑hwlh averaged over the COCO calibration split (no labels; the only input of the prior). Black: pointing of each layer’s map Fl on held-out COCO (top) and RefCOCO (bottom). ρ : Spearman correlation over layers.
Figure S3 : Label efficiency of layer tuning. A ViCrop layer is chosen on n random labeled questions of the calibration split and evaluated on held-out data (line: mean over 2000 draws; band: 10–90th percentile). Blue: WhereBits with no labels.
Figure S4 : Random V ∗ questions (seed 0). Cyan box: target; white circle / black cross: map peak on / off the target; dashed: the 1/3 crop of the primary rule (not drawn where it coincides with the panel border).
Figure S5 : Random held-out COCO questions (seed 0; two per model for LLaVA-1.5-7B, Qwen3-VL-8B, InternVL3-1B and Qwen3-VL-2B). White outline: object mask; white circle / black cross: map peak on / off the object. Maps are shown as within-image percentiles.