Attention maps cheaply show where a vision-language model (VLM) looks, but attention sinks make them unreliable for localization. Contrasting the question with a generic prompt suppresses sinks, yet only some layers localize the queried object. Existing methods find these layers with box annotations or fixed rules. We show that attention itself can identify them. A question changes how much a head attends to the image and where this attention goes. The two effects separate exactly, and the second, measured in bits on unlabeled questions, indicates which layers localize. WhereBits turns these where-to-look bits into a layer prior and combines it with per-head contrastive scores. Its three design constants were fixed once on labeled COCO data, and a new model or resolution needs only unlabeled questions. Across eight VLMs, WhereBits matches the pointing accuracy of box-tuned layers on COCO and RefCOCO and ranks target tokens better. This advantage holds on a held-out model family and at higher resolutions. Its COCO pointing accuracy is 0.72, against 0.22 for mean attention. In training-free zoom-in, its crops raise V* accuracy by 6.0 points under forced-choice evaluation.
Figures & tables
Figure 2: From attention rows to a question-specific map. (a) The inputs pair the image x or a gray image ∅ with the question t or a generic prompt g . (b,c) On a held-out COCO example (supplement), head-mean attention of Qwen3-VL-8B at layer 21 peaks in the top-left corner under both prompts (crosses). (d) Weighted per-head log-ratios at this layer peak on the queried cat (circle, target outlined). (e) Bits per layer for this question (bars) and the prior (line). (f) The final map over all layers. Heatmaps show within-image percentiles.
Figure 3: Unlabeled questions identify the layers whose maps localize. (a–c) On COCO, the layer scores from unlabeled calibration questions and the held-out pointing accuracy of each layer peak at similar depths. Both are normalized by their maximum, and ρ is their Spearman correlation. (d) Share of the 12 COCO and RefCOCO pairs in which a ViCrop layer tuned on n labeled questions reaches WhereBits (2,000 draws). (e) AUROC of the tuned layer for LLaVA-1.5-7B on COCO, with the 10th to 90th percentiles shaded.
Table 1: Held-out localization. Pointing accuracy (Pt.) and token AUROC, averaged over six models (five for V ∗ ). Bold marks the best label-free result. † Layer tuned with labels (COCO labels for V ∗ ). ‡ Reads the instruction and separator rows. § Added after the protocol was fixed.
Figure S1 : Where the peaks go (held-out COCO). (a) Share of questions whose peak lies at the map’s most frequent position. (b) Share of misses whose peak is a sink. (c) Hidden-state norm percentile of the peak token at 60% depth.
Figure S2 : Bits and per-layer pointing in every model. Blue: where-to-look bits per layer, ∑hwlh averaged over the COCO calibration split (no labels; the only input of the prior). Black: pointing of each layer’s map Fl on held-out COCO (top) and RefCOCO (bottom). ρ : Spearman correlation over layers.
Figure S3 : Label efficiency of layer tuning. A ViCrop layer is chosen on n random labeled questions of the calibration split and evaluated on held-out data (line: mean over 2000 draws; band: 10–90th percentile). Blue: WhereBits with no labels.
Figure S4 : Random V ∗ questions (seed 0). Cyan box: target; white circle / black cross: map peak on / off the target; dashed: the 1/3 crop of the primary rule (not drawn where it coincides with the panel border).
Figure S5 : Random held-out COCO questions (seed 0; two per model for LLaVA-1.5-7B, Qwen3-VL-8B, InternVL3-1B and Qwen3-VL-2B). White outline: object mask; white circle / black cross: map peak on / off the object. Maps are shown as within-image percentiles.
Visually-grounded language models (VLMs) are highly effective in linking visual and textual information, yet they often struggle with basic classification and localization tasks. While classification mechanisms have been studied more extensively, the processes that support object localization remain poorly understood. In this work, we investigate two representative families, LLaVA-1.5 and InternVL-3.5, using a suite of mechanistic interpretability tools, including token ablations, attention knockout, and causal mediation analysis. We find that localization is driven by a containerization mechanism in which object-aligned tokens define the spatial extent of the object, while the semantic arrangement of tokens within those boundaries is largely irrelevant to the predicted box. Only a very small set of attention heads mediates the causal effect for both classification and localization, concentrating in early-mid layers for LLaVA and mid-late layers for InternVL. The two tasks share some early processing but ultimately depend on largely distinct specialized heads. Overall, we provide the first layer- and head-level account of localization in VLMs, revealing narrow computational pathways that can guide future model design and grounding objectives.
Timothy Schaumlöffel, Martina G. Vilas, Gemma Roig
Goethe University Frankfurt, Germany · The Hessian Center for AI, Germany
Can internal attention patterns in Large Vision Language Models (LVLMs) identify reliable small-object boxes without fine-tuning? In this work, we provide an affirmative answer. Attention structure in LVLMs encodes grounding quality-a lightweight IoU regressor trained solely on attention maps achieves strong IoU prediction (Pearson r > 0.67). This regressor powers the regressor-based variant of our Attention-based Candidate Selection (ACS) framework, called ACS-Learned, which selects the best box from multiple sampled candidates to improve object grounding. By analyzing what the regressor learns, we reveal which transformer layers and heads are most critical and derive ACS-Free: a training-free selector that ranks candidates by attention entropy on these discriminative heads, with no learned component at inference. Experiments on COCO and Objects365 demonstrate up to 19% self-improvement on small object localization, with ACS-Free ranking best among all training-free methods, demonstrating that useful attention structure improves both localization reliability and interpretability in LVLMs.
Recent work typically assesses vision--language consistency using attention distributions of answer-side tokens. However, we observe that highest attention regions are not always consistent with the intended semantic token. This probably stems from decoding drift, where language priors from previously generated answer tokens accumulate and mismatch with visual attention. Besides the priors from previous answer tokens, we find that structural tokens, e.g., modality boundary markers, may encompass the entire context and generate high attention to areas unrelated to the target. To avoid these distortions and provide consistency evaluation for large VLMs, we adopt prompt-side semantics and propose Prompt-Vision Token Activation Map (PV-TAM). PV-TAM further incorporates a filter to remove systematic bias induced by modality boundary markers. Unlike traditional methods that evaluate overlap solely through masks while ignoring activation intensity, our metrics leverage the peak distribution of attention to measure the alignment between prompts and visual regions. In experiments, PV-TAM consistently improves both attention-based and IoU-style localization metrics over answer-side baselines on various datasets.