cs.CVNov 17, 2026

Bits, Not Boxes: Label-Free Layer Selection for Visual Localization in VLMs

Authors: Haisheng Fu, Gexin Huang, Haikai Zhao, Jie Liang, Zhenman Fang, Jingning Han, Xiaoxiao Li

Organizations: University of British Columbia · Simon Fraser University · University of Minnesota–Twin Cities · Google

Abstract

Attention maps cheaply show where a vision-language model (VLM) looks, but attention sinks make them unreliable for localization. Contrasting the question with a generic prompt suppresses sinks, yet only some layers localize the queried object. Existing methods find these layers with box annotations or fixed rules. We show that attention itself can identify them. A question changes how much a head attends to the image and where this attention goes. The two effects separate exactly, and the second, measured in bits on unlabeled questions, indicates which layers localize. WhereBits turns these where-to-look bits into a layer prior and combines it with per-head contrastive scores. Its three design constants were fixed once on labeled COCO data, and a new model or resolution needs only unlabeled questions. Across eight VLMs, WhereBits matches the pointing accuracy of box-tuned layers on COCO and RefCOCO and ranks target tokens better. This advantage holds on a held-out model family and at higher resolutions. Its COCO pointing accuracy is 0.72, against 0.22 for mean attention. In training-free zoom-in, its crops raise V* accuracy by 6.0 points under forced-choice evaluation.

Figures & tables

Explore similar work

CardsList
  1. Mechanisms of Object Localization in Vision-Language Models

    May 19, 2026Timothy Schaumlöffel, Martina G. Vilas, Gemma RoigVision-Language ModelsVLM Interpretability

  2. Self-Improving Small Object Grounding in LVLMs

    Jun 1, 2026Tianze Yang, Yucheng Shi, Ruitong Sun +2Large Vision-Language ModelsVLM Interpretability

  3. Listening makes Vision Clear for VLMs

    Jun 22, 2026Yiyang Chen, Yixin Tan, Binrui ShenVLM EvaluationVisual Attention