Whole image embeddings can under represent small objects and local attributes that distinguish relevant images for localized queries. We present LoCaR, a training free framework for localized text to image retrieval with frozen vision language models. LoCaR constructs and utilizes independent proposals (SAM2) focusing on the object so as to get a more localized representation that independently encodes each region using the frozen visual encoder. We further study paired-support adaptation as an optional residual that uses matched text region exemplars without updating the underlying encoders. We also introduce LOCFG, a Visual Genome derived benchmark designed to evaluate localized retrieval for smaller and more challenging referents. Across several vision language backbones, our results show that improving the region representation itself is the main source of gain.
Figures & tables
Figure 1: Region-canvas retrieval. (1) SAM2 generates candidate regions. (2) Each proposal defines a foreground-preserving crop with blurred background. (3) A shared frozen encoder independently embeds each canvas, and the embeddings are cached. (4) Query–region matching, optional paired-support evidence, and whole-image similarity produce the image ranking.
Region retrieval
Global / fused
Configuration
Model
R@1
R@5
R@10
R@1
R@5
R@10
Zero-shot (Global)
CLIP-B/16
–
–
–
15.15
32.83
42.89
TextRegion
9.66
26.25
37.94
17.44
37.40
48.33
TextRegion +
10.42
27.88
39.71
17.52
37.61
48.81
LoCaR
12.04
27.47
37.53
17.38
37.49
48.91
LoCaR +
12.71
28.70
39.30
17.57
37.73
49.04
Table 1: RefCOCOg retrieval (%). The right-hand columns report standalone global recall for zero-shot rows and global–region fusion otherwise. “+” adds paired support. Best and second-best values are marked within each backbone and metric. Standalone and matched fusion global channels are compared in Appendix A.2 .
Region retrieval
Global / fused
R@1
R@5
R@10
R@1
R@5
R@10
Configuration
Tr
ID
Tr
ID
Tr
ID
Tr
ID
Tr
ID
Tr
ID
(a) CLIP-B/16
Global zero-shot
–
–
–
7.01
15.80
20.76
TextRegion
5.68
13.71
18.73
8.23
18.39
24.76
TextRegion +
5.76
5.80
13.79
13.84
18.79
18.86
8.30
8.30
18.42
18.42
24.79
24.79
Table 2: LocFG retrieval (%). “+” adds paired support. Tr and ID denote transferred and in-domain settings, including their calibration choices. Identical baseline values are shown once across both columns. Separate baseline values reflect different fusion calibration or, for Qwen, different query instructions. Global zero-shot reports standalone retrieval; other right-hand entries report global–region fusion. Best and second-best values are marked within each backbone and metric across both settings. Qwen settings are detailed in Appendix A.4 .
Region
Fused
Background
Support
R@1
R@5
R@10
R@1
R@5
R@10
Blur
Without
19.26
39.19
49.51
25.99
48.48
59.54
With
19.95
40.63
51.42
25.93
48.45
59.70
White
Without
16.54
33.49
43.43
26.29
49.10
60.06
With
17.38
35.75
46.35
26.17
49.11
60.30
Table 3: SigLIP2 background ablation on RefCOCOg (%). Both configurations use the default attention-pooling readout, labeled cls in the experiment logs. The shared global channel gives 22.80/43.58/53.76 at R@1/R@5/R@10.
Region
Fused
Support
R@1
R@5
R@10
R@1
R@5
R@10
Without
10.32
26.23
36.63
23.67
45.41
56.01
With
10.62
26.60
36.93
23.72
45.46
56.08
Table 4: SigLIP2 mean-pooling ablation on RefCOCOg (%). Region canvases use blurred backgrounds and mean pooling. Support denotes the paired-support residual. The global channel is identical to that in Table 3 .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Backbone
R@1
R@5
R@10
RefCOCOg
CLIP-B/16
15.15
32.83
42.89
SigLIP2-B/16
22.80
43.58
53.76
Qwen3-VL-2B
20.17
40.56
51.92
LocFG
CLIP-B/16
7.01
15.80
20.76
SigLIP2-B/16
10.20
21.09
27.32
Qwen3-VL-2B
11.78
23.94
31.16
Appendix
Table 5: Standalone global zero-shot retrieval (%).
Instruction
Region recall
Support gain
Global R@1
Fused R@1
Incumbent (I)
10.96/24.13/31.52
+0.23/+0.43/+0.52
10.90
13.64
Selected (S)
12.27/26.43/33.91
+0.18/+0.28/+0.36
11.70
14.32
Appendix
Table 6: Qwen instruction comparison on LocFG . Region triplets are R@1/R@5/R@10 (%); gains are percentage points. Global and fused columns report matched unadapted R@1.
Instruction
β
R@1/R@5/R@10
Gain R@1/R@5/R@10
Model default
12.72
8.22/18.06/23.73
+0.14/+0.91/+1.30
asym-object
11.84
10.08/21.21/27.65
+0.11/+0.73/+0.85
asym-region
12.09
11.34/24.15/31.45
+0.08/+0.38/+0.53
asym-image (I)
11.40
10.96/24.13/31.52
+0.23/+0.43/+0.52
asym-scene-part (S)
12.24
12.27/26.43/33.91
+0.18/+0.28/+0.36
Appendix
Table 7: Qwen instruction slate on LocFG . Unadapted region recalls (%) and paired-support gains (pp). Instructions are selected on fit data.
Backbone
Gallery cache
Support bank
Combined
CLIP-B/16
78
55
133
SigLIP2-B/16
116
80
196
Qwen3-VL-2B
270
28
298
Appendix
Table 8: LoCaR cache storage (MB). Combined denotes the sum of the reported gallery cache and support bank.
Support bank
Declared source
Paired overlap
All shards via missing-split fallback
train
16.93%
Unfiltered validation
val
2.60%
Validation with shared test rows removed
val
0.00%
Appendix
Table 9: Support audit at τ=0.9999 on 3,000 RefCOCOg test queries. Paired overlap requires matching text and region features at the same support index. The 481 shared rows must be removed even when validation is loaded correctly.
LocFG (transferred)
RefCOCOg
γ
Δ R@1
Δ R@5
Δ R@10
p (R@1 / R@10)
Δ R@10
0.05
+0.12
+0.02
+0.17
0.24 / 0.19
+1.24
0.10∗
+0.08
+0.08
+0.12
0.59 / 0.50
+2.17
0.15
−0.03
−0.09
−0.16
0.88 / 0.45
+2.59
0.30 (out of band)
−0.61
−1.97
−1.94
2.8×10−3 / 3.3×10−13
+1.86
Appendix
Table 10: LocFG transfer, γ sweep , RefCOCOg-built anchors applied unchanged. Δ against γ=0 on the region channel; 2,600 images / 12,033 queries. The RefCOCOg column is the same backbone and bank in-domain, at the same γ , as a control. Every LocFG entry inside the calibrated band [0.05,0.15] is non-significant; the out-of-band row separates the two datasets in opposite directions, both significantly.
Vision-language models (VLMs) can locate an image region referred to by a text prompt and route the corresponding visual evidence to the output, yet the internal mechanism behind this behavior is not understood. Inspired by retrieval heads in large language models, we ask whether VLMs contain an analogous mechanism for visual retrieval. We answer affirmatively by introducing Visual Retrieval Heads (VRHs), a small subset of attention heads (about 1.7-2.6%) that are causally responsible for grounding text descriptions to image regions. To find them, we recast existing head-scoring methods under a unified design space over query tokens, key aggregation, and cross-sample aggregation. We then show that scoring attention from output prediction tokens with a sum over the ground-truth referent region most reliably identifies causal heads. Across eleven VLMs and five referring-expression benchmarks, masking only the top 20 VRHs reduces grounding accuracy by up to 80 percentage points, while masking the same number of random heads has little effect. Beyond replicating the causal-sparse-universal triad established for text retrieval heads, VRHs exhibit several properties not previously reported: they generalize across visual reference tasks, remaining causal on attribute, spatial, counting, and visual-math benchmarks despite being discovered through bounding-box prediction; they are functionally specific, preserving output format while corrupting localization; and they are architecturally shared, transferring causally across VLMs that share an LLM backbone but differ in vision encoder, projector, and instruction tuning.
Vision-Language Models (VLMs) still struggle on tasks requiring complex visual understanding. We argue that the core issue is not high-level reasoning, but instead failing to locate critical details in the image. Due to this shortcoming, VLMs generate often plausible but incorrect reasoning based on flawed perceptual grounding. To address this, we propose Locator-Critic (LOCI), a training-free framework that decouples visual search from evidence verification. LOCI employs a Locator agent to propose candidate visual evidence and a separate Critic agent to evaluate its relevance and sufficiency. These agents engage in an iterative refinement loop, progressively improving the evidence until it is adequate to answer the given question. This decoupled, self-correcting process yields substantial performance gains, achieving state-of-the-art results on multiple complex visual benchmarks. LOCI improves accuracy for both open-weight models like Qwen3-VL (+12.1 on V*, +5.8 on HR-Bench and +11.2 on VisualProbe-Hard) and proprietary models like Gemini 2.5 Pro (+8.9 on V*, +4.3 on HR-Bench, +4.8 on VisualProbe-Hard).
Text-to-image retrieval with frozen dual encoders degrades when the query names a small, visually subordinate object in a cluttered scene: a single global image embedding underrepresents the localized visual evidence. We present MINER, a training-free inference framework that augments a frozen dual encoder's global image embedding with a small bank of region-level embeddings and a hubness-correcting similarity rescoring, recovering visual evidence that global pooling underweights. To evaluate this setting, we introduce ROCS, a benchmark built from high-clutter subsets of Flickr30K and MS COCO whose images are re-captioned to name a single low-salience object. Experiments on CLIP, SigLIP, and SigLIP 2 show that MINER improves retrieval on every backbone, on ROCS and on the standard splits. Analyses show that these gains come primarily from broader spatial coverage rather than precise crop placement, revealing a simple and general way to recover localized evidence from frozen representations. Code: https://github.com/aalquwayfili/MINER. Dataset: https://huggingface.co/datasets/aalquwayfili/ROCS.